Illumination guide identification method for deeply-forged face in video and related equipment

Abstract

The invention provides an illumination guide identification method for a deeply-forged face in a video and related equipment, and relates to the technical field of video processing and video identification. The method comprises the following steps: extracting a face sequence from a to-be-detected video; for each frame of face image in the face sequence, constructing a corresponding three-dimensional image to obtain a three-dimensional face and a three-dimensional normal vector of each key point in a plurality of key points of the three-dimensional face; obtaining a three-dimensional illumination coefficient based on the three-dimensional normal vector of the three-dimensional face corresponding to each frame of face image and the brightness value of each frame of face image; based on the three-dimensional normal vector and the three-dimensional illumination coefficient, determining a time sequence illumination geometric feature corresponding to the face sequence; time sequence face global features are extracted from the multiple frames of face images; and based on the time sequence illumination geometric features and the time sequence face global features, judging whether the multiple frames of face images in the to-be-detected video have forged face images or not. According to the embodiment of the invention, the generalization ability of the detection of the deeply-forged video can be improved.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202411470616.5, filed on October 21, 2024, and entitled “Method and Related Device for Identifying Deep Fake Faces in Videos”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the fields of video processing technology and video identification technology, and in particular to a lighting-guided identification method for deep fake faces in videos and related equipment. Background Art

[0004] With the development of deep generative models, deep fake technology continues to improve and can generate facial images that are close to the real effect. However, if this technology is abused, it may cause serious social problems. Therefore, it is particularly important to develop effective deep fake video detection technology.

[0005] In recent years, the detection methods for deep fakes are mainly based on single-frame methods. Single-frame-based methods mainly focus on mining forgery clues at the frame level. Although single-frame detection methods have achieved certain results, with the advancement of forgery technology, it is difficult to meet the high-precision detection requirements by relying solely on single-frame information. Therefore, it is necessary to further improve the accuracy and robustness of the detection method while reducing the computational cost in order to better deal with the potential risks brought by deep fakes.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] The present disclosure provides a lighting-guided identification method and related equipment for deep fake faces in videos, which improves the accuracy and robustness of the detection method at least to a certain extent.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0009] According to one aspect of the present disclosure, a method for illumination-guided identification of deep fake faces in videos is provided, comprising:

[0010] Extracting a face sequence from the video to be detected, where the face sequence includes multiple frames of face images;

[0011] For each frame of the face image, a corresponding three-dimensional image is constructed to obtain the three-dimensional face and the three-dimensional normal vector of each key point of the three-dimensional face;

[0012] The 3D normal vector of the 3D face corresponding to each frame of the face image and the brightness value of each frame of the face image are calculated using spherical harmonic functions to obtain the 3D illumination coefficient;

[0013] Based on the 3D normal vector and the 3D illumination coefficient, the temporal illumination geometric features corresponding to the face sequence are determined;

[0014] From multiple frames of face images, temporal global features of faces are extracted, and the temporal global features of faces are used to characterize the degree of facial forgery traces;

[0015] Based on the temporal illumination geometric features and the temporal face global features, it is determined whether there are forged face images in the multiple frames of face images in the video to be detected.

[0016] In one embodiment of the present disclosure, for each frame of a face image, a corresponding three-dimensional image is constructed to obtain a three-dimensional face and a three-dimensional normal vector of each key point of a plurality of key points of the three-dimensional face, including:

[0017] For each frame of face image, construct the corresponding 3D image to obtain the 3D face and the 3D normal vector of each point;

[0018] The 3D face is input into a pre-trained 3D face key point extractor to obtain multiple key points and the 3D normal vector of each key point.

[0019] In one embodiment of the present disclosure, a spherical harmonic function is used to calculate the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image to obtain a three-dimensional illumination coefficient, including:

[0020] Eliminate points in a preset area from a plurality of key points of a three-dimensional face corresponding to the face image, where the preset area includes one or more of an eye area, a mouth area, and a nose area;

[0021] After removing the points in the preset area, the three-dimensional normal vectors of the remaining key points in the three-dimensional face and the brightness value of each frame of the face image are calculated using spherical harmonic functions to obtain the three-dimensional illumination coefficient.

[0022] In one embodiment of the present disclosure, determining temporal illumination geometric features corresponding to a face sequence based on a three-dimensional normal vector and a three-dimensional illumination coefficient includes:

[0023] According to the order of the video frames to which the face images belong in the video to be detected, the three-dimensional normal vectors of multiple key points of the three-dimensional faces corresponding to the multiple face images are spliced ​​to obtain a three-dimensional normal vector sequence;

[0024] The three-dimensional illumination coefficients of multiple face images are concatenated to obtain an illumination coefficient sequence;

[0025] By integrating the 3D normal vector sequence and the illumination coefficient sequence, the temporal illumination geometric features corresponding to the face sequence are obtained.

[0026] In one embodiment of the present disclosure, the three-dimensional normal vector sequence and the illumination coefficient sequence are integrated to obtain the temporal illumination geometric features corresponding to the face sequence, including:

[0027] For the 3D normal vector sequence, the relationship between normal vectors and the relationship between normal vector channels are learned frame by frame, and the 3D geometric feature sequence is obtained by splicing.

[0028] The 3D geometric feature sequence and the illumination coefficient sequence are spliced ​​and input into the feature extractor constructed by the multi-layer perceptron to integrate the illumination and geometric information to obtain the fused high-dimensional features.

[0029] Learn temporal information from high-dimensional features and obtain temporal illumination geometric features.

[0030] In one embodiment of the present disclosure, extracting temporal global face features from multiple frames of face images includes:

[0031] The time inconsistency between multiple frames of face images is extracted from both horizontal and vertical directions using differential operations;

[0032] Self-attention is calculated from both temporal and spatial dimensions, the extracted temporal inconsistency is modeled, and the temporal global features of the face are obtained.

[0033] In one embodiment of the present disclosure, extracting a face sequence from a video to be detected includes:

[0034] Using a face detector to extract a face image in each video frame of the video to be detected;

[0035] According to the order of the video frames to which the face images belong in the video to be detected, the extracted face images are spliced ​​to obtain a face sequence.

[0036] According to another aspect of the present disclosure, a lighting-guided identification device for deep fake faces in videos is provided, comprising:

[0037] The target extraction module is used to extract a face sequence from the video to be detected, and the face sequence includes multiple frames of face images;

[0038] A three-dimensional construction module is used to construct a corresponding three-dimensional image for each frame of the face image, and obtain a three-dimensional face and a three-dimensional normal vector of each key point of a plurality of key points of the three-dimensional face;

[0039] The illumination coefficient calculation module is used to calculate the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image by using spherical harmonic functions to obtain the three-dimensional illumination coefficient;

[0040] An illumination feature extraction module is used to determine the temporal illumination geometric features corresponding to the face sequence based on the three-dimensional normal vector and the three-dimensional illumination coefficient;

[0041] A face feature extraction module is used to extract the temporal global features of faces from multiple frames of face images. The temporal global features of faces are used to characterize the degree of facial forgery traces.

[0042] The authenticity identification module is used to determine whether there are forged face images in multiple frames of face images in the video to be detected based on temporal lighting geometric features and temporal face global features.

[0043] According to another aspect of the present disclosure, an electronic device is provided, including: a memory for storing instructions; and a processor for calling the instructions stored in the memory to implement the above-mentioned lighting-guided identification method for deep fake faces in videos.

[0044] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored, and when the computer instructions are executed by a processor, the above-mentioned lighting-guided identification method for deep fake faces in videos is implemented.

[0045] According to yet another aspect of the present disclosure, a computer program product is provided, which stores instructions, and when the instructions are executed by a computer, the computer implements the above-mentioned lighting-guided identification method for deep fake faces in videos.

[0046] According to yet another aspect of the present disclosure, there is provided a chip, comprising at least one processor and an interface;

[0047] An interface for providing program instructions or data to at least one processor;

[0048] At least one processor is used to execute program instructions to implement the above-mentioned lighting-guided identification method for deep fake faces in videos.

[0049] The embodiments of the present disclosure provide a lighting-guided identification method and related equipment for deep fake faces in videos. The normal vector can provide rich information about the surface geometry and direction, and the illumination coefficient can reflect the illumination conditions of the object surface. The embodiments of the present disclosure combine the normal vector sequence and the illumination coefficient sequence to detect fake videos, and comprehensively utilize the geometry and illumination information of the video content to capture fake traces from adjacent frame transitions, thereby improving the accuracy and robustness of model detection.

[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0052] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without any creative work.

[0053] Figure 1 A flow chart of a method for illumination-guided identification of deep fake faces in videos according to an embodiment of the present disclosure is shown;

[0054] Figure 2 Another flow chart of a method for illumination-guided identification of deep fake faces in videos according to an embodiment of the present disclosure is shown;

[0055] Figure 3 The overall framework diagram of the model in the embodiment of the present disclosure is shown;

[0056] Figure 4 A bidirectional multi-scale enhancement module in an embodiment of the present disclosure is shown;

[0057] Figure 5 The spatiotemporal self-attention module in the embodiment of the present disclosure is shown;

[0058] Figure 6 A schematic diagram of a lighting-guided identification device for deep fake faces in videos according to an embodiment of the present disclosure is shown;

[0059] Figure 7 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0061] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0062] In some embodiments, before extracting a face sequence from a video to be detected, a prompt message may be sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0063] As an optional but non-limiting implementation, the prompt information may be sent to the user in a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0064] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0065] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0066] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0067] First, an embodiment of the present disclosure provides a method for illumination-guided identification of deep fake faces in videos, which can be executed by any electronic device with computing and processing capabilities.

[0068] Figure 1 A flow chart of a method for illumination-guided identification of deep fake faces in videos according to an embodiment of the present disclosure is shown. Figure 1 As shown, the lighting-guided identification method for deep fake faces in videos provided in the embodiments of the present disclosure includes steps S101-S106.

[0069] In S101, a face sequence is extracted from a video to be detected, where the face sequence includes multiple frames of face images.

[0070] First, read the video to be detected frame by frame. This can be implemented in a variety of programming languages ​​and libraries, such as the OpenCV library in Python.

[0071] Then, for each frame of the image, a face detection algorithm is used to find the location of the face. Common methods include Haar feature cascade classifiers (proposed by Viola-Jones), deep learning methods such as MTCNN (Multi-Task Cascaded Convolutional Networks) or SSD (Single Shot MultiBox Detector) models.

[0072] After the face area is determined, it may be necessary to further align the face and crop out the face part of interest as needed.

[0073] Finally, the face images obtained at each time point are arranged in chronological order to form a face sequence. If the video is long or contains multiple people, additional logic may be required to distinguish different individuals and create independent sequences for each person.

[0074] In some embodiments, extracting a face sequence from a video to be detected can be accomplished by using a face detector to extract a face image from each video frame of the video to be detected; and splicing the extracted face images in the order of the video frames to which the face images belong in the video to be detected to obtain a face sequence.

[0075] In S102, for each frame of the face image, a corresponding three-dimensional image is constructed to obtain the three-dimensional face and the three-dimensional normal vector of each key point of the three-dimensional face.

[0076] When constructing a three-dimensional image, a single-view based method or a multi-view fusion method can be used.

[0077] Single-view based methods: If there is only one frame (i.e. single view), you can use pre-trained deep learning models, such as 3DMM (Three-Dimensional Morphable Model) or other network architectures designed specifically for 3D face reconstruction from a single image (such as Deep3DFaceRecon). These models can generate a parameterized 3D face model based on the input face image.

[0078] Multi-view fusion: If there are multiple frames of images from different angles, the multi-view geometry principle can be used to fuse the information of multiple views through feature matching and other means to improve the quality of 3D reconstruction.

[0079] The key points of a 3D face can be a series of facial positions with specific meanings defined in 3D space. Usually, these key points include the positions of facial features such as eyes, nose, mouth, as well as other important structural points such as cheek contours and eyebrows.

[0080] Eye area: including inner and outer corners of the eye, pupil center, etc. Nose: nose tip, sides of nose wings, etc. Mouth: corners of mouth, edges of upper and lower lips, etc. Facial contour: chin, highest point of cheekbones, etc. Eyebrow: starting and ending point of eyebrows, etc. Ear: main turning point of auricle (if the model contains side information).

[0081] In S103, a spherical harmonic function is used to calculate the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image to obtain a three-dimensional illumination coefficient.

[0082] In S104, based on the three-dimensional normal vector and the three-dimensional illumination coefficient, the temporal illumination geometric features corresponding to the face sequence are determined.

[0083] In S105, a temporal global face feature is extracted from multiple frames of face images, and the temporal global face feature is used to characterize the degree of facial forgery traces.

[0084] In S106, based on the temporal illumination geometric features and the temporal face global features, it is determined whether there are forged face images in the multiple frames of face images in the video to be detected.

[0085] Here, the temporal illumination geometric features and the temporal face global features can be input into a classifier composed of a multi-layer perceptron to obtain a classification result, that is, the authenticity result of the group of face images.

[0086] The normal vector can provide rich information about the surface geometry and direction, and the illumination coefficient can reflect the illumination conditions of the object surface. The disclosed embodiment combines the normal vector sequence and the illumination coefficient sequence to detect forged videos, comprehensively utilizes the geometry and illumination information of the video content, and captures forgery traces from adjacent frame transitions, thereby improving the accuracy and robustness of model detection.

[0087] In some embodiments, the above S102 constructs a corresponding three-dimensional image for each frame of facial image, obtains the three-dimensional face and the three-dimensional normal vector of each key point of the three-dimensional face, including: constructing a corresponding three-dimensional image for each frame of facial image, obtaining the three-dimensional face and the three-dimensional normal vector of each point; inputting the three-dimensional face into a pre-trained facial three-dimensional key point extractor, obtaining multiple key points and the three-dimensional normal vector of each key point.

[0088] In some embodiments, spherical harmonic functions are used to perform operations on the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image to obtain a three-dimensional illumination coefficient, including: eliminating points in a preset area from multiple key points of the three-dimensional face corresponding to the face image, the preset area including one or more of the eye area, the mouth area, and the nose area; after eliminating the points in the preset area, the three-dimensional normal vectors of the remaining key points in the three-dimensional face and the brightness value of each frame of the face image are calculated using spherical harmonic functions to obtain a three-dimensional illumination coefficient.

[0089] In some embodiments, based on the three-dimensional normal vector and the three-dimensional illumination coefficient, the temporal illumination geometric features corresponding to the face sequence are determined, including: splicing the three-dimensional normal vectors of multiple key points of the three-dimensional faces corresponding to multiple face images in the order of the video frames to which the face images belong in the video to be detected to obtain a three-dimensional normal vector sequence; splicing the three-dimensional illumination coefficients of multiple face images to obtain an illumination coefficient sequence; and combining the three-dimensional normal vector sequence and the illumination coefficient sequence to obtain the temporal illumination geometric features corresponding to the face sequence.

[0090] In some embodiments, the above S104 determines the temporal illumination geometric features corresponding to the face sequence based on the three-dimensional normal vector sequence and the illumination coefficient sequence, including: for the three-dimensional normal vector sequence, learning the relationship between normal vectors and the relationship between normal vector channels frame by frame, and splicing them to obtain a three-dimensional geometric feature sequence; splicing the three-dimensional geometric feature sequence and the illumination coefficient sequence into a feature extractor constructed by a multi-layer perceptron to integrate illumination and geometric information to obtain fused high-dimensional features; learning temporal information from the high-dimensional features to obtain temporal illumination geometric features.

[0091] In some embodiments, temporal global facial features are extracted from multiple frames of facial images, including: using differential operations to extract temporal inconsistencies between multiple frames of facial images in both horizontal and vertical directions; calculating self-attention from both temporal and spatial dimensions, modeling the extracted temporal inconsistencies, and obtaining temporal global facial features.

[0092] In view of the temporal inconsistency in the video, the disclosed embodiment uses the features of different modalities to perform analysis to improve accuracy and robustness. In addition to focusing on the discontinuity of facial forgery traces when switching across frames, attention is also paid to the continuous changes in physical features such as lighting between frames.

[0093] The disclosed embodiment uses a convolutional network to extract facial forgery trace features, reducing the spatial dimension while retaining key information. In order to deepen the learning of temporal information, a differential operation is performed on adjacent frames in the spatial dimension. Then a transformer architecture is used to learn the long-range spatiotemporal relationship in the features, in which the temporal sub-attention and spatial sub-attention are calculated in parallel to mine hidden forgery information from multiple angles.

[0094] Figure 2 A flow chart of a method for illumination-guided identification of deep fake faces in videos according to an embodiment of the present disclosure is shown. Figure 2 As shown, the lighting-guided identification method for deep fake faces in videos provided in the embodiments of the present disclosure includes steps S201-S207.

[0095] In S201, a training video containing both true and false samples is obtained, continuous frame sampling is performed on the video frames, and a face detector is used to extract the face and a small part of the background area frame by frame as a face image to obtain a face sequence;

[0096] In S202, for each frame of face image, a 3D image is constructed using a 3D face reconstruction model to obtain a 3D normal vector. The spherical harmonic illumination coefficient of each frame of face image is calculated using spherical harmonic function;

[0097] In S203, an illumination geometry feature extractor is constructed to mine the temporal inconsistency of the video in physical features from the three-dimensional normal vector and the illumination coefficient, and to construct a temporal illumination geometry feature;

[0098] In S204, a bidirectional multi-scale enhancement module is constructed, and the Xception network inserted with the bidirectional multi-scale enhancement module is used to mine the temporal inconsistency between video frames from the horizontal and vertical dimensions;

[0099] In S205, a spatiotemporal self-attention module is constructed to mine the temporal inconsistency between video frames from the temporal and spatial dimensions, and to construct a temporal global face feature;

[0100] In S206, the outputs of S203 and S205 are concatenated and input into a classifier composed of a multi-layer perceptron to obtain a classification result;

[0101] In S207, continuous face images in continuous frames in the test set are input into the trained model to obtain a probability score of the predicted authenticity, thereby determining the authenticity of the video.

[0102] In some embodiments, the specific method of acquiring the face image in S201 is:

[0103] (1) setting the number of continuous sampling frames of the video, starting from the first frame of the video, and extracting video frames from the video;

[0104] (2) Use the Dlib face detector to determine the face candidate box in the current video frame. The candidate box coordinates can be expressed as (x, y, m, n), where (x, y) is the coordinate of the upper left corner of the candidate box, and (m, n) is the coordinate of the upper left corner of the candidate box;

[0105] (3) The candidate frame range is enlarged according to the preset ratio r, and the face candidate frame becomes (x-r, y-r, w+r, h+r), and the image within the candidate frame range is captured as the face image;

[0106] (3) Concatenate the face images of each frame to obtain a face sequence.

[0107] In some embodiments, the specific method of obtaining the illumination geometry information in S202 is:

[0108] (1) For each frame in the face sequence obtained in S201, a three-dimensional image and a three-dimensional normal vector of each point are obtained using a three-dimensional reconstruction method;

[0109] (2) inputting the three-dimensional image into a pre-trained facial three-dimensional key point extractor Mediapipe to obtain 468 three-dimensional facial key points;

[0110] (3) Eliminate the eye, mouth, and nose areas, and use spherical harmonics to calculate the three-dimensional normal vectors and brightness values ​​of points in other areas to obtain the three-dimensional illumination coefficient;

[0111] (4) The three-dimensional normal vectors of each frame are concatenated to obtain a three-dimensional normal vector sequence, and the three-dimensional illumination coefficients of each frame are concatenated to obtain an illumination coefficient sequence. The three-dimensional normal vector sequence and the illumination coefficient sequence are combined to obtain illumination geometric information.

[0112] In some embodiments, the illumination geometry feature extractor S203:

[0113] (1) For the three-dimensional normal vector sequence obtained in S202, the relationship between the normal vectors and the relationship between the normal vector channels are learned frame by frame, and the three-dimensional geometric feature sequence is obtained by splicing.

[0114] (2) The three-dimensional geometric feature sequence and the illumination coefficient sequence obtained in S202 are concatenated and input into a feature extractor constructed by a multi-layer perceptron to integrate illumination and geometric information and extract fused high-dimensional features.

[0115] (3) The high-dimensional features are input into the Transformer Encoder to learn the temporal information and obtain the final temporal illumination geometric features.

[0116] In some embodiments, the specific method of the bidirectional multi-scale enhancement module in S204 is:

[0117] (1) Input the face sequence obtained in S201 into two branches respectively to mine forgery traces from multiple angles;

[0118] (2) The first branch first uses a global average pooling operation to compress the spatial dimension of the input sequence, then uses a fully connected layer to reduce the number of time channels by half, and then uses a fully connected layer to restore the time dimension, adding more nonlinear processing processes to fit the complex correlations between time series. Finally, an activation function is used to obtain the output of the first branch;

[0119] (3) The second branch first compresses in the channel dimension and resizes in the vertical direction. Adjacent frames are differentiated and convolution and pooling operations of different scales are used to learn spatiotemporal inconsistencies. The outputs of different operations are added together, the dimensions are restored by 1×1 convolution, and the vertical attention is calculated using sigmoid. The above operations are repeated in the horizontal direction to calculate the horizontal attention. The vertical and horizontal attention are added to obtain the output of the second branch.

[0120] (4) Multiplying the output of the first branch, the output of the second branch, and the input sequence, and adding the obtained result to the input sequence to highlight the part of the input sequence that is effective for true or false classification.

[0121] In some embodiments, the specific method of the spatiotemporal self-attention module in S205 is:

[0122] (1) Input the output of S204 into the temporal self-attention branch and the spatial self-attention branch respectively;

[0123] (2) The temporal self-attention branch uses 1×1 convolution to calculate the query (q). In order to reduce the amount of computation, convolution is used to compress the time dimension.

[0124] The compressed features are used to calculate the key (k) and value (v). Finally, the temporal self-attention is obtained through the self-attention calculation formula.

[0125] (3) The spatial self-attention branch uses 1×1 convolution to calculate the query (q). In order to reduce the amount of calculation, convolution is used to compress the spatial dimension. The compressed features are used to calculate the key (k) and value (v). Finally, the spatial self-attention is obtained through the self-attention calculation formula.

[0126] (4) Multiply the temporal self-attention by the spatial self-attention to obtain the final spatiotemporal self-attention.

[0127] (5) Construct a feedforward layer based on convolution operation, first map the spatiotemporal self-attention to a high-dimensional space and then to a low-dimensional space, extracting deeper features.

[0128] In some embodiments, in the above S207, the trained model is used to predict the video, and the final score is used as the prediction result of the authenticity of the face in the video.

[0129] The disclosed embodiment uses MLP and transformer encoder to construct an illumination geometry feature extractor for illumination geometry features, and obtains temporal illumination geometry features. For facial forgery trace features, a convolutional neural network and transformer are used to construct a global face feature extractor to obtain temporal global face features. Among them, differential operations are used to extract spatiotemporal inconsistencies from both horizontal and vertical directions; self-attention is calculated from both temporal and spatial dimensions to model spatiotemporal inconsistencies. The illumination geometry features and the global face features are combined and input into a classifier for training to obtain a trained classification model. The disclosed embodiment can further improve the generalization capability of detecting deep fake videos.

[0130] Please refer to the following Figure 3 -Attached Figure 5 , introduces the lighting-guided identification method for deep fake faces in videos provided by an embodiment of the present disclosure.

[0131] Step 1: Collection and preprocessing of video datasets.

[0132] First, a series of training videos containing both real and fake samples are selected, and Opencv is used to cut the video frames. The face area in the video is extracted frame by frame using the dlib algorithm, and the candidate box range is expanded according to the preset ratio r to form a complete face image sample. The video is divided into training set, validation set and test set in a ratio of 8:1:1.

[0133] Step 2: Design of multi-scale feature fusion network.

[0134] The overall framework of the proposed model is as follows Figure 3 As shown in the figure, it mainly includes an illumination geometry feature extractor, a bidirectional multi-scale enhancement module, and a spatiotemporal self-attention module. The illumination geometry feature extractor extracts illumination geometry features from the input video, and then uses transformer to learn the temporal information in the illumination geometry features. The bidirectional multi-scale enhancement module is instantiated using Xcpetion to extract multi-level features from the input video from three perspectives: horizontal, vertical, and temporal. The spatiotemporal self-attention module learns the inconsistency of forged traces from two perspectives: spatial and temporal. Finally, the detection result is obtained by combining illumination geometry features and facial forgery traces.

[0135] In some embodiments, the video sequence can be converted into a 3D face sequence and a normal vector sequence can be extracted from it using the document "Unsupervised Learning of ProbablySymmetric Deformable 3D Objects from Images in the Wild" Use spherical harmonics to calculate the illumination coefficient sequence like Figure 3 As shown in the upper part. 1 , N represents the sampling frame number and normal vector sampling number of the lighting geometry information branch respectively.

[0136] Single frame normal vector sampling point set It is processed by two shared multi-layer perceptrons (MLP) (composed of a series of fully connected layers, BN and ReLU), which successively explore the relationship between points in a single dimension and between dimensions of a single point, and obtain

[0137]

[0138] The disclosed embodiment will learn the features and the illumination coefficient I t This method not only considers the geometric information (i.e., the 3D normal vector), but also the degree to which the sampling point is affected by the illumination (i.e., the illumination coefficient), thus making the generated feature vector more comprehensive and accurate. Finally, this vector is input into another multi-layer perceptron MLP for processing to generate a 3D illumination information representation E at time t. t :

[0139]

[0140] According to the time dimension, E t Splicing to get a 3D-illumination representation sequence M is the output feature dimension. Through the multi-head self-attention block and the feedforward network block, the inconsistency of the motion of the key points in the three-dimensional space and the global illumination in the frame-by-frame conversion is learned to obtain the illumination inconsistency feature Out I .

[0141] Out I =FFN(LN(MHA(LN(E)))) (3)

[0142] The bidirectional multi-scale enhancement module mainly consists of two branches, such as Figure 5 As shown in , the attention information between different frames in the spatial domain and the attention information between frame differences in the temporal domain are captured respectively. These two modules are added to each network stage in parallel. represents the input features of the bidirectional multi-scale enhancement module, where T 2 , C, H, W represent the number of sampling frames, channels and spatial dimensions of the facial forgery branch respectively. The first branch first uses the global average pooling (GAP) operation to compress the spatial dimension of the global view, then performs two fully connected layers, and finally uses the softmax operation:

[0143]

[0144] Among them, the first fully connected layer consists of T 2 Dimensionality reduction to T 2 / 2, the second fully connected layer is upgraded to T 2 , which adds more nonlinear processing processes and can fit the complex correlations between time series.

[0145] The second branch uses the temporal difference feature maps of adjacent frames to calculate the collaborative spatiotemporal attention in both horizontal and vertical directions, such as Figure 4 The right half is shown. The left side is the vertical direction and the right side is the horizontal branch. First, the channel dimension is compressed and the dimension is adjusted to Resize the horizontal dimensions to Taking the vertical branch as an example, the feature map Diff is obtained by performing differential calculation on adjacent frames in the vertical direction. V , learning spatiotemporal inconsistency through pooling and convolution operations, the main steps are: 1) residual connection, 2) 3×1 convolution, upsampling, 3) 2×1 pooling, 3×1 convolution, upsampling, 4) 4×1 pooling, 3×1 convolution, upsampling. In the time dimension, different pooling depths are used to achieve long- and short-range information learning. Finally, the dimension is restored through 1×1 convolution, and the vertical attention Att is calculated using sigmoid. V :

[0146]

[0147] Among them, Diff H and Att H They represent the feature map obtained after the horizontal difference calculation and the horizontal attention respectively. Finally, the information learned from the two branches is integrated to obtain the output of the bidirectional multi-scale enhancement module:

[0148]

[0149] make represents the input features of the spatiotemporal self-attention module, where T 2 , C, H, and W represent the number of sampling frames, the number of channels, and the spatial dimension, respectively. The spatiotemporal self-attention module mainly uses the tansformer architecture to learn temporal inconsistencies from the two dimensions of space and time, such as Figure 5 As shown in Figure 1, the left branch is the spatial dimension and the right branch is the time dimension. Taking the spatial dimension as an example: using a 1×1 convolution instead of a linear layer to calculate Q SIn order to reduce the amount of calculation, a 3×3 convolution is used to reduce the spatial dimension of the input features (the time branch uses a one-dimensional convolution with a convolution kernel of 7 to reduce the temporal dimension of the input features), and then a 1×1 convolution is used to obtain K S 、V S . S , K S 、V S Calculate the attention, and then get the output Out of the spatial dimension branch through 1×1 convolution S The attention mechanism of the present invention can be summarized by the following formula:

[0150]

[0151] Among them, C d represents the QKV embedding dimension, Out T Indicates the output of the time dimension branch. In addition, in order to increase local information learning, the feedforward layer FeedForward Network (FFN) is changed to Locally-enhanced FeedForward Network (LeFF), mainly through 1×1 convolution, 3×3 depth-separable convolution and 1×1 convolution. After 4 layers of learning, the forged trace inconsistency feature Out is finally obtained. F .

[0152] Finally, the output features of the illumination geometry branch and the facial forgery trace branch are combined to obtain the final classification result:

[0153] Out=MLP(Cat(Out T ,Out F )) (17)

[0154] Step 3: Model training.

[0155] FaceForensics++ (FF++), Celeb-DF, and DFDC are used for training, validation, and testing. It is a large-scale deep fake detection benchmark. FaceForensics++ contains 4 fake techniques: Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT), with 1,000 videos for each fake method. These videos include high-quality (C23) and low-quality (C40) versions. The heavier the compression, the more difficult it is to distinguish the fake traces. Celeb-DF contains 540 real videos and 5,639 fake videos. Fake videos are generated using upgraded synthesis techniques, achieving better quality with fewer perceptible visual artifacts than previous datasets. DFDC contains original videos recorded from 430 hired actors, and synthesized more than 400G of fake videos through multiple deep fake generation methods.

[0156] Step 2 was built using the pytorch framework. The model was trained on an Nvidia A30 GPU for 80 rounds, with a batch size of 32. The cross entropy loss is used to measure the deviation between the probability distribution predicted by the model and the actual probability distribution of the label. The formula for the cross entropy loss is:

[0157]

[0158] Among them, y i represents the true label (0 or 1) of the i-th sample, is the predicted probability that the i-th sample belongs to the positive class. The Adam optimizer was used, and the initial learning rate was set to 0.001, which was adjusted to half of the original value every 20 iterations.

[0159] Step 4: Model testing.

[0160] The model with the largest AUC in the validation data set in step 4 is used as the test model and tested on the test data set.

[0161] Table 1

[0162]

[0163]

[0164] Internal dataset evaluation was performed on the FF++C40 dataset. The specific results are shown in Table 1. Compared with the frame-based detection method, the AUC of this patent is better than all the compared methods. Compared with the video-based detection method, the method in this paper has a significant improvement in performance. The reason behind this is that this patent not only comprehensively uses the normal vector and illumination coefficient to explore the illumination information contained in the video, but also explores facial forgery traces from the horizontal and vertical dimensions as well as the time and space dimensions, improving the model's ability to mine spatiotemporal inconsistencies.

[0165] As shown in Table 2, this patent also evaluates the proposed method on the Celeb-DF and DFDC datasets. The results show that the present invention outperforms all the compared methods. This is mainly because this patent uses illumination data of continuously sampled T1 frames and video data of sparsely sampled T2 frames to cover local and global temporal information.

[0166] Table 2

[0167]

[0168] In the embodiments of the present disclosure, the terms “first”, “second” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0169] The term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0170] Furthermore, although the steps of the method in the present disclosure are described in a particular order in the drawings, this does not require or imply that the steps must be performed in this particular order or that all the steps shown must be performed to achieve the desired results.

[0171] In some embodiments, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0172] Based on the same inventive concept, the embodiments of the present disclosure also provide a lighting guidance identification device for deep fake faces in videos, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0173] Figure 6 A schematic diagram of a lighting-guided identification device for deep fake faces in videos according to an embodiment of the present disclosure is shown. Figure 6As shown, the lighting-guided identification device 600 for deep fake faces in videos includes:

[0174] The target extraction module 601 is used to extract a face sequence from the video to be detected, where the face sequence includes multiple frames of face images;

[0175] A three-dimensional construction module 602 is used to construct a corresponding three-dimensional image for each frame of the face image, and obtain a three-dimensional face and a three-dimensional normal vector of each key point of a plurality of key points of the three-dimensional face;

[0176] The illumination coefficient calculation module 603 is used to calculate the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image by using the spherical harmonic function to obtain the three-dimensional illumination coefficient;

[0177] An illumination feature extraction module 604 is used to determine the temporal illumination geometric features corresponding to the face sequence based on the three-dimensional normal vector and the three-dimensional illumination coefficient;

[0178] The face feature extraction module 605 is used to extract the temporal global face features from multiple frames of face images, and the temporal global face features are used to characterize the degree of facial forgery traces;

[0179] The authenticity identification module 606 is used to determine whether there are forged face images in the multiple frames of face images in the video to be detected based on the temporal lighting geometric features and the temporal face global features.

[0180] In some embodiments, the illumination coefficient calculation module 603 is used to construct a corresponding three-dimensional image for each frame of the face image, obtain a three-dimensional face and a three-dimensional normal vector for each point; input the three-dimensional face into a pre-trained three-dimensional face key point extractor to obtain multiple key points and a three-dimensional normal vector for each key point.

[0181] In some embodiments, spherical harmonic functions are used to perform operations on the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image to obtain a three-dimensional illumination coefficient, including: eliminating points in a preset area from multiple key points of the three-dimensional face corresponding to the face image, the preset area including one or more of the eye area, the mouth area, and the nose area; after eliminating the points in the preset area, the three-dimensional normal vectors of the remaining key points in the three-dimensional face and the brightness value of each frame of the face image are calculated using spherical harmonic functions to obtain a three-dimensional illumination coefficient.

[0182] In some embodiments, based on the three-dimensional normal vector and the three-dimensional illumination coefficient, the temporal illumination geometric features corresponding to the face sequence are determined, including: splicing the three-dimensional normal vectors of multiple key points of the three-dimensional faces corresponding to multiple face images in the order of the video frames to which the face images belong in the video to be detected to obtain a three-dimensional normal vector sequence; splicing the three-dimensional illumination coefficients of multiple face images to obtain an illumination coefficient sequence; and combining the three-dimensional normal vector sequence and the illumination coefficient sequence to obtain the temporal illumination geometric features corresponding to the face sequence.

[0183] In some embodiments, a three-dimensional normal vector sequence and an illumination coefficient sequence are integrated to obtain temporal illumination geometric features corresponding to the face sequence, including: for the three-dimensional normal vector sequence, the relationship between normal vectors and the relationship between normal vector channels are learned frame by frame, and spliced ​​to obtain a three-dimensional geometric feature sequence; the three-dimensional geometric feature sequence and the illumination coefficient sequence are spliced ​​and input into a feature extractor constructed by a multi-layer perceptron to integrate illumination and geometric information to obtain fused high-dimensional features; and the temporal information is learned from the high-dimensional features to obtain temporal illumination geometric features.

[0184] In some embodiments, the facial feature extraction module 605 is used to extract the temporal inconsistency between multiple frames of facial images in the horizontal and vertical directions using differential operations; calculate self-attention from the temporal and spatial dimensions, model the extracted temporal inconsistency, and obtain the temporal global facial features.

[0185] In some embodiments, the target extraction module 601 is used to extract a face image in each video frame of the video to be detected using a face detector; the extracted face images are spliced ​​according to the order of the video frames to which the face images belong in the video to be detected to obtain a face sequence.

[0186] The concepts of “first”, “second”, etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0187] Regarding the lighting-guided identification device for deep fake faces in videos in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the lighting-guided identification method for deep fake faces in videos, and will not be elaborated here.

[0188] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory.

[0189] In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0190] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0191] Refer to the following Figure 7 To describe the electronic device provided by the embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0192] Figure 7 FIG. 1 is a schematic diagram showing the architecture of an electronic device 700 provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device 700 includes but is not limited to: at least one processor 710 and at least one memory 720.

[0193] The memory 720 is used to store instructions.

[0194] In some embodiments, the memory 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202 , and may further include a read-only memory unit (ROM) 7203 .

[0195] In some embodiments, the memory 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0196] In some embodiments, the memory 720 may store an operating system, which may be a real-time operating system (RTX), LINUX, UNIX, WINDOWS, or OS X.

[0197] In some embodiments, data may also be stored in the memory 720 .

[0198] As an example, the processor 710 may read data stored in the memory 720 , where the data may be stored at the same storage address as the instruction, or the data may be stored at a different storage address than the instruction.

[0199] The processor 710 is configured to call the instructions stored in the memory 720 to implement the steps described in the "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processor 710 may execute the steps of the above-mentioned illumination-guided identification method embodiment for deep fake faces in videos.

[0200] It should be noted that the processor 710 may be a general-purpose processor or a special-purpose processor. The processor 710 may include one or more processing cores, and the processor 710 executes various functional applications and data processing by running instructions.

[0201] In some embodiments, the processor 710 may include a central processing unit (CPU) and / or a baseband processor.

[0202] In some embodiments, the processor 710 may determine an instruction according to the priority identifier and / or function category information carried in each control instruction.

[0203] In the present disclosure, the processor 710 and the memory 720 may be provided separately or integrated together.

[0204] As an example, the processor 710 and the memory 720 may be integrated on a single board or a system on chip (SOC).

[0205] like Figure 7 As shown, the electronic device 700 is in the form of a general-purpose computing device. The electronic device 700 may further include a bus 730 .

[0206] Bus 730 may be a bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0207] The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 700, and / or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 750.

[0208] Furthermore, the electronic device 700 can also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 760 .

[0209] like Figure 7 As shown, the network adapter 760 communicates with other modules of the electronic device 700 via the bus 730 .

[0210] It should be understood that although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0211] It is to be understood that the structure shown in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 700. In other embodiments of the present disclosure, the electronic device 700 may include: Figure 7 More or fewer components may be shown, or some components may be combined or separated, or the components may be arranged differently. Figure 7 The components shown may be implemented in hardware, software or a combination of software and hardware.

[0212] The present disclosure also provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the lighting-guided identification method for deep fake faces in videos described in the above method embodiment is implemented.

[0213] The computer-readable storage medium in the embodiments of the present disclosure is a computer instruction that can be sent, propagated or transmitted for use by or in conjunction with an instruction execution system, apparatus or device.

[0214] As one example, computer readable storage media are non-volatile storage media.

[0215] In some embodiments, more specific examples of computer-readable storage media in the present disclosure may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, a USB flash drive, a mobile hard disk, or any suitable combination of the foregoing.

[0216] In the embodiments of the present disclosure, the computer-readable storage medium may include a data signal propagated in a baseband or as a part of a carrier wave, in which computer instructions (readable program codes) are carried.

[0217] Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

[0218] In some examples, computing instructions contained on a computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0219] The embodiments of the present disclosure also provide a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the lighting-guided identification method for deep fake faces in videos described in the above method embodiments.

[0220] The above instructions may be program codes. In specific implementation, the program codes may be written in any combination of one or more programming languages.

[0221] The programming language includes object-oriented programming languages, such as Java, C++, etc., and also includes conventional procedural programming languages, such as "C" language or similar programming languages.

[0222] The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0223] Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0224] The embodiment of the present disclosure also provides a chip, including at least one processor and an interface;

[0225] An interface for providing program instructions or data to at least one processor;

[0226] At least one processor is used to execute program instructions to implement the lighting-guided identification method for deep fake faces in videos described in the above method embodiment.

[0227] In some embodiments, the chip may further include a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0228] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0229] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein.

[0230] The present disclosure is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A lighting-guided identification method for deep fake faces in videos, characterized in that: include: Extracting a face sequence from the video to be detected, wherein the face sequence includes multiple frames of face images; For each frame of the face image, construct a corresponding three-dimensional image to obtain a three-dimensional face and a three-dimensional normal vector of each key point of a plurality of key points of the three-dimensional face; Using spherical harmonic functions to calculate the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image to obtain a three-dimensional illumination coefficient; Determining temporal illumination geometric features corresponding to the face sequence based on the three-dimensional normal vector and the three-dimensional illumination coefficient; Extracting a temporal global face feature from the multiple frames of face images, wherein the temporal global face feature is used to characterize the degree of facial forgery traces; Based on the temporal illumination geometric features and the temporal face global features, it is determined whether there are forged face images in the multiple frames of face images in the video to be detected.

2. The method according to claim 1, characterized in that: For each frame of the face image, a corresponding three-dimensional image is constructed to obtain a three-dimensional face and a three-dimensional normal vector of each key point of the three-dimensional face, including: For each frame of face image, construct the corresponding 3D image to obtain the 3D face and the 3D normal vector of each point; The three-dimensional human face is input into a pre-trained three-dimensional face key point extractor to obtain a plurality of key points and a three-dimensional normal vector of each key point.

3. The method according to claim 2, characterized in that The 3D normal vector of the 3D face corresponding to each frame of the face image and the brightness value of each frame of the face image are calculated using spherical harmonic functions to obtain a 3D illumination coefficient, including: Eliminate points in a preset area from a plurality of key points of the three-dimensional face corresponding to the face image, where the preset area includes one or more of an eye area, a mouth area, and a nose area; After removing the points in the preset area, the three-dimensional normal vectors of the remaining key points in the three-dimensional face and the brightness value of each frame of the face image are calculated using spherical harmonic functions to obtain a three-dimensional illumination coefficient.

4. The method according to claim 3, characterized in that Determining temporal illumination geometric features corresponding to the face sequence based on the three-dimensional normal vector and the three-dimensional illumination coefficient includes: splicing the three-dimensional normal vectors of the multiple key points of the three-dimensional face corresponding to the multiple face images to obtain a three-dimensional normal vector sequence according to the order of the video frames to which the face images belong in the video to be detected; splicing the three-dimensional illumination coefficients of the plurality of face images to obtain an illumination coefficient sequence; The three-dimensional normal vector sequence and the illumination coefficient sequence are combined to obtain the temporal illumination geometric features corresponding to the face sequence.

5. The method according to claim 4, characterized in that The step of synthesizing the three-dimensional normal vector sequence and the illumination coefficient sequence to obtain the temporal illumination geometric features corresponding to the face sequence includes: For the three-dimensional normal vector sequence, the relationship between normal vectors and the relationship between normal vector channels are learned frame by frame, and a three-dimensional geometric feature sequence is obtained by splicing; splicing the three-dimensional geometric feature sequence and the illumination coefficient sequence and inputting them into a feature extractor constructed by a multi-layer perceptron to integrate illumination and geometric information to obtain fused high-dimensional features; The temporal information is learned from the high-dimensional features to obtain temporal illumination geometric features.

6. The method according to claim 1, characterized in that The step of extracting temporal global features of faces from the multiple frames of face images comprises: Extracting the temporal inconsistency between the multiple frames of face images in both horizontal and vertical directions by using a differential operation; Self-attention is calculated from both temporal and spatial dimensions, the extracted temporal inconsistency is modeled, and the temporal global features of the face are obtained.

7. The method according to claim 1, characterized in that The step of extracting a face sequence from a video to be detected includes: Using a face detector to extract a face image in each video frame of the video to be detected; According to the order of the video frames to which the face images belong in the video to be detected, the extracted face images are spliced ​​to obtain a face sequence.

8. A lighting-guided identification device for deep fake faces in videos, characterized in that: include: A target extraction module is used to extract a face sequence from the video to be detected, wherein the face sequence includes multiple frames of face images; A three-dimensional construction module, for constructing a corresponding three-dimensional image for each frame of a face image, and obtaining a three-dimensional face and a three-dimensional normal vector of each key point of a plurality of key points of the three-dimensional face; An illumination coefficient calculation module, for calculating the three-dimensional normal vector of the three-dimensional face corresponding to each frame of the face image and the brightness value of each frame of the face image by using spherical harmonic functions to obtain a three-dimensional illumination coefficient; An illumination feature extraction module, used to determine the temporal illumination geometric features corresponding to the face sequence based on the three-dimensional normal vector and the three-dimensional illumination coefficient; A facial feature extraction module, used to extract temporal global facial features from the multiple frames of facial images, wherein the temporal global facial features are used to characterize the degree of facial forgery traces; The authenticity identification module is used to determine whether there are forged facial images in the multiple frames of facial images in the video to be detected based on the temporal lighting geometric features and the temporal facial global features.

9. An electronic device, characterized in that: include: A memory for storing instructions; A processor, configured to call instructions stored in the memory to implement the lighting-guided identification method for deep fake faces in videos as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the lighting-guided identification method for deep fake faces in videos as described in any one of claims 1 to 7.

Images