Face image forgery detection method and related products
By optimizing the model weights through the spatiotemporal feature extraction network, the problem of insufficient capture of temporal artifacts in facial image forgery detection in the existing technology is solved, and forgery detection with higher accuracy is achieved.
Patent Information
- Application Number
- CN202310588111.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-05-23
AI Technical Summary
The existing three-dimensional convolutional neural network model's reliance on spatial artifacts in face image forgery detection results in insufficient ability to capture temporal artifacts, poor generalization and robustness, and affects detection accuracy.
A spatiotemporal feature extraction network is adopted. By fixing the parameters of the spatial sub-network and the temporal sub-network, their weights are optimized respectively. The temporal artifact features and the spatial artifact features are combined to optimize the detection model to improve the detection accuracy.
It achieves the simultaneous capture of temporal and spatial artifacts in facial images, improves the generalization and robustness of the detection model, and improves the accuracy of forgery detection.
Smart Images
Figure CN116844243B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method for detecting forgery of facial images and related products. Background Art
[0002] For forged facial images, a trained 3D convolutional neural network model that can search for spatial and temporal information is usually used. However, because spatial artifacts are usually more obvious than temporal artifacts, current 3D convolutional neural network models rely more on spatial artifact fitting training, resulting in the model's lack of ability to capture temporal artifacts, poor generalization ability and robustness, which is not conducive to improving the accuracy of forged facial image detection. Summary of the Invention
[0003] Based on this, it is necessary to provide a method for detecting forgery of facial images and related products that can improve the accuracy of forgery detection of facial images in response to the above technical problems.
[0004] In a first aspect, the present application provides a method for detecting forgery of facial images. The method comprises:
[0005] The face images in the video training set are sequentially input into the detection model; the detection model is a spatiotemporal feature extraction network, which includes a temporal subnetwork and a spatial subnetwork;
[0006] Extract facial features from the input face image sequentially through the spatiotemporal feature extraction network;
[0007] When the parameters of the spatial sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a first composite facial feature including temporal artifact features and spatial artifact features; a first detection result of the facial image is determined based on the first composite facial feature and the facial features, and the time weight in the temporal sub-network is optimized based on the first detection result and the corresponding face label;
[0008] When the parameters of the temporal sub-network are fixed, facial features are processed based on the spatiotemporal feature extraction network to obtain a second composite facial feature containing temporal artifact features and spatial artifact features; a second detection result of the facial image is determined based on the second composite facial feature and the facial features, and the spatial weights in the spatial sub-network are optimized based on the second detection result and the corresponding facial label;
[0009] When the detection model reaches the convergence condition, the converged detection model is used to perform face forgery detection on the face image to be tested to obtain the detection result.
[0010] In one embodiment, when the parameters of the spatial sub-network are fixed, the number of times the time weights in the time sub-network are optimized is a first value;
[0011] When the parameters of the time sub-network are fixed, the number of times the time weight in the spatial sub-network is optimized is a second value; and the first value is greater than the second value.
[0012] In one embodiment, before sequentially inputting the facial images in the video training set into the detection model, the method further includes:
[0013] Get the original video set;
[0014] Performing facial key point detection on the first face image in the original video set in sequence to obtain a set of facial key points in the original video set;
[0015] Performing face tracking on the second face image in the original video set according to the face key point set to obtain a tracking result;
[0016] Based on the tracking results, the second face image is sequentially aligned to obtain a video training set.
[0017] In one embodiment, feature processing is performed on facial features based on a spatiotemporal feature extraction network to obtain a first composite facial feature including a temporal artifact feature and a spatial artifact feature, including:
[0018] Performing feature processing on facial features based on the temporal sub-network in the detection model to obtain first process features including temporal artifact features;
[0019] The first process feature is processed based on a spatial subnetwork with fixed parameters in the detection model to obtain a first face composite feature including a temporal artifact feature and a spatial artifact feature.
[0020] In one embodiment, feature processing is performed on facial features based on a spatiotemporal feature extraction network to obtain a second composite facial feature including temporal artifact features and spatial artifact features, including:
[0021] Performing feature processing on facial features based on a time sub-network with fixed parameters in the detection model to obtain a second process feature including a time artifact feature;
[0022] The second process features are processed based on the spatial sub-network in the detection model to obtain a second face composite feature including temporal artifact features and spatial artifact features.
[0023] In one embodiment, the method further includes:
[0024] If the detection result shows that the face image to be tested is authentic, the face image to be tested is matched with the preset face image;
[0025] When the face image to be tested successfully matches the preset face image, the current payment operation is confirmed to be executed.
[0026] In a second aspect, the present application also provides a facial image forgery detection device. The device comprises:
[0027] An input unit is used to sequentially input the face images in the video training set into the detection model; the detection model is a spatiotemporal feature extraction network, which includes a temporal subnetwork and a spatial subnetwork;
[0028] An extraction unit, configured to sequentially extract facial features from an input facial image through a spatiotemporal feature extraction network;
[0029] a first processing unit, configured to perform feature processing on facial features based on the spatiotemporal feature extraction network while fixing the parameters of the spatial sub-network, to obtain a first composite facial feature including a temporal artifact feature and a spatial artifact feature; determine a first detection result of the facial image based on the first composite facial feature and the facial feature, and optimize the temporal weight in the temporal sub-network based on the first detection result and the corresponding face label;
[0030] a second processing unit, configured to perform feature processing on facial features based on the spatiotemporal feature extraction network when the parameters of the temporal sub-network are fixed, to obtain a second composite facial feature including temporal artifact features and spatial artifact features; determine a second detection result of the facial image based on the second composite facial feature and the facial features, and optimize the spatial weights in the spatial sub-network based on the second detection result and the corresponding face label;
[0031] The detection unit is used to perform face forgery detection on the face image to be tested using the converged detection model when the detection model reaches the convergence condition to obtain a detection result.
[0032] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:
[0033] The face images in the video training set are sequentially input into the detection model; the detection model is a spatiotemporal feature extraction network, which includes a temporal subnetwork and a spatial subnetwork;
[0034] Extract facial features from the input face image sequentially through the spatiotemporal feature extraction network;
[0035] When the parameters of the spatial sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a first composite facial feature including temporal artifact features and spatial artifact features; a first detection result of the facial image is determined based on the first composite facial feature and the facial features, and the time weight in the temporal sub-network is optimized based on the first detection result and the corresponding face label;
[0036] When the parameters of the temporal sub-network are fixed, facial features are processed based on the spatiotemporal feature extraction network to obtain a second composite facial feature containing temporal artifact features and spatial artifact features; a second detection result of the facial image is determined based on the second composite facial feature and the facial features, and the spatial weights in the spatial sub-network are optimized based on the second detection result and the corresponding facial label;
[0037] When the detection model reaches the convergence condition, the converged detection model is used to perform face forgery detection on the face image to be tested to obtain the detection result.
[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:
[0039] The face images in the video training set are sequentially input into the detection model; the detection model is a spatiotemporal feature extraction network, which includes a temporal subnetwork and a spatial subnetwork;
[0040] Extract facial features from the input face image sequentially through the spatiotemporal feature extraction network;
[0041] When the parameters of the spatial sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a first composite facial feature including temporal artifact features and spatial artifact features; a first detection result of the facial image is determined based on the first composite facial feature and the facial features, and the time weight in the temporal sub-network is optimized based on the first detection result and the corresponding face label;
[0042] When the parameters of the temporal sub-network are fixed, facial features are processed based on the spatiotemporal feature extraction network to obtain a second composite facial feature containing temporal artifact features and spatial artifact features; a second detection result of the facial image is determined based on the second composite facial feature and the facial features, and the spatial weights in the spatial sub-network are optimized based on the second detection result and the corresponding facial label;
[0043] When the detection model reaches the convergence condition, the converged detection model is used to perform face forgery detection on the face image to be tested to obtain the detection result.
[0044] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:
[0045] The face images in the video training set are sequentially input into the detection model; the detection model is a spatiotemporal feature extraction network, which includes a temporal subnetwork and a spatial subnetwork;
[0046] Extract facial features from the input face image sequentially through the spatiotemporal feature extraction network;
[0047] When the parameters of the spatial sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a first composite facial feature including temporal artifact features and spatial artifact features; a first detection result of the facial image is determined based on the first composite facial feature and the facial features, and the time weight in the temporal sub-network is optimized based on the first detection result and the corresponding face label;
[0048] When the parameters of the temporal sub-network are fixed, facial features are processed based on the spatiotemporal feature extraction network to obtain a second composite facial feature containing temporal artifact features and spatial artifact features; a second detection result of the facial image is determined based on the second composite facial feature and the facial features, and the spatial weights in the spatial sub-network are optimized based on the second detection result and the corresponding facial label;
[0049] When the detection model reaches the convergence condition, the converged detection model is used to perform face forgery detection on the face image to be tested to obtain the detection result.
[0050] The above-mentioned facial image forgery detection and related products input the facial images in the video training set into the detection model in sequence; the detection model is a spatiotemporal feature extraction network, and the spatiotemporal feature extraction network includes a time sub-network and a space sub-network; facial features are extracted from the input facial images in sequence through the spatiotemporal feature extraction network; when the parameters of the spatial sub-network are fixed, the facial features are processed based on the spatiotemporal feature extraction network to obtain a first facial composite feature including a temporal artifact feature and a spatial artifact feature; a first detection result of the facial image is determined based on the first facial composite feature and the facial feature, and the first facial feature is detected based on the first facial composite feature. The detection results and the corresponding face labels are used to optimize the time weights in the time sub-network; when the parameters of the time sub-network are fixed, the face features are processed based on the spatiotemporal feature extraction network to obtain a second face composite feature containing temporal artifact features and spatial artifact features; the second detection result of the face image is determined based on the second face composite feature and the face features, and the spatial weights in the space sub-network are optimized based on the second detection result and the corresponding face label; when the detection model reaches the convergence condition, the face image to be tested is subjected to face forgery detection through the converged detection model to obtain the detection result. The converged detection model provided in the present application can simultaneously capture the temporal artifact features and spatial artifact features of the face image, and has higher generalization and stronger robustness. Therefore, when the face forgery detection method provided in the present application is used to perform face forgery detection on the face image to be tested through the converged detection model, it can simultaneously capture the temporal artifact features and spatial artifact features, thereby having more accurate detection results. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A diagram illustrating an application environment of a method for detecting forgery of facial images according to an embodiment;
[0052] Figure 2 1 is a flow chart of a method for detecting forgery of facial images according to an embodiment;
[0053] Figure 3 A schematic diagram of a detection model optimization process in one embodiment;
[0054] Figure 4 is a schematic flow chart of a method for detecting forgery of facial images in another embodiment;
[0055] Figure 5 is a structural block diagram of a facial image forgery detection device according to one embodiment;
[0056] Figure 6 is a structural block diagram of a facial image forgery detection device in another embodiment;
[0057] Figure 7 is a diagram of the internal structure of a computer device in one embodiment;
[0058] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0060] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0061] See also Figure 1 , Figure 1 FIG is an application environment diagram of a method for detecting forgery of a face image in an embodiment. The method for detecting forgery of a face image provided by the embodiment of the present application can be applied to the following situations: Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. In some embodiments, the facial image forgery detection method is executed by the terminal 102, and the terminal 102 inputs the facial images in the video training set into the detection model in sequence; the detection model is a spatiotemporal feature extraction network, and the spatiotemporal feature extraction network includes a time subnetwork and a space subnetwork; facial features are extracted from the input facial images in sequence through the spatiotemporal feature extraction network; when the parameters of the spatial subnetwork are fixed, the facial features are feature processed based on the spatiotemporal feature extraction network to obtain a first facial composite feature including a temporal artifact feature and a spatial artifact feature; the first detection result of the facial image is determined based on the first facial composite feature and the facial feature. According to the first detection result and the corresponding face label, the time weight in the time sub-network is optimized; when the parameters of the time sub-network are fixed, the face features are processed based on the spatiotemporal feature extraction network to obtain a second face composite feature including time artifact features and space artifact features; according to the second face composite feature and the face feature, the second detection result of the face image is determined, and according to the second detection result and the corresponding face label, the space weight in the space sub-network is optimized; when the detection model reaches the convergence condition, the terminal 102 performs face forgery detection on the face image to be tested through the converged detection model to obtain a detection result.
[0062] The terminal 102 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, IoT device, or portable wearable device. IoT device may be a smart speaker, smart TV, smart air conditioner, or smart car device. Portable wearable device may be a smart watch, smart bracelet, or head-mounted device.
[0063] The server 104 may be an independent physical server or a service node in a blockchain system. A peer-to-peer (P2P) network is formed between the service nodes in the blockchain system. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).
[0064] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0065] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus), or a network, and this application does not impose any limitation thereto.
[0066] In one embodiment, see Figure 2 , Figure 2 FIG. 1 is a flow chart of a method for detecting forgery of a face image in one embodiment. Figure 2 As shown, a method for detecting forgery of face images is provided, which can be Figure 1 The server or terminal in the embodiment is executed, or the server and the terminal are executed in collaboration, in which method the Figure 1 The following steps are used as an example to illustrate the terminal execution in .
[0067] S202, inputting the face images in the video training set into the detection model in sequence; the detection model is a spatiotemporal feature extraction network, which includes a time subnetwork and a space subnetwork.
[0068] The video training set, in a specific implementation, may refer to a plurality of trained videos including facial images. Before S202, the original video set including facial images may be pre-processed to obtain a video training set with a smaller computational load.
[0069] In a specific implementation, the face image can be a face video or a face picture.
[0070] In a specific implementation, the detection model includes an input layer and an output layer. The input layer is used to input facial images from the video training set, and the output layer is used to output the detection results of face forgery detection, which are used to indicate the authenticity of the tested facial images. Accordingly, in a specific implementation, the detection model can be used to output the detection results of face forgery detection corresponding to facial videos, or it can be used to output the detection results of face forgery detection corresponding to facial images.
[0071] The facial images in the video training set are input into the detection model in sequence. In the specific implementation, when the facial image is a facial video, each facial video is cut into a series of continuous frames, and then the continuous frames obtained by cutting each facial video are input into the detection model in sequence.
[0072] Specifically, the terminal obtains an original video set and performs face preprocessing on the original video set to obtain a video training set. The face images in the video training set are face images that have undergone face preprocessing and only include a single person identity with facial key points aligned.
[0073] S204, extracting facial features from the input facial image in sequence through a spatiotemporal feature extraction network.
[0074] Among them, facial features, in specific implementations, can refer to facial organs such as eyes, nose, lips, and the geometric relationships between them, such as the size, area, angle, and distance between facial organs.
[0075] S206, when the parameters of the spatial sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a first facial composite feature including temporal artifact features and spatial artifact features; a first detection result of the facial image is determined based on the first facial composite feature and the facial features, and the time weight in the temporal sub-network is optimized based on the first detection result and the corresponding face label.
[0076] Among them, temporal artifact features refer to features in facial images that do not conform to temporal logic. In specific implementations, they may include temporal inconsistencies in the dynamic changes of facial images. Spatial artifact features refer to features in facial images that do not conform to spatial logic. In specific implementations, they may include mixed boundary artifacts, color artifacts, etc.
[0077] In a specific implementation, the first detection result refers to a prediction result of the authenticity of the face image based on the first composite facial feature and the facial feature.
[0078] The face label corresponding to the first detection result, in a specific implementation, refers to the actual authenticity of the face image corresponding to the first detection result.
[0079] Specifically, the time weight in the time sub-network is optimized based on the first detection result and the corresponding face label, that is, the time weight in the time sub-network is optimized based on the loss between the first detection result and the corresponding face label. The purpose of the optimization is to reduce the loss between the first detection result and the corresponding face label, that is, to make the accuracy of the first detection result higher.
[0080] Time weight, in the specific implementation, refers to the time-related convolution parameter, namely K t ×1×1 convolution kernel, K t Refers to the length of the convolution kernel in time.
[0081] S208, when the parameters of the time sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a second facial composite feature including temporal artifact features and spatial artifact features; a second detection result of the facial image is determined based on the second facial composite feature and the facial features, and the spatial weight in the spatial sub-network is optimized based on the second detection result and the corresponding face label.
[0082] Among them, when the parameters of the time sub-network are fixed, if the spatial sub-network is still in a fixed state at this time, it is necessary to first release the fixed state of the spatial sub-network, and then perform subsequent steps such as feature processing of facial features based on the spatiotemporal feature extraction network.
[0083] The second detection result, in a specific implementation, refers to a prediction result of the authenticity of the face image based on the second facial composite feature and the facial feature.
[0084] The face label corresponding to the second detection result, in a specific implementation, refers to the actual authenticity of the face image corresponding to the second detection result.
[0085] Specifically, the spatial weights in the spatial subnetwork are optimized based on the second detection result and the corresponding face label. That is, the spatial weights in the spatial subnetwork are optimized based on the loss between the second detection result and the corresponding face label. The purpose of the optimization is to reduce the loss between the second detection result and the corresponding face label, that is, to make the second detection result more accurate. In a specific implementation, the loss between the first detection result, the second detection result, and the corresponding face label can be determined using a mean square error loss function, a mean absolute error loss function, a quantile loss function, a cross entropy loss function, a hinge loss function, or other loss functions.
[0086] Spatial weight, in the specific implementation, refers to the spatially related convolution parameters, that is, 1×K h×K w Convolution kernel, K h Refers to the height of the convolution kernel in space, K w Refers to the width of the convolution kernel in space.
[0087] S210: Determine whether the detection model has reached a convergence condition.
[0088] Among them, if the detection model does not meet the convergence conditions, S206 and S208 are executed again, that is, when the parameters of the spatial sub-network and the parameters of the time sub-network are fixed respectively, the time weights in the time sub-network and the spatial weights in the space sub-network are optimized respectively until the detection model can meet the convergence conditions; if the detection model meets the convergence conditions, S212 is executed.
[0089] S212, when the detection model reaches the convergence condition, performing face forgery detection on the face image to be tested using the converged detection model to obtain a detection result.
[0090] Among them, the detection model reaches the convergence condition. In the specific implementation, it means that the time weights in the time subnetwork and the spatial weights in the spatial subnetwork in the detection model have been optimized, so that the detection model can obtain accurate detection results of the authenticity of the face image to be tested.
[0091] The detection model reaches the convergence condition. In a specific implementation, after optimizing the time weight in the time subnetwork and the spatial weight in the space subnetwork, the loss between the first detection result, the second detection result and the corresponding face label is less than the preset loss value.
[0092] In a specific implementation, a face image to be tested refers to a face image whose authenticity has not yet been determined and needs to be verified. This image is used to input into the converged detection model for face forgery detection and obtain the corresponding detection results. The detection results include not only the authenticity of the face image to be tested, but also visually annotated temporal and spatial artifact features.
[0093] Specifically, after S212, the terminal obtains the detection result, and the terminal can push the detection result corresponding to the facial image to be tested to the terminal display screen. In a specific implementation, when the detection result indicates that the facial image to be tested is authentic and matches the preset facial image, if the terminal used in this application is an access control system or a terminal connected to the access control system, the detection result is used to confirm the opening of the access control system; if the terminal used in this application is a payment system or a terminal connected to the payment system, the detection result is used to confirm the payment operation.
[0094] In the above-mentioned face image forgery detection method, the face images in the video training set are sequentially input into the detection model; the detection model is a spatiotemporal feature extraction network, and the spatiotemporal feature extraction network includes a time sub-network and a space sub-network; facial features are sequentially extracted from the input face images through the spatiotemporal feature extraction network; when the parameters of the space sub-network are fixed, the face features are feature processed based on the spatiotemporal feature extraction network to obtain a first face composite feature including a time artifact feature and a space artifact feature; a first detection result of the face image is determined based on the first face composite feature and the face feature, and the first detection result is obtained based on the first detection result. The time weights in the time sub-network are optimized based on the test results and the corresponding face labels; when the parameters of the time sub-network are fixed, the face features are processed based on the spatiotemporal feature extraction network to obtain a second face composite feature containing temporal artifact features and spatial artifact features; the second detection result of the face image is determined based on the second face composite feature and the face features, and the spatial weights in the space sub-network are optimized based on the second detection result and the corresponding face label; when the detection model reaches the convergence condition, the face forgery detection is performed on the face image to be tested through the converged detection model to obtain the detection result. The converged detection model provided in the present application can simultaneously capture the temporal artifact features and spatial artifact features of the face image, and has higher generalization and stronger robustness. Therefore, when the face forgery detection method provided in the present application is performed on the face image to be tested through the converged detection model, it has more accurate detection results because it can simultaneously capture the temporal artifact features and spatial artifact features.
[0095] In one embodiment, when the parameters of the spatial sub-network are fixed, the number of times the time weights in the time sub-network are optimized is a first value;
[0096] When the parameters of the time sub-network are fixed, the number of times the time weight in the spatial sub-network is optimized is a second value; and the first value is greater than the second value.
[0097] The first value may be 5 to 20 times the second value.
[0098] Specifically, see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a detection model optimization process in one embodiment, as shown in FIG. Figure 3 As shown, the time weights in the time subnetwork and the space weights in the space subnetwork are optimized. In a specific implementation, the parameters of the space subnetwork may be fixed first, and the time weights in the time subnetwork may be optimized a first number of times. After the first number of times of optimization is completed, the parameters of the time subnetwork may be fixed again, and the time weights in the space subnetwork may be optimized a second number of times.
[0099] In this embodiment, the number of times the time weights in the time sub-network are optimized when the parameters of the spatial sub-network are fixed is greater than the number of times the time weights in the spatial sub-network are optimized when the parameters of the time sub-network are fixed. Therefore, the detection results are more dependent on the temporal artifact features obtained by the spatiotemporal feature extraction network through feature processing of facial features. By improving the feature processing capability of the detection model for temporal artifact features that are more difficult to detect, the credibility of the detection results is ensured.
[0100] In one embodiment, before sequentially inputting the facial images in the video training set into the detection model, the method further includes:
[0101] Get the original video set;
[0102] Performing facial key point detection on the first face image in the original video set in sequence to obtain a set of facial key points in the original video set;
[0103] Performing face tracking on the second face image in the original video set according to the face key point set to obtain a tracking result;
[0104] Based on the tracking results, the second face image is sequentially aligned to obtain a video training set.
[0105] Among them, the first face image in the original video set is played earlier than the second face image in the playback time sequence.
[0106] Face tracking is performed on the second face image in the original video set based on the set of facial key points. In a specific implementation, this can be to locate the face tracking result that occupies the largest area in the original video set, and other content that is not the face tracking result is not subjected to feature processing. The video training set with a reduced size reduces the computational load of the spatiotemporal feature extraction network for feature processing of facial features; based on the tracking results, face alignment processing is performed on the second face image in turn. In a specific implementation, this can be to make the face tracking result in the second face image have the same direction angle as the first face image or have only a slight error, which is beneficial to improving the efficiency of the spatiotemporal feature extraction network for feature processing of facial features.
[0107] In this embodiment, facial key point detection is performed on the first facial image in the original video set in sequence, facial tracking is performed on the second facial image in the original video set according to the facial key point set, and facial alignment is performed on the second facial image in sequence based on the tracking result to obtain a video training set. As a result, the facial images in the video training set only include a single character identity and the positions of the facial key points are aligned, thereby reducing the computational complexity of the spatiotemporal feature extraction network for feature processing of facial features while improving the feature processing efficiency.
[0108] In one embodiment, feature processing is performed on facial features based on a spatiotemporal feature extraction network to obtain a first composite facial feature including temporal artifact features and spatial artifact features, including:
[0109] Performing feature processing on facial features based on the temporal sub-network in the detection model to obtain first process features including temporal artifact features;
[0110] The first process feature is processed based on a spatial subnetwork with fixed parameters in the detection model to obtain a first face composite feature including a temporal artifact feature and a spatial artifact feature.
[0111] Among them, the first process feature is feature processed based on the spatial sub-network with fixed parameters in the detection model to obtain a first face composite feature containing temporal artifact features and spatial artifact features. In the specific implementation, it means that after the first process feature containing temporal artifact features enters the spatial sub-network, the spatial sub-network extracts the spatial features, and the spatial artifact features are extracted from the first process features, thereby obtaining the first face composite feature containing temporal artifact features and spatial artifact features; in this single optimization process, since the spatial sub-network does not include time-related parameters, the temporal artifact features will not be feature processed. Therefore, the temporal artifact features contained in the first process features and the temporal artifact features contained in the first face composite features are the same, and the temporal artifact features have not changed before and after.
[0112] In this embodiment, feature processing is performed on facial features based on the time sub-network in the detection model to obtain a first process feature including a time artifact feature, and feature processing is performed on the first process feature based on the space sub-network in the detection model with fixed parameters to obtain a first facial composite feature including a time artifact feature and a space artifact feature. By fixing only the parameters of the space sub-network, in the process of continuously optimizing the time weight of the detection model, only the time artifact feature changes in the first facial composite feature obtained, while the space artifact feature does not change, so that the time weight of the time sub-network can be continuously optimized according to the change of the time artifact feature.
[0113] In one embodiment, feature processing is performed on facial features based on a spatiotemporal feature extraction network to obtain a second composite facial feature containing temporal artifact features and spatial artifact features, including:
[0114] Performing feature processing on facial features based on a time sub-network with fixed parameters in the detection model to obtain a second process feature including a time artifact feature;
[0115] The second process features are processed based on the spatial sub-network in the detection model to obtain a second face composite feature including temporal artifact features and spatial artifact features.
[0116] In this embodiment, feature processing is performed on facial features based on the time sub-network with fixed parameters in the detection model to obtain a second process feature including a time artifact feature. Feature processing is performed on the second process feature based on the space sub-network in the detection model to obtain a second facial composite feature including a time artifact feature and a space artifact feature. By fixing only the parameters of the time sub-network, in the process of continuously optimizing the time weight of the detection model, only the spatial artifact feature changes in the second facial composite feature obtained, and the time artifact feature does not change, so that the spatial weight of the spatial sub-network can be continuously optimized according to the change of the spatial artifact feature.
[0117] In one embodiment, the method further includes:
[0118] If the detection result shows that the face image to be tested is authentic, the face image to be tested is matched with the preset face image;
[0119] When the face image to be tested successfully matches the preset face image, the current payment operation is confirmed to be executed.
[0120] Among them, the preset facial image, in a specific implementation, can be a facial image pre-entered as identity identification information by the user who wants to perform the current payment operation.
[0121] The current payment operation, in a specific implementation, may be a payment operation involving money transfer that a user intends to perform on a shopping software, payment software, or other software on a terminal.
[0122] In this embodiment, when the detection result shows that the facial image to be tested is authentic and the facial image to be tested successfully matches the preset facial image, the current payment operation is confirmed to be executed, thereby ensuring that the current payment operation is an authorized operation, thereby ensuring the security of the current payment operation.
[0123] The following is a detailed example of how to apply the above-mentioned method for detecting forgery of facial images. Figure 4 , Figure 4 FIG. 1 is a flow chart of a method for detecting forgery of a face image in another embodiment. Figure 4As shown, the user hopes to train a detection model that can capture both temporal artifact features and spatial artifact features. Therefore, the user uses the terminal to input the face images in the video training set into the detection model in sequence. The detection model is a spatiotemporal feature extraction network. The spatiotemporal feature extraction network includes a temporal subnetwork and a spatial subnetwork. The face features are extracted from the input face images in sequence through the spatiotemporal feature extraction network. Specifically, the optimization process of the detection model is as follows: in an optimization cycle, when the parameters of the spatial subnetwork are first fixed, the face features are processed based on the spatiotemporal feature extraction network to obtain a first face composite feature containing temporal artifact features and spatial artifact features; a first detection result of the face image is determined based on the first face composite feature and the face feature. The time weight in the time sub-network is optimized according to the first detection result and the corresponding face label, and the number of times the time weight in the time sub-network is optimized is a first value; after completing the time weight optimization for the first number of times, the fixed state of the spatial sub-network is released, and then the parameters of the time sub-network are fixed, and the face features are processed based on the spatiotemporal feature extraction network to obtain a second face composite feature including temporal artifact features and spatial artifact features; the second detection result of the face image is determined according to the second face composite feature and the face feature, and the spatial weight in the spatial sub-network is optimized according to the second detection result and the corresponding face label, and the number of times the spatial weight of the spatial sub-network is optimized is a second value, and the first value is greater than the second value. After completing the second numerical number of spatial weight optimization, determine whether the detection model has reached the convergence condition. If the detection model has reached the convergence condition, it is determined that the detection model optimization is completed. The user inputs the face image to be tested into the converged detection model on the terminal to perform face forgery detection and obtain the detection result. If the detection model has not reached the convergence condition, the optimization process of the next optimization cycle is started, that is, a new round of optimization of the time weight in the time subnetwork for the first numerical number of times and optimization of the spatial weight in the spatial subnetwork for the second numerical number of times is performed until the detection model reaches the convergence condition and can be used for face forgery detection to obtain the detection result.
[0124] The solution of the present application optimizes the temporal weight by fixing the parameters of the spatial sub-network respectively, and optimizes the spatial weight by fixing the parameters of the temporal sub-network, and finally obtains a convergent detection model that can simultaneously capture the temporal artifact features and spatial artifact features of facial images, thereby improving the accuracy of the detection results corresponding to facial forgery detection.
[0125] Based on the same inventive concept, embodiments of the present application also provide a facial image forgery detection device for implementing the aforementioned facial image forgery detection method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of the facial image forgery detection device provided below can be found in the aforementioned limitations of the facial image forgery detection method, and will not be further elaborated here.
[0126] In one embodiment, see Figure 5 , Figure 5 The structural block diagram of a facial image forgery detection device in one embodiment is as follows: Figure 5 As shown, a facial image forgery detection device is provided, comprising: an input unit 1002, an extraction unit 1004, a first processing unit 1006, a second processing unit 1008 and a detection unit 1010, wherein:
[0127] Input unit 1002, used to input the face images in the video training set into the detection model in sequence; the detection model is a spatiotemporal feature extraction network, which includes a time subnetwork and a space subnetwork;
[0128] Extraction unit 1004, configured to extract facial features from the input facial image in sequence through a spatiotemporal feature extraction network;
[0129] A first processing unit 1006 is configured to perform feature processing on facial features based on the spatiotemporal feature extraction network while fixing the parameters of the spatial sub-network to obtain a first composite facial feature including a temporal artifact feature and a spatial artifact feature; determine a first detection result of the facial image based on the first composite facial feature and the facial feature; and optimize the temporal weight in the temporal sub-network based on the first detection result and the corresponding face label;
[0130] A second processing unit 1008 is configured to perform feature processing on facial features based on the spatiotemporal feature extraction network when the parameters of the temporal sub-network are fixed, to obtain a second composite facial feature including temporal artifact features and spatial artifact features; determine a second detection result of the facial image based on the second composite facial feature and the facial features; and optimize the spatial weights in the spatial sub-network based on the second detection result and the corresponding facial label;
[0131] The detection unit 1010 is used to perform face forgery detection on the face image to be tested using the converged detection model when the detection model reaches a convergence condition to obtain a detection result.
[0132] In one embodiment, when the parameters of the spatial sub-network are fixed, the first processing unit 1006 optimizes the time weights in the time sub-network a number of times equal to a first value;
[0133] When the parameters of the time sub-network are fixed, the second processing unit 1008 optimizes the time weights in the space sub-network a second number of times; the first number is greater than the second number.
[0134] In one embodiment, see Figure 6 , Figure 6 FIG. 1 is a structural block diagram of a facial image forgery detection device in another embodiment. Figure 6 As shown, in one embodiment, the apparatus further includes a preprocessing unit 1000, which is configured to perform the following steps before sequentially inputting the face images in the video training set into the detection model:
[0135] Get the original video set;
[0136] Performing facial key point detection on the first face image in the original video set in sequence to obtain a set of facial key points in the original video set;
[0137] Performing face tracking on the second face image in the original video set according to the face key point set to obtain a tracking result;
[0138] Based on the tracking results, the second face image is sequentially aligned to obtain a video training set.
[0139] In one embodiment, the first processing unit 1006 includes:
[0140] A first time module is used to perform feature processing on facial features based on the time sub-network in the detection model to obtain a first process feature including a time artifact feature;
[0141] The first spatial module is used to perform feature processing on the first process feature based on the spatial sub-network with fixed parameters in the detection model to obtain a first facial composite feature including a temporal artifact feature and a spatial artifact feature.
[0142] In one embodiment, the second processing unit 1008 includes:
[0143] A second time module is used to perform feature processing on facial features based on a time sub-network with fixed parameters in the detection model to obtain a second process feature including a time artifact feature;
[0144] The second spatial module is used to perform feature processing on the second process feature based on the spatial sub-network in the detection model to obtain a second facial composite feature including a temporal artifact feature and a spatial artifact feature.
[0145] In one embodiment, Figure 5 As shown, the above apparatus further includes an execution unit 1012, and the execution unit 1012 is configured to:
[0146] If the detection result shows that the face image to be tested is authentic, the face image to be tested is matched with the preset face image;
[0147] When the face image to be tested successfully matches the preset face image, the current payment operation is confirmed to be executed.
[0148] Each module in the aforementioned facial image forgery detection device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.
[0149] In one embodiment, a computer device is provided, which may be a server. Figure 7 , Figure 7 The internal structure diagram of a computer device in one embodiment can be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store face label data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting forgery of a face image is implemented.
[0150] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for detecting forgery of a facial image is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0151] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0152] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.
[0153] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0154] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.
[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.
[0157] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0158] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.
Claims
1. A method for detecting forgery of a face image, characterized in that: The method comprises: Inputting the face images in the video training set into the detection model in sequence; the detection model is a spatiotemporal feature extraction network, which includes a time subnetwork and a space subnetwork; Extracting facial features from the input facial images in sequence through the spatiotemporal feature extraction network; When the parameters of the spatial sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a first composite facial feature including a temporal artifact feature and a spatial artifact feature; a first detection result of the facial image is determined based on the first composite facial feature and the facial feature, and a time weight in the temporal sub-network is optimized based on the first detection result and the corresponding face label; When the parameters of the temporal sub-network are fixed, feature processing is performed on the facial features based on the spatiotemporal feature extraction network to obtain a second composite facial feature including a temporal artifact feature and a spatial artifact feature; a second detection result of the facial image is determined based on the second composite facial feature and the facial feature, and the spatial weights in the spatial sub-network are optimized based on the second detection result and the corresponding face label; When the detection model reaches a convergence condition, face forgery detection is performed on the face image to be tested using the converged detection model to obtain a detection result.
2. The method according to claim 1, characterized in that When the parameters of the spatial sub-network are fixed, the number of times the time weights in the time sub-network are optimized is a first value; When the parameters of the time sub-network are fixed, the number of times the time weight in the space sub-network is optimized is a second value; and the first value is greater than the second value.
3. The method according to claim 1, characterized in that Before sequentially inputting the facial images in the video training set into the detection model, the method further includes: Get the original video set; Performing facial key point detection on the first face image in the original video set in sequence to obtain a set of facial key points in the original video set; Performing face tracking on the second face image in the original video set according to the face key point set to obtain a tracking result; Based on the tracking result, face alignment processing is performed on the second face image in sequence to obtain the video training set.
4. The method according to claim 1, wherein The facial features are processed based on the spatiotemporal feature extraction network to obtain a first composite facial feature including a temporal artifact feature and a spatial artifact feature, including: Performing feature processing on the facial features based on the time sub-network in the detection model to obtain a first process feature including a time artifact feature; The first process feature is subjected to feature processing based on the spatial subnetwork with fixed parameters in the detection model to obtain the first composite face feature including the temporal artifact feature and the spatial artifact feature.
5. The method according to claim 1, wherein The facial features are processed based on the spatiotemporal feature extraction network to obtain a second facial composite feature including a temporal artifact feature and a spatial artifact feature, including: Performing feature processing on the facial features based on the time sub-network with fixed parameters in the detection model to obtain a second process feature including a time artifact feature; The second process feature is subjected to feature processing based on the spatial sub-network in the detection model to obtain the second face composite feature including the temporal artifact feature and the spatial artifact feature.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: If the detection result indicates that the face image to be tested is authentic, matching the face image to be tested with a preset face image; When the face image to be tested successfully matches the preset face image, the execution of the current payment operation is confirmed.
7. A facial image forgery detection device, characterized in that: The device comprises: An input unit, configured to sequentially input facial images from a video training set into a detection model; the detection model is a spatiotemporal feature extraction network comprising a temporal subnetwork and a spatial subnetwork; An extraction unit, configured to extract facial features from an input facial image in sequence through the spatiotemporal feature extraction network; a first processing unit, configured to perform feature processing on the facial features based on the spatiotemporal feature extraction network while fixing the parameters of the spatial sub-network to obtain a first composite facial feature including a temporal artifact feature and a spatial artifact feature; determine a first detection result of the facial image based on the first composite facial feature and the facial feature, and optimize the temporal weight in the temporal sub-network based on the first detection result and the corresponding face label; a second processing unit, configured to perform feature processing on the facial features based on the spatiotemporal feature extraction network while fixing the parameters of the temporal sub-network to obtain a second composite facial feature comprising a temporal artifact feature and a spatial artifact feature; determine a second detection result of the facial image based on the second composite facial feature and the facial feature, and optimize the spatial weights in the spatial sub-network based on the second detection result and the corresponding face label; The detection unit is used to perform face forgery detection on the face image to be tested by using the converged detection model when the detection model reaches a convergence condition to obtain a detection result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.