Region classification-based counterfeited face tracing method and device, medium and equipment

Through the fake face traceability method based on region classification, using coarse-grained and fine-grained network models combined with attention fusion modules, the facial image features are extracted and decision-level fusion is carried out, which solves the low accuracy and complex forgery traceability in the existing technology. The problem of insufficient strategic resistance has achieved higher traceability accuracy and fine-grained forgery identification.

CN118942133AActive Publication Date: 2024-11-12WUHAN UNIV

Patent Information

Application Number
CN202410940639.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-12
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing forgery traceability algorithms are not accurate in terms of accuracy and are difficult to resist complex forgery strategies, especially when full-picture forgery and regional forgery coexist.

Method used

Using a fake face traceability method based on region classification, face analysis and feature extraction of face pictures through coarse-grained traceability network model, combined with channel attention and spatial attention fusion modules to fusion feature level and decision-making level to generate coarse-grained Trace the decision vector and make fake type judgments at the fine-grained level.

Benefits of technology

It improves the accuracy of face traceability, enhances the ability to resist complex forgery strategies, and can conduct accurate fine-grained forgery traceability when full-picture forgery and regional forgery coexist.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942133A_ABST
    Figure CN118942133A_ABST
Patent Text Reader

Abstract

The invention discloses a counterfeited face tracing method and device based on region classification, a medium and equipment. The method comprises the following steps: acquiring a face picture, wherein the face picture comprises a regional forged picture, a global forged picture and a real picture; performing face analysis on the face picture to obtain a face region picture, a background region picture and a connection region picture; performing traceability feature extraction on the face picture to obtain a global feature and a global feature decision vector; traceability feature extraction is carried out on the face region picture, the background region picture and the connection region picture to obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors and connection region feature decision vectors; performing feature level fusion and decision level fusion on the features and the feature decision vector to obtain a coarse-grained traceability decision vector; and inputting the coarse-grained traceability decision vector into a classifier, and predicting a traceability result. According to the invention, the accuracy of face traceability can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of security technology, and in particular to a method, device, storage medium and electronic device for tracing the source of a forged face based on region classification. Background Art

[0002] With the continuous advancement of face forgery technology in recent years, forgery face defense technology has become increasingly critical. Forgery face defense technology includes detection and tracing: forgery detection is to judge the authenticity of face images, which is a binary classification task, and forgery tracing is to determine the specific forgery model of forged face images, which is a multi-classification task. Compared with forgery detection, forgery tracing can provide more accurate forgery-related information for judicial investigation, which is of great significance for improving the credibility of physical evidence identification.

[0003] With the development of deep learning in the field of computer vision, research on forgery tracing algorithms has made significant progress and achieved remarkable results. For example, forgery tracing is based on the unique "fingerprint" left by the generative adversarial network (GAN). For another example, due to the observation that the generative adversarial network (GAN) structure leaves a globally consistent fingerprint in the process of generating images, while the traces left by the model weights are different in different regions, DNA-Det is proposed to trace the architecture of forged images to their source models, as well as the feature decomposition based on the attention mechanism to perform three-class face manipulation detection method (TFMD).

[0004] The aforementioned counterfeit tracing algorithms have poor accuracy and are not able to resist complex counterfeiting strategies. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for tracing the source of a forged face based on region classification, which can improve the accuracy of face tracing.

[0006] The embodiment of the present application provides a method for tracing the source of a forged face based on region classification, comprising: Acquire a face picture, wherein the face picture includes a regional forged picture, a global forged picture, and a real picture; Constructing a coarse-grained traceability network model, inputting the face image into the trained coarse-grained traceability network model, and predicting the traceability result; The regional forged image, the global forged image and the real image are input into a trained coarse-grained traceability network model to predict the traceability result, including: Performing face analysis on the face image to obtain a face area image, a background area image, and a connection area image; Performing source feature extraction on the face image to obtain global features and a global feature decision vector, and performing source feature extraction on the face region image, the background region image, and the connection region image, respectively, to obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors; Performing feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector and the connection region feature decision vector to obtain a coarse-grained tracing decision vector; The coarse-grained tracing decision vector is input into a classifier to predict the tracing result.

[0007] Furthermore, in the forged face tracing method based on region classification, the coarse-grained tracing model includes a face parsing module, and the face parsing module is embedded with a BiSeNet network; The face analysis of the face image to obtain a face area image, a background area image and a connection area image includes: Inputting the face image into the BiSeNet network to obtain a face contour mask; Performing an erosion operation on the face contour mask to obtain a face region mask, and obtaining a face region image based on the face region mask and the face image; Performing an expansion operation on the face contour mask and negating it to obtain a background area mask, and obtaining a background area image based on the background area mask and the face image; A logical AND operation is performed on the inversion of the face contour mask and the inversion of the background area mask to obtain a connected area mask, and a connected area image is obtained based on the connected area mask and the face image.

[0008] Furthermore, in the forged face tracing method based on region classification, the coarse-grained tracing model includes a region classification fusion module, the region classification fusion module includes a feature fusion module and a decision fusion module, and the feature fusion module includes a channel attention fusion submodule and a spatial attention fusion submodule; The step of performing feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector, and the connection region feature decision vector to obtain a coarse-grained tracing decision vector includes: Inputting the face region feature, the background region feature, and the connection region feature into the channel attention fusion submodule to perform channel attention feature fusion to obtain three-region channel attention fusion features; Inputting the three-region channel attention fusion features and the global features into the spatial attention fusion submodule to perform spatial attention feature fusion, and obtaining a first decision vector of feature-level fusion; The global feature decision vector, the face area feature decision vector, the background area feature decision vector, the connection area feature decision vector and the first decision vector are input into the decision fusion module for decision fusion to obtain a coarse-grained tracing decision vector of decision-level fusion.

[0009] Furthermore, in the forged face tracing method based on regional classification, the coarse-grained tracing network model includes a tracing feature extraction module, which is used to extract tracing features from face images, and the tracing feature extraction module includes a global tracing feature extractor, a face region tracing feature extractor, a background region tracing feature extractor, and a connection region tracing feature extractor; The trained coarse-grained traceability network model is trained based on a coarse-grained overall loss function, which is:

[0010] in, is the legitimacy loss function, is the cross entropy loss function of the background region traceability feature extractor, is the cross entropy loss function of the face region traceability feature extractor, is the cross entropy loss function of the connected region tracing feature extractor, is the cross entropy loss function of the global provenance feature extractor.

[0011] Furthermore, in the forged face tracing method based on region classification, the legitimacy loss function is:

[0012] in, , and are the probability values ​​of the true labels of the background region tracing feature extractor, the face region tracing feature extractor, and the connection region tracing feature extractor, respectively. , and are the probability values ​​of false labels of the background region tracing feature extractor, the face region tracing feature extractor and the connection region tracing feature extractor, respectively. is the probability value of the true and false mixed labels of the connected area tracing feature extractor.

[0013] Furthermore, the forged face tracing method based on region classification, after the step of inputting the coarse-grained tracing decision vector into the classifier to predict the tracing result, comprises: If the traceability result is regional forgery or global forgery, then regional forgery and global forgery judgment are performed; The regional and full-image forgery judgment includes: Construct a fine-grained traceability network model, which includes a coarse-grained traceability feature fusion module, a regional forgery fine-grained traceability module, and a full-map forgery fine-grained traceability module. The coarse-grained traceability feature fusion module includes a coarse-grained fusion submodule and a coarse-grained weight fusion submodule. Inputting the global features, the face region features, the background region features and the connection region features into a trained coarse-grained fusion module for feature fusion to obtain a coarse-grained fusion feature; Inputting the global features into the regional forgery fine-grained tracing module or the full-image forgery fine-grained tracing module to extract tracing features to obtain fine-grained features; Inputting the coarse-grained fusion feature and the fine-grained feature into the coarse-grained and fine-grained weight fusion submodule for weight fusion to obtain a weight fusion feature; The weighted fusion features are input into the classifier to obtain specific tracing results.

[0014] Furthermore, in the forged face tracing method based on regional classification, the fine-grained tracing network model includes a regional forged fine-grained tracing network model and a full-image forged fine-grained tracing network model, and the fine-grained tracing network model is trained based on a two-stage loss function; The first-stage loss function is:

[0015] in, is the consistency loss, is the weighted cross entropy loss function of the regional forgery fine-grained provenance network model, Forge the weighted cross entropy loss function of the fine-grained traceability network model for the entire graph; The second stage loss function is:

[0016] in, Predict the sample belongs to the category for the coarse-grained traceability network model The probability of Forging a fine-grained provenance network model for the entire image predicts that the sample belongs to a category The probability of Forging a fine-grained provenance network model for a region predicts that the sample belongs to a category The probability of , , are the values ​​of the corresponding true labels in their respective true label vectors, is the two-stage loss function.

[0017] The embodiment of the present application also provides a forged face tracing device based on region classification, comprising: An acquisition module, used to acquire a face picture, wherein the face picture includes a regional forged picture, a global forged picture and a real picture; A face analysis module, used for performing face analysis on the face image to obtain a face area image, a background area image and a connection area image; A traceability feature extraction module is used to perform traceability feature extraction on the face image to obtain global features and global feature decision vectors, and to perform traceability feature extraction on the face area image, the background area image and the connection area image to obtain face area features, background area features, connection area features, face area feature decision vectors, background area feature decision vectors and connection area feature decision vectors; A region classification fusion module is used to perform feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector and the connection region feature decision vector to obtain a coarse-grained tracing decision vector; The prediction module is used to input the coarse-grained tracing decision vector into the classifier to predict the tracing result.

[0018] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored. The instructions are suitable for being loaded by a processor to execute any of the forged face tracing methods based on region classification described above.

[0019] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any of the above-mentioned methods for tracing the source of a forged face based on regional classification.

[0020] The method, device, storage medium and electronic device for tracing the forged face based on region classification provided by the present application respectively extract the features of the face image and the features of the three regions, fuse the two at the feature level and the decision level to obtain a coarse-grained tracing decision vector, and predict the tracing result through the coarse-grained tracing decision vector. The present application makes full use of the complementary information at different levels such as the region and the whole image, the decision level and the feature level, thereby realizing coarse-grained forged face tracing, improving the accuracy of face tracing, and having strong resistance to complex forgery strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0022] Figure 1 A flowchart of a method for tracing the origin of a forged face based on region classification provided in an embodiment of the present application.

[0023] Figure 2 Another flowchart of the forged face tracing method based on region classification provided in an embodiment of the present application.

[0024] Figure 3 A flowchart of the face parsing process provided in an embodiment of the present application.

[0025] Figure 4 A flow chart for generating channel attention fusion features provided in an embodiment of the present application.

[0026] Figure 5 A flowchart for generating a first decision vector provided in an embodiment of the present application.

[0027] Figure 6 A flowchart for generating a coarse-grained tracing decision vector provided in an embodiment of the present application.

[0028] Figure 7 A flowchart for determining regional forgery and full-image forgery provided in an embodiment of the present application.

[0029] Figure 8 A flowchart for generating coarse-grained fusion features provided in an embodiment of the present application.

[0030] Fig. 9 A flowchart for fine-grained feature generation provided in an embodiment of the present application.

[0031] Fig.10 A schematic diagram of the structure of a forged face tracing device based on region classification provided in an embodiment of the present application.

[0032] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0033] Fig.12 Another schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0035] The embodiment of the present application provides a method, device, storage medium and electronic device for tracing the source of a fake face based on regional classification. The embodiment of the present application provides a device for tracing the source of a fake face based on regional classification, which can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0036] Current forgery tracing algorithms mostly focus on either full-image forgery or regional forgery, ignoring the coexistence of full-image forgery and regional forgery, and failing to consider the differences in the technical principles of these two basic forgery types, making it difficult to adapt to actual application scenarios where both types of forgery coexist. In addition, the accuracy of current forgery tracing algorithms is poor, and their ability to resist complex forgery strategies is insufficient.

[0037] In order to solve the above technical problems, the present application provides a method, device, storage medium and electronic device for tracing the source of forged faces based on regional classification. Figure 1 and Figure 2 , Figure 1 A flowchart of a method for tracing the origin of a forged face based on regional classification provided in an embodiment of the present application. Figure 2 Another flow chart of a forged face tracing method based on region classification provided in an embodiment of the present application, which is applied to an electronic device, and the forged face tracing method based on region classification includes the following steps: S1, obtaining face images, which include regional forged images, global forged images and real images.

[0038] Specifically, some face images are obtained from video files. Several frames of the video files are randomly sampled as image data. All images are unified to a scale of 224×224 and saved in jpg format.

[0039] In order to reduce the impact of external factors on the algorithm, all images were cropped to only leave the face. All data were processed with a random compression factor of 50 to 90. The random compression factor was used to process the data to simulate the common compression conditions in actual network communication. This can make the training set and test set closer to the real environment when training the model, which can improve the generalization ability and practicality of the model.

[0040] S2, build a coarse-grained tracing network model, input the face image into the trained coarse-grained tracing network model, and predict the tracing result.

[0041] Specifically, the coarse-grained tracing network model includes a face parsing module, a tracing feature extraction module and a regional classification fusion module. First, the face parsing module obtains three-region images: face region image, background region image and connection region image. Secondly, the tracing feature extraction module extracts the features of the three-region images. Finally, the regional classification fusion module fuses the three-region feature features of the global branch and the regional branch to obtain a coarse-grained tracing decision vector, which is input into the classifier to predict the tracing result.

[0042] Step S2 includes the following steps: S21, performing face analysis on the face image to obtain a face area image, a background area image, and a connection area image.

[0043] In one embodiment, step S2 is implemented by a face analysis module, and the face analysis module is embedded with a BiSeNet network. Figure 3 A flowchart of the face analysis process provided in the embodiment of the present application is as follows: Figure 3 As shown, step S21 includes the following steps: Perform face analysis on the face image to obtain the face area image, background area image and connection area image, including: S211, input the face image into the BiSeNet network (Bilateral Segmentation Network) to obtain the face contour mask.

[0044] It can be specifically expressed by the following formula:

[0045] in, It's a face picture. is the face contour mask.

[0046] S212, performing an erosion operation on the face contour mask to reduce the boundary, obtaining a face region mask, ensuring that the face region mask does not include an edge portion, and obtaining a face region image based on the face region mask and the face image.

[0047] It can be specifically expressed by the following formula:

[0048]

[0049] in, is a corrosion operation, is the face region mask, is a pixel-by-pixel multiplication operation, The face area image.

[0050] S213, dilate the face contour mask and invert it to obtain a background area mask, and obtain a background area image based on the background area mask and the face image.

[0051] It can be specifically expressed by the following formula:

[0052]

[0053] in, is the expansion operation, is the negation operation, is the background area mask, The background area image.

[0054] S214, performing a logical AND operation on the inversion of the face contour mask and the inversion of the background area mask to obtain a connected area mask, and obtaining a connected area image based on the connected area mask and the face image.

[0055] It can be specifically expressed by the following formula:

[0056]

[0057] in, is the logical AND operation, is the connection area mask, This is a picture of the connection area.

[0058] S22, perform source feature extraction on the face image to obtain global features and global feature decision vectors, perform source feature extraction on the face region image, background region image and connection region image respectively, and obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors and connection region feature decision vectors.

[0059] Specifically, step S22 is implemented by a traceability feature extraction module. The traceability feature extraction module includes 12 convolutional layers, 5 pooling layers, and 3 fully connected layers. Each convolution block includes a convolutional layer, batch normalization, and a RULU activation function. After passing through the conv12 layer of the network, a 14×14×512-dimensional feature map (including face area features, background area features, and connection area features) is obtained. This feature map is used as the intermediate feature of the traceability feature extraction module. After passing through the fully connected layer fc2, a 1×1×512-dimensional vector (including face area feature decision vector, background area feature decision vector, and connection area feature decision vector) is output. The structure of the traceability feature extraction module is shown in Table 1 below.

[0060] Table 1 Structure of the traceability feature extraction module Number of layers Layer Name Operation Type Input size Output size Number of channels 1-2 conv1conv2 Conv+BN+ReLU 224×224 224×224 64 3 pool1 MaxPool 224×224 112×112 64 4-5 conv3conv4 Conv+BN+ReLU 112×112 112×112 128 6 pool2 MaxPool 112×112 56×56 128 7-8 conv5conv6 Conv+BN+ReLU 56×56 56×56 256 9 pool3 MaxPool 56×56 28×28 256 10-11 conv7conv8 Conv+BN+ReLU 28×28 28×28 512 12 pool4 MaxPool 28×28 14×14 512 13-14 conv9conv10 Conv+BN+ReLU 14×14 14×14 512 15-16 conv11conv12 Conv+BN+ReLU 14×14 14×14 512 17 pool5 MaxPool 14×14 7×7 512 18 fc1 FC+ReLU 7×7×512 4096 N / A 19 fc2 FC 4096 512 N / A 20 fc3 FC+Softmax 512 C N / A The coarse-grained provenance network model is a two-branch network model. The global branch includes a provenance feature extraction module, which extracts the provenance features of the face image through the global provenance feature extractor to obtain the global features, and outputs the global feature decision vector through the fully connected layer. The regional branch includes a provenance feature extraction module, which extracts the face region features, background region features, and connection region features through the face region provenance feature extractor, background region provenance feature extractor, and connection region provenance feature extractor to obtain the face region features, background region features, and connection region features, and outputs the face region feature decision vector, background region feature decision vector, and connection region feature decision vector.

[0061] S23, performing feature-level fusion and decision-level fusion on the global features, face region features, background region features, connection region features, global feature decision vectors, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors to obtain a coarse-grained tracing decision vector; Specifically, step S23 is implemented by a regional classification fusion module, which includes a feature fusion module and a decision fusion module, and the feature fusion module includes a channel attention fusion submodule and a spatial attention fusion submodule. In one embodiment, step S23 includes the following steps: S231, input the face region features, background region features, and connection region features into the channel attention fusion submodule to perform channel attention feature fusion to obtain three-region channel attention fusion features.

[0062] Figure 4 For a flow chart of channel attention fusion feature generation provided in the embodiment of the present application, please refer to Figure 4 , step S231 can be specifically expressed by the following formula:

[0063] in, is the background area feature, is the face region feature, Connecting regional features, is the channel attention fusion operation, It is the three-region channel attention fusion feature.

[0064] S232, input the three-region channel attention fusion features and the global features into the spatial attention fusion submodule for spatial attention feature fusion, and obtain the first decision vector of feature-level fusion.

[0065] Figure 5 For a flowchart of generating the first decision vector provided in the embodiment of the present application, please refer to Figure 5 , step S232 can be specifically expressed by the following formula:

[0066] in, is a global feature, is the spatial attention feature fusion operation, is the first decision vector of feature fusion.

[0067] S233, input the global feature decision vector, the face area feature decision vector, the background area feature decision vector, the connection area feature decision vector and the first decision vector into the decision fusion module for decision fusion, and obtain a coarse-grained tracing decision vector of decision-level fusion.

[0068] Figure 6 The flowchart of the coarse-grained tracing decision vector generation provided in the embodiment of the present application can be found in Figure 6 , step 233 can be specifically expressed by the following formula:

[0069] in, is the global feature decision vector, is the background region feature decision vector, is the face region feature decision vector, is the connection region feature decision vector, is the decision fusion network (the decision fusion module uses the decision fusion network), is the coarse-grained provenance decision vector.

[0070] S24, input the coarse-grained tracing decision vector into the classifier to predict the tracing result.

[0071] Specifically, the classifier includes multiple fully connected layers, and the classifier determines whether the face image is a real image, a regional forgery, or a full-image forgery.

[0072] The above describes the application process of the coarse-grained provenance network model. The following describes the training process of the coarse-grained provenance network model: (1) Obtain a dataset of face images, including regional forged images, global forged images, and real images, and annotate them with real labels; (2) Input the face image dataset into the coarse-grained traceability network model to obtain the predicted traceability results; (3) Iterative training is performed based on the traceability results, true labels, and the coarse-grained overall loss function to obtain a trained coarse-grained traceability network model.

[0073] In one embodiment, the coarse-grained overall loss function is:

[0074] in, is the legitimacy loss function, is the cross entropy loss function of the background region traceability feature extractor, is the cross entropy loss function of the face region traceability feature extractor, is the cross entropy loss function of the connected region tracing feature extractor, is the cross entropy loss function of the global provenance feature extractor.

[0075] In one embodiment, the legitimacy loss function is:

[0076] in, , and are the probability values ​​of the true labels of the background region tracing feature extractor, the face region tracing feature extractor, and the connection region tracing feature extractor, respectively. , and are the probability values ​​of false labels of the background region tracing feature extractor, the face region tracing feature extractor and the connection region tracing feature extractor, respectively. is the probability value of the true and false mixed labels of the connected area tracing feature extractor.

[0077] In addition to the legitimacy loss of the three regional traceability feature extractors, these traceability feature extractors also require independent cross entropy losses. At the same time, the global traceability feature extractor of the global branch does not interact with these three regional traceability feature extractors, and it only requires its own cross entropy loss function. The four cross entropy loss functions are as follows: The loss function of the background area tracing feature extractor is:

[0078] The loss function of the face region traceability feature extractor is:

[0079] The loss function of the connected region traceability feature extractor is:

[0080] The loss function of the global provenance feature extractor is:

[0081] in, , , , The background region traceability feature extractor predicts that the sample belongs to the category The probability that the face region traceability feature extractor predicts that the sample belongs to the category The probability of the connected region tracing feature extractor predicting that the sample belongs to the category The probability that the global traceability feature extractor predicts that the sample belongs to the category The probability of is the true label, if the sample belongs to the category ,but =1 if the value is set to 0, otherwise it is 0.

[0082] In one embodiment, after step S2, the method further includes the following steps: S3, if the tracing result is regional forgery or global forgery, regional forgery and global forgery judgment are performed.

[0083] Figure 7 The flowchart for determining regional forgery and full image forgery provided in the embodiment of the present application is as follows: Figure 7 As shown, the fine-grained traceability network model is also a parallel network model, which has a regional forged fine-grained traceability network model and a full-map forged fine-grained traceability network model. It should be noted that when the regional forged fine-grained traceability network model is true, the full-map forged fine-grained traceability network model should also be identified as true, and vice versa. Similarly, the regional forged fine-grained traceability network model also includes a traceability feature extraction module and a classifier, and the full-map forged fine-grained traceability network model also includes a traceability feature extraction module and a classifier.

[0084] Specifically, is the input face image, is the classification function of the coarse-grained provenance network, which classifies images into real (Real), regional forgery (RegionalFake), or full-image forgery (EntireFake). The result determines whether to enter fine-grained traceability or .if , then the classification ends. , then we enter the fine-grained classification of regional forgeries. , then enter the fine-grained classification of full-image forgery . Define the overall traceability function for:

[0085] Regional and full-map forgery judgment includes: S31, construct a fine-grained traceability network model. The fine-grained traceability network model includes a coarse-fine-grained traceability feature fusion module, a regional forgery fine-grained traceability module and a full-map forgery fine-grained traceability module. The coarse-fine-grained traceability feature fusion module includes a coarse-grained fusion sub-module and a coarse-fine weight fusion sub-module.

[0086] S32, inputting the global features, face region features, background region features and connection region features into the trained coarse-grained fusion module for feature fusion to obtain coarse-grained fusion features.

[0087] Figure 8 The flowchart of the coarse-grained fusion feature generation provided in the embodiment of the present application is as follows: Figure 8 As shown, step S32 can be expressed by the following formula:

[0088] in, represents the coarse-grained fusion submodule, Represents coarse-grained fusion features.

[0089] S33, inputting the global features into the regional forgery fine-grained tracing module or the whole-image forgery fine-grained tracing module to extract the tracing features and obtain the fine-grained features.

[0090] S34, inputting the coarse-grained fusion features and the fine-grained features into the coarse-grained and fine-grained weight fusion submodule for weight fusion to obtain weight fusion features.

[0091] Fig. 9 The flowchart of the fine-grained feature generation provided in the embodiment of the present application is as follows: Fig. 9 As shown, step S34 can be expressed by the following formula:

[0092] in, The fine-grained features generated by the regional forged fine-grained traceability module or the full-image forged fine-grained traceability module, is the coarse and fine weight fusion submodule, is the weighted fusion feature.

[0093] S35, input the weighted fusion features into the classifier to obtain specific tracing results.

[0094] The above describes the application process of the fine-grained traceability network model. The fake face traceability network model includes the coarse-grained traceability network model and the fine-grained traceability network model. The following describes the training process of the fake face traceability network model: (1) Obtain a dataset of face images, including regional forged images, global forged images, and real images, and annotate them with real labels; (2) Input the face image dataset into the fake face tracing network model to obtain the predicted tracing results; (3) Iterative training is performed based on the tracing results, true labels, and the overall loss function to obtain a trained fake face tracing network model.

[0095] In one embodiment, the overall loss function includes two stages of loss functions.

[0096] The first-stage loss function is:

[0097] in, is the consistency loss, is the weighted cross entropy loss function of the regional forgery fine-grained provenance network model, Weighted cross entropy loss function for forging fine-grained provenance network model for the whole graph.

[0098] Among them, the consistency loss is:

[0099] in, is the indicator function, which means when Belongs to the collection Returns 1 if yes, otherwise returns 0. and They are the softmax output vectors of the full-image forgery traceability network model and the regional forgery traceability network model, which contain the prediction probabilities of each category. , and They represent the full-image forged model set, the regional forged model set and the true result set respectively. is the probability that the regional forgery tracing network prediction model is a full-map forgery, is the probability that the full-image forgery traceability network prediction model is a regional forgery, and It is the probability that the regional forged fine-grained provenance network model and the full-map forged fine-grained provenance network model predict the true result.

[0100] The weighted cross entropy loss emphasizes the importance of the fine-grained traceability network model to the real and coarse-grained forgery types, and improves the accuracy of reverse correction by assigning higher weights to these two categories. The weighted cross entropy of the two fine-grained traceability networks is:

[0101] in, is the weighted cross entropy loss function of the regional forgery fine-grained provenance network model, is the number of output categories of the network, Yes Category The weights of the whole-image forgery and authenticity are set to 2 in the regional forgery fine-grained traceability network model, and the weights of the other categories are set to 1. The fine-grained traceability network model predicts that the sample belongs to the category probability. is the corresponding category in the true label vector If the sample belongs to the category ,but , otherwise 0. is the weighted cross entropy loss function of the full-graph forged fine-grained provenance network, is the weighted cross entropy loss function of the forged fine-grained provenance network model, is the number of output categories of the network, Yes Category The weight of The fine-grained traceability network model predicts that the sample belongs to the category The probability of is the corresponding category in the true label vector The value of .

[0102] The second stage loss function is:

[0103] in, Predict the sample belongs to the category for the coarse-grained traceability network model The probability of Forging a fine-grained provenance network model for the entire image predicts that the sample belongs to a category The probability of Forging a fine-grained provenance network model for a region predicts that the sample belongs to a category The probability of , , are their corresponding true labels respectively, is the two-stage loss function.

[0104] The method for tracing the forged face based on regional classification provided by the present application extracts the features of the face image and the features of the three regions respectively, fuses the two at the feature level and the decision level to obtain a coarse-grained tracing decision vector, and predicts the tracing result through the coarse-grained tracing decision vector. The present application makes full use of the complementary information at different levels such as the region and the whole image, the decision level and the feature level, thereby realizing the coarse-grained forged face tracing and improving the accuracy of face tracing. In addition, the present application also utilizes the distribution difference of the forged traces of the two forged types in the image area, and considers the guiding role of the two basic forged type classifications on the classification of the specific forged model, adopts a strategy of combining coarse-fine tracing to adapt to the actual application scenario where the two forged types coexist, and can perform accurate fine-grained forged tracing.

[0105] According to the method described in the above embodiment, the present embodiment will be further described from the perspective of a forged face tracing device based on regional classification. The forged face tracing device based on regional classification can be implemented as an independent entity or integrated in an electronic device. The electronic device can be a terminal, a server or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0106] See also Fig.10 , Fig.10 The forged face tracing device based on region classification provided by the embodiment of the present application is specifically described, and is applied to an electronic device. The forged face tracing device based on region classification may include: An acquisition module is used to acquire face images, including regional forged images, global forged images, and real images; A face analysis module is used to analyze the face image to obtain a face area image, a background area image and a connection area image; A traceability feature extraction module is used to perform traceability feature extraction on a face image to obtain global features and a global feature decision vector, and to perform traceability feature extraction on a face region image, a background region image, and a connection region image to obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors; The regional classification fusion module is used to perform feature-level fusion and decision-level fusion on global features, face region features, background region features, connection region features, global feature decision vectors, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors to obtain a coarse-grained tracing decision vector. The prediction module is used to input the coarse-grained traceability decision vector into the classifier to predict the traceability result.

[0107] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0108] In addition, an embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Fig.11 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0109] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0110] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions: Acquire a face picture, wherein the face picture includes a regional forged picture, a global forged picture, and a real picture; Constructing a coarse-grained traceability network model, inputting the face image into the trained coarse-grained traceability network model, and predicting the traceability result; The regional forged image, the global forged image and the real image are input into a trained coarse-grained traceability network model to predict the traceability result, including: Performing face analysis on the face image to obtain a face area image, a background area image, and a connection area image; Performing source feature extraction on the face image to obtain global features and a global feature decision vector, and performing source feature extraction on the face region image, the background region image, and the connection region image, respectively, to obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors; Performing feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector and the connection region feature decision vector to obtain a coarse-grained tracing decision vector; The coarse-grained tracing decision vector is input into a classifier to predict the tracing result.

[0111] The electronic device can implement the steps in any embodiment of the forged face tracing method based on region classification provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any forged face tracing method based on region classification provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0112] Fig.12 The specific structural block diagram of the electronic device provided in the embodiment of the present invention is shown, and the electronic device can be used to implement the forged face tracing method based on regional classification provided in the above embodiment. The electronic device 500 can be a terminal, a server or other equipment, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other equipment.

[0113] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0114] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0115] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0116] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0117] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0118] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0119] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0120] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: Acquire a face picture, wherein the face picture includes a regional forged picture, a global forged picture, and a real picture; Constructing a coarse-grained traceability network model, inputting the face image into the trained coarse-grained traceability network model, and predicting the traceability result; The regional forged image, the global forged image and the real image are input into a trained coarse-grained traceability network model to predict the traceability result, including: Performing face analysis on the face image to obtain a face area image, a background area image, and a connection area image; Performing source feature extraction on the face image to obtain global features and a global feature decision vector, and performing source feature extraction on the face region image, the background region image, and the connection region image, respectively, to obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors; Performing feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector and the connection region feature decision vector to obtain a coarse-grained tracing decision vector; The coarse-grained tracing decision vector is input into a classifier to predict the tracing result.

[0121] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0122] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the forged face tracing method based on regional classification provided by the embodiment of the present invention.

[0123] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0124] Since the instructions stored in the storage medium can execute the steps in any embodiment of the forged face tracing method based on region classification provided in the embodiments of the present invention, the beneficial effects that can be achieved by any forged face tracing method based on region classification provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0125] The above is a detailed introduction to a method, device, storage medium and electronic device for tracing the source of a forged face based on regional classification provided in an embodiment of the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for tracing the source of forged faces based on regional classification, characterized in that: The method comprises: Acquire a face picture, wherein the face picture includes a regional forged picture, a global forged picture, and a real picture; Constructing a coarse-grained traceability network model, inputting the face image into the trained coarse-grained traceability network model, and predicting the traceability result; The regional forged image, the global forged image and the real image are input into a trained coarse-grained traceability network model to predict the traceability result, including: Performing face analysis on the face image to obtain a face area image, a background area image, and a connection area image; Performing source feature extraction on the face image to obtain global features and a global feature decision vector, and performing source feature extraction on the face region image, the background region image, and the connection region image, respectively, to obtain face region features, background region features, connection region features, face region feature decision vectors, background region feature decision vectors, and connection region feature decision vectors; Performing feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector and the connection region feature decision vector to obtain a coarse-grained tracing decision vector; The coarse-grained tracing decision vector is input into a classifier to predict the tracing result.

2. The method for tracing the forged face based on regional classification according to claim 1 is characterized in that: The coarse-grained tracing model includes a face parsing module, and the face parsing module is embedded with a BiSeNet network; The face analysis of the face image to obtain a face area image, a background area image and a connection area image includes: Inputting the face image into the BiSeNet network to obtain a face contour mask; Performing an erosion operation on the face contour mask to obtain a face region mask, and obtaining a face region image based on the face region mask and the face image; Performing an expansion operation on the face contour mask and negating it to obtain a background area mask, and obtaining a background area image based on the background area mask and the face image; A logical AND operation is performed on the inversion of the face contour mask and the inversion of the background area mask to obtain a connected area mask, and a connected area image is obtained based on the connected area mask and the face image.

3. The method for tracing the forged face based on regional classification according to claim 1 is characterized in that: The coarse-grained traceability model includes a regional classification fusion module, the regional classification fusion module includes a feature fusion module and a decision fusion module, and the feature fusion module includes a channel attention fusion submodule and a spatial attention fusion submodule; The step of performing feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector, and the connection region feature decision vector to obtain a coarse-grained tracing decision vector includes: Inputting the face region feature, the background region feature, and the connection region feature into the channel attention fusion submodule to perform channel attention feature fusion to obtain three-region channel attention fusion features; Inputting the three-region channel attention fusion features and the global features into the spatial attention fusion submodule to perform spatial attention feature fusion, and obtaining a first decision vector of feature-level fusion; The global feature decision vector, the face area feature decision vector, the background area feature decision vector, the connection area feature decision vector and the first decision vector are input into the decision fusion module for decision fusion to obtain a coarse-grained tracing decision vector of decision-level fusion.

4. The method for tracing the forged face based on regional classification according to claim 1 is characterized in that: The coarse-grained traceability network model includes a traceability feature extraction module, which is used to extract traceability features from face images. The traceability feature extraction module includes a global traceability feature extractor, a face area traceability feature extractor, a background area traceability feature extractor, and a connection area traceability feature extractor. The trained coarse-grained traceability network model is trained based on a coarse-grained overall loss function, which is: in, is the legitimacy loss function, is the cross entropy loss function of the background region traceability feature extractor, is the cross entropy loss function of the face region traceability feature extractor, is the cross entropy loss function of the connected region tracing feature extractor, is the cross entropy loss function of the global provenance feature extractor.

5. The method for tracing the forged face based on regional classification according to claim 4 is characterized in that: The legitimacy loss function is: in, , and are the probability values ​​of the true labels of the background region tracing feature extractor, the face region tracing feature extractor, and the connection region tracing feature extractor, respectively. , and are the probability values ​​of false labels of the background region tracing feature extractor, the face region tracing feature extractor and the connection region tracing feature extractor, respectively. is the probability value of the true and false mixed labels of the connected area tracing feature extractor.

6. The method for tracing the forged face based on regional classification according to claim 1 is characterized in that: After the step of inputting the coarse-grained tracing decision vector into the classifier to predict the tracing result, the method further comprises: If the traceability result is regional forgery or global forgery, then regional forgery and global forgery judgment are performed; The regional and full-image forgery judgment includes: Construct a fine-grained traceability network model, which includes a coarse-grained traceability feature fusion module, a regional forgery fine-grained traceability module, and a full-map forgery fine-grained traceability module. The coarse-grained traceability feature fusion module includes a coarse-grained fusion submodule and a coarse-grained weight fusion submodule. Inputting the global features, the face region features, the background region features and the connection region features into a trained coarse-grained fusion module for feature fusion to obtain a coarse-grained fusion feature; Inputting the global features into the regional forgery fine-grained tracing module or the full-image forgery fine-grained tracing module to extract tracing features to obtain fine-grained features; Inputting the coarse-grained fusion feature and the fine-grained feature into the coarse-grained and fine-grained weight fusion submodule for weight fusion to obtain a weight fusion feature; The weighted fusion features are input into the classifier to obtain specific tracing results.

7. The method for tracing the forged face based on regional classification according to claim 6 is characterized in that: The fine-grained provenance network model includes a regional forgery fine-grained provenance network model and a full-graph forgery fine-grained provenance network model, and the fine-grained provenance network model is trained based on a two-stage loss function; The first-stage loss function is: in, is the consistency loss, is the weighted cross entropy loss function of the regional forgery fine-grained provenance network model, Forge the weighted cross entropy loss function of the fine-grained traceability network model for the entire graph; The second stage loss function is: in, Predict the sample belongs to the category for the coarse-grained traceability network model The probability of Forging a fine-grained provenance network model for the entire image predicts that the sample belongs to a category The probability of Forging a fine-grained provenance network model for a region predicts that the sample belongs to a category The probability of , , are the values ​​of the corresponding true labels in their respective true label vectors, is the two-stage loss function.

8. A forged face tracing device based on region classification, characterized in that: include: An acquisition module, used to acquire a face picture, wherein the face picture includes a regional forged picture, a global forged picture and a real picture; A face analysis module, used for performing face analysis on the face image to obtain a face area image, a background area image and a connection area image; A traceability feature extraction module is used to perform traceability feature extraction on the face image to obtain global features and global feature decision vectors, and to perform traceability feature extraction on the face area image, the background area image and the connection area image to obtain face area features, background area features, connection area features, face area feature decision vectors, background area feature decision vectors and connection area feature decision vectors; A region classification fusion module is used to perform feature-level fusion and decision-level fusion on the global features, the face region features, the background region features, the connection region features, the global feature decision vector, the face region feature decision vector, the background region feature decision vector and the connection region feature decision vector to obtain a coarse-grained tracing decision vector; The prediction module is used to input the coarse-grained tracing decision vector into the classifier to predict the tracing result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method for tracing the forged face based on region classification as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the method for tracing the forged face based on regional classification as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • GAN fingerprint enhancement method for counterfeited face traceability

    CN114973363A

  • Deep counterfeit image detection method based on clustering decision and related equipment

    CN116580462A

  • Adversarial transfer learning-based face beauty prediction method and device

    WO2021052159A1

Cited By

  • Face forgery detection method, system and device and medium

    CN122200821A