A deepfake image detection method and system based on fine-grained features

Abstract

This invention discloses a method and system for deepfake image detection based on fine-grained features. The method includes extracting features using a backbone network for image detection, the backbone network comprising four residual convolutional blocks; dividing the extracted global features horizontally into several blocks to construct fine-grained features, and using these fine-grained features to guide the output features of the residual convolutional blocks, constrained by a loss function; pooling the fine-grained features and global features, and training the pooled output features using a classification loss function. This invention, without requiring complex attention mechanisms or introducing additional training parameters, enables the model to acquire more robust features, significantly improving the accuracy of cross-domain deepfake detection.

Description

Technical Field

[0001] This invention relates to the technical field of image detection, and in particular to a method and system for detecting deepfake images based on fine-grained features. Background Technology

[0002] The increasing resolution of generated forged images has led to their misuse and raised concerns about forgery technology. Consequently, deepfake detection technology has attracted attention from both academia and industry. Existing generative adversarial networks (GANs) are divided into conditional GANs and unconditional GANs. Unconditional GANs produce images with far lower realism than those generated by conditional GANs, making unconditional GAN ​​forgeries extremely easy to identify. However, conditional GANs produce very high-resolution and realistic forgeries, requiring more representative feature information. Current deepfake algorithms primarily rely on global information to detect real and forged images, neglecting fine-grained local features. Over-reliance on global features leads to the loss of important local discriminative information.

[0003] As the quality of forged images generated by conditional generative adversarial networks (GANs) increases, the performance of existing deepfake detection algorithms is declining. Deepfake detection technology primarily utilizes neural networks to extract features from real and forged images, classifying them to distinguish between the two types of images. Previous forgery methods generated images based on the entire image's features; however, current forgery methods, focusing on details, produce more realistic forgeries, thus requiring detection and recognition to pay more attention to fine-grained features.

[0004] Existing deepfake detection techniques primarily classify images based on global feature information. However, many existing forged images only alter a very small local region. Therefore, while considering global feature information, local feature information also needs to be taken into account. Learning both global and local feature information is beneficial for obtaining discriminative feature information. Existing forgery detection algorithms utilize facial landmark technology to obtain local regions and thus acquire local feature information; however, these techniques require additional algorithms and cannot form an end-to-end detection algorithm. Summary of the Invention

[0005] To address the aforementioned technical problems in the existing technology, this invention proposes a deep forgery image detection method and system based on fine-grained features to solve these problems.

[0006] According to one aspect of the present invention, a method for detecting deepfake images based on fine-grained features is proposed, comprising:

[0007] S1: Features are extracted using the backbone network for image detection. The backbone network consists of four residual convolutional blocks.

[0008] S2: Divide the extracted global features into several blocks along the horizontal plane, construct fine-grained features, and use the fine-grained features to guide the output features of the residual convolution block, using the loss function for constraint;

[0009] S3: Pool the fine-grained features and global features, and then use classification loss to constrain the training of the pooled features.

[0010] In some specific embodiments, the backbone network is a ResNet50 network, which includes four feature extraction stages: Stage 0, Stage 1, Stage 2, and Stage 3.

[0011] In some specific embodiments, features F∈R are obtained after Stage 4. H×W×C Its feature extraction expression is as follows: F = (B ResNet50 {i i |θ1, θ2, ..., θ n}), where B ResNet50 Represents the backbone network ResNet50, I i Represents the input image, θ1, θ2, ..., θ n The parameters represent the residual convolutional block.

[0012] In some specific embodiments, the input image is 224*224, and S2 specifically includes processing the features F∈R obtained in S1. H ×W×C Divide it horizontally into 7 parts, numbered F i The feature size obtained is F∈R for (i = 0, 1, 2, 3, 4, 5, 6). 14×14×2048 , will F i (i = 0, 1) are constructed as the first fine-grained feature F 1 F i (i = 2, 3) are constructed as the second fine-grained feature F 2 F i (i = 4, 5, 6) are constructed as the third fine-grained feature F 3 F i , (i = 0, 1, 2, 3, 4, 5, 6) are global features.

[0013] In some specific embodiments, feature F 1 F 2 F 3 Both F and F use global max pooling and global average pooling, and train the features output by each pooling using a classification loss.

[0014] In some specific embodiments, feature F 1 F2 and F 3 These are used to guide the output features of Stage 1, Stage 2, and Stage 3 respectively, and are constrained using L2 loss.

[0015] In some specific embodiments, the classification loss is specifically the label smoothing regularization cross-entropy loss function. Where ε∈[0,1], K represents the number of face categories, and p(k) represents the probability of predicting that the face belongs to category k.

[0016] According to a second aspect of the invention, a computer-readable storage medium is provided on which one or more computer programs are stored, which, when executed by a computer processor, implement any of the methods described above.

[0017] According to a third aspect of the present invention, a deepfake image detection system based on fine-grained features is proposed, the system comprising:

[0018] The feature extraction unit is configured to extract features using a backbone network for image detection, the backbone network comprising four residual convolutional blocks.

[0019] The fine-grained feature construction unit is configured to divide the extracted global features into several blocks along the horizontal direction, construct fine-grained features, and use the fine-grained features to guide the output features of the residual convolution block, using the loss function constraint.

[0020] Constraint Unit: Configured to perform constraint training on the pooled features and global features through pooling, and on the pooled output features through classification loss.

[0021] In some specific embodiments, the backbone network is a ResNet50 network, which includes four feature extraction stages: Stage 0, Stage 1, Stage 2, and Stage 3. After Stage 4, the feature F∈R is obtained. H×W×C Its feature extraction expression is as follows: F = (B ResNet50 {I i |θ1, θ2, ..., θ n}), where B ResNet50 Represents the backbone network ResNet50, I i Represents the input image, θ1, θ2, ..., θ n The parameters represent the residual convolutional block.

[0022] In some specific embodiments, the input image is 224*224, and the features F∈R H×W×C Divide it horizontally into 7 parts, numbered F i The feature size obtained is F∈R for (i = 0, 1, 2, 3, 4, 5, 6).14×14×2048 , will F i (i = 0, 1) are constructed as the first fine-grained feature F 1 F i (i = 2, 3) are constructed as the second fine-grained feature F 2 F i (i = 4, 5, 6) are constructed as the third fine-grained feature F 3 F i , (i = 0, 1, 2, 3, 4, 5, 6) are global features.

[0023] In some specific embodiments, feature F 1 F 2 and F 3 These are used to guide the output features of Stage 1, Stage 2, and Stage 3 respectively, and L2 loss is used for constraints to optimize the feature F. 1 F 2 F 3 Both F and F use global max pooling and global average pooling, and train the features output by each pooling using a classification loss.

[0024] In some specific embodiments, the classification loss is specifically the label smoothing regularization cross-entropy loss function. Where ε∈[0,1], K represents the number of face categories, and p(k) represents the probability of predicting that the face belongs to category k.

[0025] This invention proposes a deepfake detection method and system based on fine-grained features. The feature map output from the last convolutional layer is divided into multiple horizontal spatial bars, and then classification operations are performed independently on horizontal bars of different scales. For each independent horizontal bar, both averaging and max pooling strategies are applied. Average pooling can perceive the global information of the spatial bar, while max pooling aims to extract the most discriminative information. The fused feature obtained by combining the two pooling methods has even greater discriminative power. This invention can increase the model's feature acquisition to be more robust without involving complex attention mechanisms or introducing additional training parameters. The invention is validated on publicly available datasets, showing a significant improvement in cross-domain deepfake detection accuracy. Attached Figure Description

[0026] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Other features, objects, and advantages of this application will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0027] Figure 1 This is a flowchart of a deep forgery image detection method based on fine-grained features according to an embodiment of this application;

[0028] Figure 2 This is an algorithm framework diagram of a deep forgery image detection method based on fine-grained features, according to a specific embodiment of this application.

[0029] Figure 3 This is a framework diagram of a deep forgery image detection system based on fine-grained features, according to one embodiment of this application.

[0030] Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0031] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] A method for detecting deep forged images based on fine-grained features according to an embodiment of this application, Figure 1 A flowchart illustrating a deepfake image detection method based on fine-grained features according to an embodiment of this application is shown. Figure 1 As shown, the method includes:

[0034] S101: Features are extracted using a backbone network for image detection. The backbone network consists of four residual convolutional blocks.

[0035] In a specific embodiment, the backbone network is a ResNet50 network, which includes four feature extraction stages: Stage 0, Stage 1, Stage 2, and Stage 3. After Stage 4, the feature F∈R is obtained. H×W×C Its feature extraction expression is as follows: F = (B ResNet50 {I i |θ1, θ2, ..., θ n}), where B ResNet50 Represents the backbone network ResNet50, I i Represents the input image, θ1, θ2, ..., θ n The parameters represent the residual convolutional block.

[0036] S102: Divide the extracted global features into several blocks along the horizontal direction, construct fine-grained features, and use the fine-grained features to guide the output features of the residual convolution block, using the loss function for constraint.

[0037] In a specific embodiment, the input image is 224*224, and the features F∈R obtained in S101 are... H×W×C Divide it horizontally into 7 parts, numbered F i The feature size obtained is F∈R for (i = 0, 1, 2, 3, 4, 5, 6). 14×14×2048 , will F i (i = 0, 1) are constructed as the first fine-grained feature F 1 F i (i = 2, 3) are constructed as the second fine-grained feature F 2 F i (i = 4, 5, 6) are constructed as the third fine-grained feature F 3 F i , (i = 0, 1, 2, 3, 4, 5, 6) are global features.

[0038] In a specific embodiment, feature F 1 F 2 F 3 Both F and F employ global max pooling and global average pooling, respectively, and train the features output by each pooling operation using a classification loss. Global average pooling can perceive global information about the spatial bars, while max pooling aims to extract the most discriminative information.

[0039] In a specific embodiment, feature F 1 F 2 and F 3 These are used to guide the output features of Stage 1, Stage 2, and Stage 3 respectively, and L2 loss is used for constraints. This allows the network to gradually learn fine-grained features.

[0040] S103: Pool the fine-grained features and global features, and then use the classification loss to constrain the training of the pooled features.

[0041] In a specific embodiment, the classification loss is specifically the label smoothing regularization cross-entropy loss function. Where ε∈[0,1], K represents the number of face categories, and p(k) represents the probability of predicting that the face belongs to category k.

[0042] Figure 2The diagram illustrates the algorithmic framework of a deep forgery image detection method based on fine-grained features according to a specific embodiment of the present invention. The entire framework uses ResNet50 as the backbone network and includes the following steps:

[0043] Step S1: This invention uses ResNet50, commonly used in deepfake detection algorithms, as the backbone network, such as... Figure 2 As shown, it contains 4 residual convolutional blocks. The features F∈R are obtained after ResNet50 Stage 4. H×W×C Its feature extraction expression is as follows: F = (B ResNet50 {I i |θ1, θ2, ..., θ n}), where B ResNet50 Represents the backbone network ResNet50, I i Represents the input image, θ1, θ2, ..., θ n The parameters represent the residual convolutional block.

[0044] Step S2: Apply the features F∈R obtained in step S1 H×W×C Dividing the image horizontally into several parts, the input image for this invention is 224*224, and the final feature size is F∈R. 14×14×2048 Divide it into 7 parts horizontally.

[0045] Step S3: Based on the 7 features divided in step S3, number them as F i (i = 0, 1, 2, 3, 4, 5, 6), F i (i = 0, 1) are constructed as the first fine-grained feature F 1 F i (i = 2, 3) are constructed as the second fine-grained feature F 2 F i (i = 4, 5, 6) are constructed as the third fine-grained feature F 3 F i (i = 0, 1, 2, 3, 4, 5, 6) are global features, i.e., F ∈ R in step S2. H×W×C .

[0046] Step S4: Transfer the feature F obtained in step S3 1 F 2 F 3 Each feature, F, is processed using global max pooling and global average pooling. Global average pooling captures global information about the spatial bars, while max pooling aims to extract the most discriminative information. Then, the features output from each pooling step are trained using a classification loss.

[0047] Step S5: To enhance the guidance of fine-grained features in the overall network training, this invention will use F...1 F 2 F 3 This is used to guide the output features of Stage 1, Stage 2, and Stage 3 in ResNet50, and L2 loss is used for constraint, so that the network gradually learns fine-grained features.

[0048] Step S6: The algorithm uses a label-smoothed regularized cross-entropy loss function for constraints. The label-smoothed regularized cross-entropy loss function is shown in the following formula: Where ε∈[0,1], K is the number of face categories, and p(k) represents the probability of predicting that the face belongs to category k.

[0049] Step S7: The entire framework incorporates fine-grained features and multiple constraint losses during training, allowing the network to focus on as many salient features as possible throughout the training process. During inference, only the output of the classification layer corresponding to feature F is used as the final judgment result.

[0050] This invention proposes a deepfake detection algorithm based on fine-grained features. The feature map output from the last convolutional layer is divided into multiple horizontal spatial bars, and then classification operations are performed independently on these bars at different scales. For each independent horizontal bar, both averaging and max pooling strategies are applied. Average pooling can capture global information from the spatial bars, while max pooling aims to extract the most discriminative information. The fused feature obtained by combining the two pooling methods is even more discriminative. This invention achieves more robust feature acquisition without involving complex attention mechanisms or introducing additional training parameters. The invention is validated on publicly available datasets, demonstrating a significant improvement in cross-domain deepfake detection accuracy.

[0051] Continue to refer to Figure 3 , Figure 3 A framework diagram of a deepfake image detection system based on fine-grained features according to an embodiment of the present invention is shown. The system specifically includes a feature extraction unit 301, a fine-grained feature construction unit 302, and a constraint unit 303. The feature extraction unit 301 is configured to extract features using an image detection backbone network, which includes four residual convolutional blocks. The fine-grained feature construction unit 302 is configured to divide the extracted global features horizontally into several blocks to construct fine-grained features, and use these fine-grained features to guide the output features of the residual convolutional blocks, constrained by a loss function. The constraint unit 303 is configured to pool the fine-grained features and global features, and constrain the pooled output features using a classification loss.

[0052] The following is for reference. Figure 4It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0053] like Figure 4 As shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of system 400. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0054] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a liquid crystal display (LCD) and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card and a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0055] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0056] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0057] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0058] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0059] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: extract features using an image detection backbone network, the backbone network including four residual convolutional blocks; divide the extracted global features horizontally into several blocks to construct fine-grained features, and use the fine-grained features to guide the output features of the residual convolutional blocks, constrained by a loss function; pool the fine-grained features and the global features, and constrain the pooled output features using a classification loss.

[0060] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for detecting deepfake images based on fine-grained features, characterized in that, include: S1: Features are extracted using an image detection backbone network, which includes four residual convolutional blocks. The backbone network is a ResNet50 network, which includes four feature extraction stages: Stage 1, Stage 2, Stage 3, and Stage 4. Features are obtained after Stage 4. Its feature extraction expression is as follows: ,in, Represents the backbone network ResNet50. Represents the input image. The parameters representing the residual convolutional block; S2: The extracted global features are divided into several blocks horizontally to construct fine-grained features, and these fine-grained features are used to guide the output features of the residual convolutional blocks, constrained by a loss function; the input image is... S2 specifically includes taking the features obtained in S1 Divide horizontally into 7 parts, numbered as follows , The obtained feature size is ,Will The features are constructed as the first fine-grained features. , The features are constructed as a second fine-grained feature. , The features are constructed as a third fine-grained feature. , For global features; S3: Pool the fine-grained features and the global features, and then constrain the pooled output features using a classification loss.

2. The deepfake image detection method based on fine-grained features according to claim 1, characterized in that, Features , , and Both methods employ global max pooling and global average pooling, with the features output by each pooling method constrained by a classification loss during training.

3. The deepfake image detection method based on fine-grained features according to claim 1, characterized in that, The feature , and These are used to guide the output features of Stage 1, Stage 2, and Stage 3 respectively, and are constrained using L2 loss.

4. The method for detecting deepfake images based on fine-grained features according to claim 1 or 2, characterized in that, The classification loss is specifically the cross-entropy loss function of label smoothing regularization. ,in, , Represents the number of face categories. Indicates that the prediction belongs to a category The probability of.

5. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that, When the one or more computer programs are executed by a computer processor, they perform the method described in any one of claims 1 to 4.

6. A deepfake image detection system based on fine-grained features, characterized in that, The system includes: The feature extraction unit is configured to extract features using a backbone network for image detection. The backbone network comprises four residual convolutional blocks and is a ResNet50 network. The ResNet50 network includes four feature extraction stages: Stage 1, Stage 2, Stage 3, and Stage 4. Features are obtained after Stage 4. Its feature extraction expression is as follows: ,in, Represents the backbone network ResNet50. Represents the input image. The parameters representing the residual convolutional block; A fine-grained feature construction unit is configured to divide the extracted global features horizontally into several blocks to construct fine-grained features, and use these fine-grained features to guide the output features of the residual convolutional blocks, constrained by a loss function; the input image is... , the features Divide horizontally into 7 parts, numbered as follows , The obtained feature size is ,Will The features are constructed as the first fine-grained features. , The features are constructed as a second fine-grained feature. , The features are constructed as a third fine-grained feature. , For global features; Constraint Unit: Configured to perform pooling on the fine-grained features and the global features, and to constrain the pooled output features through classification loss training.

7. The deepfake image detection system based on fine-grained features according to claim 6, characterized in that, The feature , and These are used to guide the output features of Stage 1, Stage 2, and Stage 3 respectively, and L2 loss is used for constraint, thus shaping the features. , , and Both methods employ global max pooling and global average pooling, with the features output by each pooling method constrained by a classification loss during training.

8. The deepfake image detection system based on fine-grained features according to claim 6, characterized in that, The classification loss is specifically the cross-entropy loss function of label smoothing regularization. ,in, , Represents the number of face categories. Indicates that the prediction belongs to a category The probability of.

Images