A fake image recognition method based on local-global feature fusion and joint loss learning
Patent Information
- Application Number
- CN202610854498.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-04-25
- Filing Date
- 2026-06-13
- Publication Date
- 2026-09-11
AI Technical Summary
然而，现有基于CLIP的检测方法存在以下显著缺陷：首先，它们通常仅利用CLIP视觉编码器提取的全局高维特征，而完全忽略了包含伪造痕迹的局部特征
本发明所提出的一种基于局部-全局特征融合与联合损失学习的伪造图像识别方法在提高伪造图像检测准确率和泛化性方面具有显著优势，在公开的AIGC图像数据集上取得了优异的性能，通过测试得到的识别准确率可达到约99.09%（针对GAN数据集）与98.73%（针对Diffusion数据集），相比未引入局部特征增强模块、局部与全局特征融合模块和联合损失函数的基础模型，本发明方法的准确率和精确率明显提升；
Smart Images
Figure CN122737589A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of AIGC image forgery detection, and particularly relates to a forgery image recognition method based on local-global feature fusion and joint loss learning. Background Technology
[0002] With the rapid development of generative artificial intelligence (AIGC), image generation techniques based on generative adversarial networks (GANs) and diffusion models have achieved breakthroughs in realism. These technologies have driven the application of generated images in entertainment, art, and virtual digital humans, but have also brought security risks such as the spread of fake images and the generation of misinformation. These issues have made AIGC fake image detection a hot topic today.
[0003] In recent years, large-scale vision-language models (such as CLIP) have been increasingly used in deepfake detection and GAN image recognition tasks due to their cross-modal alignment capabilities. The highly generalized high-dimensional semantic representations obtained by pre-training on massive amounts of image-text alignment data can improve cross-model generalization capabilities to some extent. However, existing CLIP-based detection methods have the following significant drawbacks: First, they typically only utilize the global high-dimensional features extracted by the CLIP visual encoder, completely ignoring local features containing forgery traces. In deepfake detection tasks, the most discriminative forgery clues (such as edge artifacts, structural inconsistencies, and multi-scale local blurring) are often hidden in local regions, and these subtle traces are easily overlooked by global semantic features. Second, in terms of training strategies, traditional detection models typically employ the standard binary cross-entropy (BCE) classification loss function. However, relying solely on a single classification loss only distinguishes between real and forged images at the prediction level (i.e., in terms of output probability), lacking explicit constraints on the model's feature space distribution structure. This results in loosely distributed similar images in the feature space, leading to low robustness of the features learned by the model. When a model encounters a completely new generative architecture that it has never seen before (such as various variant diffusion models), its generalization performance often drops sharply due to the unstable decision boundary.
[0004] In summary, existing technologies have the following shortcomings in the AIGC image forgery recognition task: (1) they rely excessively on global semantic features and lack the ability to capture fine-grained forgery traces at multiple local scales; (2) the single binary cross-entropy loss cannot effectively optimize the feature distribution space, resulting in weak generalization ability across generative models. To solve the above problems, a novel model structure and joint loss optimization scheme are needed to improve the accuracy and generalization of AIGC image forgery recognition. Summary of the Invention
[0005] To address the shortcomings of the aforementioned background technologies, this invention proposes an AIGC-based image forgery recognition method based on local-global feature fusion and joint loss learning. This method innovatively extracts global features and local patch tokens from images based on a frozen CLIP visual encoder and LoRA fine-tuning. By constructing local attention and multi-scale dilated convolution branches, it enhances the missing structural information and fine-grained texture features in the global semantic features. Furthermore, a residual fusion strategy is designed, enabling the model to not only grasp the overall semantics of the image but also focus on local forgery details. Simultaneously, in terms of optimization strategies, it abandons the single classification loss and proposes a fusion of classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss. The aim is to propose a novel model structure and joint loss optimization scheme to improve the accuracy and generalization of AIGC image forgery recognition.
[0006] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution: A method for identifying forged images based on local-global feature fusion and joint loss learning specifically includes the following steps: Collect and organize real images, and generate fake images based on the AIGC generation paradigm; The AIGC image forgery recognition dataset is constructed based on real images and forged images, and the AIGC image forgery recognition dataset is divided into training set, validation set and test set. An AIGC image forgery detection model is constructed, comprising a text generation and enhancement module, a visual-text feature extraction module, a local feature enhancement module, a local-global feature fusion module, and a joint loss optimization module. The original images in the training set are input into the text generation and enhancement module, which extracts semantic descriptions from the original images and constructs enhanced text using general prompts for true and false categories; the original images include real images and fake images. The enhanced text and the original images in the training set are input into the visual-text feature extraction module, which consists of a text encoder and a visual encoder; the text encoder extracts enhanced text features from the enhanced text, and the visual encoder extracts global features and local patch features from the original images; The local patch features are input into the local feature enhancement module. The local attention branch and the multi-scale dilated convolution branch of the local feature enhancement module are used to enhance the features, resulting in local attention enhanced features and multi-scale structural features. Local features are obtained by fusing local attention enhancement features with multi-scale structural features. These local features are then input into a local-global feature fusion module, where they are fused with global features using a learnable residual fusion strategy to obtain fused output features. A linear classifier maps the fused output features of each image to the predicted forgery probability of that image. The enhanced text features, global features, local patch features, and fused output features are input into the joint loss optimization module to calculate the joint loss. The joint loss is obtained by weighted summation of classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss. The model is trained iteratively with the goal of minimizing the joint loss, and the parameters are updated. In each iteration, the performance of the AIGC image forgery detection model is evaluated using a validation set. When the stopping condition is met, the model parameters are saved to obtain the trained AIGC image forgery detection model. The trained AIGC image forgery detection model is then used to identify AIGC forged images.
[0007] Furthermore, the training set, validation set, and test set specifically include: The training set includes real images obtained from public datasets, as well as fake images generated using a single adversarial generative network; the validation set is from the same source as the training set and has the same distribution; the test set includes fake images generated by at least two different generative adversarial networks or diffusion models, as well as real images of the same number as the fake images.
[0008] Furthermore, the extraction of global features and local patch features from the original image via the visual encoder specifically includes: Feature extraction is performed using a pre-trained CLIP visual encoder. With the main parameters of the pre-trained CLIP visual encoder frozen, a low-rank adaptive module (LoRA) is introduced to fine-tune the parameters. The original images from the training set are input into the fine-tuned CLIP visual encoder to extract one-dimensional features representing the image context and patch markers. Using the one-dimensional features as global features, the patch markers are reshaped into a two-dimensional feature map, serving as local patch features.
[0009] Furthermore, the step of inputting local patch features into the local feature enhancement module, and performing feature enhancement through the local attention branch and multi-scale dilated convolution branch of the local feature enhancement module to obtain local attention-enhanced features and multi-scale structural features specifically includes: In the local attention branch, the local patch features are first subjected to a depthwise separable convolution operation to extract local spatial context and channel information; then, channel-level local attention weights are generated through the Sigmoid activation function; the generated channel-level local attention weights are multiplied element-wise with the local patch features to obtain the local attention enhancement features. In the multi-scale dilated convolution branch, three sets of 3×3 convolution operations with different dilation rates are used to process the local patch features of the input in parallel. Then, the outputs of different branches are added and aggregated element by element to obtain multi-scale structural features.
[0010] Furthermore, the process of fusing local attention enhancement features with multi-scale structural features to obtain local features, and then inputting these local features into a local-global feature fusion module, where they are fused with global features using a learnable residual fusion strategy to obtain the fused output features, is as follows: We use 1×1 convolution to concatenate and reduce the dimensionality of local attention enhancement features and multi-scale structural features to obtain fused features; then we perform global average pooling and layer normalization on the local features in sequence to align the dimensions and obtain the final local features. A learnable residual fusion strategy is adopted. A learnable scalar fusion weight is generated by the Sigmoid function. The final local feature is multiplied by the scalar fusion weight and then added to the global feature by residual addition to obtain the fused output feature.
[0011] Furthermore, the classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss are specifically as follows: Classification loss: A binary cross-entropy loss is constructed based on the true label and the predicted probability of forgery for each image, and used as the classification loss; Image-text contrast loss: A contrastive learning mechanism is used to calculate the one-way contrast loss from global features to enhanced text features and the one-way contrast loss from enhanced text features to global features; the mean of the two one-way contrast losses is taken to obtain the image-text contrast loss. Weakly supervised localization loss: Local patch features are mapped and processed with a Sigmoid activation function to obtain a class activation map; for a forged image, the mean response of the top K% region of its class activation map response value is calculated, and 1 is subtracted from the mean response value to obtain the localization error of the forged image; for a real image, all response values of its class activation map are normalized to obtain the probability distribution of the class activation map; the negative value of the information entropy of this probability distribution is calculated as the localization loss of the real image; the localization loss of the real image is weighted and summed with the localization loss of the forged image to obtain the weakly supervised localization loss; Class-level feature alignment loss: Obtain the feature centers of the fused output features of the real image and the fake image respectively, calculate the square of the Euclidean distance between the fused output feature of each image and its feature center; then take the mean of the squares of the Euclidean distances and sum them to obtain the class-level feature alignment loss.
[0012] Furthermore, this application also discloses an application system for the aforementioned method of forged image recognition based on local-global feature fusion and joint loss learning, comprising a dataset construction module, an AIGC image forgery detection model, and a training and recognition module, wherein: The dataset construction module is used to construct a dataset containing real images and fake images, and divides it into a training set, a validation set and a test set; The AIGC image forgery detection model obtains the fused output features of real and forged images through a text generation and enhancement module, a visual-text feature extraction module, a local feature enhancement module, and a local-global feature fusion module. Based on the fused output features, it obtains the predicted forgery probability value. Then, through the joint loss optimization module, it calculates the joint loss obtained by weighted summation of classification loss, image-text comparison loss, weak supervision localization loss, and class-level feature alignment loss. The training and recognition module iteratively trains the AIGC image forgery detection model with the goal of minimizing the joint loss, and finally uses the trained model to realize AIGC forgery image recognition.
[0013] An electronic device is also disclosed, comprising a memory and a processor, wherein: Memory is used to store computer programs that can run on a processor; A processor is configured to, while running the computer program, execute a forged image recognition method based on local-global feature fusion and joint loss learning as described above.
[0014] A computer-readable storage medium is also disclosed, which stores computer instructions for causing a processor to execute and implement a forged image recognition method based on local-global feature fusion and joint loss learning as described above.
[0015] Based on the above technical solution, the present invention has at least the following beneficial effects: The forgery image recognition method proposed in this invention based on local-global feature fusion and joint loss learning has significant advantages in improving the accuracy and generalization of forgery image detection. It has achieved excellent performance on the publicly available AIGC image dataset, with recognition accuracy reaching approximately 99.09% (for the GAN dataset) and 98.73% (for the Diffusion dataset). Compared with the basic model that does not introduce local feature enhancement module, local and global feature fusion module and joint loss function, the accuracy and precision of the method proposed in this invention are significantly improved. The method proposed in this invention addresses the shortcomings of traditional models that focus only on global semantics while ignoring minute artifacts by fusing global and local features and a joint loss function, thereby improving the accuracy of image forgery detection. By freezing the main parameters of the traditional CLIP model, it introduces local attention and multi-scale dilated convolution as feature enhancement modules, solving the model's inability to focus on specific forgery details at multiple scales, allowing for a deeper understanding of the image. The joint loss function not only optimizes the distribution of the underlying feature space through feature alignment loss but also guides the model to automatically focus on forged regions using weakly supervised localization loss, completely resolving the core pain points of traditional single classification loss: "weak generalization ability and low accuracy in detecting unfamiliar forgery types." Combined with these modules, the model exhibits stronger robustness on unknown generated model test sets, maintaining stable classification performance even under complex forgery interference. Therefore, the method proposed in this invention achieves higher accuracy and generalization in AIGC image forgery recognition. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the AIGC image forgery detection model structure in the method proposed in this invention; Figure 2 This is a flowchart illustrating a forgery image recognition method based on local-global feature fusion and joint loss learning proposed in this invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention.
[0019] like Figure 2 As shown, this invention proposes a method for identifying forged images based on local-global feature fusion and joint loss learning, which specifically includes:
[0020] Collect and organize real images, and generate fake images based on the AIGC generation paradigm;
[0021] The AIGC image forgery recognition dataset is constructed based on real images and forged images, and the AIGC image forgery recognition dataset is divided into training set, validation set and test set.
[0022] In a preferred embodiment, the training set, validation set, and test set specifically include:
[0023] The training set includes real images obtained from public datasets and fake images generated using a single adversarial generative network. In this embodiment, the real images in the training set are from images of four specific object categories in the LSUN (Large-scale SceneUnderstanding) dataset, while the fake images in the training set are generated based on the Progressive Adversarial Generative Network (ProGAN) generative model. A corresponding number of fake images are generated for each category, forming a balanced training set.
[0024] The validation set and the training set are from the same source and have the same distribution.
[0025] The test set includes forged images generated by various generative adversarial networks or diffusion models (i.e., covering GAN models, DALL-E, LDM, and various variant models), as well as real images of equal quantity to the number of forged images. The test set is used to subsequently evaluate the accuracy and generalization ability of the forgery detection model; therefore, multiple different generation paradigms are selected.
[0026] Construction as Figure 1 The AIGC image forgery detection model shown includes a text generation and enhancement module, a visual-text feature extraction module, a local feature enhancement module, a local-global feature fusion module, and a joint loss optimization module.
[0027] The original images in the training set are input into the text generation and enhancement module. Semantic descriptions are extracted from the original images, and enhanced text is constructed using general hints for true and false categories (such as "Camera" for true images and "Deepfake" for false images). The original images include real images and fake images.
[0028] The enhanced text and the original images in the training set are input into the visual-text feature extraction module, which consists of a text encoder and a visual encoder; the text encoder extracts enhanced text features from the enhanced text, and the visual encoder extracts global features and local patch features from the original images;
[0029] In a preferred embodiment, the CLIP:ViT-L / 14 model is used as the visual-text feature extraction module, i.e., the text encoder uses a pre-trained CLIP text encoder, and the visual encoder uses a pre-trained CLIP visual encoder; the extraction of global features and local patch features from the original image through the visual encoder specifically includes:
[0030] Feature extraction is performed using a pre-trained CLIP visual encoder. With the main parameters of the pre-trained CLIP visual encoder frozen, a low-rank adaptive module (LoRA) is first introduced to fine-tune the parameters. The original images in the training set are then used for feature extraction. The fine-tuned CLIP visual encoder is input to encode a high-order one-dimensional feature representing the overall image context and patch tokens; the one-dimensional feature is then used as the global feature. In order to restore the spatial correspondence, the patch markers are reshaped into two-dimensional feature maps as local patch features. The formula is expressed as: ; in, This indicates a reshaping operation. This indicates the feature extraction operation of the CLIP visual encoder.
[0031] The local patch features are input into the local feature enhancement module, and feature enhancement is performed through the local attention branch LA and the multi-scale dilated convolution branch MDC of the local feature enhancement module to obtain local attention enhanced features and multi-scale structural features.
[0032] In a preferred embodiment, this step specifically includes:
[0033] Traditional CLIP models focus on global semantics in image forgery detection tasks, neglecting local structure and multi-scale details. Therefore, without compromising the original semantic expressive power, this application uses a dual-branch feature enhancement strategy to supplement the missing structural information in high-level semantic features.
[0034] In the local attention branch, the local patch features of the input are first processed. Perform depthwise separable convolution operations Depthwise separable convolution operations include depthwise convolution and pointwise convolution, which are used to capture local spatial context and integrate channel information;
[0035] Then activate the Sigmoid function Generate channel-level local attention weights The generated channel-level local attention weights are multiplied element-wise with the local patch features to obtain the local attention enhancement features. The formula is expressed as: ; ; In the local attention branch, the local attention mechanism models each channel independently, adaptively amplifying feature channels that are sensitive to high-frequency traces of forgery, while suppressing irrelevant background features.
[0036] In the multi-scale dilated convolution branch, three groups with different dilation rates (in this embodiment, the dilation rate is...) are used. The 3×3 convolution operation processes the local patch features of the input in parallel, and then the outputs of different branches are summed element-wise to obtain multi-scale structural features. The formula is expressed as: ; in, ; indicates the rate of expansion. The 3×3 convolution operation; since different types of forgery operations may manifest as anomalies of different scales in the image, such as small edge misalignment or large area blur, this application introduces multi-scale dilated convolution to effectively increase the receptive field.
[0037] Local features are obtained by fusing local attention enhancement features with multi-scale structural features. These local features are then input into a local-global feature fusion module, where they are fused with global features using a learnable residual fusion strategy to obtain fused output features. A linear classifier maps the fused output features of each image to a predicted forgery probability value for that image. ;
[0038] In this preferred embodiment, the specific process for obtaining the fused output features is as follows:
[0039] To eliminate computational redundancy, 1×1 convolution is first used. The local attention enhancement features and multi-scale structural features are concatenated and dimensionality reduced to obtain local features. The formula is expressed as: ; Then, global average pooling is performed sequentially on the local features. and layer normalization Alignment dimensions yield the final local enhancement features. The formula is expressed as: ; A learnable residual fusion strategy is adopted, and learnable scalar parameters are introduced. A learnable scalar fusion weight is generated using the Sigmoid function. The final local enhancement feature is multiplied by the scalar fusion weight, and then added to the global feature as a residual to obtain the fused output feature. The formula is expressed as: .
[0040] The enhanced text features, global features, local patch features, and fused output features are input into the joint loss optimization module to calculate the joint loss. The joint loss is obtained by weighted summation of classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss.
[0041] In this preferred embodiment, the classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss are specifically as follows:
[0042] Classification loss: based on the true label of each image Construct a binary cross-entropy loss using the predicted spoofing probability as the classification loss. The formula is expressed as: ; in, The total number of images, Image indexing; classification loss ensures basic accuracy in recognition and classification.
[0043] Image-text contrast loss: A contrastive learning mechanism is used to calculate the one-way contrast loss from global features to enhanced text features. One-way contrastive loss from enhancing text features to global features The formula is expressed as: ; ; in, For N images, the one with Different image indexes; , These are the enhanced text features for the i-th and j-th images, respectively. , These are the global features of the i-th image and the j-th image, respectively; for transpose, for transpose;
[0044] The image-text contrast loss is obtained by taking the average of the two one-way contrast losses. Image-text contrast loss encourages visual and text images with the same semantics to move closer to each other in the representation space.
[0045] Weakly supervised localization loss: a two-dimensional feature map obtained by mapping local patch features. The class activation map is obtained after 1×1 convolution mapping and processing with the Sigmoid activation function. For a forged image, calculate the mean response value of the top K% region in its class activation map, and subtract this mean response value from 1 to obtain the localization error of the forged image. For a real image, normalize all response values of its class activation map to obtain the probability distribution Q of the class activation map; calculate the negative of the information entropy of this probability distribution as the localization loss of the real image. The localization loss of the real image is weighted and then summed with the localization loss of the forged image to obtain the weakly supervised localization loss used to locate the forged region in the forged image. The formula is expressed as: ; ; ; in, This indicates that the top k% of response values are selected; Let represent the probability distribution of the class activation map corresponding to the i-th image; Weighting is applied to the location loss;
[0046] Class-level feature alignment loss: The feature centers of the fused output features from the real image and the fake image are obtained separately, denoted as... , Calculate the squared Euclidean distance between the fused output feature of each image and its feature center; then average the squared Euclidean distances and sum them to obtain the class-level feature alignment loss. The formula is expressed as: ; in, For real image set The fused output features of the i-th image. Forging image sets The Middle The fusion output features of the images; , They are respectively , The total number of images in the dataset; class-level feature alignment loss forces images of the same class to be more compact in the feature space.
[0047] The joint loss It is obtained by weighted summation of classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss; the formula is expressed as: ; in, , , , These are the weighting coefficients corresponding to each type of loss.
[0048] The model is trained iteratively with the goal of minimizing the joint loss, and the parameters are updated. In each iteration, the performance of the AIGC image forgery detection model is evaluated using a validation set. When the stopping condition is met, the model parameters are saved to obtain the trained AIGC image forgery detection model. The trained AIGC image forgery detection model is then used to identify AIGC forged images.
[0049] Furthermore, this embodiment also provides an application system for the aforementioned forged image recognition method based on local-global feature fusion and joint loss learning, comprising a dataset construction module, an AIGC image forgery detection model, and a training and recognition module, wherein:
[0050] The dataset construction module is used to construct a dataset containing real images and fake images, and divides it into a training set, a validation set and a test set;
[0051] The AIGC image forgery detection model obtains the fused output features of real and forged images through a text generation and enhancement module, a visual-text feature extraction module, a local feature enhancement module, and a local-global feature fusion module. Based on the fused output features, it obtains the predicted forgery probability value. Then, through the joint loss optimization module, it calculates the joint loss obtained by weighted summation of classification loss, image-text comparison loss, weak supervision localization loss, and class-level feature alignment loss.
[0052] The training and recognition module iteratively trains the AIGC image forgery detection model with the goal of minimizing the joint loss, and finally uses the trained model to realize AIGC forgery image recognition.
[0053] An electronic device is also disclosed, comprising a memory and a processor, wherein: Memory is used to store computer programs that can run on a processor; A processor is configured to, while running the computer program, execute a forged image recognition method based on local-global feature fusion and joint loss learning as described above.
[0054] A computer-readable storage medium is also disclosed, which stores computer instructions for causing a processor to execute and implement a forged image recognition method based on local-global feature fusion and joint loss learning as described above.
[0055] In summary, this invention provides an efficient and generalized method for detecting forged images. Under the constraint of limited training data, it can more fully mine the key detailed features of real / forged images at multiple scales. At the same time, it introduces multiple loss mechanisms to deeply optimize the feature space, further enhancing the intra-class compactness and inter-class separation of real and forged images, and significantly improving the recognition accuracy and generalization robustness of AIGC forged images from unknown generative models.
[0056] In this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to at least one embodiment or example described in connection with a specific feature, structure, material, or characteristic. These specific features, structures, materials, or characteristics may be combined in a suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples and their features described in this specification.
[0057] The logic and / or steps shown in the flowchart or otherwise described can be viewed as a sequence of executable instructions for implementing logical functions. These instructions may be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. Such systems, apparatus, or devices include processor systems or other systems capable of receiving and executing instructions.
[0058] The above embodiments detail the principles and implementation methods of the present invention, and illustrate its working principle using specific examples. These examples are only used to help understand the method and core ideas of the present invention. Furthermore, based on the ideas of the present invention, actual implementation methods and application scope may vary. Therefore, the content of this specification should not be construed as limiting the present invention.
Claims
1. A method for fake image identification based on local-global feature fusion and joint loss learning, characterized in that, Specifically, the following steps are included: Collect and organize real images, and generate fake images based on the AIGC generation paradigm; The AIGC image forgery recognition dataset is constructed based on real images and forged images, and the AIGC image forgery recognition dataset is divided into training set, validation set and test set. An AIGC image forgery detection model is constructed, comprising a text generation and enhancement module, a visual-text feature extraction module, a local feature enhancement module, a local-global feature fusion module, and a joint loss optimization module. The original images in the training set are input into the text generation and enhancement module, which extracts semantic descriptions from the original images and constructs enhanced text using general prompts for true and false categories; the original images include real images and fake images. The enhanced text and the original images in the training set are input into the visual-text feature extraction module, which consists of a text encoder and a visual encoder; the text encoder extracts enhanced text features from the enhanced text, and the visual encoder extracts global features and local patch features from the original images; The local patch features are input into the local feature enhancement module. The local attention branch and the multi-scale dilated convolution branch of the local feature enhancement module are used to enhance the features, resulting in local attention enhanced features and multi-scale structural features. Local features are obtained by fusing local attention enhancement features with multi-scale structural features. These local features are then input into a local-global feature fusion module, where they are fused with global features using a learnable residual fusion strategy to obtain fused output features. A linear classifier maps the fused output features of each image to the predicted forgery probability of that image. The enhanced text features, global features, local patch features, and fused output features are input into the joint loss optimization module to calculate the joint loss. The joint loss is obtained by weighted summation of classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss. The model is trained iteratively with the goal of minimizing the joint loss, and the parameters are updated. In each iteration, the performance of the AIGC image forgery detection model is evaluated using a validation set. When the stopping condition is met, the model parameters are saved to obtain the trained AIGC image forgery detection model. The trained AIGC image forgery detection model is then used to identify AIGC forged images. 2.The method of claim 1, wherein, The training set, validation set, and test set specifically include: The training set includes real images obtained from public datasets, as well as fake images generated using a single adversarial generative network; the validation set is from the same source as the training set and has the same distribution; the test set includes fake images generated by various different generative adversarial networks or diffusion models, as well as real images of the same number as fake images.
3. The forged image recognition method based on local-global feature fusion and joint loss learning according to claim 1, characterized in that, The extraction of global features and local patch features from the original image using a visual encoder specifically includes: Feature extraction is performed using a pre-trained CLIP visual encoder. With the main parameters of the pre-trained CLIP visual encoder frozen, a low-rank adaptive module (LoRA) is introduced to fine-tune the parameters. The original images from the training set are input into the fine-tuned CLIP visual encoder to extract one-dimensional features representing the image context and patch markers. Using the one-dimensional features as global features, the patch markers are reshaped into a two-dimensional feature map, serving as local patch features.
4. The forged image recognition method based on local-global feature fusion and joint loss learning according to claim 1, characterized in that, The process of inputting local patch features into the local feature enhancement module, and performing feature enhancement through the local attention branch and multi-scale dilated convolution branch of the local feature enhancement module to obtain local attention-enhanced features and multi-scale structural features specifically includes: In the local attention branch, the local patch features are first subjected to a depthwise separable convolution operation to extract local spatial context and channel information; then, channel-level local attention weights are generated through the Sigmoid activation function; the generated channel-level local attention weights are multiplied element-wise with the local patch features to obtain the local attention enhancement features. In the multi-scale dilated convolution branch, three sets of 3×3 convolution operations with different dilation rates are used to process the local patch features of the input in parallel. Then, the outputs of different branches are added and aggregated element by element to obtain multi-scale structural features.
5. The forged image recognition method based on local-global feature fusion and joint loss learning according to claim 1, characterized in that, The process involves fusing local attention enhancement features with multi-scale structural features to obtain local features, which are then input into a local-global feature fusion module. Through a learnable residual fusion strategy, these local features are fused with global features to obtain the fused output features. Specifically: We use 1×1 convolution to concatenate and reduce the dimensionality of local attention enhancement features and multi-scale structural features to obtain fused features; then we perform global average pooling and layer normalization on the local features in sequence to align the dimensions and obtain the final local features. A learnable residual fusion strategy is adopted. A learnable scalar fusion weight is generated by the Sigmoid function. The final local feature is multiplied by the scalar fusion weight and then added to the global feature by residual addition to obtain the fused output feature.
6. The forged image recognition method based on local-global feature fusion and joint loss learning according to claim 1, characterized in that, The classification loss, image-text comparison loss, weakly supervised localization loss, and class-level feature alignment loss are as follows: Classification loss: A binary cross-entropy loss is constructed based on the true label and the predicted probability of forgery for each image, and used as the classification loss; Image-text contrast loss: A contrastive learning mechanism is used to calculate the one-way contrast loss from global features to enhanced text features and the one-way contrast loss from enhanced text features to global features; the mean of the two one-way contrast losses is taken to obtain the image-text contrast loss. Weakly supervised localization loss: Local patch features are processed by mapping and activation functions to obtain a class activation map; for a forged image, the mean response of the top K% region of its class activation map response value is calculated, and 1 is subtracted from the mean response value to obtain the localization error of the forged image; for a real image, all response values of its class activation map are normalized to obtain the probability distribution of the class activation map; the negative value of the information entropy of this probability distribution is calculated as the localization loss of the real image; the localization loss of the real image is weighted and summed with the localization loss of the forged image to obtain the weakly supervised localization loss; Class-level feature alignment loss: Obtain the feature centers of the fused output features of the real image and the fake image respectively, calculate the square of the Euclidean distance between the fused output feature of each image and its feature center; then take the mean of the squares of the Euclidean distances and sum them to obtain the class-level feature alignment loss.
7. A forged image recognition system based on local-global feature fusion and joint loss learning, wherein the system performs the method as described in any one of claims 1-6, characterized in that, include: The dataset construction module, the AIGC image forgery detection model, and the training and recognition module include: The dataset construction module is used to construct a dataset containing real images and fake images, and divides it into a training set, a validation set and a test set; The AIGC image forgery detection model obtains the fused output features of real and forged images through a text generation and enhancement module, a visual-text feature extraction module, a local feature enhancement module, and a local-global feature fusion module. Based on the fused output features, it obtains the predicted forgery probability value. Then, through the joint loss optimization module, it calculates the joint loss obtained by weighted summation of classification loss, image-text comparison loss, weak supervision localization loss, and class-level feature alignment loss. The training and recognition module iteratively trains the AIGC image forgery detection model with the goal of minimizing the joint loss, and finally uses the trained model to realize AIGC forgery image recognition.
8. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: Memory is used to store computer programs that can run on a processor; A processor, configured to, while running the computer program, execute a forgery image recognition method based on local-global feature fusion and joint loss learning as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement a forgery image recognition method based on local-global feature fusion and joint loss learning as described in any one of claims 1-6.