An example synthetic image detection method and system for poly and cross-artifact simulation

Abstract

The application discloses a kind of instance synthesis image detection methods and systems of aggregation and orthogonal artifact simulation, comprising: the preprocessed image is obtained to input image uniform size preprocessing, is sent into pre-training CLIP visual encoder, and the picture feature of multiple intermediate layers is extracted;Mapping feature is obtained by mapping multiple layer features to the same feature space, configuring lightweight scoring network, calculating the effective weight of each layer mapping feature, weighted sum all mapping features, output uniform aggregation feature;In the training phase, enable orthogonal constraint controlled feature artifact simulation to force image detection model to learn general fake features, obtain the completed classifier of training;In the inference phase, close orthogonal artifact simulation disturbance module, uniform aggregation feature is directly sent into classifier, and the probability that image is real image or synthetic image is output, and detection is completed.The application closes disturbance module when reasoning, only uses aggregation feature to complete rapid detection, so as to give consideration to detection performance, generalization ability and deployment efficiency.

Description

Technical Field

[0001] This invention relates to the field of synthetic image detection technology, and more specifically to a synthetic image detection method and system based on instance aggregation and orthogonal artifact simulation. Background Technology

[0002] With the rapid development of image generation technologies such as generative adversarial networks and diffusion models, synthetic images are constantly improving in terms of realism, detail consistency, and semantic plausibility. It is becoming increasingly difficult for ordinary users and even professional reviewers to determine the authenticity of images with the naked eye alone. Therefore, reliable synthetic image detection technology has become a crucial link in network content governance, identity authentication, digital forensics, and multimedia security.

[0003] Existing synthetic image detection methods can be broadly categorized into spatial domain detection, frequency domain detection, generator fingerprint detection, and detection methods utilizing large-scale pre-trained models. Spatial domain methods typically learn the differences between real and fake images directly from image texture, edges, local patterns, or convolutional features; frequency domain methods use Fourier transform, wavelet transform, and other techniques to find spectral anomalies left by the generation process; generator fingerprint methods attempt to capture stable traces left by specific generation models or upsampling structures; and in recent years, methods have also leveraged pre-trained visual models such as CLIP and ViT to obtain stronger semantic priors and transfer capabilities.

[0004] While the aforementioned methods can achieve good results when the training and test sets come from the same or similar generators, they still have significant shortcomings in real-world applications. First, the detector is prone to learning artifacts specific to the generator used for training, rather than learning stable forgery cues common to all synthesized images. Detection accuracy drops significantly when test images come from unseen GANs, diffusion models, or commercially available closed-source models. Second, simply increasing the variety of training generators can expand the data distribution, but this approach is passive, time-consuming, and costly, making it difficult to keep up with the rapid iteration of new generative models. Furthermore, many advanced generative models do not publicly disclose their training interfaces or sample sources. Third, existing feature augmentation or data augmentation methods often directly add unstructured perturbations to images or features, easily destroying already subtle forgery traces and failing to specifically distinguish changes related to real / fake detection as well as changes related to image content, background, and style. Fourth, some methods rely solely on deep semantic features, easily ignoring shallow, fine-grained artifacts; while methods relying solely on shallow or frequency cues are easily affected by image content, compression, and post-processing. Summary of the Invention

[0005] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a synthetic image detection method based on instance aggregation and orthogonal artifact simulation. The core technical problem solved by this invention is: how to prevent the synthetic image detector from overfitting to the specific artifacts of a certain training generator when it is impossible to obtain all unknown generator samples in advance, and instead learn general forgery features that are still effective for unknown GANs, diffusion models and compressed images.

[0006] To achieve the above-mentioned technical objectives, in a first aspect, the present invention provides a method for detecting synthetic images based on instance aggregation and orthogonal artifact simulation, comprising the steps of: Obtain the input image and perform uniform size preprocessing on the input image to obtain a preprocessed image; The preprocessed image is fed into the pre-trained CLIP visual encoder, and image features are extracted through multiple intermediate layers of the visual encoder: the shallow layer preserves texture and subtle edge artifacts, the deep layer stores global semantic information of objects and scenes, and retains all layer features for subsequent fusion. A learnable projection layer is set up to map multi-layer features of different dimensions and scales to the same feature space to obtain mapped features. Each layer is configured with a lightweight scoring network to dynamically calculate the effective weights of the mapped features of each layer for a single input image. All the mapped features are weighted and summed to output a unified aggregated feature that takes into account both shallow fine-grained artifacts and high-level semantics. During the training phase, class centers are constructed based on the unified aggregated features and the feature subspace is orthogonally decomposed. Orthogonal constraint controlled feature artifact simulation is enabled to force the image detection model to learn general forgery features, thus obtaining a trained classifier. During the inference phase, the orthogonal artifact simulation perturbation module is turned off. The image under test only undergoes preprocessing, CLIP multi-layer feature extraction, and instance-level aggregation to obtain unified aggregated features, which are directly fed into the classifier to output the probability that the image is a real image or a synthetic image, thus completing the detection.

[0007] Compared with the prior art, the beneficial effects of the present invention include: This invention addresses the shortcomings of insufficient generalization ability and inadequate training sample distribution coverage in related technologies for synthetic image detection. It proposes a synthetic image detection method based on instance-level feature aggregation and feature-level artifact simulation under orthogonal constraints. Instead of passively collecting more generator data, this method actively simulates diverse and challenging artifact samples in the feature space during the training phase, thereby expanding the artifact distribution seen by the detector. Specifically, a pre-trained visual encoder is first used to extract multi-layer intermediate features from the input image. An instance-level aggregation module adaptively selects and fuses shallow artifact cues and deep semantic cues for different images, resulting in more comprehensive aggregated features. Then, a true category center and a fake category center are established in this aggregated feature space. A discrimination direction from the true center to the fake center is defined, and the feature space is decomposed into a fake-related discriminative subspace and a non-discriminative subspace orthogonal to it. Finally, the random Gaussian perturbation is decomposed into these two subspaces. In the orthogonal subspace, the changes of non-real and false factors such as content, background and style are simulated. Difficult perturbations that make the samples close to the opposite class are generated according to the class label in the discrimination direction. In this way, the classifier is trained to learn a smoother, larger-margin, and less generator-dependent true and false discrimination boundary.

[0008] This invention adaptively fuses multi-layer visual representations on an input instance basis, taking into account both subtle low-level artifacts and high-level semantic context. It utilizes real / fake category centers to construct orthogonal subspaces, giving feature perturbations a clear direction for true / false discrimination and a direction for non-discriminatory changes. During training, it generates diverse and difficult samples through controlled feature perturbations, and during inference, it disables the perturbation module, using only aggregated features to achieve rapid detection, thus balancing detection performance, generalization ability, and deployment efficiency.

[0009] According to some embodiments of the present invention, a preprocessed image is obtained by performing uniform size preprocessing on the input image, including the following steps: The input image is processed to a uniform size by scaling it to a set size, and then randomly cropped during the training phase and centered and cropped to the target size during the inference phase. The input image is normalized by normalizing each pixel channel according to the mean and variance of the CLIP pre-trained model to obtain a normalized image. The normalized image is divided into multiple fixed-size image blocks. Each image block is flattened into a vector and mapped to an embedding block through a linear embedding layer. The embedding block is added to the category label and position code to form the input sequence of layer 0.

[0010] According to some embodiments of the present invention, the preprocessed image is fed into a pre-trained CLIP visual encoder, and image features are extracted through multiple intermediate layers of the visual encoder, including the following steps: The preprocessed image is sequentially processed Each visual block includes: multi-head self-attention, feedforward network, layer normalization, and residual connection; Image features are extracted from multiple intermediate visual patches, denoted as... , ... The shallow features include texture, edges, and subtle artifacts of local noise, while the deep features include object semantics, scene content, and global structure.

[0011] According to some embodiments of the present invention, a learnable projection layer is set to map multi-layer features of different dimensions and scales to the same feature space to obtain mapped features, including the following steps: Let the first Layer features are , dimension Set the corresponding learnable projection layer The learnable projection layer consists of a linear layer, a normalization layer, and an activation function. Mapping to a unified dimension The latent space is mapped to features.

[0012] According to some embodiments of the present invention, a lightweight scoring network is configured for each layer, and the effective weights of the mapping features of each layer are dynamically calculated for a single input image. All the mapping features are then weighted and summed to output a unified aggregated feature that takes into account both shallow fine-grained artifacts and high-level semantics. The steps include: Each layer constructs a lightweight scoring network, and the mapping features are input into the lightweight scoring network to obtain scalar scores; The scalar scores of all layers are Softmax normalized to obtain the corresponding weight scores. The multi-layer mapping features are weighted and summed to output the unified aggregated features that take into account both shallow fine-grained artifacts and high-level semantics.

[0013] According to some embodiments of the present invention, constructing class centers and orthogonally decomposing feature subspaces based on the unified aggregated features during the training phase includes the following steps: Construct two learnable category centers: real image centers and fake image centers; During training, the center constraint loss is used to make the features of real samples closer to the center of the real image and the features of fake samples closer to the center of the fake image, so that real samples and fake samples form a clear structure in the feature space. The vector pointing from the center of the real image to the center of the fake image is used to define the direction of authenticity discrimination. The overall feature space is divided into two orthogonal subspaces: a discriminative subspace, which only affects the authenticity attribute of the image; and an orthogonal non-discriminative subspace, which only corresponds to changes in image content, background, style, color, etc., which are unrelated to authenticity. Two sets of projection operators are generated: discriminative directional projection and orthogonal directional projection. Random perturbations are decomposed by these two sets of projection operators before use, so that the perturbations do not change all feature dimensions aimlessly, but serve the two objectives of "generating difficult true and false samples" and "expanding the non-discriminative change distribution".

[0014] According to some embodiments of the present invention, enabling orthogonal constraint controlled feature artifact simulation forces an image detection model to learn general forgery features, resulting in a trained classifier, including the following steps: Random noise is sampled from the standard Gaussian distribution, and Bernoulli variables are sampled to determine whether a perturbation is applied to the current sample. Random coefficients between 0 and 1 are then sampled to control the perturbation intensity in the discrimination direction. Random noise is first decomposed into a discrimination direction component and an orthogonal direction component according to the discrimination direction. The orthogonal direction component is used to simulate changes in content, background, style, etc. that are weakly correlated with the true and false discrimination, so that the detector can keep these changes smooth and stable. The discrimination direction component is controlled by the class-related sign. Real samples are slightly pushed to the fake side, and fake samples are slightly pushed to the real side, thus generating difficult samples that are closer to the classification boundary. Based on the enhanced features generated from the difficult samples, the image detection model learns the enhanced features of the difficult samples to obtain a trained classifier.

[0015] According to some embodiments of the present invention, based on the generated enhanced features of the difficult samples, the image detection model learns the enhanced features of the difficult samples to obtain a trained classifier, including the following steps: The generated enhanced features are fed into the classifier. The classifier performs a linear operation on the enhanced features by combining its own weights and bias parameters, and outputs the corresponding probabilities of the image belonging to the real image and the fake image. The classification loss is calculated based on the true and false probabilities predicted by the model for the real image label: if it is a real image, the loss is calculated using only the probability prediction of the real image; if it is a fake image, the loss is calculated using only the probability prediction of the fake image. The overall deviation between the model's true and false judgment result and the real label is measured by the cross-entropy formula. For each image's aggregated features, calculate the squared distance from the aggregated feature to the feature center of its own category. Take the average of the distance values ​​of all samples in the batch to obtain the category center constraint loss. Set a fixed weight coefficient to adjust the proportion of the class center constraint loss, add the classification loss to the weighted class center constraint loss, and obtain the total loss value used for model training. Using the total loss as the optimization objective, the backpropagation algorithm is executed to synchronously update all learnable parameters in the model, resulting in a trained classifier.

[0016] In a second aspect, the present invention provides a synthetic image detection system based on instance aggregation and orthogonal artifact simulation, which applies the synthetic image detection method based on instance aggregation and orthogonal artifact simulation as described in any one of the first aspects.

[0017] The core technical problem solved by this invention is: how to prevent the synthetic image detector from overfitting to the specific artifacts of a particular trained generator, and instead learn general forgery features that remain effective for unknown GANs, diffusion models, and compressed images, when all unknown generator samples cannot be obtained in advance. Compared with existing methods, the significant features of this invention are: first, it adaptively fuses multi-layer visual representations on an input instance basis, taking into account both subtle low-level artifacts and high-level semantic context; second, it constructs an orthogonal subspace using real / forged category centers, giving feature perturbations a clear direction for real / forged discrimination and a direction for non-discriminatory changes; and third, it generates diverse and difficult samples through controlled feature perturbations during training, and disables the perturbation module during inference, using only aggregated features to complete fast detection, thus balancing detection performance, generalization ability, and deployment efficiency.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein the abstract drawings are to be completely consistent with one of the drawings in the specification: Figure 1 This is a flowchart of a synthetic image detection process based on instance aggregation and orthogonal artifact simulation, provided as an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned figures are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Reference Figure 1 , Figure 1 This is a flowchart of a synthetic image detection process based on instance aggregation and orthogonal artifact simulation, provided as an embodiment of the present invention.

[0023] Overall Process Overview: The input of this invention is an image to be detected, and the output is a judgment result indicating whether the image is a real image or a synthetic image. The overall process is divided into a training phase and an inference phase. In the training phase, in addition to normal feature extraction and classification, an artifact simulation module under orthogonal constraints is enabled to construct perturbation samples on aggregated features, allowing the detector to encounter more diverse and harder-to-distinguish artifact patterns during training. In the inference phase, this perturbation module is disabled, retaining only image preprocessing, feature extraction, instance-level aggregation, and the classifier, thus avoiding additional generator sampling costs. For ease of understanding, a specific example from input image to judgment result output is given below.

[0024] In one embodiment, the synthetic image detection method based on instance aggregation and orthogonal artifact simulation includes the following steps: acquiring an input image; performing uniform-size preprocessing on the input image to obtain a preprocessed image; feeding the preprocessed image into a pre-trained CLIP visual encoder, extracting image features through multiple intermediate layers of the visual encoder: shallow layers preserve texture and subtle edge artifacts, deep layers store global semantic information of objects and scenes, and retain all layer features for subsequent fusion; setting a learnable projection layer to map multi-layer features of different dimensions and scales to the same feature space to obtain mapped features; configuring a lightweight scoring network for each layer to dynamically evaluate a single input image. The effective weights of each layer of mapping features are calculated, and all mapping features are weighted and summed to output a unified aggregated feature that takes into account both shallow fine-grained artifacts and high-level semantics. During the training phase, class centers are constructed based on the unified aggregated feature, and the feature subspace is orthogonally decomposed. Orthogonal constraint controlled feature artifact simulation is enabled to force the image detection model to learn general forgery features, resulting in a trained classifier. During the inference phase, the orthogonal artifact simulation perturbation module is turned off, and the image to be tested only undergoes preprocessing, CLIP multi-layer feature extraction, and instance-level aggregation to obtain unified aggregated features, which are directly fed into the classifier to output the probability that the image is a real image or a synthetic image, thus completing the detection.

[0025] Specific example: Suppose a user uploads a JPEG image of a face to be detected. The system will first The image is scaled to 256x256 and then centered to 224x224. Following this, pixels are normalized and segmented into image patches according to the CLIP visual encoder requirements, resulting in the visual encoder input sequence. After the encoder outputs multiple intermediate layer features, the instance-level aggregation module maps these features to the same latent space and calculates layer weights to obtain the aggregated feature z. The classifier outputs two probabilities for z, such as the probability of the true image. Synthetic image probability .because If the value is high, the system outputs a judgment result stating "This image is a composite image," and can simultaneously provide 0.88 as a forgery confidence score. If the output is... , If the system outputs the correct image, then the image is considered a real image. This case corresponds to the complete chain of the inference stage; the artifact simulation in the training stage is only used to improve the generalization ability of the aforementioned classifier.

[0026] 1. Image preprocessing and multi-layer visual feature extraction: For the input image First, the image is resized to a uniform size, for example, scaled to 256 x 256. Then, it is randomly cropped during training and centered to 224 x 224 during inference. Subsequently, each pixel channel is normalized according to the mean and variance of the CLIP pre-trained model to obtain a normalized image. If a CLIP visual encoder with a ViT structure is used, then... The image is divided into multiple fixed-size blocks, such as 16 x 16 blocks. Each block is flattened into a vector, mapped to an embedding block through a linear embedding layer, and added to the class label and position code to form the input sequence of layer 0. The sequence passes through... Each Vision Transformer (ViT) block consists of a multi-head self-attention network, a feedforward network, layer normalization, and residual connections, and can be represented as follows: This invention extracts features from multiple intermediate visual blocks, denoted as... , ... Shallow features mainly include subtle artifacts such as texture, edges, and local noise, while deep features mainly include object semantics, scene content, and global structure, providing multiple layers of information for subsequent aggregation.

[0027] 2. Instance-level aggregation module: Features from different layers reside in different representation spaces, and direct addition or concatenation can lead to inconsistencies in information scale. Therefore, this invention first performs explicit mapping on the features of each layer. Let the first layer be... Layer features are , dimension Set the learnable projection layer corresponding to this layer. For example, it consists of linear layers, normalized layers, and activation functions. Mapping to a unified dimension Potential space: ,in and For learnable parameters, Representation layer normalization, To unify the representation space of the first Layer features.

[0028] After mapping is completed, this invention calculates an importance score related to the instance for each layer of mapped features. Specifically, it will... Input Lightweight Rating Network For example, a two-layer MLP (Multilayer Perceptron) yields a scalar score. ,in This is a non-linear activation function. Then, the scores of all layers are normalized using Softmax to obtain the weights of the l-th layer. Finally, the multi-layer mapping features are weighted and summed to obtain the aggregated features. For example, when four intermediate layers are selected with scores of 0.2, 0.6, -0.1, and 1.0, the weights after Softmax are approximately 0.19, 0.28, 0.14, and 0.39, indicating that the current sample relies more on the features of the fourth and second layers; the system calculates accordingly. This calculation process enables each input image to adaptively select the most discriminative hierarchical information.

[0029] The key to this module is its "instance-level" nature. This means that forgery cues in different images may be distributed across different layers: some image artifacts manifest in low-level textures, while others rely more on semantic inconsistencies or global structural anomalies. Through the aforementioned projection, scoring, normalization, and weighted summation steps, the model can dynamically calculate layer weights for each input image, adaptively selecting the most useful layer features for the current image. This avoids both using only deep features and losing subtle artifacts, and using only low-level features and lacking semantic context, thus providing a more stable and comprehensive feature space for subsequent artifact simulation.

[0030] 3. Class center and orthogonal subspace decomposition: After obtaining the aggregated feature z, this invention maintains two learnable category centers: the real image center and the ground truth center. and fake image center During training, a center constraint loss is used to bring the features of real samples closer to the center of the real image, and the features of fake samples closer to the center of the fake image. This allows real and fake samples to form a clearer structure in the feature space, rather than relying entirely on the last layer of the classifier to learn on its own.

[0031] Based on these two category centers, this invention defines a unit vector pointing from the true center to the fake center. That is, the direction for judging truth from falsehood. Along... When moving, features will more clearly move from the real side to the fake side, or from the fake side to the real side. Orthogonal directions tend to represent factors less relevant to true / false classification, such as image content, background, shooting style, color distribution, or other semantic changes. Therefore, this invention constructs two projection operators: a discriminative direction projection operator and a method for determining the projection direction. Orthogonal projection Before being used, the random perturbation is decomposed by these two projection operators, so that the perturbation no longer changes all feature dimensions aimlessly, but serves two objectives respectively: "generating difficult true and false samples" and "expanding the non-discriminatory change distribution".

[0032] 4. Controlled feature perturbation and artifact simulation: During the training phase, this invention samples random noise from a standard Gaussian distribution. And sample Bernoulli variables Determine whether to apply a perturbation to the current sample, and then resample random coefficients between 0 and 1. Control the intensity of the perturbation in the direction of discrimination. Random noise is first determined according to the direction of discrimination. Decompose: Determine the directional components as The orthogonal direction components are The orthogonal direction component is used to simulate changes in content, background, style, etc., that are weakly correlated with the true / false distinction, thus allowing the detector to remain smooth and stable in the face of these changes. The discriminative direction component, on the other hand, uses class-related sign control: real samples are slightly pushed toward the fake side, and fake samples are slightly pushed toward the real side, thereby generating hard samples that are closer to the classification boundary.

[0033] Specifically, due to Pointing from the real center to the fake center, therefore setting the category symbol. When the sample is a real image When the sample is a fake image The direction disturbance is calculated as follows: The complete disturbance is .like The enhanced feature is ;like ,but ,in Let be the total perturbation intensity. Using a simplified example in a two-dimensional feature space: assuming... , ,but Aggregated features of a certain forged sample Random noise , , , Because this sample is a fake sample, ;at this time , , , ,final This enhanced feature is closer to the true side than the original feature and has orthogonal transformations, thus it can be used as a difficult training sample.

[0034] 5. Training objectives and reasoning process: During training, the classifier applies the augmented features Perform true / false prediction. Specifically, first calculate the classifier logit vector. Then, the true probability is obtained through Softmax. and the probability of forgery If the tag Representing real images, If the image is forged, then the cross-entropy loss is... Meanwhile, the class center constraint loss continues to constrain sample features to be close to the corresponding class center, which can be written as... Where B is the batch size. The center of the sample's category. The final training objective is... , The loss weights are centered and constrained. For example, if the classifier outputs a fake sample... Then the classification loss for this sample is If the squared distance from its aggregated features to the forgery center is 0.30 and The center constraint term contributes 0.003, and the total loss is approximately 0.131. The model updates the parameters of the visual adaptation layer, aggregation module, category center, and classifier through backpropagation and optimizers such as Adam.

[0035] During inference, noise is no longer sampled, and artifact simulation is no longer performed. The image to be tested only needs to undergo the same preprocessing and CLIP visual encoder to obtain multi-layer features. Then, the instance-level aggregation module generates z, which is directly fed into the classifier to output the probability of being real or fake. For example, the image to be detected, I_0, is obtained by the classifier... , When the system determines it to be a composite image, if Greater than the preset threshold or greater than If the image does not match the description, it is considered a real image. Since artifact simulation only works during the training phase, the computational cost of this invention during the deployment phase is close to that of a typical feature-based classification detector, but its generalization ability comes from the richer feature distribution constructed during the training phase.

[0036] 6. The reason why this invention can solve the technical problem: First, the instance-level aggregation module incorporates both shallow, subtle artifacts and deep semantic cues into the detection representation, reducing missed detections caused by insufficient features in a single layer. Second, orthogonal subspace decomposition breaks down perturbations into "true / false discrimination directions" and "non-discriminative directions," preventing unstructured noise from destroying key artifacts while allowing the model to see more content, background, and style variations. Third, the category-related discrimination direction perturbation actively constructs difficult samples close to the classification boundary, effectively increasing the training difficulty in the feature space and prompting the classifier to learn a larger safety margin. Fourth, the aforementioned artifact simulation does not require additional collection of unknown generator images, making it suitable for real-world scenarios where new GANs, new diffusion models, and closed-source commercial models are rapidly emerging.

[0037] The effect after improvement: The experiments of this invention verify the effectiveness from three perspectives: cross-generator generalization, JPEG compression robustness, and module ablation. In the experiments, the training set uses images generated by ProGAN in ForenSynths, and sets up training scenarios with 1, 2, and 4 semantic categories; the test set covers unseen GAN models and diffusion models. Evaluation metrics include Accuracy (Acc.) and Average Precision (AP), where accuracy reflects the overall proportion of correct judgments, and average precision reflects the comprehensive detection capability under different thresholds.

[0038] In the diffusion model testing scenario, the detector is trained on GAN-generated data. The test set includes images generated by diffusion models such as DALL-E, GLIDE, ADM, and LDM. The generation mechanisms of these models differ significantly, making this a more challenging cross-paradigm generalization test. As shown in Table 1, this invention achieves an average accuracy of 93.8% and an average precision of 98.6% in training class 1, 95.2% and 98.7% in training class 2, and 96.7% and 98.1% in training class 4. Compared to the second-best method, this invention improves the average accuracy by 2.1%, 5.2%, and 3.8% in the three training settings, respectively. This indicates that this invention learns not ProGAN-specific artifacts, but rather general forgery cues that can be transferred to modern diffusion models.

[0039] Table 1 Results of the generalization experiment on the diffusion model

[0040] In the GAN cross-generator testing scenario, the model was still trained using ProGAN, but the tests covered various unseen generative models such as StyleGAN, StyleGAN2, BigGAN, CycleGAN, StarGAN, GauGAN, and DeepFake. As shown in Table 2, this invention achieved an average accuracy of 97.6% and an average precision of 99.6% for training with 1 class, an average accuracy of 97.1% and an average precision of 99.8% for training with 2 classes, and an average accuracy of 96.7% and an average precision of 97.2% for training with 4 classes. Compared with the second-best method, the average accuracy was improved by 3.0%, 2.4%, and 1.3%, respectively. It maintained a significant advantage, especially when the number of training classes was small, indicating that orthogonal artifact simulation in the feature space can alleviate the overfitting problem caused by limited training sources.

[0041] Table 2 Results of generalization experiments on GAN

[0042] In JPEG compression robustness tests, the present invention also maintains strong performance. As shown in Tables 3 and 4, under two training settings, an average accuracy of 80.9% is achieved in quality-aware scenarios, an improvement of 5.5% compared to the second-best method. Under four training settings, an average accuracy of 85.0% is achieved in quality-aware scenarios, exceeding the second-best method by 8.5%. This indicates that the features obtained by the present invention do not rely on fragile artifacts easily destroyed by compression, but rather possess better compression invariance.

[0043] Table 3 Generalization experiment results under the two training settings in the quality-aware scenario.

[0044] Table 4. Generalization Experiment Results under Four Training Settings in a Quality-Aware Scenario

[0045] Ablation experiments further demonstrate the necessity of each module. As shown in Table 5, the baseline detector's average accuracy across generators using GANs is 92.1%; this improves to 95.0% after adding instance-level aggregation; it reaches 94.8% when using artifacts to simulate relevant perturbations but without instance-level aggregation; it reaches 94.4% when retaining instance-level aggregation and controlled perturbations but removing orthogonal subspace decomposition; and the complete method achieves 97.6%. These results indicate that instance-level aggregation can provide a more suitable basic feature space for perturbations and classification, orthogonal subspace decomposition can avoid confusion in perturbation direction, and controlled feature perturbations can explicitly generate diverse and challenging training samples. The combination of these three elements is necessary to achieve the best generalization effect.

[0046] Table 5 Ablation Experiment Results of Key Modules

[0047] Therefore, this invention significantly improves the generalization ability and practical value of synthetic image detectors when facing unknown generation models, diffusion models and compressed images, without relying on additional unknown generator data.

[0048] The preprocessing of the input image to obtain a preprocessed image includes the following steps: scaling the input image to a set size, then randomly cropping it during the training phase and center-cropping it to the target size during the inference phase; normalizing each pixel channel of the input image according to the mean and variance of the CLIP pre-trained model to obtain a normalized image; dividing the normalized image into multiple fixed-size image blocks, flattening each image block into a vector, mapping it to an embedding block through a linear embedding layer, and adding the embedding block with the category label and position code to form the input sequence of layer 0.

[0049] Furthermore, the preprocessed image is fed into a pre-trained CLIP visual encoder, and image features are extracted through multiple intermediate layers of the visual encoder, including the following steps: the preprocessed image sequentially passes through... There are 3 visual blocks, each consisting of: multi-head self-attention, a feedforward network, layer normalization, and residual connections; image features are extracted from multiple intermediate visual blocks, denoted as . , ... The shallow features include texture, edges, and subtle artifacts of local noise, while the deep features include object semantics, scene content, and global structure.

[0050] Furthermore, a learnable projection layer is set up to map multi-layer features of different dimensions and scales to the same feature space to obtain mapped features, including the following steps: Let the first... Layer features are , dimension Set the corresponding learnable projection layer The learnable projection layer consists of a linear layer, a normalization layer, and an activation function. Mapping to a unified dimension The latent space is mapped to features.

[0051] Furthermore, a lightweight scoring network is configured for each layer. For a single input image, the effective weights of the mapping features of each layer are dynamically calculated. All mapping features are weighted and summed to output a unified aggregated feature that takes into account both shallow fine-grained artifacts and high-level semantics. The steps include: constructing a lightweight scoring network for each layer, inputting the mapping features into the lightweight scoring network to obtain scalar scores; performing Softmax normalization on the scalar scores of all layers to obtain corresponding weight scores; performing weighted summation on the mapping features of multiple layers to output a unified aggregated feature that takes into account both shallow fine-grained artifacts and high-level semantics.

[0052] Furthermore, during the training phase, class centers are constructed based on unified aggregated features, and the feature subspace is orthogonally decomposed. This includes the following steps: constructing two learnable class centers: a real image center and a fake image center; during training, using center constraint loss, real sample features are made closer to the real image center, and fake sample features are made closer to the fake image center, so that real and fake samples form a clear structure in the feature space; defining the true / false discrimination direction using the vector pointing from the real image center to the fake image center, and splitting the overall feature space into two orthogonal subspaces: a discriminative subspace, which only affects the true / false attributes of the image; and an orthogonal non-discriminative subspace, which only corresponds to changes in image content, background, style, color, etc., which are unrelated to true / false; generating two sets of projection operators, discriminative direction projection and orthogonal direction projection, and random perturbations are decomposed by these two sets of projection operators before use, so that the perturbations do not aimlessly change all feature dimensions, but serve the two goals of "generating difficult true / false samples" and "expanding the distribution of non-discriminative changes".

[0053] Furthermore, enabling orthogonal constraint controlled feature artifact simulation forces the image detection model to learn general forgery features, resulting in a trained classifier. This involves the following steps: sampling random noise from a standard Gaussian distribution and sampling Bernoulli variables to determine whether to apply a perturbation to the current sample; then sampling random coefficients between 0 and 1 to control the intensity of the perturbation in the discrimination direction; the random noise is first decomposed into discrimination direction components and orthogonal direction components according to the discrimination direction. The orthogonal direction components are used to simulate changes in content, background, style, etc., that are weakly correlated with the real / fake distinction, thus allowing the detector to remain smooth and stable in response to these changes. The discrimination direction components are controlled by class-related signs, with real samples being slightly pushed towards the forgery side and forgery samples being slightly pushed towards the real side, thereby generating hard samples that are closer to the classification boundary; based on the hard samples, enhancement features are generated, and the image detection model learns the enhancement features of the hard samples to obtain a trained classifier.

[0054] Furthermore, based on the generation of enhanced features from difficult samples, the image detection model learns the enhanced features of difficult samples to obtain a trained classifier. This includes the following steps: feeding the generated enhanced features into the classifier, which performs linear operations on the enhanced features using its own weights and bias parameters, outputting the corresponding probabilities of the image belonging to the real image and the fake image; calculating the classification loss based on the image's true label and the model's predicted true / fake probabilities; calculating the loss using only the true image prediction probability for real images and only the fake image prediction probability for fake images; measuring the deviation between the model's true / fake judgment result and the true label using the cross-entropy formula; calculating the squared distance from the aggregated features of each image to the feature center of its own category, averaging the distances of all samples in the batch to obtain the category center constraint loss; adjusting the proportion of the category center constraint loss using fixed weight coefficients, adding the classification loss to the weighted category center constraint loss to obtain the total loss value used for model training; and executing the backpropagation algorithm with the total loss as the optimization objective, synchronously updating all learnable parameters in the model to obtain the trained classifier.

[0055] Specifically, during the training phase, the generated enhanced features are fed into the classifier. The classifier performs a linear operation on the enhanced features using its own weights and bias parameters, outputting an original score vector. Then, a Softmax operation is performed on the score vector to calculate the probability that the image belongs to either a real image or a fake image. The label number 0 represents a real image, and the number 1 represents a fake image.

[0056] The classification loss is calculated based on the true and false probabilities predicted by the model for real images. For real images, only the true image probability is used to calculate the loss; for fake images, only the fake image probability is used. The overall deviation between the model's true / false judgment and the true label is measured using the cross-entropy formula. For each image in a batch, the aggregated features are used to calculate the squared distance from the feature center of its own category. The average distance values ​​of all samples in the batch are then used to obtain the category center constraint loss. The purpose of the category center constraint loss is to constrain real image features to be closer to the real category center and fake image features to be closer to the fake category center.

[0057] If the squared distance between a feature of a forged sample and the center of the forged category is 0.30, and the center constraint loss weight is 0.01, the value of the center constraint loss part corresponding to this sample is 0.003.

[0058] By setting a fixed weight coefficient to adjust the proportion of the class center constraint loss, the classification loss is added to the weighted class center constraint loss to obtain the total loss value used for model training. In the example above, the total sample loss is approximately 0.131.

[0059] With the total loss as the optimization objective, the backpropagation algorithm is executed with the help of optimizers like Adam to simultaneously update all learnable parameters in the model, including visual adaptation layer parameters, instance-level aggregation module parameters, feature centers for real and fake objects, classifier weights and bias parameters, and continuously iterate to optimize the model's detection capabilities.

[0060] In one embodiment, the synthetic image detection system using instance aggregation and orthogonal artifact simulation employs the synthetic image detection method using instance aggregation and orthogonal artifact simulation as described above.

[0061] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0063] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described terminal embodiment, causing the processor to perform the synthetic image detection method of instance aggregation and orthogonal artifact simulation in the above-described embodiment.

[0064] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information transmission medium.

[0065] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

[0066] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting synthetic images using instance aggregation and orthogonal artifact simulation, characterized in that, Including the following steps: Obtain the input image and perform uniform size preprocessing on the input image to obtain a preprocessed image; The preprocessed image is fed into a pre-trained CLIP visual encoder, which extracts image features through multiple intermediate layers of the visual encoder: The shallow layer preserves texture and eliminates artifacts at subtle edges, while the deep layer stores global semantic information of objects and scenes, retaining all layer features for subsequent fusion. A learnable projection layer is set up to map multi-layer features of different dimensions and scales to the same feature space to obtain mapped features. Each layer is configured with a lightweight scoring network to dynamically calculate the effective weights of the mapped features of each layer for a single input image. All the mapped features are weighted and summed to output a unified aggregated feature that takes into account both shallow fine-grained artifacts and high-level semantics. Two learnable category centers are defined: the real image center. and fake image center ; Based on the two category centers, define a unit vector pointing from the real center to the fake center. That is, the direction for judging truth from falsehood; along When moving, the features will more clearly move from the real side to the fake side, or from the fake side to the real side; and Orthogonal directions represent factors that are less related to true / false classification. Therefore, two projection operators are constructed: discriminative direction projection and orthogonal direction projection. Random perturbations are decomposed by these two projection operators before use, so that the perturbations no longer aimlessly change all feature dimensions, but serve the two objectives of "generating difficult true / false samples" and "expanding non-discriminative change distribution". During the training phase, class centers are constructed based on the unified aggregated features and the feature subspace is orthogonally decomposed. Orthogonal constraint controlled feature artifact simulation is enabled to force the image detection model to learn general forgery features, thus obtaining a trained classifier. Random noise is sampled from a standard Gaussian distribution. And sample Bernoulli variables Determine whether to apply a perturbation to the current sample, and then resample random coefficients between 0 and 1. Control the intensity of the disturbance in the discrimination direction; random noise is first determined according to the discrimination direction. Decompose: Determine the directional components as The orthogonal direction components are The orthogonal direction component is used to simulate changes in content, background, style and real / fake discrimination that are weakly correlated, so that the detector can remain smooth and stable in the face of these changes; the discrimination direction component is controlled by class-related sign: real samples are slightly pushed toward the fake side, and fake samples are slightly pushed toward the real side, so as to generate difficult samples that are closer to the classification boundary. During the inference phase, the orthogonal artifact simulation perturbation module is turned off. The image under test only undergoes preprocessing, CLIP multi-layer feature extraction, and instance-level aggregation to obtain unified aggregated features, which are directly fed into the classifier to output the probability that the image is a real image or a synthetic image, thus completing the detection.

2. The synthetic image detection method based on instance aggregation and orthogonal artifact simulation according to claim 1, characterized in that, The input image is preprocessed to a uniform size to obtain a preprocessed image, including the following steps: The input image is processed to a uniform size by scaling it to a set size, and then randomly cropped during the training phase and centered and cropped to the target size during the inference phase. The input image is normalized by normalizing each pixel channel according to the mean and variance of the CLIP pre-trained model to obtain a normalized image. The normalized image is divided into multiple fixed-size image blocks. Each image block is flattened into a vector and mapped to an embedding block through a linear embedding layer. The embedding block is added to the category label and position code to form the input sequence of layer 0.

3. The synthetic image detection method based on instance aggregation and orthogonal artifact simulation according to claim 1, characterized in that, The preprocessed image is fed into a pre-trained CLIP visual encoder, and image features are extracted through multiple intermediate layers of the visual encoder, including the following steps: The preprocessed image is sequentially processed Each visual block includes: multi-head self-attention, feedforward network, layer normalization, and residual connection; Image features are extracted from multiple intermediate visual patches, denoted as... , ... The shallow features include texture, edges, and subtle artifacts of local noise, while the deep features include object semantics, scene content, and global structure.

4. The synthetic image detection method based on instance aggregation and orthogonal artifact simulation according to claim 3, characterized in that, Each layer is configured with a lightweight scoring network. For a single input image, the effective weights of the mapped features at each layer are dynamically calculated. All mapped features are then weighted and summed to output a unified aggregated feature that balances shallow, fine-grained artifacts with high-level semantics. The steps include: Each layer constructs a lightweight scoring network, and the mapping features are input into the lightweight scoring network to obtain scalar scores; The scalar scores of all layers are Softmax normalized to obtain the corresponding weight scores. The multi-layer mapping features are weighted and summed to output the unified aggregated features that take into account both shallow fine-grained artifacts and high-level semantics.

5. The synthetic image detection method based on instance aggregation and orthogonal artifact simulation according to claim 4, characterized in that, Based on the generated enhanced features from the difficult samples, the image detection model learns the enhanced features from the difficult samples to obtain a trained classifier, including the following steps: The generated enhanced features are fed into the classifier, which performs a linear operation on the enhanced features by combining its own weights and bias parameters, and outputs the corresponding probabilities of the image belonging to the real image and the fake image. Calculate the classification loss based on the true and false probabilities of the image's real label and the model's prediction: If it is a real image, calculate the loss using only the probability prediction of the real image. If it is a fake image, only the probability of the fake image is predicted to calculate the loss, and the overall deviation between the model's real and fake judgment results and the real label is measured by the cross-entropy formula. For each image's aggregated features, calculate the squared distance from the aggregated feature to the feature center of its own category. Take the average of the distance values ​​of all samples in the batch to obtain the category center constraint loss. Set a fixed weight coefficient to adjust the proportion of the class center constraint loss, add the classification loss to the weighted class center constraint loss, and get the total loss value used for model training. Using the total loss as the optimization objective, the backpropagation algorithm is executed to synchronously update all learnable parameters in the model, resulting in a trained classifier.

6. A synthetic image detection system based on instance aggregation and orthogonal artifact simulation, characterized in that, The synthetic image detection method using instance aggregation and orthogonal artifact simulation as described in any one of claims 1-5 is applied.

Images