Open world image tracing method and device based on semi-supervised learning

By combining heterogeneous networks and the DBSCAN clustering algorithm to generate hard pseudo-labels and using the mutual mean teaching framework to optimize soft pseudo-labels, the problem of insufficient adaptability of existing image source tracing methods in open-world scenarios is solved, and efficient identification and classification of unknown generation methods is achieved.

CN120912949APending Publication Date: 2025-11-07ZHEJIANG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510941355.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-09
Publication Date
2025-11-07

AI Technical Summary

Technical Problem

Existing image tracing methods struggle to adapt to unknown categories in open-world scenarios, rely on large amounts of labeled data, and have insufficient generalization performance, making it difficult to effectively distinguish between real and fake images.

Method used

We employ a semi-supervised learning approach, combining heterogeneous networks (CLIP-ResNet and CLIP-ViT) for pre-training, using DBSCAN clustering to generate hard pseudo-labels, and generating soft pseudo-labels through a cross-means teaching framework. We also integrate multiple loss functions to optimize the model, thereby improving its adaptability to unknown generation methods and its classification performance.

Benefits of technology

It significantly improves the ability to trace the source of images generated by unknown methods, enhances robustness and adaptability in open-world scenarios, can efficiently distinguish images generated by multiple methods, and supports accurate classification and probability estimation of known and unknown generation methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120912949A_ABST
    Figure CN120912949A_ABST
Patent Text Reader

Abstract

The invention discloses an open world image traceability method and device based on semi-supervised learning, and the method comprises the steps: obtaining to-be-traced image data which comprises a labeled image and an unlabeled image; pre-training is carried out through a heterogeneous network, a feature embedding vector of a labeled image is extracted, and the feature extraction capability is optimized in combination with category classification loss and ternary loss; on the basis of semi-supervised learning, utilizing a DBSCAN clustering algorithm to cluster the feature embedding vectors of the label-free images, generating noise hard and pseudo labels, generating soft pseudo labels through cooperation of two networks of a mutual mean teaching framework, and fusing a hard classification loss, a soft classification loss and a soft triple loss optimization model; and reasoning an input image to be traced, extracting a feature embedding vector by using the trained average model, and outputting probability distribution of each generation method by a classifier. According to the invention, images generated by multiple generation methods can be efficiently distinguished, the traceability in an open world scene is improved, and the robustness in a complex scene is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multimedia forensics, and particularly relates to an open-world image tracing method and device based on semi-supervised learning. BACKGROUND

[0002] With the wide application of deep learning in the field of computer vision and image processing, image tracing technology has become a research hotspot in recent years. In particular, the rapid development of generative models such as generative adversarial networks makes it possible to generate high-quality fake images that are almost indistinguishable from real images in terms of resolution, texture and details. This technology has positive applications in fields such as artistic creation, data enhancement and virtual reality, but it has also raised serious concerns due to its potential malicious use.

[0003] Some existing image tracing methods mainly rely on supervised learning to distinguish real and fake images by analyzing the generation characteristics of images. However, these methods generally have the problem of insufficient generalization ability. With the continuous progress of fake technology, new generation models can generate highly realistic images, resulting in a significant decline in the detection performance of traditional methods when facing unknown generation methods. In addition, existing methods usually require a large amount of labeled data, while in an open-world scenario, unlabeled data dominates, which further limits the applicability of the model. SUMMARY

[0004] The present application overcomes the above-mentioned shortcomings of the prior art and provides an open-world image tracing method and device based on semi-supervised learning, aiming to solve the problem that traditional image tracing methods are difficult to adapt to unknown categories in an open-world scenario, rely on a large amount of labeled data, and have insufficient generalization performance. The present application combines semi-supervised learning technology with supervised pre-training and mutual average teaching framework to design an efficient image feature extraction and classification system, which realizes accurate identification of image origin through double network collaborative training and pseudo-label optimization, especially in the case of unknown categories.

[0005] The method comprises:

[0006] S1. Obtain image data to be traced, the image data comprising labeled images and unlabeled images, the labeled images being images generated by known generation methods, and the unlabeled images being images generated by unknown generation methods. Preprocess the image data to be traced and divide it into a training set and a test set, the test set containing a closed set and an open-world subset.

[0007] S2. Pre-train through a heterogeneous network to extract feature embedding vectors of the labeled images, and optimize the feature extraction capability in combination with a class classification loss and a ternary loss.

[0008] S3. Based on semi-supervised learning, the feature embedding vectors of the unlabeled images are clustered using the DBSCAN clustering algorithm to generate noise hard pseudo labels, and the soft pseudo labels are generated by the mutual mean teaching framework of the two networks, and the model is optimized by fusing the hard classification loss, soft classification loss and soft triplet loss.

[0009] S4. The input image to be traced is inferred, and the feature embedding vector is extracted using the trained average model, and the classifier outputs the probability distribution of each generation method.

[0010] Optionally, the process of obtaining the image data to be traced in step S1 comprises:

[0011] S11. Preprocess the input image to obtain a standardized image sample, specifically including normalizing and data enhancing the input image to generate a standardized image sample.

[0012] S12. Construct a dataset through multiple image generation techniques, including but not limited to GAN model-based and diffusion model-based image generation methods. Classify the labeled images according to known generation methods and unknown generation methods, the known generation methods include various generation model-based image generation techniques, and the unknown generation methods include generation model-based images other than the known ones.

[0013] S13. Divide the dataset into a training set and a test set, and the test set is divided into a closed set and an open world. For the closed set test set, the generation model used is consistent with the training set. For the open world dataset, the generation method is different from the training set. Based on the standardized image sample and the classified dataset, the model is trained and tested

[0014] Optionally, the process of pre-training through a heterogeneous network to extract feature embedding vectors of labeled images, combining class classification loss and triplet loss to optimize feature extraction capability in step S2 comprises:

[0015] S21. Use CLIP-ResNet and CLIP-ViT networks to extract feature embedding vectors of labeled images, respectively, to generate a feature representation set, and the feature embedding vectors are processed by network parameters and a learnable classifier.

[0016] S22. Based on the extracted feature embedding vectors, apply class classification loss to optimize the supervised classifier, so that the model can accurately distinguish the classes of known generation methods, and the classification loss is based on cross-entropy calculation.

[0017] S23. Apply triplet loss to optimize feature extraction capability, select the hardest positive and negative samples in each batch through a set sampling strategy, compare the features of positive and negative samples based on feature distance, and finally fuse the classification loss and triplet loss to complete pre-training.

[0018] Optionally, the step S3 based on semi-supervised learning uses the DBSCAN clustering algorithm to cluster the feature embedding vectors of the unlabeled images, generates noise hard pseudo labels, and generates soft pseudo labels through the mutual mean teaching framework of two networks, and the process of fusing hard classification loss, soft classification loss and soft triplet loss to optimize the model includes:

[0019] S31. The feature embedding vectors of the unlabeled images are extracted using the pre-trained heterogeneous CLIP network to generate a feature representation set of the unlabeled data.

[0020] S32. The feature embedding vectors of the unlabeled images are clustered based on the DBSCAN clustering algorithm to generate hard pseudo labels for preliminary labeling of the image categories of unknown generation methods.

[0021] S33. Through the mutual mean teaching framework, the soft pseudo labels are generated by the time average model of the two networks, the hard classification loss, the soft classification loss and the soft triplet loss are fused, and the adaptability of the model to unknown generation methods is improved by optimizing the feature extractor and the classifier.

[0022] Optionally, the step S4 of inferring the input image to be traced uses the trained average model to extract the feature embedding vectors, and the process of the classifier outputting the probability distribution of each generation method includes:

[0023] S41. Select the best trained average model, which is generated based on the optimized heterogeneous CLIP network parameters in the semi-supervised learning process.

[0024] S42. Feature extraction is performed on the input image to be traced using the selected average model to generate feature embedding vectors.

[0025] S43. Based on the feature embedding vectors, the probability is normalized by the classifier to generate a probability distribution supporting known and unknown generation methods, and the class and probability value of each generation method are output.

[0026] The second aspect of the application relates to an open world image tracing device based on semi-supervised learning, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the open world image tracing method based on semi-supervised learning.

[0027] The third aspect of the application relates to a computer readable storage medium, characterized in that a program is stored thereon, which is executed by a processor to implement the open world image tracing method based on semi-supervised learning.

[0028] The present application uses a two-stage learning framework of pre-training and semi-supervised learning to extract multi-dimensional features of images by a heterogeneous CLIP network (CLIP-ResNet based on a convolutional neural network and CLIP-ViT based on a visual Transformer), generates hard pseudo labels by combining a DBSCAN clustering algorithm, and dynamically optimizes the fusion of soft and hard labels by a mutual mean teaching framework, so as to realize the identification of image generation methods, including known generation methods and unknown generation methods. The present application overcomes the limitations of the existing image tracing methods, such as high dependence on large-scale labeled data and poor adaptability to unknown generation methods, can efficiently distinguish images generated by various generation methods, improves the tracing ability of emerging generation technologies in an open world scenario, enhances the robustness in a complex scenario, and provides an efficient and reliable solution for image authenticity identification, intellectual property protection, digital forensics and social media content verification.

[0029] Compared with the prior art, the present application has the following advantages:

[0030] The present application proposes an open world image tracing method based on semi-supervised learning, which significantly improves the tracing ability of images generated by unknown generation methods by combining heterogeneous networks, DBSCAN clustering algorithm and mutual mean teaching framework, and breaks through the limitation of traditional methods limited to closed set scenarios. In the feature extraction process, the generated feature embedding vector has high discriminability and generalization ability, which can effectively distinguish the image features of known and unknown generation methods. For unlabeled data, the present application generates hard pseudo labels by using DBSCAN clustering, and generates soft pseudo labels by using a mutual mean teaching framework, and fuses hard classification loss, soft classification loss and soft triplet loss, to further optimize the adaptability and classification performance of the model to unknown generation methods. In addition, the present application outputs the probability distribution of each generation method by the trained average model, which not only supports accurate classification of known generation methods, but also can estimate the probability of unknown generation methods, thereby enhancing the interpretability and application flexibility of the tracing results.

[0031] The present application is suitable for the fields of digital forensics, content authenticity verification and generated image tracing, and has important value in the field of anti-deepfake and other generative AI technologies, and provides efficient and robust technical support for network security and content supervision. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a supervised pre-training model framework of the present application;

[0033] Figure 2 is a model framework of the method of the present application;

[0034] Figure 3 is a flowchart of the method of the present application;

[0035] Figure 4 is the reasoning framework diagram of the method of the present application.

[0036] DETAILED DESCRIPTION

[0037] The present application will be further illustrated below in conjunction with specific examples, which are implemented on the premise of the technical solutions of the present application, and it should be understood that these examples are only used to illustrate the present application and not to limit the scope of the present application. Before the technical solutions provided in the present application are specifically described, a brief description of the related background involved in the present application is provided.

[0038] Example 1

[0039] The present embodiment provides an open-world image tracing method based on semi-supervised learning.

[0040] With the rapid development of generative artificial intelligence technology, image generation methods based on generative adversarial networks, diffusion models and other technologies are increasingly popular and widely used in content creation, digital media and other fields. However, these technologies also pose significant challenges to image authenticity verification and tracing, especially in open-world scenarios, where images may be generated by unknown generative methods, and traditional tracing methods based on closed-set assumptions are difficult to effectively cope with. In addition, malicious use of fake images generated by generative AI technology poses a serious threat to network security, content regulation and other fields, and efficient tracing technology is urgently needed to identify the source of image generation.

[0041] Currently, for image tracing problems, existing technologies mostly rely on supervised learning, training models with labeled data of known generative methods, and extracting texture, frequency or statistical features of images for classification. However, these methods face significant performance degradation when dealing with images from unknown generative methods in open-world scenarios. In addition, traditional methods often lack effective use of unlabeled data, and the model generalization ability is insufficient, which cannot adapt to the diversity and rapid iteration of generative models.

[0042] To solve the above problems in the prior art, the present application proposes an open-world image tracing method based on semi-supervised learning, which extracts features through a heterogeneous network for image data containing known and unknown generative methods, generates pseudo-labels by combining DBSCAN clustering and mutual mean teaching framework, and optimizes model performance by fusing multiple types of loss functions. This method can accurately trace the image generation method in an open-world scenario, support the probability distribution output of known and unknown generative methods, and significantly improve the robustness and adaptability of tracing. This method is suitable for digital forensics, content authenticity verification and adversarial deep forgery scenarios, and has strong anti-interference ability and wide application prospects.

[0043] The architecture of the semi-supervised learning method provided in the present application will be briefly described as follows.

[0044] Referring to Figure 1 is pre-trained, the feature embedding vectors of the labeled images are extracted, and the feature extraction capability is optimized in combination with the category classification loss and the triplet loss, including:

[0045] By using the image data to supervise the pre-training of the encoder, the feature extraction capability of the model is constructed, providing a reliable foundation for subsequent semi-supervised learning.

[0046] The encoder adopts a heterogeneous CLIP network structure, including two branches of CLIP-ResNet and CLIP-ViT, which respectively extract the feature embedding vectors of the images. Specifically, CLIP-ResNet extracts local features of the image using a convolutional neural network, while CLIP-ViT captures global semantic information of the image based on the Transformer structure. During the training process, only the labeled data (x i ,y i ) is used, and the goal is to optimize the encoder parameters θ and the learnable target domain classifier C t , so that it can effectively distinguish images generated by known methods

[0047] The training process includes the following steps:

[0048] S101, feature extraction: for each labeled image x i , the feature embedding vectors are extracted by CLIP-ResNet and CLIP-ViT respectively:

[0049]

[0050] where θ represents the network parameters, N s represents the batch size of the labeled data.

[0051] S102, classification loss calculation: the feature embedding vectors are classified by the target domain classifier C t , and the category classification loss is calculated. The cross-entropy loss function is used:

[0052]

[0053] where y is the true label of the image, N t represents the number of labeled samples in the current batch, and represents the cross-entropy loss.

[0054] S103, triplet loss calculation: to further enhance the discriminability of the feature embedding, the triplet loss is introduced.

[0055] By optimizing the distance relationship between samples in the feature space, the intra-class compactness and inter-class separability are improved:

[0056]

[0057] Total loss calculation: combine the classification loss and the triplet loss to define the total loss function:

[0058]

[0059] where λ is the weight coefficient of the triplet loss (usually set to 1.0) to balance the contribution of the two losses.

[0060] The optimization process uses the AdamW optimizer with an initial learning rate of 0.001 and a cosine annealing scheduling strategy to dynamically adjust the learning rate to ensure the stability of the training. The training goal is to extract high-discriminative feature embedding vectors through multiple iterations, so as to provide a high-quality initial model for subsequent semi-supervised learning.

[0061] Through the above supervised training, the model can effectively distinguish the source of images of known categories, laying the foundation for unknown category recognition in open world scenarios.

[0062] Refer to Figure 1 the model framework of the semi-supervised learning-based open world image provenance method for semi-supervised learning.

[0063] Using the mutual mean teaching framework, through double network collaborative training and pseudo-label optimization, the model adapts to unknown categories in open world scenarios and improves its efficiency in utilizing unlabeled data.

[0064] Based on the pre-trained heterogeneous CLIP network, combined with the DBSCAN clustering algorithm and the mutual mean teaching framework, process the unlabeled images and optimize the model parameters. The specific process includes the following steps:

[0065] S201, feature extraction: use the pre-trained CLIP-ResNet and CLIP-ViT networks to extract feature embedding vectors of unlabeled images where i represents the index of the unlabeled image. The feature extraction process maintains consistency to ensure alignment with the feature representation in the supervised training phase.

[0066] S202, hard pseudo-label generation: based on the DBSCAN clustering algorithm, perform unsupervised clustering on the feature embedding vectors to generate hard pseudo-labels The clustering parameters include the distance threshold ε and the minimum number of samples minPts, defined as:

[0067]

[0068] Where ∈ = 0.5 represents the maximum distance within a cluster, and minPts = 5 represents the minimum number of samples per cluster. This process clusters unlabeled data into potential categories. After clustering, we count the number of effective clusters and add it to the number of true categories in the source domain to obtain the target number of categories for the overall classification task. Subsequently, we dynamically adjust the classifier structure of the model: retaining the original source domain classification weights unchanged, while using the cluster centers as the initial weights for new pseudo-classes to achieve smooth expansion of the classifier. However, due to the presence of noise and unknown categories, hard pseudo-labels may contain errors.

[0069] S203. Soft Pseudo-Label Generation: To reduce the amplification effect of pseudo-label errors, a time-averaging model in the mutual averaging teaching framework is used to generate reliable soft pseudo-labels. Using the two pre-trained heterogeneous networks and their time-averaging model, the update formula is as follows:

[0070] E (T) [θ1]=αE (T-1) [θ1]+(1-α)θ1

[0071] E (T) [θ2]=αE (T-1) [θ2]+(1-α)θ2

[0072] Then, the two time-averaging models are respectively derived from... and Generate strong soft pseudo-labels, where α is the ensemble momentum in the range [0, 1], and the soft label is the probability, expressed as E. (T) [θ2] Average model extraction Feature F, after C t The classification layer obtains the final predicted probabilities. These soft pseudo-labels are used to supervise the training of another network, enhancing the robustness of the model.

[0073] S204. Calculation of total loss:

[0074] The soft pseudo-label loss is defined as:

[0075]

[0076] The pseudo-label predictions of the two networks are supervised by using past average models of the other networks, thus better avoiding error amplification.

[0077] Soft triplet loss: To further improve the discriminative power of feature embeddings, soft triplet loss is introduced to optimize the feature space structure of unlabeled data. Based on soft pseudo-labels generated by the time-averaged model, the most difficult positive and negative samples in each mini-batch are selected. The soft triplet loss is defined as follows:

[0078]

[0079] Finally, the hard pseudo-label and the soft pseudo-label are used for training, and the total loss function is represented as:

[0080]

[0081] wherein is a weighting parameter. The design of the above loss function fully utilizes the collaborative training mechanism of the mutual mean teaching framework, and through the joint optimization of the soft pseudo-label and the hard pseudo-label, the noise and instability caused by hard clustering are alleviated, and the generalization performance and traceability accuracy of the model are significantly improved.

[0082] Referring to Figure 3 the flowchart of the open-world image traceability method based on semi-supervised learning

[0083] S101 data input and preprocessing

[0084] The labeled image data and unlabeled image data are input. The input images are preprocessed, including normalization and data augmentation, to generate standardized image samples. The data set is divided into a training set and a test set, and the test set contains a closed set subset and an open-world subset.

[0085] S102 hard pseudo-label generation

[0086] First, the pre-trained heterogeneous network is used to extract the feature embedding vector of the unlabeled image to represent the unlabeled image index. Ensure that the feature extraction process is consistent with the feature representation of the labeled data, and provide a unified feature space for subsequent clustering and pseudo-label generation.

[0087] Second, the DBSCAN clustering algorithm is applied to the feature embedding vector of the unlabeled data to generate a hard pseudo-label clustering parameter. The number of effective clustering classes is counted, and the number of real classes of the labeled data is added to obtain a hard label.

[0088] S103 soft pseudo-label generation

[0089] The mutual mean teaching framework is adopted to generate soft pseudo-labels through the time average model of the two heterogeneous networks. The time average model generates a probability distribution for the unlabeled sample, which is used as a soft pseudo-label for the two networks to cross-supervise the training of the two networks and alleviate the noise in the hard pseudo-label.

[0090] S104 loss function calculation

[0091] The total loss function is calculated, which includes the supervised classification loss, the hard pseudo-label loss, the soft pseudo-label loss, and the soft triplet loss

[0092] S105 model optimization

[0093] Based on the total loss function, the parameters of the two heterogeneous networks are updated alternately by gradient descent method. After each round of training, the parameters of the time-averaged model are updated to generate more reliable soft pseudo labels. The optimization process ensures model convergence and performance through hyperparameter adjustment. Determine whether the preset training rounds or convergence conditions are reached. If not, continue to return to the next round of iterative training; if so, end the training and save the optimized average model parameters.

[0094] Referring to Figure 4 Inference framework diagram of the open-world image provenance method based on semi-supervised learning

[0095] Provenance analysis is performed on the input image, feature extraction and classification are performed using the trained model, and the probability of the image generation method is output

[0096] To improve the stability and accuracy of inference, the average model with better validation performance is selected for inference. Compared with directly using the model with the current parameters, the average model can effectively smooth the parameter fluctuations by integrating the weights of multiple time steps in the training process, showing higher robustness and generalization ability.

[0097] The inference process includes the following steps:

[0098] S401, feature extraction: after preprocessing, the input image is sent to the trained average network to extract the feature embedding vector F input The feature extraction process is consistent with the training stage.

[0099] S402, classification prediction: the feature embedding vector F input is normalized by the classifier to output the probability distribution of N generation methods:

[0100] P = softmax(C t (F input ))

[0101] Where C t is the target domain classifier, N is the total number of classes (including known and unknown generation methods), and P is the probability distribution vector, representing the likelihood of the input image belonging to each generation method.

[0102] S403, result output: classification of known generation methods and unknown generation methods. The known generation methods include 8 categories of real, VQDM, ProGAN, GauGAN, StyleGAN2, Midjourney, Wukong, ADM, etc.; the unknown generation methods include 7 categories of Glide, whichfaceisreal, DALL-E2, CycleGAN, stable_diffusion_v_1_5, StarGAN, BigGAN, etc. Similar samples of unknown categories can be clustered together, and unknown categories can be classified.

[0103] The inference module adopts a batch processing mode to optimize the calculation efficiency and can efficiently cope with large-scale image tracing requirements.

[0104] Embodiment 2

[0105] This embodiment relates to an open world image tracing device based on semi-supervised learning, comprising a memory and one or more processors, the memory has stored executable code, and the one or more processors execute the executable code to implement the open world image tracing method based on semi-supervised learning of embodiment 1.

[0106] Embodiment 3

[0107] This embodiment relates to a computer readable storage medium, characterized in that a program is stored thereon, and the program is executed by a processor to implement the open world image tracing method based on semi-supervised learning of embodiment 1.

Claims

1. An open-world image provenance method based on semi-supervised learning, characterized in that, The method comprises the following steps: S1. Obtain image data to be traced, which includes labeled images and unlabeled images, the labeled images are images generated by known generation methods, and the unlabeled images are images generated by unknown generation methods; pre-process the image data to be traced and divide it into a training set and a test set, the test set includes a closed set and an open world subset; S2. Pre-training through a heterogeneous network, extracting feature embedding vectors of labeled images, combining class classification loss and triple loss to optimize feature extraction capability; S3. Based on semi-supervised learning, using DBSCAN clustering algorithm to cluster feature embedding vectors of unlabeled images, generating noise hard pseudo labels, and generating soft pseudo labels through mutual mean teaching framework of two networks, and optimizing the model by fusing hard classification loss, soft classification loss and soft triple loss; S4. Inference on the input image to be traced, extracting feature embedding vectors using the trained average model, and the classifier outputs the probability distribution of each generation method.

2. The method of claim 1, wherein, The process of obtaining image data to be traced in step S1 comprises: S11. Pre-process the input image to obtain standardized image samples, which specifically includes normalizing and data enhancing the input image to generate standardized image samples; S12. Construct a dataset through multiple image generation techniques, including but not limited to GAN model-based and diffusion model-based image generation methods; classify the labeled images according to known generation methods and unknown generation methods, the known generation methods include various generation model-based image generation techniques, and the unknown generation methods include generation model-based images other than the known ones; S13. Divide the dataset into a training set and a test set, the test set is divided into a closed set and an open world, for the closed set test set, the generation model used is consistent with the training set, for the open world dataset, the generation method is different from the training set, based on the standardized image samples and the classified dataset, model training and testing are performed.

3. The method of claim 1, wherein, The process of pre-training through a heterogeneous network, extracting feature embedding vectors of labeled images, and combining class classification loss and triple loss to optimize feature extraction capability in step S2 comprises: S21. Use CLIP-ResNet and CLIP-ViT networks to extract feature embedding vectors of labeled images, generate feature representation sets, and the feature embedding vectors are processed by network parameters and learnable classifiers; S22. Based on the extracted feature embedding vectors, apply class classification loss to optimize the supervised classifier, so that the model can accurately distinguish the categories of known generation methods, and the classification loss is calculated based on cross entropy; S23. Apply triple loss to optimize feature extraction capability, select the most difficult positive and negative samples in each batch through a set sampling strategy, compare the features of positive and negative samples based on feature distance, and finally fuse classification loss and triple loss to complete pre-training.

4. The method of claim 1, wherein, The semi-supervised learning based step S3 uses the DBSCAN clustering algorithm to cluster the feature embedding vectors of the unlabeled images, generates noise hard pseudo labels, and generates soft pseudo labels through the mutual mean teaching framework of two networks, and the process of fusing hard classification loss, soft classification loss and soft triplet loss to optimize the model includes: S31. Using a pre-trained heterogeneous CLIP network to extract feature embedding vectors of unlabeled images, generating a feature representation set of unlabeled data; S32. Based on the DBSCAN clustering algorithm, the feature embedding vectors of the unlabeled images are clustered to generate hard pseudo labels, which are used to preliminarily label the image categories of unknown generation methods; S33. Through the mutual mean teaching framework, the time average model of the two networks is used to generate soft pseudo labels, and the hard classification loss, soft classification loss and soft triplet loss are fused to improve the adaptability of the model to unknown generation methods by optimizing the feature extractor and the classifier.

5. The method of claim 1, wherein, The step S4 described above is the process of inferring the input image to be traced, using the trained average model to extract the feature embedding vector, and the classifier outputs the probability distribution of each generation method, which includes: S41. Select the best trained average model, which is generated based on the optimized parameters of the heterogeneous CLIP network in the semi-supervised learning process; S42. Using the selected average model to extract features from the input image to be traced, generating a feature embedding vector; S43. Based on the feature embedding vector, the probability is normalized through the classifier to generate a probability distribution supporting known and unknown generation methods, and the class and probability value of each generation method are output.

6. An apparatus for open-world image provenance based on semi-supervised learning, characterized in that, A memory and one or more processors, the memory having stored therein executable code, the one or more processors executing the executable code to implement the semi-supervised learning based open world image tracing method of any one of claims 1-5.

7. A computer readable storage medium characterized in that, A program is stored thereon, which is executed by a processor to implement the semi-supervised learning based open world image tracing method of any one of claims 1-5.