Generative image tracing method and device and related equipment

Abstract

The invention discloses a generative image traceability method and device and related equipment. The method comprises the following steps: acquiring an image to be traced and a corresponding prompt text; an image to be traced and a prompt text are input to generate an image traceability network, the network is generated based on a CLIP model, the network comprises a feature extraction module, a feature fusion module and a classification module, the feature extraction module comprises at least two image encoders sharing the same parameters, feature extraction is carried out to obtain feature vectors, and the feature vectors are classified; the feature fusion module fuses the obtained feature vectors, and the classification module determines a generation source according to the fused feature vectors; and outputting a generation source of the to-be-traced image. By adopting the method, the images generated by different models can be distinguished, and the source of the generated image can be accurately traced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to a generative image tracing method, device and related equipment. Background Art

[0002] In the field of image generation and copyright protection, with the continuous advancement of deep learning technology, especially the emergence of image generation models such as diffusion models, generative adversarial networks (GANs) and variational autoencoders (VAEs), it is possible to create extremely realistic and diverse images, which has brought revolutionary impacts to visual arts, multimedia content production, etc. However, with the continuous advancement and rapid development of technology, it has also brought about a series of ethical and copyright issues, such as image forgery and copyright infringement. Therefore, tracing the source of generated images has become particularly important.

[0003] At present, the main method to solve the problem of the source of generated images is to simplify it into an image classification task, that is, to distinguish images generated by different models by training classifiers. However, this method has great limitations and is prone to misclassification of images of unknown sources. In particular, when faced with diverse generation models and complex image content, classifiers often cannot accurately identify the true source of the image.

[0004] Therefore, how to quickly and accurately identify the true source of the generated image and improve the accuracy and robustness of tracking is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present invention provide a generative image tracing method, apparatus and related equipment, which improve the accuracy and robustness of generative image tracing by integrating contrastive learning and twin network architecture and combining a powerful pre-training model.

[0006] In order to solve the above technical problems, an embodiment of the present application provides a generative image tracing method, comprising the following steps: obtaining an image to be traced and a prompt text corresponding to the image to be traced; inputting the image to be traced and the prompt text into a generative image tracing network, the generative image tracing network is generated based on a CLIP model, the generative image tracing network comprises a feature extraction module, a feature fusion module and a classification module, the feature extraction module comprises at least two image encoders, the at least two image encoders share the same parameters, the at least two image encoders are respectively used to extract features from the image to be traced and the image generated according to the prompt text to obtain feature vectors, the feature fusion module is used to fuse the feature vectors to obtain a fused feature vector, the classification module is used to determine the generation source of the image to be traced according to the fused feature vector; and outputting the generation source of the image to be traced.

[0007] In a possible implementation, before the pest image to be identified is input into the pest identification network, the method also includes: selecting an integrated CLIP model as an initial generation image tracing network; obtaining a sample training set, wherein the sample training set includes multiple sample images, and the sample images include real images and images generated based on different generation models; grouping the sample images based on the same prompt text, each group containing at least two sample images; and training the initial generation image tracing network using multiple groups of sample images.

[0008] In another possible implementation, obtaining a sample training set includes: selecting a T2I model, wherein the T2I model includes GLIDE, LDM, SDv2.1, Mid-journey, and DALLE-E; determining text prompts, wherein types of the text prompts include daily scenes, abstract concepts, and artistic styles; and for each text prompt, generating a plurality of sample images using the T2I model.

[0009] In another possible implementation, after obtaining the sample training set, the method further includes: performing size normalization and color space normalization on the sample image; performing data enhancement operations on the standardized sample image, wherein the data enhancement operations include rotation, scaling, and flipping.

[0010] In another possible implementation, the initial generative image tracing network is trained using multiple groups of sample images, including: randomly selecting two sample images from any group of sample images and inputting them into a CLIP image encoder that shares the same parameters for feature extraction to obtain two corresponding feature vectors, wherein the feature vector includes deep semantic information; calculating the absolute difference between the two feature vectors, wherein the absolute difference includes a discriminative feature; constructing a contrastive learning objective function, wherein the objective function is used to maximize the representation similarity between similar sample images and minimize the similarity between non-similar sample images; and iteratively optimizing the objective function based on the feature vectors and the absolute difference, adjusting the model parameters of the generative image tracing network, and obtaining a trained generative image tracing network.

[0011] In another possible implementation, the contrastive learning objective function is as follows:

[0012]

[0013] Among them, L represents the target value of contrastive learning, x i and x j Represents a pair of input images, f represents the feature extraction function, sim represents the function used to measure the similarity of two feature vectors, and τ represents the temperature parameter.

[0014] In another possible implementation, the iterative optimization of the objective function and the adjustment of the model parameters of the generated image tracing network include: optimizing the objective function using a gradient descent optimization algorithm, and the gradient descent optimization algorithm is as follows:

[0015]

[0016] Among them, θ t represents the model parameters at the current time step t, θ t+1 represents the model parameters of the optimized time step t, η represents the learning rate, represents the gradient operator, and L() represents the contrastive learning objective function.

[0017] In order to solve the above technical problems, an embodiment of the present application further provides a generative image tracing device, comprising: an acquisition module, used to acquire an image to be traced and a prompt text corresponding to the image to be traced; an image tracing module, used to determine the generation source of the image to be traced, the image tracing module comprising a feature extraction module, a feature fusion module and a classification module, the feature extraction module comprising at least two image encoders, the at least two image encoders share the same parameters, the at least two image encoders are respectively used to extract features from the image to be traced and the image generated according to the prompt text to obtain a feature vector, the feature fusion module is used to fuse the feature vectors to obtain a fused feature vector, the classification module is used to determine the generation source of the image to be traced according to the fused feature vector; an output module is used to output the generation source of the image to be traced.

[0018] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0019] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0020] The generative image tracing method provided in an embodiment of the present invention uses the CLIP model to extract the deep semantics and content features of the image, thereby enhancing the expressiveness of the image features, thereby having better robustness when processing complex and diverse image generation tasks. In addition, the contrastive learning mechanism is combined with the twin network, and the inherent semantic and content-related characteristics between images generated under the same text prompt are utilized. The constructed contrastive learning objective function is used to learn and distinguish the differential expressions between images generated by different models, thereby effectively improving the accurate tracing of the source of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 It is a schematic diagram of an exemplary scenario in which the present application can be applied.

[0023] Figure 2 It is the overall flow chart of the generative image tracing method of the present application.

[0024] Figure 3 This is an example diagram of the generated image dataset of this application.

[0025] Figure 4 It is a structural schematic diagram of the detection model of this application.

[0026] Figure 5 This is a schematic diagram of visualization after t-SNE dimensionality reduction in this application.

[0027] Figure 6 It is a schematic diagram of the tracking performance of the present application using different I2T models.

[0028] Figure 7 It is a schematic diagram of the effect of the number of generated images on the tracing accuracy of the present application.

[0029] Figure 8 It is a schematic diagram of the traceability performance after adding different types of noise in the present application.

[0030] Fig. 9 It is a structural diagram of the generative image tracing network of the present application.

[0031] Fig.10 It is a structural schematic diagram of the image tracing device generated in the present application.

[0032] Fig.11It is a schematic diagram of the structure of the computer device of the present application. DETAILED DESCRIPTION

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] In order to better understand the contents of the embodiments of the present application, some terms and related technologies involved in the embodiments of the present application are first explained to facilitate understanding by those skilled in the art.

[0036] Data rights confirmation refers to determining the ownership of data, including who owns the rights, usage rights, and income rights generated by the data. In the digital age, data rights confirmation becomes particularly important, especially when it comes to AI-generated content, because the content created by AI may involve multiple contributions and potential copyright issues.

[0037] Traceability refers to tracking and verifying the true source and generation process of digital content (such as images) to ensure that data generated by a specific algorithm or model can be identified. This is crucial for copyright protection, content review, and maintaining the authenticity and credibility of digital content.

[0038] Text-to-image models (T2I) are artificial intelligence models that can automatically generate corresponding images based on the text description provided by the user. For example, if you input "a sunset scene on a beach", the model can generate a picture that matches the description. Such models are usually based on deep learning techniques such as generative adversarial networks (GANs) and diffusion models.

[0039] Contrastive learning is a machine learning method that optimizes the model by learning the similarities and differences between data samples. For example, in the embodiment of the present application, contrastive learning is used to learn to generate difference representations between images, so as to distinguish images from different sources and achieve effective traceability.

[0040] Siamese networks are composed of two or more neural networks that share weights and are often used to compare the similarity of input data. In an embodiment of the present application, the Siamese network is used to process input images and determine whether they come from the same model or system by comparing their feature representations.

[0041] The CLIP model (contrastive language-image pre-training, CLIP) is a large pre-training model that aims to map text and images into the same semantic space through contrastive learning, so that the model can understand the image content and match it with the text description, thereby improving the quality of image generation and the relevance of the text description.

[0042] DreamBooth is a personalization technology used to fine-tune the text-to-image generation model so that it can better generate high-quality content relevant to a specific user based on their images. It trains the model using specific prompts (such as descriptions of users' personal photos) to enhance the model's understanding and generation capabilities for specific instances.

[0043] Diffusion models are generative models that generate new data samples by gradually adding noise to the data and then learning to remove the noise. In the field of image generation, diffusion models can generate high-quality images, especially those that are rich in details and close to real photo-level images.

[0044] Adversarial generation refers to the use of adversarial processes in adversarial networks (such as GANs) to generate data that is difficult to distinguish between real and fake, such as images or audio. Adversarial generation techniques are very effective in improving the realism of generated content, but at the same time may raise concerns about forgery and copyright infringement.

[0045] Deep adversarial generation method refers to a method that uses deep learning technology to generate adversarial samples or solve specific problems (such as the scarcity of hospital text data). It generates data samples with higher quality or specific attributes through adversarial training of generative models and discriminative models.

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] See also Figure 1 ,like Figure 1 As shown, in this scenario, the system 100 may include a system 110 for generating image tracing, a network 120, and an image database 130. The network 120 is used as a medium for providing a communication link between the system 110 for generating image tracing and the image database 130. The network 120 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0048] The user may use the generated image tracing system 110 to interact with the image database 130 via the network 120 to receive or send information, etc.

[0049] The image tracing system 110 may be deployed on a server that provides image tracing, for example, a backend server that provides image tracing for sample images stored in a server that deploys the image database 130 .

[0050] The image database 130 may be deployed in an edge computing device or a central computing device, for example, in a database server.

[0051] It should be noted that the generative image tracing method provided in the embodiment of the present application is executed by a server, and accordingly, the generative image tracing device is arranged in the server.

[0052] It should be understood that Figure 1 The number of generated image tracing systems, networks, and image databases in the embodiment is only for illustration. Any number of generated image tracing systems, networks, and image databases may be provided according to implementation requirements.

[0053] See also Figure 2 , Figure 2 An overall flow chart of a generative image tracing method provided by an embodiment of the present invention is shown and described in detail as follows.

[0054] S201, obtaining an image to be traced and a prompt text corresponding to the image to be traced.

[0055] Specifically, the image to be traced may be an image to be traced stored in an image database, or an image to be traced obtained through a web page, or an image to be traced obtained through other methods. The present application does not limit the method for obtaining the image to be traced.

[0056] Furthermore, if the acquired image to be traced has no relevant or direct prompt text, the image to be traced can be processed by an I2T model, such as a gpt2-image-captioning model, to obtain a prompt text corresponding to the image to be traced.

[0057] S202: Input the image to be traced and the prompt text to generate an image traceability network.

[0058] Specifically, after the image to be traced is obtained, it is input into a trained generated image source tracing network for tracing, so as to obtain the generation source of the image to be traced.

[0059] It should be noted that the generated image tracing network does not have the ability to trace the image before formally processing the image. It needs to be trained with relevant data in advance so that it has the ability to trace the image after the training is completed, and then it can be deployed on the relevant server for image tracing.

[0060] In a possible implementation, an integrated CLIP model is selected as the initial generative image tracing network, and a sample training set is obtained. The sample training set includes multiple sample images, and the sample images include real images and images generated based on different generative models. The sample images are grouped based on the same prompt text, and each group contains at least two sample images. The initial generative image tracing network is trained using multiple groups of sample images.

[0061] Specifically, the initial generative image tracing network can be built based on the CLIP model architecture. In addition, when obtaining the sample training set, it is necessary to build a dataset containing real images and a variety of fake images (i.e., images generated by different generative models) to ensure coverage of a wide range of image types and sources. It should be understood that the construction of the dataset is the most important foundation, which is crucial to ensuring the generalization performance of the model and improving the tracing accuracy. For example, some publicly available datasets (such as flickr, etc.) can be selected, which contain a large number of pictures and corresponding text descriptions.

[0062] In another possible implementation, a T2I model is selected, wherein the T2I model includes GLIDE, LDM, SDv2.1, Mid-journey, and DALLE-E; a text prompt is determined, wherein the types of the text prompt include daily scenes, abstract concepts, and artistic styles; and for each text prompt, a plurality of sample images are generated using the T2I model.

[0063] Specifically, the most advanced T2I models can be pre-selected, such as GLIDE, LDM, SDv2.1, StableDiffusion (SDv1.42, SDv2.13), Mid-journey, DALLE, etc. These models cover different generation algorithms and technical paths, such as diffusion models, variational autoencoders, and Transformer architectures, ensuring the breadth and technical diversity of the dataset.

[0064] Furthermore, a large number of different types of text prompts are selected for each model, covering a variety of topics such as daily scenes, abstract concepts, and artistic styles. This approach not only increases the diversity of images, but also enables the model to have better adaptability and traceability accuracy when facing various complex and specific text descriptions. For each selected text prompt, the above T2I model is used to generate a fixed number (e.g., 100) of synthetic images to ensure that each model generates multiple images for the same prompt text for subsequent analysis.

[0065] For example, Figure 3 As shown in the figure, each model generates the same number of images (such as 100) for the same text prompt, such as "In the garden, a white cat rests under the cherry tree. The sun casts mottled shadows through the flying petals." This ensures that the number of samples is sufficient to support statistical analysis in a balanced and standardized experimental environment, and ensures fairness in comparative analysis, because each group of images is generated under the same language guidance, and theoretically there should be a certain internal connection between them.

[0066] It should be noted that after the dataset is constructed, images from the same prompt text can be grouped, each group contains multiple images, and in the subsequent model training process, two images in the group are randomly selected as input for training in each iterative training.

[0067] Furthermore, the constructed dataset can be divided into a training dataset, a validation dataset and a test dataset. The ratio of the division can be 7:2:1 or other ratios. The training set is used to train the generated image tracing network; the validation set is used to evaluate the performance and generalization ability of the network during the training process, so as to select hyperparameters and tune the network; the test set is used to finally evaluate the performance of the network to verify its accuracy and robustness on unseen data.

[0068] In another possible implementation, after obtaining the sample training set, the sample images are subjected to size normalization and color space normalization, and the normalized sample images are subjected to data enhancement operations, where the data enhancement operations include rotation, scaling, and flipping.

[0069] Specifically, after constructing the data set, relevant preprocessing is required to optimize the data quality in order to prepare for subsequent model training. First, all images are resized and standardized in color space, such as converting to RGB color mode and adjusting to the same resolution to ensure that images from different sources are consistent before being input into the model. At the same time, in order to enhance the robustness of the model, the image data is lightly augmented, such as rotating, scaling, flipping, etc., so as to increase the diversity of the training set without changing the essential content of the image.

[0070] In another possible implementation, two sample images from any group of sample images are randomly selected and respectively input into a CLIP image encoder that shares the same parameters for feature extraction to obtain two corresponding feature vectors, wherein the feature vector includes deep semantic information; the absolute difference between the two feature vectors is calculated, wherein the absolute difference includes a discriminative feature; a contrastive learning objective function is constructed, wherein the objective function is used to maximize the representation similarity between similar sample images and minimize the similarity between non-similar sample images; based on the feature vectors and the absolute difference, the objective function is iteratively optimized to adjust the model parameters of the generative image tracing network to obtain a trained generative image tracing network.

[0071] Specifically, two images generated according to the same prompt text are respectively input into the CLIP image encoder sharing the same parameters for feature extraction. The CLIP image encoder is a pre-trained deep neural network that can convert images into high-dimensional feature representations to obtain high-dimensional feature vectors. These feature vectors represent the main visual information of the image and can be used to compare the similarity between two images.

[0072] It should be understood that the CLIP model uses large-scale cross-modal alignment learning to enable the model to seamlessly understand the relationship between images and text. During the training process, the CLIP model processes images and related text descriptions at the same time, and then learns highly consistent multimodal representations. Through this pre-training method, CLIP's image encoder can directly extract feature vectors rich in semantic information from images without relying on fine-tuning of any specific task. This not only improves the versatility of the model and enhances the model's understanding of complex and abstract concepts, but also greatly enhances its performance in various downstream tasks, effectively improving the generalization performance when facing novel or unseen generation technologies.

[0073] It should also be understood that since both CLIP image encoders share the same parameters and weights, they both process the image in the same way.

[0074] For example, Figure 4As shown in the figure, a preprocessed training image pair is used as the input of the generated image tracing network. The prompt text of the image pair is the same. First, the image pair is input into the CLIP image encoder for deep analysis. The process can be abstracted into the following formula:

[0075] f(I)=CLIP En (I)

[0076] Where I represents the input image, CLIP En The image encoding part of CLIP converts the image into a high-dimensional feature vector f. This feature vector is not just a description at the pixel level, but contains deep semantic information such as the theme, emotion, object category, etc. of the image. This conversion enables the machine to "see" the image in a more abstract and human-understandable way, providing a solid foundation for subsequent analysis.

[0077] It should be noted that the above two CLIP image encoders can be regarded as a Siamese (twin) network structure. Of course, the twin network can also contain more sub-networks with shared weights. Through the processing of the twin network, two feature vectors can be obtained, that is, f1 = CLIP En (I1) and f2 = CLIP En (I2), then the absolute difference between the two eigenvectors is calculated. This absolute difference can highlight the difference between the two images. This difference measurement method enhances the model's sensitivity to subtle changes, which is especially important when judging whether the images come from the same generative model. In this way, the model can more accurately predict the source of image generation based on these difference features, providing strong technical support for application fields such as image authenticity identification and copyright protection.

[0078] Finally, the new feature vector (i.e., the absolute difference) is input into a classification layer, which can determine whether the two images come from the same text-to-image generation model. If they are generated by the same model, the output result is "yes", otherwise the output is "no".

[0079] It is worth mentioning that in the above-mentioned generative image tracing network, based on the contrastive learning mechanism, a detailed objective function is designed to optimize the model's ability to identify the source of generated images. This objective function not only enhances the model's capture of the similarities between homologous images, but also ensures its sensitivity to the differences between different source images, thereby achieving the purpose of effectively tracing the source of the image generation model.

[0080] Furthermore, the objective function is constructed around the core concept of contrastive learning, which is to maximize the representation similarity between similar samples and minimize the similarity between dissimilar samples, so as to promote the model to learn effective image feature representation. For example, in practical application scenarios, for each pair of input images x i and x j , the goal of contrastive learning can be mathematically expressed by the following formula:

[0081]

[0082] Among them, L represents the target value of contrastive learning, x i and x j Represents a pair of input images, f represents the feature extraction function, which maps the image to a high-dimensional feature space. This can be done using the image encoding part of the CLIP model. sim is a function used to measure the similarity of two feature vectors. A common choice is cosine similarity, which measures the directional consistency of two vectors rather than their length or scale. τ represents the temperature parameter, which is used to adjust the steepness of the softmax function, thereby affecting the emphasis on the similarity gap between positive and negative samples in contrastive learning. A smaller τ value will make the model prefer extreme similarity distributions, while a larger τ value allows the model to have greater tolerance in distinguishing positive and negative samples. The terms in the numerator represent the maximization of the similarity between images from the same source, and the denominator normalizes this similarity with the similarity of all other images (including positive and negative samples) to ensure that the model not only focuses on positive samples, but also distinguishes negative samples.

[0083] It should be understood that in order to optimize the objective function, it is necessary to construct positive and negative sample pairs. The positive sample pairs consist of images generated by the same text prompt, while the negative sample pairs come from different text prompts or model generation. Therefore, the dataset needs to be carefully designed to ensure that it contains enough samples to cover various situations. By continuously iteratively optimizing the objective function, the model gradually learns to extract key features that can effectively distinguish the source of the image. This process involves optimization algorithms such as gradient descent to gradually adjust the model parameters to reduce the objective function value. In actual operation, in order to ensure the stability and efficiency of model training, it may be necessary to carefully adjust the loss function, such as by adjusting the temperature parameter τ, or introducing additional regularization terms to avoid overfitting.

[0084] Through the construction and optimization of the above-mentioned contrastive learning framework, for images of the same source, even if they differ in details, the model can capture their shared semantic and visual features and give a higher similarity score. For images from different sources, even if they are visually similar, the model can accurately distinguish them based on subtle feature differences, thereby achieving effective tracing of the source of the generated image.

[0085] In addition, the back-propagation algorithm plays a central role in the optimization process of the model, which allows the error to be back-propagated along the structure of the neural network, starting from the output layer, and calculating the gradient of each parameter with respect to the loss function.

[0086] Exemplarily, first, the gradient of the contrastive learning objective function with respect to the model parameter θ is calculated The objective function L aims to maximize the similarity between similar image pairs while minimizing the similarity between image pairs from different sources. The gradient calculation reflects how to fine-tune the model parameters to more accurately separate these image pairs. The gradient descent optimization algorithm updates the model parameters based on the direction and size of the gradient. The update rule can be expressed by the following formula:

[0087]

[0088] Among them, θ t represents the model parameters at the current time step t, θ t+1 represents the model parameters of the optimized time step t, η represents the learning rate, which controls the step size of parameter update. Choosing a suitable learning rate is crucial to avoid oscillation and slow convergence. represents the gradient operator, and L() represents the contrastive learning objective function.

[0089] In practical applications, batch gradient descent is usually used instead of single sample gradient descent to reduce noise and accelerate convergence, which means that before each parameter update, the average gradient on a small batch of data is calculated. At the same time, in order to avoid overfitting, regularization terms such as L1 or L2 regularization can be added to the objective function, which helps to promote the sparsity of parameters or reduce their amplitude, thereby improving the generalization ability of the model. As training proceeds, the learning rate can be appropriately reduced to prevent over-adjustment when approaching the optimal solution, ensuring that the model can converge to a good local minimum point.

[0090] In order to intuitively show the differences between image features learned through contrastive learning, the t-distributed stochastic neighbor embedding (t-SNE) method can be used to reduce the dimensionality of the high-dimensional feature vector extracted from the CLIP model. t-SNE is a nonlinear dimensionality reduction technology that is particularly good at maintaining local structures in high-dimensional space and showing the relative distance and clustering relationship between data points in low-dimensional space.

[0091] Specifically, a certain number of images are selected from each model and generated using the same text prompt. The CLIP model is then used to extract the semantic and content features of each image to form a set of feature vectors. The feature vectors are used as input and the t-SNE algorithm is used for dimensionality reduction. By setting an appropriate perplexity parameter to balance the preservation of local and global structures, the point cloud after dimensionality reduction can clearly distinguish the image features generated by different models. For example, Figure 5 As shown, from Figure 5 As can be seen in Figure 1, even in a complex feature space, the feature differences learned through contrastive learning can still be clearly distinguished in the t-SNE dimensionality reduction diagram. Image features from the same model tend to cluster together, forming relatively compact and distinguishable areas, while image features generated by different models are scattered in different locations, showing good separation. This visualization not only verifies that the contrastive learning method can effectively capture the essential differences between images generated by different models, but also provides an intuitive way for manual inspection, which is conducive to quickly identifying the potential source of the image.

[0092] After the above training process, the initial generated image tracing network has the ability to generate image tracing and can trace the image to be traced input into the network to determine the generation source of the image.

[0093] Optionally, after the model training is completed, the accuracy and generalization ability of the image generation source tracing algorithm based on contrastive learning and twin networks can be further evaluated and verified by designing relevant experiments. For example, using BLIP as a text extractor, the tracing accuracy of images generated by various image generation models (including LDM and SDv2.1 models involved in the training, and GLIDE models not involved in the training) is evaluated; by replacing different text extractors (such as trying gpt2-image-captioning, etc.), the adaptability and tracing accuracy of the model to different description methods are studied, revealing the impact of image description quality on tracing effect; exploring the impact of changes in the number of images output by a single generation model on tracing accuracy, and verifying the stability of the model under different data densities; by adding natural noise (such as JPEG compression distortion, Gaussian blur, etc.) to the test images, the robustness of the model in the face of complex real-world situations is evaluated.

[0094] See Table 1, which is the detailed information of the training data set. Table 1 is as follows:

[0095] Table 1 Details of the training dataset

[0096]

[0097] Table 1 shows the tracing performance of LDM, SD-v2.1 and GLIDE models on the test data set. It can be seen from the results in the table that even when facing the GLIDE model that did not appear in the training, the tracing algorithm still maintains a high accuracy (AUC is about 0.88), indicating that this method has good generalization ability.

[0098] See also Figure 6 , Figure 6 This paper describes the changes in tracing performance when using different I2T models for content extraction. When using the gpt2-image-captioning model as a content extractor, the tracing accuracy of the SD-v2.1 model decreases. This may be related to the low accuracy of the text description generated by the gpt2 model, while the BLIP model has a higher accuracy in generating text descriptions. Therefore, when using the BLIP model as a content extractor, the tracing accuracy of each model does not change significantly, which also shows that high-quality image descriptions play a vital role in improving tracing accuracy. See Figure 7 , Figure 7 The effect of changing the number of generated images on the tracing accuracy of the GLIDE model is shown. Figure 7 It can be seen that when the number of references of the generated images is increased, the tracing results do not change significantly, whether for Acc or AUC, indicating that the model is robust to the internal variability of a single generated model. Figure 8 , Figure 8 Describes the comparison of adding different types of noise (such as uncompressed (such as Figure 8 Uncompressed in), JPEG compression (such as Figure 8 JPEG75 and JPEG50 in ), Gaussian blur (such as Figure 8 The traceability performance after GaussB3 and GaussB1) in Figure 8 It can be seen that the detection model can still maintain a high accuracy rate when facing the noise interference commonly seen in real network communication, which shows the robustness of the model.

[0099] In summary, these experiments not only verify the accuracy of the generative image tracing method, but also its good generalization performance in the face of different descriptors, different generation models, different numbers of generated images and in the presence of noise interference.

[0100] S203: Output the generation source of the image to be traced.

[0101] Specifically, after the training process described in step S202, the initial generated image tracing network will have the ability to generate image tracing after a series of training. At this time, the input image to be traced can be directly processed to identify the generation source of the image to be traced.

[0102] For example, see Fig. 9 ,like Fig. 9 As shown in the figure, when it is necessary to trace the source of an unknown image, the text content of the image is first extracted through a self-description or image-to-text (I2T) model (such as Blip, gpt2, etc.), such as "a painting of a white cat sitting on a green surface covered with flowers", and then the text description is input into the T2I model of interest to generate an image. The newly generated image is then compared with the original query image input into the detection model. The detection model can make a judgment. If the original query image is generated by the relevant T2I model, then the newly generated image and the original query image will show strong semantic similarity. At this time, the detection model will give a positive judgment. Otherwise, if the two do not match, the detection model will give a negative judgment.

[0103] In this embodiment, by adopting a method combining contrastive learning and twin networks, for the images generated by the T2I model, the inherent semantic and content-related characteristics between the images generated under the same text prompt are used, and the differential expressions between the images generated by different models are learned and distinguished through a carefully designed contrastive learning objective function, so as to effectively track the generation source of the image. In the specific implementation, the input image to be traced is compared with the reference image generated by the research model using the same text prompt. If the image is derived from the same model, their semantic and visual features are highly consistent. Otherwise, the difference is obvious. The contrastive learning function grasps these differences, and then accurate image source tracking can be achieved. In addition, the semantic and content features of the image are extracted using the pre-trained CLIP model, which can further enhance the accuracy of tracking. It overcomes the problem of false attribution caused by simplifying the tracking problem into an image classification task in the traditional solution, can effectively deal with the challenges of image forgery and copyright infringement, and promote the healthy and orderly development of the image generation industry.

[0104] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0105] Fig.10 FIG. 2 is a block diagram showing a principle of a generative image tracing device 200 corresponding to the above embodiment. Fig.10 As shown, the generative image tracing device 200 includes an acquisition module 210, an image tracing module 220 and an output module 230. Each functional module is described in detail as follows.

[0106] The acquisition module 210 is used to acquire the image to be traced and the prompt text corresponding to the image to be traced.

[0107] The image tracing module 220 is used to determine the generation source of the image to be traced. The image tracing module includes a feature extraction module, a feature fusion module and a classification module. The feature extraction module includes at least two image encoders. The at least two image encoders share the same parameters. The at least two image encoders are respectively used to extract features of the image to be traced and the image generated according to the prompt text to obtain feature vectors. The feature fusion module is used to fuse the feature vectors to obtain a fused feature vector. The classification module is used to determine the generation source of the image to be traced based on the fused feature vector.

[0108] The output module 230 is used to output the generation source of the image to be traced.

[0109] For the specific limitations of the generative image tracing device, please refer to the limitations of the generative image tracing method above, which will not be repeated here. Each module in the above-mentioned generative image tracing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0110] To solve the above technical problems, the present application also provides a computer device. Fig.11 , Fig.11 This is a basic structural block diagram of the computer device in this embodiment.

[0111] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 300 having components connected to the memory 310, the processor 320, and the network interface 330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0112] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0113] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store the operating system and various application software installed in the computer device 300, such as program codes for controlling electronic files, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or will be output.

[0114] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to run the program code stored in the memory 310 or process data, such as running the program code for controlling the electronic file.

[0115] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0116] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the above method.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0118] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A generative image tracing method, characterized in that: The method comprises: Acquire the image to be traced and the prompt text corresponding to the image to be traced; Input the image to be traced and the prompt text into a generated image tracing network, the generated image tracing network is generated based on the CLIP model, the generated image tracing network includes a feature extraction module, a feature fusion module and a classification module, the feature extraction module includes at least two image encoders, the at least two image encoders share the same parameters, the at least two image encoders are respectively used to extract features from the image to be traced and the image generated according to the prompt text to obtain feature vectors, the feature fusion module is used to fuse the feature vectors to obtain a fused feature vector, and the classification module is used to determine the generation source of the image to be traced according to the fused feature vector; Output the generation source of the image to be traced.

2. The method according to claim 1, characterized in that Before inputting the image to be traced into a generation image tracing network, the method further includes: Select the integrated CLIP model as the initial generation image tracing network; Acquire a sample training set, wherein the sample training set includes a plurality of sample images, and the sample images include real images and images generated based on different generation models; Grouping the sample images based on the same prompt text, each group including at least two sample images; The initial generated image tracing network is trained using multiple groups of sample images.

3. The method according to claim 2, characterized in that The step of obtaining a sample training set comprises: Selecting a T2I model, wherein the T2I model includes GLIDE, LDM, SDv2.1, Mid-journey, and DALLE-E; determining text prompts, wherein types of the text prompts include everyday scenes, abstract concepts, and artistic styles; For each text prompt, a plurality of sample images are generated using the T2I model.

4. The method according to claim 2 or 3, characterized in that After obtaining the sample training set, the method further includes: Performing size standardization and color space standardization on the sample image; The standardized sample images are subjected to data enhancement operations, including rotation, scaling, and flipping.

5. The method according to claim 4, characterized in that The method of training the initially generated image tracing network using multiple groups of sample images includes: Randomly select two sample images from any group of sample images and input them into CLIP image encoders sharing the same parameters for feature extraction to obtain two corresponding feature vectors, wherein the feature vectors include deep semantic information; Calculating an absolute difference between the two feature vectors, wherein the absolute difference contains a discriminative feature; Constructing a contrastive learning objective function, the objective function being used to maximize the representation similarity between similar sample images and minimize the similarity between non-similar sample images; Based on the feature vector and the absolute difference, the objective function is iteratively optimized, and the model parameters of the generated image tracing network are adjusted to obtain a trained generated image tracing network.

6. The method according to claim 5, characterized in that The contrastive learning objective function is as follows: Among them, L represents the target value of contrastive learning, x i and x j Represents a pair of input images, f represents the feature extraction function, sim represents the function used to measure the similarity of two feature vectors, and τ represents the temperature parameter.

7. The method according to claim 6, characterized in that The iterative optimization of the objective function and the adjustment of the model parameters of the generated image tracing network include: The objective function is optimized using a gradient descent optimization algorithm, which is as follows: Among them, θ t represents the model parameters at the current time step t, θ t+1 represents the model parameters of the optimized time step t, η represents the learning rate, represents the gradient operator, and L() represents the contrastive learning objective function.

8. A generative image tracing device, characterized in that: The device comprises: An acquisition module, used for acquiring the image to be traced and the prompt text corresponding to the image to be traced; An image tracing module is used to determine the generation source of the image to be traced. The image tracing module includes a feature extraction module, a feature fusion module and a classification module. The feature extraction module includes at least two image encoders. The at least two image encoders share the same parameters. The at least two image encoders are used to extract features of the image to be traced and the image generated according to the prompt text, respectively, to obtain feature vectors. The feature fusion module is used to fuse the feature vectors to obtain a fused feature vector. The classification module is used to determine the generation source of the image to be traced according to the fused feature vector. The output module is used to output the generation source of the image to be traced.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images