A model provenance method and system for open environments

Abstract

The application provides a model tracing method in an open environment, comprising the following steps: constructing a closed set sample by using a known image and a corresponding known image generation model category; constructing an enhanced model by using a convolutional neural network, generating an open set sample of a corresponding unknown image generation model category based on the closed set sample and the enhanced model; training a task model by using the closed set sample and the open set sample, and predicting an image generation model of a given image by using the trained task model. The application also provides a model tracing system in an open environment and a data processing device for model tracing in an open environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of digital image forensics, and particularly relates to an open set model tracing method and system based on progressive open space expansion. BACKGROUND

[0002] Existing model tracing work has achieved ideal tracing effect on a fixed limited known model set, however, most of the work does not consider a large number of unknown models existing in the real world, and these methods often identify unknown models as a certain known model with high confidence.

[0003] A direct solution to the open set model tracing problem is to apply existing open set identification methods to model tracing. Existing open set identification methods can be roughly divided into discriminative or generative based methods, the performance of discriminative based open set identification methods is highly dependent on the closed set classifier, simply migrating this kind of method cannot improve the feature learning of the generated model fingerprint. The generative based open set identification method simulates the open space, making the feature space of the closed set class more compact, thereby improving the detection effect of unknown samples. However, existing generative based open set identification work often only uses a single generator or mechanism to simulate open set samples or features, resulting in synthesized samples lacking diversity. A single generator can generate open set samples of different semantics, but its fingerprint is fixed, so it is not suitable for the expansion of the open space of model tracing.

[0004] To solve the above problems, the present application proposes an open set model tracing method based on progressive open space expansion for the open set model tracing task, which gradually simulates the potential open space of unknown models by gradually increasing the enhanced model, and distinguishes between known and unknown models while tracing known models. Thus, the performance of the model tracing algorithm in the open set environment is improved. SUMMARY

[0005] To solve the above problems, the present application proposes an open environment model tracing method, including: constructing a closed set sample with a known image and a corresponding known image generation model class; constructing an enhanced model based on a convolutional neural network, and generating an open set sample of the corresponding unknown image generation model class based on the enhanced model and the closed set sample; training a task model based on the closed set sample and the open set sample, and predicting the image generation model of a given image through the trained task model.

[0006] The open environment model tracing method disclosed by the present application, wherein for any closed set sample, the closed set sample is input into the enhanced model, and N enhanced models and corresponding N open set samples are obtained through N-step training.

[0007] The open environment model tracing method disclosed by the present application, wherein the loss function of the enhanced model is Lrecons is the reconstruction loss of the generated open set sample and the input image in the pixel domain, x is the closed set sample, is the open set sample generated by the current enhanced model, is the open set sample generated by a randomly selected old enhanced model, L div is the diversity loss, α, β are hyperparameters, represents the feature embedding of the new open set sample, represents the feature embedding of the old open set sample, z represents the feature embedding of the input image, F cos is the cosine similarity function, d is the distance threshold to avoid the complete coincidence of the open set sample feature and the closed set sample feature.

[0008] The model tracing method in an open environment according to the present application, wherein the enhanced model has two layers of convolution, and the weight size of each layer of convolution is 3*32*3*3.

[0009] The model tracing method in an open environment according to the present application, wherein the loss function of the task model L cls (x) is the cross-entropy loss function of the known image generation model category, L metric is the metric loss for distinguishing each unknown image generation model category, L metric The triplet loss function is adopted.

[0010] The model tracing method in an open environment according to the present application, wherein the task model comprises a feature extractor and a K-way classification head, the given image is taken as the input of the task model, the confidence of the given image corresponding to K known image generation model types is obtained, if the maximum value in the confidence is greater than a threshold θ, then the given image is classified into the known image generation model category corresponding to the maximum value, otherwise, the given image is classified into the unknown image generation model category.

[0011] The present application also provides a model tracing system in an open environment, comprising: a closed set sample acquisition module, configured to construct a closed set sample with a known image and a corresponding known image generation model category; an open set sample acquisition module, configured to construct an enhanced model with a convolutional neural network, and generate an open set sample corresponding to an unknown image generation model category based on the closed set sample and the enhanced model; a training and prediction module, configured to train a task model with the closed set sample and the open set sample, and predict the image generation model of a given image through the trained task model.

[0012] The model tracing system of the open environment, wherein the task model comprises a feature extractor and a K-way classification head, the training and prediction module takes the given image as the input of the task model, obtains the confidence of the given image corresponding to K known image generation model types, if the maximum value in the confidence is greater than a threshold value theta, then the given image is classified into the known image generation model category corresponding to the maximum value, otherwise, the given image is classified into an unknown image generation model category.

[0013] The application further provides a computer readable storage medium storing computer executable instructions, wherein when the computer executable instructions are executed, the open environment model tracing method is realized.

[0014] The application further provides a data processing device comprising the computer readable storage medium, when the processor of the data processing device calls and executes the computer executable instructions in the computer readable storage medium, the open environment model tracing is realized. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 It is a model tracing method flowchart of the application.

[0016] Figure 2 It is a data processing device schematic diagram of the application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below with reference to the drawings. It should be understood that the specific implementation method described herein is only used to explain the application, and is not used to limit the application.

[0018] The main technical problem solved by the application is how to effectively simulate the open set space and improve the performance of the model tracing algorithm in the open set environment.

[0019] The model tracing method of the application adopts an open space simulation method based on a convolution small model. In order to simulate the open space of unknown models, an intuitive method is to train a large number of models close to real models. However, the training cost of considering all types of real models is too high. In order to solve this problem, the application adopts a small number of parameter convolution small models, and the small models receive the generated images of known models as input, maintain the semantics of the input images but disturb the traces of the images. Through this method, the open set space located at the boundary of the known model fingerprint can be simulated at the cost of a small number of parameters.

[0020] In addition, an open space expansion method based on progressive training is also adopted. In order to enrich the simulated open set space, it is necessary to expand the simulated open set space. An intuitive method is to independently train a large number of small models. However, simply increasing the number of small models may cause the expanded open set spaces of different small models to overlap, and the open set space expansion efficiency is not high. In order to solve this problem, the present application proposes an open space expansion method based on progressive training, which improves the efficiency of open space expansion by constraining the difference between the newly expanded open set space and the old expanded space.

[0021] The specific implementation method of the present application includes a task model and N enhanced models. The task model includes a feature extractor and a K-way classification head, K being the number of known classes. An enhanced model is newly trained for each epoch. As shown in Figure 1 the training process of the task model and the N enhanced models is introduced as follows:

[0022] Step S1, constructing a closed set sample x with known images and corresponding known image generation model classes;

[0023] Step S2, enhanced model training: the open set sample generated for the current enhanced model, the open set sample generated for a randomly selected old enhanced model. The training loss of the current enhanced model is:

[0024]

[0025] wherein L recons is the reconstruction loss of the generated open set sample and the input image in the pixel domain, L div is a diversity loss, which constrains the distance between the open set sample generated by the new enhanced model and the open set sample generated by the old enhanced model in the feature space to be as large as possible, so as to ensure that the new enhanced model can effectively expand the open set space outside the old expanded open set space. Specifically, L div is calculated as follows:

[0026]

[0027] wherein, and are the feature embeddings of the new open set sample and the old open set sample, z is the feature embedding of the input image, F cos is a cosine similarity function. The first term of L div constrains the similarity between the old open set sample and the new open set sample to be as low as possible. However, simply constraining the new open set sample and the old open set sample to be dissimilar may cause the enhanced model to generate simple open set samples that are also far away from the boundary of the known model. In order to avoid this situation, the second term of L divThe second constraint restricts the feature similarity between the new open set sample and the known sample as high as possible, while using a distance threshold d to avoid the feature of the open set sample and the feature of the known sample from being completely overlapped.

[0028] Step S3, training the task model: given a closed set sample x, a new open set sample and an old generated open set sample The loss function L of the task model task is:

[0029]

[0030] Wherein, L cls (x) is the cross-entropy loss of known class classification, which is calculated by L metric The constraint restricts the task model to distinguish between known samples and open set samples, while distinguishing between different known classes and open set sample classes. In order to avoid model forgetting old open set samples, the present application simultaneously calculates L metric on old and new open set samples. Specifically, L metric Adopting triplet loss, unlike previous work which regards all open set samples as K+1 classes, the present application assigns different open set classes to open set samples obtained by inputting different classes of known samples into the enhanced model, and randomly selects triplets in the image pool containing known samples and open set samples to calculate the triplet loss.

[0031] Step S4, open environment model prediction: given a given image, first input the given image into the feature extractor and classification head to obtain the predicted confidence of each class. If the maximum confidence is greater than the threshold θ, the image is classified as the model corresponding to the maximum confidence, otherwise it is recognized as an unknown model.

[0032] Figure 2 is a schematic diagram of the data processing device of the present application. As Figure 2As shown, the embodiment of the present application further provides a computer readable storage medium, and a data processing apparatus. The computer readable storage medium of the present application stores computer executable instructions, and when the computer executable instructions are executed by a processor of the data processing apparatus, the image generation model of an open environment is traced. Those skilled in the art can understand that all or part of the steps in the above method can be completed by a program instructing related hardware (for example, a processor, an FPGA, an ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. All or part of the steps of the above embodiment can also be implemented by using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, for example, by an integrated circuit to implement its corresponding function, or can be implemented in the form of a software function module, for example, by a processor executing a program / instruction stored in a memory to implement its corresponding function. The embodiment of the present application is not limited to any specific form of combination of hardware and software.

[0033] The model open set tracing method based on the progressive open space expansion of the present application simulates the potential open space of an unknown model by a method of gradually increasing the enhanced model. The present application considers three open set scenarios including an unseen random seed, an unseen structure and an unseen data set. A large number of experimental results show that the method of the present application is superior to existing model tracing methods and open set identification methods.

[0034] The above embodiments are only used to illustrate the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and all equivalent technical solutions also belong to the scope of the present application. The patent protection scope of the present application should be defined by the claims.

Claims

1. A method for model provenance in open environments, characterized in that, Comprise: A closed set sample is constructed by known images and corresponding known image generation model categories; An enhanced model is constructed by a convolutional neural network, and an open set sample corresponding to an unknown image generation model category is generated based on the closed set sample and the enhanced model; the loss function of the enhanced model , is a reconstruction loss of the generated open set sample and the input image in the pixel domain, is a closed set sample, is an open set sample generated by the current enhanced model, is an open set sample generated by a randomly selected old enhanced model, is a diversity loss, , , is a hyperparameter, denotes the feature embedding of the new open set sample, denotes the feature embedding of the old open set sample, and z denotes the feature embedding of the input image, is a cosine similarity function, is a distance threshold value to avoid complete overlap of open set sample features and closed set sample features; A task model is trained based on the closed set sample and the open set sample, and an image generation model of a given image is predicted by the trained task model.

2. The open environment model provenance method of claim 1, wherein, For any closed set sample, the closed set sample is input into the enhanced model, and N enhanced models and corresponding N open set samples are obtained through N-step training.

3. The open environment model provenance method of claim 1, wherein, The enhanced model has two layers of convolution, and the weight size of each layer of convolution is 3x32x3x3.

4. The open environment's model provenance method of claim 1, wherein, The loss function of the task model , The cross-entropy loss function of the model class for the known image, The metric loss for distinguishing the model classes of the unknown images, Adopt the triplet loss function.

5. The open environment model provenance method of claim 1, wherein, The task model includes a feature extractor and a K-way classification head, takes the given image as input of the task model, obtains a confidence of the given image corresponding to K known image generation model types, and if a maximum value in the confidence is greater than a threshold , then the given image is classified as a known image generation model category corresponding to the maximum value, otherwise, the given image is classified as an unknown image generation model category.

6. A model provenance system for open environments, characterized in that, Comprise: A closed set sample acquisition module is configured to construct a closed set sample by known images and corresponding known image generation model categories; An open set sample acquisition module is configured to construct an enhanced model by a convolutional neural network, and generate an open set sample corresponding to an unknown image generation model category based on the closed set sample and the enhanced model; the loss function of the enhanced model , is the reconstruction loss of the generated open set sample and the input image in the pixel domain, is the closed set sample, is the open set sample generated by the current enhanced model, is the open set sample generated by a randomly selected old enhanced model, is the diversity loss, , , is a hyperparameter, represents the feature embedding of the new open set sample, represents the feature embedding of the old open set sample, and z represents the feature embedding of the input image, is a cosine similarity function, is a distance threshold to avoid complete overlap between the open set sample features and the closed set sample features; A training and prediction module is configured to train a task model based on the closed set sample and the open set sample, and predict an image generation model of a given image by the trained task model.

7. The open environment's model provenance system of claim 6, wherein, The task model comprises a feature extractor and a K-way classification head, the training and prediction module inputs the given image into the task model to obtain a confidence of K known image generation model types corresponding to the given image If a maximum value in the confidence is greater than a threshold value, the given image is classified as a known image generation model category corresponding to the maximum value, otherwise, the given image is classified as an unknown image generation model category.

8. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, cause the processor to perform operations comprising: When the computer executable instructions are executed, the model tracing method in an open environment as claimed in any one of claims 1-5 is implemented.

9. A data processing apparatus comprising the computer readable storage medium as claimed in claim 8, when the processor of the data processing apparatus invokes and executes the computer executable instructions in the computer readable storage medium, the model tracing in an open environment is implemented.

Images