Model reversing method, system, device and equipment for multi-modal pre-training model

Abstract

The invention provides a model reversing method, system, device and equipment for a multi-modal pre-training model, and belongs to the field of artificial intelligence security and privacy protection. A multi-modal pre-training model generally comprises an image encoder and a text encoder, and the association between cross-modal data (such as images and texts) is learned through training. According to the model reversing method for the multi-modal pre-training model, through embedding space alignment, adversarial fine-tuning of an image encoder and diffusion model refining, reconstruction from text prompting to high-fidelity training images is achieved, visual detail and semantic accuracy is remarkably improved, and a new tool is provided for evaluating model privacy risks.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence security and privacy protection, and specifically relates to model reverse engineering and attack technology, and more specifically to a model reverse engineering method, system, device and equipment for multimodal pre-training models. Background Art

[0002] In recent years, large-scale pre-trained models, represented by large language models (LLMs), text-to-image generation models (such as diffusion models), and multimodal models that learn cross-modal associations (for example, models trained on massive image-text pairs through contrastive learning), have achieved breakthrough progress and widespread application in many artificial intelligence tasks such as natural language processing, computer vision, and multimodal understanding and generation. The core capability of such multimodal models lies in learning a powerful, unified or aligned representation space that captures deep semantic associations between different modalities through joint training on large-scale, diverse cross-modal data pairs (such as image-text pairs). This enables the model to perform complex tasks such as image retrieval, image description generation, text-guided image generation and editing, and visual question answering.

[0003] However, while these powerful, large models offer convenience, they also introduce new and serious security and privacy risks. Research has demonstrated that deep learning models, including LLMs and various generative models, can inadvertently "memorize" portions of their training data during training, even including sensitive or private information. By crafting specific queries or inputs (i.e., prompts or probes), attackers can potentially induce or reconstruct these memorized training data fragments from the model output. Conditional models, particularly those that require specific inputs (such as text prompts) to guide their output (e.g., text-to-image generation models or multimodal models that perform cross-modal tasks), exhibit a stronger tendency to memorize training data. The design goal of multimodal models is to explicitly learn the close association between one modality (e.g., an image) and its corresponding other modality (e.g., a text description). This inherent cross-modal conditional dependency could theoretically further exacerbate the model's memorization of training data and increase the risk of leaking training data from one modality (e.g., an image) through input from another modality (e.g., text).

[0004] Model inversion (MI) attacks are a class of techniques designed to recover the training data from the parameters or behavior of a trained model. Early MI research focused on relatively simple models (such as classifiers) and low-dimensional or structured data. Its effectiveness is often limited for modern large-scale multimodal models that process high-dimensional, complex data (such as natural images). Recently, with increasing attention to the privacy issues of large models, some research has begun to apply MI techniques to complex multimodal scenarios, and has initially demonstrated the feasibility of reconstructing images that may have been paired with a given text prompt during training.

[0005] Despite this, existing inverse methods for such multimodal models still face significant technical challenges in practical applications, especially in terms of the quality and fidelity of the reconstructed images:

[0006] (1) Non-robust feature interference problem: Multimodal models under standard training (especially their image encoder part) may over-rely on some "non-robust" features (such as small texture changes or high-frequency noise) during the learning process, which are not sensitive to the human visual system but play a key role in the model's internal prediction. When performing gradient-based inverse optimization to reconstruct the image, these non-robust features may cause the gradient signal during the optimization process to be extremely noisy and unstable, making it difficult for the optimization to converge to a visually clear and reasonable image. The final reconstruction result is often full of artifacts or of low quality.

[0007] (2) The problem of cross-modal embedding space differences: The text encoder in the model usually focuses more on capturing abstract semantic concepts. Although the generated text embedding is semantically related to the corresponding image, it often lacks the rich visual details contained in the image embedding, such as the specific spatial layout of the object, precise color and texture, lighting conditions, etc. If the original text embedding is directly used as the optimization target to guide image reconstruction, due to this inherent information difference, it is difficult to restore a visually realistic and detailed image, and it may even introduce text-related visual artifacts in the reconstructed image (such as the direct appearance of text).

[0008] (3) The problem of generally low fidelity of reconstructed images: The combination of the above two factors, coupled with the inherent limitations of traditional optimization-based image generation methods in generating high-resolution, high-complexity natural images, results in the images reconstructed by existing model inverse methods. Although they may roughly match the target text prompts in semantics, they are often far from the actual training images in terms of visual quality, detail presentation, and overall realism, and their fidelity is low.

[0009] Therefore, there is an urgent need to develop a more advanced and effective technical solution that can overcome the above challenges and achieve high-quality, high-fidelity reconstruction of multimodal pre-trained model training data (especially images). Such a technology can not only more accurately assess the privacy leakage risks of these models, but also provide a powerful analytical tool for understanding the internal representation and memory mechanisms of the models. Summary of the Invention

[0010] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a model inversion method, system, device and equipment for multimodal pre-training models.

[0011] The present invention provides a model inversion method for a multimodal pre-trained model. The multimodal pre-trained model includes at least an original image encoder and an original text encoder, and has the following characteristics, including the following steps: S10, using an independent auxiliary image dataset to perform adversarial perturbation training on the original image encoder to obtain a fine-tuned image encoder; S20, after using the original text encoder to calculate the original text embedding of the target text prompt, using the pre-trained embedding space alignment transformation to map the original text embedding to the target embedding space aligned with its corresponding image embedding, to obtain the aligned target embedding; S30, after creating an image variable, iteratively updating the pixel value of the image variable based on a gradient optimization algorithm, so that the similarity between the current image embedding obtained by inputting the image variable into the fine-tuned image encoder and the aligned target embedding is maximized, thereby generating a preliminary reconstructed image; S40, using a pre-trained conditional generative model that can accept conditional input and generate high-quality images, with the target text prompt and / or the aligned target embedding as conditions, the preliminary reconstructed image is denoised and conditionally denoised to generate a final reconstructed image.

[0012] In the model inversion method for a multimodal pre-training model provided by the present invention, it may also have the following features: wherein, step S10 includes the following steps: S11, after extracting a batch of images from an independent auxiliary image dataset, calculating their embedded representation under the original image encoder or the current image encoder, and recording the embedded representation as a first embedded representation; S12, generating adversarial perturbations for the images extracted in step S11, and adding the adversarial perturbations to the corresponding images to form adversarial image samples; S13, inputting the adversarial image samples into the original image encoder or the current image encoder to obtain an embedded representation, and recording the embedded representation as a second embedded representation; S14, after defining a loss function that can minimize the difference between the first embedded representation and the second embedded representation, calculating the gradient according to the loss function and using it to update the parameters of the original image encoder or the current image encoder to fine-tune it; S15, repeating steps S11 to S14 until the robustness index of the current image encoder reaches the expected level or converges, and obtaining a fine-tuned image encoder.

[0013] The model inverse method for the multimodal pre-trained model provided by the present invention may also have the following features: wherein, in step S20, the pre-trained embedding space alignment transformation is learned in the following manner: A10, preparing an independent auxiliary data set of image-text pairs with paired data; A20, using a fine-tuned image encoder to extract the embedding representations of all images in the auxiliary data set to form an image embedding set; A30, using the original text encoder to extract the embedding representations of all corresponding texts in the auxiliary data set to form a text embedding set corresponding to the image embedding set; A40, learning a transformation operation that acts on the text embedding set and can make it as close as possible to the corresponding image embedding set, and the transformation operation is the pre-trained embedding space alignment transformation.

[0014] The model inverse method for the multimodal pre-training model provided by the present invention may also have the following features: wherein, the transformation operation in step A40 includes a linear transformation, the linear transformation is a transformation matrix, and the transformation matrix minimizes the average distance between the transformed text embedding set and its corresponding image embedding set.

[0015] In the model inverse method for multimodal pre-training models provided by the present invention, it can also have the following features: wherein, step S30 includes the following sub-steps: S31, starting from a random noise image, a blurred image or an average image, creating an image variable to be optimized; S32, inputting the current image variable into the fine-tuning image encoder to obtain the current image embedding; S33, calculating the optimization target loss to measure the dissimilarity between the current image embedding and the aligned target embedding; S34, calculating the regularization loss that can inject prior knowledge of the image and encourage the generated image to have better visual characteristics; S35, after obtaining the total loss function by weighted combination of the optimization target loss and the regularization loss, calculating its gradient with respect to the pixel value of the current image variable according to the total loss function; S36, using the optimizer to update the pixel value of the current image variable according to the gradient in step S35; S37, repeating steps S32 to S36 until the preset number of iterations is reached or the total loss function converges to generate a preliminary reconstructed image.

[0016] The model inverse method for the multimodal pre-trained model provided by the present invention may also have the following features: wherein, in step S40, the pre-trained conditional generative model includes a text-to-image diffusion model, GAN or VAE, and the diffusion model includes Stable Diffusion or Imagen.

[0017] In the model inverse method for multimodal pre-trained models provided by the present invention, it can also have the following characteristics: wherein, the pre-trained conditional generative model is a text-to-image diffusion model, and step S40 includes the following sub-steps: S41, adding forward diffusion noise to the preliminary reconstructed image, wherein the degree of added noise corresponds to the time step of the diffusion process; S42, running the reverse denoising process of the pre-trained conditional generative model, and in each denoising / generation step, using the target text prompt and / or the aligned target embedding for guidance to ensure that the pre-trained conditional generative model always moves in a direction consistent with the text semantics while removing noise and generating details; S43, repeating steps S41 to S42 until the optimization target loss no longer decreases, and finally obtaining a high-quality image.

[0018] The present invention also provides a model inversion system for a multimodal pre-training model, which includes at least an original image encoder and an original text encoder, and has the following characteristics: a model inversion method for a multimodal pre-training model using any of the aforementioned items, including: an image encoder fine-tuning module, which is used to perform adversarial perturbation training on the original image encoder using an independent auxiliary image dataset; an aligned target embedding generation module, which is used to use the original text encoder to calculate the original text embedding of the target text prompt, and then use the pre-trained embedding space alignment transformation to map the original text embedding to the target embedding space aligned with its corresponding image embedding to obtain the aligned target embedding; a preliminary image reconstruction module, which is used to create an image variable, iteratively update the pixel value of the image variable based on a gradient optimization algorithm, so that the similarity between the current image embedding obtained in the image variable input fine-tuning image encoder and the aligned target embedding is maximized, thereby generating a preliminary reconstructed image; a final image reconstruction module, which uses a pre-trained conditional generative model that can accept conditional input and generate high-quality images, and performs noise and conditional denoising on the preliminary reconstructed image based on the target text prompt and / or the aligned target embedding to generate a final reconstructed image.

[0019] The present invention also provides a model inversion device for a multimodal pre-training model, which has the following characteristics: it uses the aforementioned model inversion system for a multimodal pre-training model, including: a memory for storing the model inversion system for a multimodal pre-training model and intermediate data generated during the processing; a processor for executing a model inversion method for a multimodal pre-training model according to the model inversion system for a multimodal pre-training model in the memory; an input interface connected to the processor for receiving a target text prompt; and an output interface connected to the processor for outputting a final reconstructed image.

[0020] The present invention also provides a model inversion device for a multimodal pre-training model, which has the following characteristics: it uses the aforementioned model inversion device for a multimodal pre-training model.

[0021] Functions and effects of the invention

[0022] According to a model inversion method, system, apparatus, and device for a multimodal pre-trained model, the multimodal pre-trained model includes at least one original image encoder and one original text encoder, and has the following characteristics. The model inversion method for the multimodal pre-trained model includes the following steps: S10, using an independent auxiliary image dataset to perform adversarial perturbation training on the original image encoder to obtain a fine-tuned image encoder; S20, after using the original text encoder to calculate the original text embedding of the target text prompt, using a pre-trained embedding space alignment transformation to map the original text embedding to a target embedding space aligned with its corresponding image embedding, to obtain an aligned target embedding; S30, after creating an image variable, iteratively updating the pixel value of the image variable based on a gradient optimization algorithm to maximize the similarity between the current image embedding obtained by inputting the image variable into the fine-tuned image encoder and the aligned target embedding, thereby generating a preliminary reconstructed image; S40, using a pre-trained conditional generative model that can accept conditional input and generate high-quality images, denoising and conditionally denoising the preliminary reconstructed image based on the target text prompt and / or the aligned target embedding, to generate a final reconstructed image.

[0023] Therefore, the present invention addresses the core issues commonly faced by existing techniques when attempting to perform model inversion attacks on multimodal pre-trained models (especially those that have learned image-text associations) to reconstruct training images, such as poor visual quality, missing details, and insufficient realism (i.e., low fidelity) of reconstructed images. The invention aims to provide an innovative method and system that, starting from a given text prompt, can more efficiently, accurately, and with higher fidelity reconstruct an image that is highly semantically related to the text prompt and has visual quality close to the actual training data, compared to existing techniques.

[0024] The present invention provides a model inversion method, system, device, and apparatus for a multimodal pre-trained model, which has the following beneficial effects:

[0025] (1) Significantly improve the quality and fidelity of reconstructed images: By introducing an adversarial fine-tuning step, the interference of non-robust features on the optimization process is effectively suppressed, making the initial reconstruction more stable and accurate. Combined with the alignment of the embedding space, it provides a target signal that is more suitable for visual reconstruction for optimization. Most importantly, the use of a powerful pre-trained generative model for final refinement greatly improves the visual realism, detail richness, and overall aesthetic quality of the reconstructed images, making the reconstruction results significantly better than existing methods in multiple objective image quality and similarity evaluation indicators (such as SSIM, LPIPS, and FID, etc.) as well as subjective visual evaluation.

[0026] (2) Effectively overcome the core challenges of the existing technology: The present invention specifically designs solutions to address the three major challenges mentioned in the background technology: adversarial fine-tuning alleviates the problem of non-robust features; embedding alignment compensates for the difference in visual information between text and image embeddings; and generative model refinement significantly improves the final reconstruction fidelity.

[0027] (3) Enhanced assessment of model privacy risks: This invention provides a model inversion technique with stronger attack capabilities and better reconstruction results. This method can more deeply and accurately explore and assess the degree of memory of training data and potential privacy leakage risks of large multimodal models (especially models trained based on contrastive learning, etc.), providing a more effective risk assessment tool for model developers, deployers, and regulators.

[0028] (4) The method is highly versatile: Although the image-text modality is used as an example for detailed explanation, the core ideas of the present invention (adversarial fine-tuning encoder, cross-modal embedding alignment, optimized reconstruction, and generative model refinement) have certain versatility and are expected to be extended to other types of multimodal models and reverse scenarios of data modality combinations.

[0029] (5) The present invention has important application value in multiple fields. In terms of privacy security assessment, enterprises or research institutions can use this technology to detect whether there is a risk of training data leakage in large multimodal models (such as CLIP, Stable Diffusion, etc.), and help identify whether the model has memorized sensitive image data (such as medical images, facial information), so as to formulate more effective privacy protection strategies. In the field of intellectual property protection, this technology can help identify whether AI-generated content uses copyrighted training data, providing technical support for digital copyright management. In terms of model security testing, researchers can use this method to evaluate the resistance of different model architectures to data leakage, and promote the development of more privacy-protective AI systems. In digital forensic investigations, law enforcement agencies can use this technology to trace the source of training data for suspicious AI-generated content, providing new technical means for cybercrime investigations. In addition, in the field of AI ethics research, this method provides a quantitative tool for evaluating model memory effects, which helps to establish more standardized AI data usage guidelines. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of a model inversion method for a multimodal pre-training model according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a model inversion method for a multimodal pre-training model according to an embodiment of the present invention;

[0032] Figure 3 1 is a schematic diagram of the connection relationship of a model inversion system for a multimodal pre-training model according to an embodiment of the present invention;

[0033] Figure 4 It is a schematic diagram of the connection relationship of the model inversion device for the multimodal pre-training model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the model inversion method, system, device and equipment for a multimodal pre-training model of the present invention.

[0035] <Example>

[0036] Figure 1 This is a flowchart of a model inversion method for a multimodal pre-training model according to an embodiment of the present invention.

[0037] like Figure 1As shown, this embodiment provides a model inversion method for a multimodal pre-training model, where the multimodal pre-training model includes at least one original image encoder and one original text encoder, including the following steps:

[0038] S10, adversarial fine-tuning of the image encoder, including the following sub-steps S11 to S15:

[0039] S11: prepare an auxiliary image dataset that is independent of the target model training data and the data to be subsequently reconstructed. After extracting a batch of images from the auxiliary image dataset, calculate their embedded representation under the original image encoder or the current image encoder, and record the embedded representation as the first embedded representation.

[0040] S12, generating adversarial perturbations for the image extracted in step S11, and adding the adversarial perturbations to the corresponding images to form adversarial image samples.

[0041] S13: Input the adversarial image sample into the original image encoder or the current image encoder to obtain an embedded representation, and record the embedded representation as a second embedded representation.

[0042] In steps S12 and S13 above, the goal of the adversarial perturbation in step S12 is to maximize the change in the second embedding representation output by the image after passing through the original image encoder or the current image encoder in step S13, while maintaining a certain perturbation strength. This can be achieved through a gradient-based attack method, which seeks input perturbations that significantly deviate from the embedding vector.

[0043] S14, after defining a loss function that can minimize the difference between the first embedding representation and the second embedding representation (for example, the distance between the vector representations of the two), the gradient is calculated according to the loss function and the parameters of the original image encoder or the current image encoder are updated to fine-tune it.

[0044] S15, repeating steps S11 to S14 until the current image encoder has been sufficiently adversarially fine-tuned (e.g., trained on an auxiliary dataset for several rounds) until its robustness index reaches a desired level or converges, thereby obtaining a fine-tuned image encoder. The fine-tuned image encoder is expected to have better robustness and smoother optimization characteristics.

[0045] S20, cross-modal embedding space alignment, includes the following sub-steps S21-S22:

[0046] S21, for the target text prompt subsequently used for image reconstruction, the original text encoder is used to calculate the original text embedding of the target text prompt.

[0047] S22, uses the pre-trained embedding space alignment transformation to map the original text embedding to the target embedding space aligned with its corresponding image embedding, and obtains the aligned target embedding.

[0048] In step S22, it is recognized that although text embeddings and image embeddings are semantically aligned, they have inherent differences in expressing specific visual details. Therefore, it is necessary to learn a transformation to "move" the text embedding to a position in the embedding space that is closer to its corresponding image embedding, thereby providing a better target signal with more potential visual information for subsequent image reconstruction. In step S22, the pre-trained embedding space alignment transformation is learned through the following steps A10 to A40:

[0049] A10, prepare another independent auxiliary dataset containing paired data (e.g., image-text pairs) (also unrelated to the target reconstruction data).

[0050] A20, utilizes a fine-tuned image encoder to extract embedding representations of all images in the auxiliary dataset to form a set of image embeddings.

[0051] A30, uses the original text encoder to extract the embedding representations of all corresponding texts in the auxiliary dataset, forming a text embedding set corresponding to the image embedding set.

[0052] A40, based on the paired image and text embedding representations from steps A20 and A30, learns a transformation operation. This transformation operation is applied to the text embedding, with the goal of making the transformed text embedding as close as possible to the corresponding image embedding. This transformation operation is the pre-trained embedding space alignment transformation.

[0053] Specifically in this embodiment, an effective and commonly used method is to learn a linear transformation (for example, by learning a transformation matrix) so that the average distance (for example, Euclidean distance or cosine distance) between the transformed text embedding and the corresponding image embedding is minimized over the entire auxiliary dataset. This transformation relationship can be obtained by solving a linear regression problem.

[0054] S30, initial image reconstruction, includes the following sub-steps S31 to S37:

[0055] S31, creating an image variable to be optimized. This can start from a random noise image (for example, pixel values ​​are sampled from a Gaussian distribution or a uniform distribution), or from a blurred image, an average image, etc.

[0056] S32, inputting the current image variable into the fine-tuning image encoder to obtain the current image embedding.

[0057] S33, calculate an optimization target loss. This loss function measures the dissimilarity between the current image embedding and the aligned target embedding. The optimization goal is to maximize the cosine similarity between the two, or minimize the distance between them (such as L2 distance).

[0058] S34 calculates one or more regularization losses. These losses are used to inject prior knowledge about the image into the optimization process, encouraging the generated image to have better visual properties, such as smoothness and a reasonable color distribution. Common regularization terms include total variation (TV), which can suppress high-frequency noise in the image and make the image smoother. Constraints on the range or distribution of image pixel values ​​can also be added.

[0059] S35, after obtaining the total loss function by weighted combination of the optimization target loss and the regularization loss (the weight of the regularization loss needs to be adjusted appropriately), the gradient of the pixel value with respect to the current image variable is calculated according to the total loss function.

[0060] S36, using an optimizer (such as AdamW) to update the pixel value of the current image variable according to the gradient in step S35.

[0061] S37: Using a gradient-based optimization algorithm (e.g., Adam, AdamW, or L-BFGS), repeat steps S32 to S36 to iteratively modify the pixel values ​​of the current image variable until a predetermined number of iterations is reached or the total loss function converges (e.g., the change is less than a threshold), thereby generating a preliminary reconstructed image. This image is aligned with the target at the embedding level, but may still lack visual quality and detail.

[0062] S40, generative model refinement, includes the following sub-steps S41 to S43:

[0063] S41 adds a certain amount of forward diffusion noise to the preliminary reconstructed image. The degree of noise addition (corresponding to the time step of the diffusion process) is a key hyperparameter that controls the degree of preservation of the original image structure and the space allowed for model creativity.

[0064] S42: Run the reverse denoising process of the pre-trained conditional generative model. In each denoising / generation step, the target text hint and / or the aligned target embedding are used for guidance to ensure that the pre-trained conditional generative model is always moving in a direction consistent with the text semantics while removing noise and generating details. Techniques such as Classifier-Free Guidance (CFG) can be used to control the strength of the conditional guidance.

[0065] The pre-trained conditional generative model includes a text-to-image diffusion model, GAN, or VAE, and the diffusion model includes Stable Diffusion or Imagen. Specifically, in this embodiment, the pre-trained conditional generative model is a diffusion model.

[0066] In this step, the rich natural image prior knowledge and powerful generation capabilities contained in the pre-trained conditional generative model are used to repair defects in the initial reconstructed image, improve its visual quality, add realistic details, and maintain semantic consistency with the target text prompt.

[0067] S43, repeating steps S41 and S42 until the optimization target loss no longer decreases, ultimately obtaining a high-quality image. This image is expected to have significantly improved visual fidelity, rich details, and good overall quality, and be highly relevant in content to the original target text prompt.

[0068] Figure 3 It is a schematic diagram of the connection relationship of a model inverse system for a multimodal pre-training model according to an embodiment of the present invention.

[0069] like Figure 3 As shown, this embodiment also provides a model inverse system 100 for a multimodal pre-training model, which uses the model inverse method for a multimodal pre-training model provided by this embodiment, including an image encoder fine-tuning module 10, an aligned target embedding generation module 20, a preliminary reconstructed image module 30 and a final reconstructed image module 40.

[0070] The image encoder fine-tuning module 10 is used to perform adversarial perturbation training on the original image encoder using an independent auxiliary image dataset according to the method of step S10.

[0071] The input of the image encoder fine-tuning module 10 is the multimodal pre-trained model and the auxiliary image dataset, and the output is the fine-tuned image encoder.

[0072] The aligned target embedding generation module 20 is used to calculate the original text embedding of the target text prompt using the original text encoder according to the method of step S20, and then use the pre-trained embedding space alignment transformation to map the original text embedding to the target embedding space aligned with its corresponding image embedding to obtain the aligned target embedding.

[0073] The input of the aligned target embedding generation module 20 is the fine-tuned image encoder, the original text encoder, the auxiliary dataset and the target text prompt, and the output is the aligned target embedding.

[0074] The preliminary image reconstruction module 30 is used to create an image variable according to the method of step S30, and then iteratively update the pixel value of the image variable based on the gradient optimization algorithm to maximize the similarity between the current image embedding obtained by inputting the image variable into the fine-tuning image encoder and the aligned target embedding, thereby generating a preliminary reconstructed image.

[0075] The input of the preliminary reconstructed image module 30 is the aligned target embedding and the fine-tuned image encoder, and the output is the preliminary reconstructed image.

[0076] The final reconstructed image module 40 is used to use a pre-trained conditional generative model that can accept conditional input and generate high-quality images in accordance with the method of step S40, and to perform noise removal and conditional denoising on the preliminary reconstructed image based on the target text prompt and / or aligned target embedding to generate the final reconstructed image.

[0077] The input of the final reconstructed image module 40 is the preliminary reconstructed image, the target text prompt and / or the aligned target embedding, and the pre-trained conditional generative model, and the output is the final reconstructed image.

[0078] Figure 4 It is a schematic diagram of the connection relationship of the model inversion device for the multimodal pre-training model according to an embodiment of the present invention.

[0079] like Figure 4 As shown, this embodiment further provides a model inversion device 200 for a multimodal pre-training model, which uses a model inversion system 100 for a multimodal pre-training model of this embodiment.

[0080] A model inversion device 200 for a multimodal pre-training model in this embodiment includes a memory 50 , a processor 60 , an input interface 70 , and an output interface 80 .

[0081] The memory 50 is used to store the code, model, data set and intermediate data generated during the processing of the model inverse system 100 for the multimodal pre-training model (such as the fine-tuned encoder, alignment transformation relationship, preliminary reconstructed image, etc.).

[0082] The processor 60 is connected to the memory 50, and is used to execute the model inverse method for the multimodal pre-training model according to the model inverse system 100 for the multimodal pre-training model in the memory 50. The number of processors 60 is one or more, which can be a CPU, a GPU or a dedicated AI accelerator. The processor 60 is configured to control the model inverse system 100 for the multimodal pre-training model to execute steps S10 to S40 of the model inverse method for the multimodal pre-training model according to the instructions in the memory 50. In this embodiment, the memory 50 cooperates with the processor 60 to load and run the target multimodal model and the pre-trained generative model, and perform computing tasks such as adversarial fine-tuning, linear regression (or other methods for learning alignment transformations), gradient optimization, and reasoning (refining) of the generative model.

[0083] The input interface 70 is connected to the processor 60 and is used to receive a target text prompt.

[0084] The output interface 80 is connected to the processor 60 and is used to output the final reconstructed image.

[0085] This embodiment also provides a model inversion device for a multimodal pre-training model, which uses the model inversion apparatus 200 for a multimodal pre-training model in this embodiment.

[0086] <Test Example>

[0087] For the ViT-B-16 model, by comparing the baseline method with the AFT+DR+EA combination method of this embodiment that integrates Embedding Alignment (EA), Diffusion Refinement (DR) and Adversarial Fine-Tuning (AFT), it can be observed that the latter shows significant performance improvement on all three evaluation datasets.

[0088] Specifically, on the Laion 2B Subset dataset, the baseline method has an SSIM of 0.0421, a LPIPS of 0.9733, a CS of 0.4064, a SSCD of 0.0101, and a HS (%) of 0.0000; while the AFT+DR+EA method of this embodiment optimizes these indicators to 0.1509 (SSIM), 0.8189 (LPIPS), 0.4617 (CS), 0.0653 (SSCD), and 4.2000 (HS%), respectively.

[0089] On the Furniture Object Dataset, the AFT+DR+EA method of this embodiment (SSIM: 0.1985, LPIPS: 0.8200, CS: 0.4859, SSCD: 0.0553, HS (%): 1.4000) also far exceeds the baseline method (SSIM: 0.0484, LPIPS: 0.9910, CS: 0.3686, SSCD: 0.0193, HS (%): 0.0000).

[0090] On the Flickr30k Subset dataset, the various indicators of AFT+DR+EA of this embodiment (SSIM: 0.0647, LPIPS: 0.8500, CS: 0.5022, SSCD: 0.0602, HS (%): 3.8000) are also better than the Baseline (SSIM: 0.0292, LPIPS: 0.9765, CS: 0.4447, SSCD: 0.0112, HS (%): 0.0000), which fully demonstrates the effectiveness of the AFT+DR+EA combination strategy of this embodiment in improving various performance indicators of the model.

[0091] When the complete four-stage method proposed in this embodiment is tested on multiple standard datasets and different types of multimodal models (such as image encoder models based on ViT or ResNet architectures), compared with using only partial steps (such as direct optimization or refinement) or other existing baseline model inverse methods, the reconstructed images of this embodiment have achieved significant improvements in commonly used objective evaluation indicators (such as SSIM, LPIPS, CLIP-Score, etc.), and also show higher visual quality and detail fidelity through subjective evaluation by the human eye, which strongly demonstrates the advanced nature and effectiveness of the method of the present invention.

[0092] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A model inversion method for a multimodal pre-trained model, wherein the multimodal pre-trained model comprises at least one original image encoder and one original text encoder, characterized in that: The following steps are involved: S10, performing adversarial perturbation training on the original image encoder using an independent auxiliary image dataset to obtain a fine-tuned image encoder; S20, after calculating the original text embedding of the target text prompt using the original text encoder, map the original text embedding to the target embedding space aligned with the corresponding image embedding using a pre-trained embedding space alignment transformation to obtain an aligned target embedding; S30, after creating an image variable, iteratively updating the pixel values ​​of the image variable based on a gradient optimization algorithm so as to maximize the similarity between the current image embedding obtained by inputting the image variable into the fine-tuning image encoder and the aligned target embedding, thereby generating a preliminary reconstructed image; S40, using a pre-trained conditional generative model that can accept conditional input and generate high-quality images, conditioned on the target text prompt and / or the aligned target embedding, performing noise removal and conditional denoising on the preliminary reconstructed image to generate a final reconstructed image.

2. The model inversion method for a multimodal pre-training model according to claim 1, characterized in that: in, Step S10 includes the following steps: S11, extracting a batch of images from an independent auxiliary image dataset, calculating their embedded representations under the original image encoder or the current image encoder, and recording the embedded representation as a first embedded representation; S12, generating an adversarial perturbation for the image extracted in step S11, and adding the adversarial perturbation to the corresponding image to form an adversarial image sample; S13, inputting the adversarial image sample into the original image encoder or the current image encoder to obtain an embedded representation, and recording the embedded representation as a second embedded representation; S14, after defining a loss function that can minimize the difference between the first embedded representation and the second embedded representation, calculate the gradient according to the loss function and use it to update the parameters of the original image encoder or the current image encoder to fine-tune it; S15 , repeating steps S11 to S14 until the robustness index of the current image encoder reaches a desired level or converges, thereby obtaining a fine-tuned image encoder.

3. The model inversion method for a multimodal pre-training model according to claim 1, characterized in that: in, In step S20, the pre-trained embedding space alignment transformation is learned by: A10, prepare an independent auxiliary dataset of image-text pairs with paired data; A20, extracting embedding representations of all images in the auxiliary dataset using the fine-tuned image encoder to form an image embedding set; A30, extracting embedding representations of all corresponding texts in the auxiliary dataset using the original text encoder to form a text embedding set corresponding to the image embedding set; A40, learning a transformation operation that acts on the text embedding set and can make it as close as possible to the corresponding image embedding set, the transformation operation being the pre-trained embedding space alignment transformation.

4. The model inversion method for a multimodal pre-training model according to claim 3, characterized in that: in, The transformation operation in step A40 includes a linear transformation, wherein the linear transformation is a transformation matrix. The transformation matrix minimizes the average distance between the transformed text embedding set and its corresponding image embedding set.

5. The model inversion method for a multimodal pre-training model according to claim 1, Its characteristics are: in, Step S30 includes the following sub-steps: S31, starting from a random noise image, a blurred image or an average image, creating an image variable to be optimized; S32, inputting the current image variable into the fine-tuning image encoder to obtain a current image embedding; S33, calculating the optimized target loss to measure the dissimilarity between the current image embedding and the aligned target embedding; S34, calculates the regularization loss that can inject prior knowledge of the image and encourage the generated image to have better visual properties; S35, after obtaining a total loss function by weighted combination of the optimization target loss and the regularization loss, calculating the gradient of the total loss function with respect to the pixel value of the current image variable; S36, using an optimizer to update the current pixel value of the image variable according to the gradient in step S35; S37 , repeating steps S32 to S36 until a preset number of iterations is reached or the total loss function converges, to generate a preliminary reconstructed image.

6. The model inversion method for a multimodal pre-training model according to claim 1, characterized in that: in, In step S40, the pre-trained conditional generative model includes a text-to-image diffusion model, GAN or VAE, The diffusion model includes Stable Diffusion or Imagen.

7. The model inversion method for a multimodal pre-training model according to claim 5, Its characteristics are: Wherein, the pre-trained conditional generative model is a text-to-image diffusion model, and step S40 includes the following sub-steps: S41, adding forward diffusion noise to the preliminary reconstructed image, wherein the degree of adding noise corresponds to the time step of the diffusion process; S42, running a reverse denoising process of the pre-trained conditional generative model, using the target text prompt and / or the aligned target embedding to guide in each denoising / generation step to ensure that the pre-trained conditional generative model always moves in a direction consistent with the text semantics while removing noise and generating details; S43, repeating steps S41 to S42 until the optimization target loss no longer decreases, and finally obtaining a high-quality image.

8. A model inverse system for a multimodal pre-trained model, wherein the multimodal pre-trained model comprises at least one original image encoder and one original text encoder, characterized in that: A model inversion method for a multimodal pre-training model according to any one of claims 1 to 7, comprising: An image encoder fine-tuning module, configured to perform adversarial perturbation training on the original image encoder using an independent auxiliary image dataset; an aligned target embedding generation module for computing a raw text embedding of a target text prompt using the raw text encoder, and mapping the raw text embedding to a target embedding space aligned with its corresponding image embedding using a pre-trained embedding space alignment transformation to obtain an aligned target embedding; a preliminary image reconstruction module, configured to create an image variable and iteratively update the pixel values ​​of the image variable based on a gradient optimization algorithm so as to maximize the similarity between the current image embedding obtained by inputting the image variable into the fine-tuning image encoder and the aligned target embedding, thereby generating a preliminary reconstructed image; The final reconstructed image module uses a pre-trained conditional generative model that can accept conditional input and generate high-quality images. It noises and conditionally denoises the preliminary reconstructed image based on the target text prompt and / or the aligned target embedding to generate a final reconstructed image.

9. A model inversion device for a multimodal pre-training model, characterized in that: The model inversion system for the multimodal pre-training model according to claim 8 includes: A memory for storing the model inversion system for the multimodal pre-training model and intermediate data generated during the processing; A processor, configured to execute the model inversion method for the multimodal pre-training model according to the model inversion system for the multimodal pre-training model in the memory; an input interface, connected to the processor, for receiving the target text prompt; and An output interface is connected to the processor and is used to output the final reconstructed image.

10. A model inversion device for a multimodal pre-trained model, characterized in that: The model inversion device for the multimodal pre-training model described in claim 9 is used.

Images