Model inversion attack method based on multi-stage generation and antagonistic semantic optimization

Abstract

The invention discloses a model inversion attack method based on multi-stage generation and antagonistic semantic optimization, belongs to the technical field of image processing, and solves the technical problems of single generation model, difficulty in detail reconstruction evaluation and semantic drift in multi-stage generation in the existing model inversion attack technology. According to the method, low-resolution semantic reconstruction, antagonistic semantic optimization and high-resolution detail completion are progressive, the limitation of a single generation model is broken through, two different generation models are adopted, and a generator with higher semantic reconstruction capability is used for performing reconstruction in the aspect of emphasizing semantic consistency under low resolution; a semantic-guided potential space mapping and antagonism refining module is added between the semantic reconstruction stage and the detail completion stage, a mapping process between potential spaces of different generation models is dynamically optimized, and high-fidelity transmission of core semantic information in transition from the semantic reconstruction stage to the detail completion stage is ensured; and high-quality private image reconstruction with both semantic correctness and detail fidelity is completed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and more particularly relates to a model inversion attack method based on multi-stage generation and adversarial semantic optimization. BACKGROUND

[0002] At present, artificial intelligence technology develops rapidly, which brings about extensive application and also brings about severe security and privacy challenges. Among them, model inversion attack (MIA) is a particularly serious privacy threat. Attackers can use the access right to the published model to reversely deduce and reconstruct the original private data used to train the model. In the field of biometric recognition such as face recognition, once the attacker successfully reconstructs the clear face image of the user, it can be used for malicious activities such as illegal authentication and identity fraud, which poses a great threat to personal property safety and even the social trust system. Therefore, it is crucial to deeply study the mechanism of model inversion attack and evaluate its threat for building a safer and more reliable artificial intelligence system.

[0003] The existing model inversion attack technology gradually changes from directly optimizing in the input space (such as the pixel space of an image) to introducing a generator to optimize the generated image. The more mainstream method usually uses a pre-trained generative adversarial network (GAN) as a generator to continuously optimize in its latent space to obtain the optimal latent vector.

[0004] However, the existing method is generally limited by the inherent defects of a single generation model. Since the optimization strategy of a single generator is fixed, it focuses on either semantic consistency reconstruction and ignores image details or focuses on image detail reconstruction and lacks semantic information, making it difficult to balance semantic consistency at low resolution and detail consistency at high resolution. Moreover, in terms of high-resolution detail reconstruction, the existing method lacks effective measurement of detail levels and only stays at the level of prior loss and identification loss, making it difficult to serve as a powerful aid for high-resolution detail reconstruction. The existing model inversion attack method based on a single generator is difficult to achieve a balance between image details and semantic consistency, which has become an important factor limiting the attack performance. In addition, due to the huge differences in structure, distribution and characteristics of the latent spaces of different generation models, the “semantic drift” across models easily causes loss or distortion of key semantic information (especially identity features). This loss of semantic fidelity in the cross-model information transmission process has become a core technical bottleneck limiting the performance of multi-stage generation. SUMMARY

[0005] The present application is directed to the technical problems existing in the prior art, and provides a model inversion attack method based on multi-stage generation and adversarial semantic optimization.

[0006] To solve the above technical problems, the present application provides the following technical solutions: The model inversion attack method based on multi-stage generation and adversarial semantic optimization comprises the following steps: S1, model pre-training stage Load the generator model used in the low-resolution semantic reconstruction stage G GAN w Optimize based on the W space to realize the reconstruction in the semantic consistency level; After loading is completed G GAN w Initialize the generator model with stronger detail generation capability used in the high-resolution detail completion stage, and construct a conditional diffusion model G Diff l S2, low-resolution semantic reconstruction stage Take the pre-trained model G GAN w as the core, utilize the excellent decoupling characteristics of the W space of the StyleGAN model, optimize in the W space to lock the target category semantic structure, and use the generator with stronger semantic reconstruction capability as the core demand of semantic consistency; S3, adversarial semantic optimization stage Adopt the semantic-guided latent space mapping and adversarial refining module, which comprises a mapping network and a semantic refiner, wherein the mapping network is responsible for semantic transmission of cross-model latent vectors, and the semantic refiner optimizes the mapping network parameters by judging the consistency between the intermediate layer features of the generated image and the initial semantic vector; The mapping network and the semantic refiner are used to complete the semantic alignment of the latent space in an adversarial manner, and through the transmission of semantics, the judgment of distortion and the reverse optimization, after multiple iterations, the mapping network generates a latent vector with high semantic fidelity m under the guidance of the semantic refiner. S4, high-resolution detail completion stage Select a generator with better detail generation performance to carry out reconstruction work, and introduce feature extraction as part of the loss function to evaluate the detail reconstruction effect; including: Input and basic model selection, take the semantic latent vector m as the input, and select the conditional diffusion model as the core reconstruction model; Multiple rounds of optimization and feature evaluation, through multiple iterations to optimize the latent vector​​​​​m , feature extraction is introduced in the process to evaluate the effect of detail reconstruction; Dynamic weighted composite loss function controls quality, and loss function balances semantics, cross-model, and details.

[0007] Preferably, in step S1, semantic reconstruction is performed based on StyleGAN, a low-resolution base image is generated using the latent space of StyleGAN which has the feature disentanglement property, and optimization is performed based on W space to realize reconstruction at the semantic consistency level. w The vector is obtained by mapping the Z space random vector through the mapping network G mapping (·) get, to z Vector decoupling w Vector: w = G mapping ( z ; θ map ) In the formula, θ map is the mapping network parameter.

[0008] Preferably, after loading G GAN ( w ), initialization of the generator model with stronger detail generation capability used in the high-resolution detail completion stage is performed, and image detail reconstruction is performed based on the conditional diffusion model, which is used for image detail reconstruction in S4. Construct a conditional diffusion model G Diff ( l ), the core is to make the model learn the data distribution related to the target label through conditional guidance, so as to generate samples conforming to the characteristics of the target label. The public data set X pub The sample x 0 is input into the target model f θ After obtaining the predicted label f θ ( x 0), after label embedding to match the feature dimension of the diffusion model, add timestamp encoding and perform feature fusion, and then participate in the noise prediction task of the diffusion model as a conditional input.

[0009] Preferably, in step S2, the pre-trained model G GAN ( wUsing the StyleGAN model's excellent feature decoupling properties in the W space as its core, the model optimizes in the W space to lock in the semantic structure of the target category. The Poincaré loss is used as the optimization loss function, which is:

[0010] In the formula, arccosh(·) represents the inverse hyperbolic cosine function, used to calculate the geodesic distance between two points in hyperbolic space. v 1 represents the normalized prediction confidence level. v 2 represents the one-hot vector used for the truth value. To avoid division by zero, the number 1 in the formula is replaced with 0.9999.

[0011] Preferably, the optimization process uses the Adam optimizer with a learning rate of [missing information]. η = 10 -4 Iteratively update the vector w, using the following formula:

[0012] After the iteration, a low-resolution image is obtained. X low = G GAN ( w * ).

[0013] Preferably, in step S3, the semantic refiner is represented as R It is a fixed semantic and perceptual feature extractor that calculates the semantic features of the image generated in the low-resolution stage as the target semantic features, and calculates the semantic features of the image generated by the current latent vector as the current semantic features. The mean squared error of both is used to measure the semantic loss in this mapping process. The loss function is expressed as:

[0014] In the formula, G GAN ( w * (Image generated by the low-resolution stage generator) M α ( w * To utilize mapping networks M α The resulting latent vector, G Diff ( M α ( w * The latent vector is the image generated by the high-resolution stage generator; Two images using the target model f θ The intermediate features are obtained and processed by the semantic refiner. R Extract its semantic feature vector.

[0015] Preferably, the parameters of the mapping network are updated based on the calculated semantic refinement loss. α The optimization process uses the Adam optimizer, expressed by the following formula:

[0016] After multiple iterations, guided by the semantic refiner, the mapping network's mapping ability is enhanced, ultimately generating latent vectors with extremely high semantic fidelity. m = M α ( w * ).

[0017] Preferably, in step S4, during the multi-round optimization process, a dynamically weighted composite loss function is used to collaboratively optimize the generation process. The composite loss function includes: The Poincaré loss function used to maintain semantic correctness is one; Feature consistency loss function two is used to address cross-model feature distribution offset. The third high-level feature loss function is used to constrain the alignment of generated image details with private data; The loss of detail consistency is measured by extracting intermediate layer features to maintain the semantic consistency between the low-resolution and high-resolution images, while the reconstruction effect at the detail level is measured by extracting high-level features.

[0018] Preferably, feature consistency loss is used to address cross-model feature distribution shift, by calculating... G GAN ( w )and G Diff ( l The L2 distance between the generated image and the intermediate layer features of the target model is used as the feature consistency loss, and the loss function is:

[0019] In the formula, N To extract the number of intermediate layers; This is the intermediate layer feature extraction function, which takes the generated image as input and outputs the intermediate features of the corresponding layer of the target model; Feat n |for the first n The total number of elements in the layer feature map is determined by the feature map size. H ×W × C The calculation divided by the total number of elements can avoid the numerical deviation of the loss value caused by the size difference of the feature map; The high-level feature loss is used to constrain the diffusion model to generate images whose details are aligned with the private data, and the L2 distance between the high-level features of the diffusion image and the approximate high-level features of the public data set is calculated as the high-level feature loss The loss function is:

[0020] In the formula, | Feat high The total number of elements of the high-level feature map is The mean value of the features of the target model at each layer is calculated as the approximate high-level feature by using the public auxiliary data set X pub FeatFetcher high The calculation formula is: .

[0021] Preferably, in step S4, the weights of the three types of loss are integrated to balance the semantic constraint, cross-model alignment and detail fidelity, and the total loss function of the present stage is obtained, and the calculation formula is:

[0022] In the formula, α, β and γ α , β , λ are hyperparameters respectively adjusting the proportion of the three losses, and the latent vector is iteratively updated, and the optimization process adopts the Adam optimizer, and the updating method is as follows:

[0023] In the formula, α, β and γ η is the learning rate, and the value is 5*10 -5 , which is suitable for the high dimensionality of the latent vector of the diffusion model; After the optimization is completed, the high-resolution image can be generated X mid = G Diff ( l * ).

[0024] Compared with the prior art, the present application has the following beneficial effects: ​The embodiment of the present application provides a model inversion attack method based on multi-stage generation and adversarial semantic optimization, solves the problems that the existing model inversion attack technology is limited to a single generation model, details of a reconstructed image and semantic consistency are difficult to balance, and detail reconstruction is difficult to evaluate, and the like. The embodiment of the present application breaks through the limitation of a single generation model by taking low-resolution semantic reconstruction, adversarial semantic optimization, and high-resolution detail completion as a progression, adopts two different generation models, uses a generator with stronger semantic reconstruction capability at a low resolution to perform reconstruction focusing on semantic consistency, adds a semantic-guided latent space mapping and adversarial refinement module between the two, dynamically optimizes the mapping process between the latent spaces of the two different generation models, ensures that core semantic information can be accurately transmitted from the semantic reconstruction stage to the detail completion stage, and moreover, uses a generator with stronger detail generation capability at a high resolution to perform reconstruction focusing on image details, and introduces feature extraction to evaluate the effect of the reconstructed image in terms of details, and the reconstruction is performed in stages for different reconstruction tasks with different focuses, so that the balance between image details and semantic consistency is achieved. Thus, the limitation of a traditional single generation model and the limitation of detail reconstruction evaluation are broken through, semantic consistency reconstruction and image detail reconstruction are realized in stages, the balance between the two is achieved, and high-quality private image reconstruction with semantic correctness and detail fidelity is completed. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative labor.

[0026] Figure 1 The flowchart of the embodiment of the model inversion attack method based on multi-stage generation and adversarial semantic optimization of the present application. DETAILED DESCRIPTION

[0027] In order to make the technical problems, technical solutions and beneficial effects of the present application more clearly understood, the following will further describe the present application in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0028] EMBODIMENT Please refer to Figure 1 The embodiment of the present application provides a model inversion attack method based on multi-stage generation and adversarial semantic optimization, which comprises the following steps: S1, model pre-training stage Load the generator model used in the low-resolution semantic reconstruction stage GGAN w , based on W space optimization, to achieve semantic consistency level reconstruction; Load complete G GAN w After, the initialization of the generator model with stronger detail generation ability used in the high-resolution detail completion stage, the construction of the conditional diffusion model G Diff l ; S2, low-resolution semantic reconstruction stage With the pre-trained model G GAN w As the core, using the excellent feature decoupling characteristics of the StyleGAN model W space, optimizing in W space to lock the target category semantic structure, using the generator with stronger semantic reconstruction ability with semantic consistency as the core appeal; S3, adversarial semantic optimization stage Adopt semantic guided latent space mapping and adversarial refining module, including mapping network and semantic refiner, wherein the mapping network is responsible for semantic transmission of cross-model latent vectors, and the semantic refiner optimizes the mapping network parameters by judging the consistency of the generated image intermediate layer features and the initial semantic vector; Use mapping network and semantic refiner to complete latent space semantic alignment adversarially, through semantic transmission, distortion discrimination and reverse optimization, after multiple iterations, the mapping network generates high semantic fidelity latent vector m under the guidance of the semantic refiner; S4, high-resolution detail completion stage Select a generator with better detail generation performance to carry out reconstruction work, and introduce feature extraction as part of the loss function to evaluate the detail reconstruction effect; including: Input and base model selection, with semantic latent vector m as input, select the conditional diffusion model as the core reconstruction model; Multiple rounds of optimization and feature evaluation, through multiple iterations to optimize the latent vector m , feature extraction is introduced in the process to evaluate the detail reconstruction effect; Dynamic weighted composite loss function controls quality, using loss function to balance semantics, cross-model, and details.

[0029] ​​​​The embodiment of the present application provides a model inversion attack method based on multi-stage generation and adversarial semantic optimization, which solves the problems that the existing model inversion attack technology is limited to a single generation model, resulting in difficulty in balancing the consistency of image details and semantics, and difficulty in evaluating detail reconstruction. The embodiment of the present application takes low-resolution semantic reconstruction, adversarial semantic optimization and high-resolution detail completion as a progression, breaks through the limitations of a single generation model, uses two different generation models, and uses a generator with stronger semantic reconstruction capability at low resolution to focus on semantic consistency reconstruction. The semantic guided latent space mapping and adversarial refinement module is added between the two, which dynamically optimizes the mapping process between the latent spaces of the two different generation models, ensures that the core semantic information can be accurately transmitted from the semantic reconstruction stage to the detail completion stage, and moreover, a generator with stronger detail generation capability is used at high resolution to focus on image detail reconstruction, and feature extraction is introduced to evaluate the effect of the reconstructed image in the detail level, and the reconstruction is carried out in stages for different reconstruction tasks, so that the balance between image details and semantic consistency is realized. Thus, the limitations of traditional single generation model and the limitations of detail reconstruction evaluation are broken through, the semantic consistency reconstruction and image detail reconstruction are realized in stages, the balance between the two is achieved, and the high-quality private image reconstruction with semantic correctness and detail fidelity is completed.

[0030] Specifically, the model inversion attack method based on multi-stage generation and adversarial semantic optimization provided by the embodiment of the present application comprises the following steps: S1, model pre-training stage loading a generator model with stronger semantic reconstruction capability used in the low-resolution semantic reconstruction stage G GAN ( w ), which is based on StyleGAN (Style-Based Generative Adversarial Network, Style-Based Generative Adversarial Network) for semantic reconstruction. The core function of StyleGAN is to generate a low-resolution basic image using the latent space (W space) with feature disentangling characteristics. Based on the W space, the image style can be quickly controlled, and the reconstruction in the semantic consistency level is realized.

[0031] w The vector is obtained by a mapping network z N(0,1) from the Z space random vector (subject to standard normal distribution G mapping (·), and z the vector is decoupled to obtain w vector: w = Gmapping z θ map wherein, θ map is a mapping network parameter.

[0032] Further, after the loading is completed G GAN w , initialization of a generator model with stronger detail generation capability used in a high-resolution detail completion stage is performed, and the stage is based on a conditional diffusion model for image detail reconstruction, the conditional diffusion model having stronger detail generation capability and image fidelity, and being used for image detail level reconstruction in the high-resolution detail completion stage.

[0033] The conditional diffusion model is constructed G Diff l , and the core is to make the model learn the data distribution related to the target label through "conditional guidance", so as to generate samples conforming to the characteristics of the target label. The prediction label X pub obtained after the sample x 0 in the public data set f θ is input into the target model f θ x 0) is added with a timestamp code after being embedded with a label to match the feature dimension of the diffusion model, and then participates in the noise prediction task of the diffusion model after feature fusion.

[0034] S2, low-resolution semantic reconstruction stage is the core to quickly lock the semantic structure of the target category, and lays the foundation for subsequent detail generation. In this stage, the pre-trained model G GAN w is used as the core, and the excellent feature decoupling characteristics of the W space of the StyleGAN model are used for optimization in the W space to lock the semantic structure of the target category. In order to avoid the problem of gradient disappearance (optimization process stagnation, unable to continue to approach the target) easily occurring under high iteration times of traditional cross-entropy loss, Poincaré loss is used as the loss function for optimization.

[0035] The Poincaré loss maps the output confidence of the target model and the target category one-hot vector to hyperbolic space, and optimizes the W vector by calculating the geodesic distance (straight line distance in hyperbolic space) of the two. The specific loss function is:

[0036] ​​​​​​​where arccosh(·) denotes the inverse hyperbolic cosine function used to calculate the geodesic distance between two points in hyperbolic space, v 1 denotes the normalized prediction confidence, v 2 denotes the one-hot vector for the true value, the number 1 in the formula is replaced by 0.9999 to avoid the case of division by zero, and the optimization stability is ensured.

[0037] The optimization process uses the Adam optimizer, the learning rate η = 10 -4 , iteratively updates the w vector, and gradually makes the model output approach the target category. The update formula is:

[0038] After the iteration is completed, the low-resolution image X low = G GAN w * .

[0039] The purpose of the low-resolution semantic reconstruction stage is to quickly determine the core semantic structure of the target category, laying the foundation for subsequent detail generation. Although the image obtained at this time lacks details, it already has the core semantic structure of the target category. This stage solves key pain points such as category deviation and gradient disappearance in traditional semantic reconstruction. Through modular stage design, it provides precise semantic anchor points for subsequent high-resolution generation links, and provides important support for the efficiency and quality improvement of the whole process of image generation.

[0040] S3, adversarial semantic optimization stage At present, there is a semantic drift problem in the semantic transfer between different generation models. The semantic drift problem refers to the phenomenon that the encoded core semantic information obtained in the low-resolution semantic reconstruction stage (especially the identity features of the target) is lost, distorted or weakened when converted to the latent vector in the adversarial semantic optimization stage due to the fundamental differences in model architecture, latent space characteristics and training mechanism between different generation models. Therefore, the image generated by the generator model with stronger detail generation capability finally has rich details and high resolution, but the identity deviates from the target locked in the low-resolution semantic reconstruction stage, and the final reconstruction effect is not good.

[0041] In order to overcome the semantic drift problem, the embodiment of the present application designs a latent space mapping and adversarial refining module based on semantic guidance which can ensure that the semantic information maintains high fidelity in the cross-model transmission process. On the basis of the traditional mapping network, a "semantic refiner" is introduced as a supervisor to reduce the semantic loss caused by mapping through adversarial training.

[0042] ​Specifically, the double module structure includes a mapping network and a semantic refiner. The mapping network is responsible for the semantic transmission between the latent vectors of the two generation models, converting the semantic vector of the low-resolution model into a vector that can be recognized by the high-resolution model, which is the "transmitter". The semantic refiner acts as an adversarial discriminator, optimizing the parameters of the mapping network to obtain a better mapping strategy by judging whether the intermediate layer features of the generated image are consistent with the initial semantic vector, supervising the transmission effect of the mapping network, judging whether the semantics are distorted and optimizing in the opposite direction, which is the "supervisor" and "corrector". Through the dynamic and adversarial refining process, it is ensured that the semantic information transmission from the low-resolution stage to the high-resolution stage is not only a one-way "injection", but also a "alignment" process that has been checked and optimized, thereby significantly improving the identity consistency and semantic correctness of the final generated image.

[0043] Furthermore, this stage mainly includes: Transmit semantics: the mapping network first converts the semantic vector of the low-resolution model into the latent vector of the high-resolution model.

[0044] Judge distortion: in order to make an accurate judgment, since the image details may interfere with the judgment of the core semantics, the semantic refiner needs to judge not only according to the generated image, but also according to the intermediate layer features of the target model.

[0045] The semantic refiner is represented as R which is a fixed semantic and perceptual feature extractor, calculates the semantic features of the image generated in the low-resolution stage as the target semantic features, and calculates the semantic features of the image generated by the current latent vector as the current semantic features. The mean square error of the two is used to measure the semantic loss in this mapping process, and the loss function formula is represented as:

[0046] In the formula, G GAN is the image generated by the low-resolution stage generator, w * M α is the latent vector obtained by using the mapping network w * M α G Diff is the image generated by the high-resolution stage generator from the latent vector, M α w * Two images use the target model f ​​​​​θ get the intermediate features, pass through the semantic refiner R extract its semantic feature vector.

[0047] In this embodiment, the semantic refiner extracts the "core semantics of low-resolution images" (as the "standard target"), and extracts the "semantics of images generated by the mapping vector of the high-resolution model" (as the "current result"); then by calculating the "mean square error" of the two, quantifying the semantic loss (the greater the error, the more serious the semantic drift).

[0048] Reverse optimization: use the calculated "semantic loss" to update the parameters of the mapping network through the Adam optimizer, so that the mapping network is more accurate next time, and gradually reduces the semantic distortion.

[0049] Specifically, according to the calculated semantic refinement loss, update the parameters of the mapping network α , the optimization process uses the Adam optimizer, and the formula is:

[0050] After multiple iterations, the mapping network becomes more and more powerful under the guidance of the semantic refiner, and finally generates a latent vector with extremely high semantic fidelity m = M α ( w * ), thereby effectively overcoming the semantic drift problem of multi-stage generation.

[0051] S4, high-resolution detail completion stage In this embodiment, in order to supplement the high-resolution fine texture on the basis of semantic reconstruction, this stage gives the reconstruction based on the conditional diffusion model with stronger detail generation capability, and introduces feature extraction to evaluate the detail reconstruction effect. The whole process revolves around "using low-resolution semantics as the basis, using diffusion model to supplement details, and using loss function to control quality", which specifically includes: S401, input and basic model selection The latent vector obtained by the mapping network and the semantic adversarial refinement in the adversarial semantic optimization stage m as input, select a conditional diffusion model with stronger detail generation capability as the core reconstruction model, because the diffusion model performs better in generating high-resolution, high-detail images; after multiple rounds of optimization, high-resolution is obtained.

[0052] S402, multiple rounds of optimization and feature evaluation To achieve high-precision reconstruction at the detail level, the embodiment proposes to measure the detail consistency loss by extracting intermediate layer features to maintain the semantic consistency of the images generated in the low-resolution stage and the high-resolution stage, and to measure the reconstruction effect at the detail level by extracting high-level features to directly evaluate whether the details of the high-resolution image are real.

[0053] S3, dynamically weighted composite loss function controls quality In the multi-round optimization process, a dynamically weighted composite loss function is used to optimize the generation process cooperatively. The composite loss function includes: Poincare loss function one for maintaining semantic correctness; feature consistency loss function two for solving cross-model feature distribution offset; high-level feature loss function three for constraining the alignment of generated image details and private data.

[0054] Through the cooperation of the three, the final generated high-resolution image has both semantic correctness and detail fidelity, thereby realizing high-fidelity private image reconstruction.

[0055] In the embodiment, since two generators G GAN ( w ) and G Diff ( l ) are used, the feature consistency loss is needed to solve the cross-model feature distribution offset, and the L2 distance of the generated images in the intermediate layer features of the target model is calculated G GAN ( w ) and G Diff ( l ) as the feature consistency loss, and the loss function is:

[0056] In the formula, N is the number of extracted intermediate layers; is the intermediate layer feature extraction function, the input is the generated image, and the output is the intermediate feature of the corresponding layer of the target model; Feat n is the total number of elements of the n layer feature map, which is calculated by the feature map size H × W × C , and divided by the total number of elements to avoid loss value deviation caused by feature map size difference.

[0057] The details of the generated image are aligned with private data by using a high-level feature loss constrained diffusion model. This is achieved by calculating the high-level features of the diffused image and approximate high-level features statistically derived from public datasets. The L2 distance is used as the high-level feature loss, and the loss function is:

[0058] In the formula, | Feat high | represents the total number of elements in the high-level feature map. To utilize public auxiliary datasets X pub The mean of features at each layer of the statistical target model is used as the high-level approximate feature, and then extracted using a feature extraction function. FeatFetcher high (·) Calculate, the formula is: .

[0059] By combining the weights of the three types of losses and balancing semantic constraints, cross-model alignment, and detail fidelity, the total loss function for this stage is obtained, and the calculation formula is as follows:

[0060] In the formula, α , β , λ The hyperparameters are defined as follows, and the weights of the three losses are adjusted accordingly. The latent vector is iteratively updated using the Adam optimizer, and the update method is as follows:

[0061] In the formula, η The learning rate is 5 × 10. -5 It adapts to the high-dimensional characteristics of the latent vectors in the diffusion model; High-resolution images can be generated after optimization. X mid = G Diff ( l * ).

[0062] The resulting high-resolution image combines low-resolution semantics with high-resolution details, achieving high-fidelity private image reconstruction.

[0063] Based on the above, the embodiment of the present application solves the problems that the single generation model in the prior art cannot simultaneously consider the macro semantic and micro details of the image and the semantic drift in the multi-stage generation by progressive generation and optimization. First, a low-resolution stage generator is used to quickly build a semantic outline, and a semantic alignment of the latent vectors of the two generation models is realized through an adversarial semantic optimization module. Then, a conditional diffusion model is used to finely carve details, and a dynamic composite loss function is used for constraint, so that the whole attack process is more targeted and efficient, and the quality of the reconstructed image of the model inversion attack is significantly improved.

[0064] The present application provides a model inversion attack method based on multi-stage generation and adversarial semantic optimization, which solves the problems that the existing model inversion attack technology is limited by a single generation model, resulting in difficulty in considering the consistency of details and semantics of the reconstructed image, difficulty in evaluating the details reconstruction, and semantic drift in multi-stage generation. The present application divides the reconstruction process of high-fidelity images into three progressive stages of low-resolution semantic reconstruction, adversarial semantic optimization, and high-resolution detail completion, breaks through the limitations of a single generation model, uses two different generation models, uses a generator with stronger semantic reconstruction capability in low resolution for reconstruction focusing on semantic consistency; adds a semantic-guided latent space mapping and adversarial refinement module between the two, dynamically optimizes the mapping process between the latent spaces of the two different generation models, ensures that the core semantic information (especially the identity information) can be accurately transmitted from the semantic reconstruction stage to the detail completion stage; and in the high-resolution scene, the key goal is to present details, a generator with stronger detail generation capability is used for reconstruction focusing on image details, and feature extraction is introduced to evaluate the effect of the reconstructed image in terms of details, and the reconstruction is performed in stages for different reconstruction tasks, realizing the consideration of image details and semantic consistency. Thus, the limitations of traditional single generation model and the limitations of detail reconstruction evaluation are broken through, and the problem of semantic drift in multi-stage generation is also solved, the semantic consistency reconstruction and image detail reconstruction are realized in stages, the balance between the two is achieved, and high-quality private image reconstruction with semantic correctness and detail fidelity is completed.

[0065] Moreover, the present application can break through the limitation of traditional single generation model and fixed optimization space, make full use of the advantages of different generation models, effectively evaluate the detail reconstruction effect through feature extraction, solve the problem that the existing loss function is difficult to measure the detail reconstruction effect, solve the problem of "semantic drift" in multi-stage generation, realize higher fidelity private image reconstruction, and the attack accuracy, KNN distance and F1 score of the present application are effectively and obviously improved compared with the baseline method. The present application overcomes the defects of existing model inversion attack methods, such as single generation model, difficulty in evaluating detail reconstruction effect, difficulty in balancing semantics and details, etc., and has obvious effect in improving private image reconstruction quality and identity semantic consistency, and provides a stronger technical tool for evaluating the privacy security risk of artificial intelligence model.

[0066] In the description of the present application, it should be understood that the orientation or positional relationship indicated by terms such as "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0067] In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0068] The above only describes the preferred embodiments of the present application and is not intended to limit the present application, and any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A model inversion attack method based on multi-stage generation and adversarial semantic optimization, characterized in that, Includes the following steps: S1, Model Pre-training Stage The generator model used in the low-resolution semantic reconstruction stage G GAN ( w Based on the W space, optimization is performed to achieve reconstruction at the semantic consistency level; Loading complete G GAN ( w After that, the generator model with stronger detail generation capabilities used in the high-resolution detail completion stage is initialized, and the conditional diffusion model is constructed. G Diff ( l ); S2, Low-resolution semantic reconstruction stage With pre-trained models G GAN ( w With the StyleGAN model as the core, the model utilizes the excellent feature decoupling properties of the W space to optimize the semantic structure of the target category in the W space, with semantic consistency as the core requirement, and uses a generator with stronger semantic reconstruction capabilities. S3, Adversarial Semantic Optimization Stage A semantically guided latent space mapping and adversarial refining module is adopted, including a mapping network and a semantic refiner. The mapping network is responsible for the semantic transfer of latent vectors across models, and the semantic refiner optimizes the mapping network parameters by judging the consistency between the intermediate layer features of the generated image and the initial semantic vector. By adversarially aligning latent space semantics using a mapping network and a semantic refiner, and through semantic propagation, distortion discrimination, and reverse optimization, the mapping network, guided by the semantic refiner, generates latent vectors with high semantic fidelity after multiple iterations. m ; S4, High-resolution detail enhancement stage A generator with superior detail generation performance is selected for reconstruction, and feature extraction is incorporated as part of the loss function to evaluate the detail reconstruction effect; including: Input and base model selection, based on semantic latent vectors m As input, the conditional diffusion model was selected as the core reconstruction model; Multi-round optimization and feature evaluation: The latent vector is optimized through multiple rounds of iteration. m Feature extraction is introduced during the process to evaluate the effect of detail reconstruction; Quality is controlled by a dynamically weighted composite loss function, which balances semantics, cross-model alignment, and details.

2. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 1, characterized in that, In step S1, semantic reconstruction is performed based on StyleGAN. A low-resolution base image is generated using the latent space of StyleGAN, which has the feature deentanglement property. Optimization is performed based on the W space to achieve reconstruction at the semantic consistency level. w Vectors are random vectors in Z-space through a mapping network G mapping (·) obtain, for z Vector decoupling yields w vector: w = G mapping ( z ; θ map ) In the formula, θ map To map network parameters.

3. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 2, characterized in that, Loading complete G GAN ( w After that, the generator model with stronger detail generation capabilities is initialized for the high-resolution detail completion stage, and image detail reconstruction is performed based on the conditional diffusion model for image detail level reconstruction in S4. Constructing a conditional diffusion model G Diff ( l The core idea is to use conditional guidance to enable the model to learn the data distribution related to the target label, thereby generating samples that conform to the characteristics of the target label, and using public datasets. X pub medium sample x 0 Input Target Model f θ The predicted labels obtained later f θ ( x After label embedding to match the feature dimensions of the diffusion model, timestamp encoding is added and features are fused, and then used as conditional input to participate in the noise prediction task of the diffusion model.

4. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 3, characterized in that, In step S2, using the pre-trained model G GAN ( w Using the StyleGAN model's excellent feature decoupling properties in the W space as its core, the model optimizes in the W space to lock in the semantic structure of the target category. The Poincaré loss is used as the optimization loss function, which is: In the formula, arccosh(·) represents the inverse hyperbolic cosine function, used to calculate the geodesic distance between two points in hyperbolic space. v 1 represents the normalized prediction confidence level. v 2 represents the one-hot vector used for the truth value. To avoid division by zero, the number 1 in the formula is replaced with 0.9999.

5. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 4, characterized in that, The optimization process uses the Adam optimizer with a learning rate of [missing information]. η = 10 -4 Iteratively update the vector w, using the following formula: After the iteration, a low-resolution image is obtained. X low = G GAN ( w * ).

6. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 5, characterized in that, In step S3, the semantic refiner is represented as R It is a fixed semantic and perceptual feature extractor that calculates the semantic features of the image generated in the low-resolution stage as the target semantic features, and calculates the semantic features of the image generated by the current latent vector as the current semantic features. The mean squared error of both is used to measure the semantic loss in this mapping process. The loss function is expressed as: In the formula, G GAN ( w * (Image generated by the low-resolution stage generator) M α ( w * To utilize mapping networks M α The resulting latent vector, G Diff ( M α ( w * The latent vector is the image generated by the high-resolution stage generator; Two images using the target model f θ Obtain its intermediate features, After semantic refiner R Extract its semantic feature vector.

7. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 6, characterized in that, Update the parameters of the mapping network based on the calculated semantic refinement loss. α The optimization process uses the Adam optimizer, expressed by the following formula: After multiple iterations, guided by the semantic refiner, the mapping network's mapping ability is enhanced, ultimately generating latent vectors with extremely high semantic fidelity. m = M α ( w * ).

8. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 7, characterized in that, In step S4, during the multi-round optimization process, a dynamically weighted composite loss function is used to collaboratively optimize the generation process. The composite loss function includes: The Poincaré loss function used to maintain semantic correctness is one; Feature consistency loss function two is used to address cross-model feature distribution offset. The third high-level feature loss function is used to constrain the alignment of generated image details with private data; The loss of detail consistency is measured by extracting intermediate layer features to maintain the semantic consistency between the low-resolution and high-resolution images, while the reconstruction effect at the detail level is measured by extracting high-level features.

9. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 8, characterized in that, Feature consistency loss addresses cross-model feature distribution shifts by calculating... G GAN ( w )and G Diff ( l The L2 distance between the generated image and the features in the intermediate layer of the target model is used as the feature consistency loss, and the loss function is: In the formula, N To extract the number of intermediate layers; This is the intermediate layer feature extraction function, which takes the generated image as input and outputs the intermediate features of the corresponding layer of the target model; Feat n |for the first n The total number of elements in the layer feature map is determined by the feature map size. H × W × C Calculation by dividing by the total number of elements can avoid numerical deviations in loss caused by differences in feature map size; The details of the generated image are aligned with private data by using a high-level feature loss constrained diffusion model. This is achieved by calculating the high-level features of the diffused image and the approximate high-level features statistically derived from the public dataset. The L2 distance is used as the high-level feature loss, and the loss function is: In the formula, | Feat high | represents the total number of elements in the high-level feature map. To utilize public auxiliary datasets X pub The mean of features at each layer of the statistical target model is used as the high-level approximate feature, and then extracted using a feature extraction function. FeatFetcher high (·) Calculate, the formula is: 。 10. The model inversion attack method based on multi-stage generation and adversarial semantic optimization according to claim 9, characterized in that, In step S4, the weights of the three types of losses are combined, and semantic constraints, cross-model alignment, and detail fidelity are balanced to obtain the total loss function for this stage. The calculation formula is as follows: In the formula, α , β , λ The hyperparameters are defined as follows, and the weights of the three losses are adjusted accordingly. The latent vector is iteratively updated using the Adam optimizer, and the update method is as follows: In the formula, η The learning rate is 5 × 10. -5 It adapts to the high-dimensional characteristics of the latent vectors in the diffusion model; High-resolution images can be generated after optimization. X mid = G Diff ( l * ).

Images