High-mobility image reconstruction method based on integration method
Through the integration method, adaptive weight adjustment and dynamic model update strategy, the potential vector is optimized to generate highly transferable reconstructed images, which solves the problem of insufficient migrationability of reconstructed images on the black box model by white box model, reducing computing and economic overhead.
Patent Information
- Application Number
- CN202510427032.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-07
- Publication Date
- 2025-07-11
AI Technical Summary
The existing white box model inversion attack methods have insufficient migration ability to reconstruct images on black box models and require a large amount of access, resulting in excessive computational and economic overhead.
The image reconstruction method based on the integration method is adopted, and an adversarial network is generated by collecting auxiliary image data to train and generate a local model pool, and the adaptive weight adjustment and dynamic model update strategies are used to optimize potential vectors to generate highly transferable reconstructed images.
Improves the migration of reconstructed images, enables capturing more model features, reduces access requirements for unseen models, and reduces computational and economic overhead.
Smart Images
Figure CN120298525A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and deep learning security, and in particular to a highly transferable image reconstruction method based on an ensemble method. Background Art
[0002] In recent years, with the wide application of deep neural network models in fields such as image processing, data privacy and security issues have received increasing attention. As an important technical means, model inversion attacks can obtain a representative approximation reconstruction of the training data of the model by accessing the model, that is, to achieve image reconstruction. This image reconstruction technology of model inversion attacks can not only help to reveal the privacy and security issues of deep neural networks, but also be used as a means to evaluate the robustness of the model, which is helpful for designing more effective defense strategies and promoting the development and application of privacy protection technologies.
[0003] According to the attacker's knowledge of the background of the target model, model inversion attacks can be divided into white-box attacks and black-box attacks. In white-box attacks, the attacker can access the structure and parameters of the target model; while black-box attacks limit this access. Existing white-box model inversion attacks usually use generative adversarial networks to learn the prior knowledge of images, and transform the model inversion attack into an optimization problem of the latent vector of the generator in the input generative adversarial network. However, since most existing models work like black boxes, the white-box model inversion attack method is greatly limited in practical applications.
[0004] For the black-box setting, existing research has proposed various methods including methods for calculating gradient approximations. But these methods have some common deficiencies: on the one hand, they usually use a single model as the attack target to obtain the attack result, which will lead to overfitting of the reconstructed image to the attacked target and is difficult to transfer to other black-box models that have not been seen and provide the same service as the attacked model; on the other hand, in order to successfully implement the attack, these methods all require a large number of accesses to the model. When the attacker hopes to attack a model that has not been seen and provides the same service as the attacked model, the existing attack results cannot be reused, and a complete training and attack process must be performed on this model, which will result in huge computational overhead and economic overhead. Therefore, how to design an image reconstruction method to obtain a highly transferable reconstructed image has become an urgent problem to be solved. Summary of the Invention
[0005] The purpose of the present invention is to overcome the deficiencies of the prior art, and propose a highly transferable image reconstruction method based on an ensemble method, which solves the problem of insufficient transferability of the reconstructed images obtained by existing model inversion attack methods. The reconstructed images obtained by using the method of the present invention can capture more features, and the transferability is significantly improved.
[0006] To achieve the above object, the technical solution provided by the present invention is: a highly migratable image reconstruction method based on an integration method, comprising the following steps:
[0007] 1) Collect auxiliary image data approximating the distribution of privacy image data to train a generative adversarial network, thereby capturing the prior knowledge of images; collect local models performing the same task to form a local model pool, and randomly select a preset number of local models from it to form a local model set;
[0008] 2) Perform an integrated model inversion attack with the generative adversarial network. During the attack, through an adaptive weight adjustment strategy, adaptively and reasonably allocate weights to all local models in the local model set, so as to make the attack direction more accurate and accelerate the capture of features concerned by all local models in the local model set;
[0009] 3) After a preset number of attack iterations, adopt a dynamic model update strategy. This strategy constructs a new local model set adapted to the current attack requirements more pertinently according to the attacked situation of the current local model set, and at the same time enhances the diversity of attack targets, so that the reconstructed image can capture more features concerned by local models;
[0010] 4) Continuously perform attacks using the adaptive weight adjustment strategy and the dynamic model update strategy until a specified number of local model sets are attacked to obtain a highly migratable reconstructed image.
[0011] Further, in step 1), after collecting auxiliary image data approximating the distribution of privacy image data, in order to ensure that the generative adversarial network can correctly capture the prior knowledge of the image distribution, first preprocess the auxiliary image data. The specific preprocessing operations include: performing a central cropping operation on the auxiliary image data to remove the background, and then uniformly adjusting the size of all auxiliary image data to 64×64; using the preprocessed auxiliary image data to train the generative adversarial network, which can capture the prior of the image distribution; the generative adversarial network includes a generator G that can generate realistic images from latent vectors and a discriminator D for discriminating whether an image is a real image; collect all local models that perform the same task and whose structures and parameters are accessible to form a local model pool F pool , and then from the local model pool F pool randomly select k local models to form an initial local model set
[0012] Further, the specific operation steps of step 2) are as follows:
[0013] First, if the latent vector z has not been initialized, sample a latent vector z from the standard normal distribution N(0, 1); otherwise, continue to use the existing latent vector z. Use the generator G in the pre-trained generative adversarial network to map the latent vector z to the image space and generate the reconstructed image G(z). Then, denote the current local model set, and use all the local models in the current local model set as the attack target together. By inputting the reconstructed image G(z) into each local model in the local model set , the output result of each local model can be obtained, and the identity loss L iden (z) can be calculated using this result. The identity loss L iden (z) can reflect the matching degree between the reconstructed image G(z) and the target class c, that is:
[0014]
[0015] In the formula, ω i represents the loss weight of the local model f i . ω i ≥0 and k represents the number of local models, and L i (z) refers to the loss of the i-th local model f i in the local model set, which is expressed as:
[0016] L i (z) = L(f i (G(z)), c)
[0017] In the formula, L(·, ·) is the cross-entropy or other appropriate loss function;
[0018] The loss L i of the local model f i (z) can characterize the capture situation of the features that the local model f i focuses on by the reconstructed image. The larger the L i (z), the worse the capture situation of the features that the local model f i focuses on by the reconstructed image. The adaptive weight adjustment strategy assigns higher loss weights to the local models with larger losses during the attack process, making the attack process pay more attention to these local models with large losses, thereby guiding the attack in a more accurate direction and accelerating the capture of the features that these local models focus on by the reconstructed image to improve the transferability of the reconstructed image. The adaptive weight adjustment strategy also uses softmax to provide smooth and flexible weight adjustment for each local model. The weight obtained by the local model f i through adaptive weight adjustment can be expressed as:
[0019]
[0020] Wherein, α is a hyperparameter used to control the degree of weight smoothing, and L j (z) represents the loss of the j-th local model in the local model ensemble;
[0021] To ensure that the reconstructed image is realistic enough and close to the real image in distribution, the model inversion attack also needs to introduce the prior loss L prior (z) from the discriminator D in the generative adversarial network:
[0022] L prior (z) = -D(G(z))
[0023] The weights of the identity loss and the prior loss are adjusted with the hyperparameter λ, and the total loss L total (z) is expressed as:
[0024] L total (z) = L prior (z) + λ·L iden (z)
[0025] Using the total loss L total the latent vector z can be optimized to obtain the latent vector z that can minimize the total loss * , and the optimization objective is expressed as:
[0026] z* = argmin z L total (z).
[0027] Furthermore, in step 3), after a preset number of attack iterations, a dynamic model update strategy needs to be adopted to construct and replace the local model ensemble, so that the reconstructed image captures the features concerned by more local models in the local model pool F pool and thus improve the transferability of the reconstructed image;
[0028] Let represent the current local model ensemble. The dynamic model update strategy first calculates the classification loss of each local model in the current local model ensemble for the reconstructed image generated by the current latent vector; then, selects the local model f with the largest classification loss max as part of the new local model ensemble , and retaining this local model helps the reconstructed image capture more features and ensure that the direction of subsequent attacks is more accurate;
[0029] Then, the dynamic model update strategy randomly samples k local models from the local model pool F pool to obtain a set of local models F samp, together with f max constitute a new set of local models;
[0030] The new local models obtained through the dynamic model update strategy are determined based on the attacked situation of the current set of local models so that the new local models obtained in this way can adapt to the changing attack requirements, and the new set of local models can be expressed as:
[0031]
[0032] Furthermore, in step 4), by performing an integrated model inversion attack on the set of local models using an adaptive weight adjustment strategy, and constructing and replacing the set of local models using the dynamic model update strategy every preset number of attack iterations, the latent vector z can be gradually optimized; after sequentially performing attacks on N sets of local models, the final latent vector can be obtained; by inputting the final latent vector into the generator G, a reconstructed image with high transferability can be generated.
[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0034] 1. The present invention realizes a high-transferability image reconstruction method by using an integration method. Aiming at the problem of insufficient transferability of the reconstructed images obtained by the existing model inversion attacks for image reconstruction, the present invention improves the model inversion attacks. First, collect auxiliary image data to train a generative adversarial network; collect local models to form a local model pool, construct a set of local models from the local model pool, and perform an integrated model inversion attack on all local models in the set of local models with the help of the generative adversarial network; during the attack process, use an adaptive weight adjustment strategy to reasonably allocate weights to all local models in the set of local models, guide the attack in a more accurate direction, and accelerate the capture of the features that each local model in the set of local models focuses on by the reconstructed image; every preset number of attack iterations, use the dynamic model update strategy to construct and replace the set of local models; until a specified number of sets of local models are attacked, a reconstructed image with high transferability is obtained.
[0035] 2. Compared with other model inversion attack methods, the reconstructed images obtained by the method of the present invention capture more features that the model focuses on, and the transferability is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a framework diagram of the method of the present invention.
[0037] Figure 2 is a schematic diagram of the adaptive weight adjustment of the method of the present invention. Detailed implementation manners
[0038] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the implementation manners of the present invention are not limited thereto.
[0039] As Figure 1 and Figure 2 shown, this embodiment discloses a highly migratable image reconstruction method based on an integration method, including the following steps:
[0040] 1) After collecting auxiliary image data approximate to the distribution of private image data, in order to ensure that the generative adversarial network can correctly capture the prior knowledge of the image distribution, the auxiliary image data is first preprocessed. The specific preprocessing operations include: performing a central cropping operation on the auxiliary image data to remove the background, and then uniformly adjusting the size of all auxiliary image data to 64×64. Using the preprocessed auxiliary image data to train the generative adversarial network can capture the prior of the image distribution. The generative adversarial network includes a generator G that can generate realistic images from latent vectors and a discriminator D for discriminating whether an image is a real image. Collect all local models that perform the same task and have accessible structures and parameters to form a local model pool F pool , and then randomly select k local models from the local model pool F pool to form an initial local model set
[0041] 2) Based on the obtained generative adversarial network and local model set, the attack process of the local model set as shown in Figure 1 can be realized. First, if the latent vector z has not been initialized, sample a latent vector z from the standard normal distribution N(0, 1), otherwise continue to use the existing latent vector z. Use the generator G in the pre-trained generative adversarial network to map the latent vector z to the image space to generate a reconstructed image G(z). Then, let represent the current local model set, and use all the local models in the current local model set as the attack target together. By inputting the reconstructed image G(z) into each local model in the local model set for classification, the output result of each local model can be obtained, and the identity loss L iden (z) can be calculated using this result. The identity loss L iden (z) can reflect the matching degree between the reconstructed image G(z) and the target category c, that is:
[0042]
[0043] In the formula, ω i represents the local model fi Loss weight, ω i ≥0 and k represents the number of local models, L i (z) refers to the loss of the i-th local model f in the local model set i can be expressed as:
[0044] L i (z) = L(f i (G(z)), c)
[0045] In the formula, L(·, ·) can be cross-entropy or other suitable loss functions.
[0046] Local model f i The loss L i (z) can characterize the capture of the features that the local model f i is concerned about. The larger L i (z) indicates that the capture of the features that the local model f i is concerned about is worse. The adaptive weight adjustment strategy assigns higher loss weights to the local models with larger losses during the attack process, making the attack process pay more attention to these local models with large losses, thereby guiding the attack in a more accurate direction and accelerating the capture of the features that the reconstructed image is concerned about by these local models to improve the transferability of the reconstructed image. Specifically, the adaptive weight adjustment strategy also uses softmax to provide smooth and flexible weight adjustment for each local model. The weight obtained by the local model f i through adaptive weight adjustment can be expressed as:
[0047]
[0048] In the formula, α is a hyperparameter used to control the smoothness of the weights, and L j (z) represents the loss of the j-th local model in the local model set.
[0049] To ensure that the reconstructed image is realistic enough and close to the real image in distribution, the model inversion attack also needs to introduce the prior loss L prior (z) from the discriminator D in the generative adversarial network:
[0050] L prior (z) = -D(G(z))
[0051] Use the hyperparameter λ to adjust the weights of the identity loss and the prior loss. The total loss L total (z) can be expressed as:
[0052] L total (z) = L prior (z) + λ·Liden (z)
[0053] Using the total loss L total the latent vector z can be optimized to obtain the latent vector z that can minimize the total loss * , and the optimization objective can be expressed as:
[0054] z * = argmin z L total (z)
[0055] 3) After the preset number of attack iterations, a dynamic model update strategy needs to be adopted to construct and replace the local model set, so that the reconstructed image captures the features concerned by more local models in the local model pool F pool and thus improve the transferability of the reconstructed image.
[0056] Taking to represent the current local model set, the dynamic model update strategy first calculates the classification loss of each local model in the current local model set for the reconstructed image generated by the current latent vector. Then, select the local model f with the largest classification loss among them max as a part of the new local model set . Retaining this local model helps the reconstructed image capture more features and ensure that the direction of subsequent attacks is more accurate.
[0057] Next, the dynamic model update strategy randomly samples k local models from the local model pool F pool to obtain a set of local models F samp , which together with f max constitute the new local model set. Random sampling can effectively avoid the bias caused by the fixed local model set, improve the local model diversity, and reduce the overfitting problem.
[0058] The new local model obtained through the dynamic model update strategy is determined based on the attacked situation of the current local model set , so that the new local model obtained can adapt to the changing attack requirements, and the new local model set can be expressed as:
[0059]
[0060] 4) By adopting an adaptive weight adjustment strategy on the local model set for integrated model inversion attacks, and constructing and replacing the local model set with a dynamic model update strategy every preset number of attack iterations, the latent vector z can be gradually optimized. After sequentially performing attacks on N local model sets, the final latent vector can be obtained. Inputting the final latent vector into the generator G can generate a reconstructed image with high transferability.
[0061] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.
Claims
1. A highly transferable image reconstruction method based on an integration method, characterized in that, Including the following steps: 1) Collect auxiliary image data that approximates the distribution of privacy image data to train a generative adversarial network, thereby capturing the prior knowledge of images; Collect local models that perform the same task to form a local model pool, randomly select a preset number of local models from it to form a local model set; 2) Perform an integrated model inversion attack with the help of the generative adversarial network. During the attack, through an adaptive weight adjustment strategy, adaptively and reasonably allocate weights to all local models in the local model set, so that the attack direction is more accurate and accelerate the capture of features that all local models in the local model set focus on; 3) After a preset number of attack iterations, adopt a dynamic model update strategy. This strategy constructs a new local model set that adapts to the current attack requirements more pertinently according to the attacked situation of the current local model set, and at the same time enhances the diversity of attack targets, so that the reconstructed image can capture more features that local models focus on; 4) Continuously adopt the adaptive weight adjustment strategy and the dynamic model update strategy for attack until a specified number of local model sets are attacked, and obtain a reconstructed image with high transferability.
2. The high-migrability image reconstruction method based on an integration method according to claim 1, characterized in that In step 1), after collecting auxiliary image data approximated to the distribution of privacy image data, in order to ensure that the generative adversarial network can correctly capture the prior knowledge of the image distribution, the auxiliary image data is preprocessed first. The specific preprocessing operations include: performing a central cropping operation on the auxiliary image data to remove the background, and then uniformly adjusting the size of all auxiliary image data to 64×64; training the generative adversarial network with the preprocessed auxiliary image data to capture the prior of the image distribution. The generative adversarial network includes a generator G that can generate realistic images from latent vectors and a discriminator D for discriminating whether an image is a real image; collecting all local models that perform the same task and have accessible structures and parameters to form a local model pool F pool , and then from the local model pool F pool randomly select k local models to form an initial local model set 3. The highly transferable image reconstruction method based on the integration method according to claim 2, characterized in that The specific operation steps of step 2) are as follows: First, if the latent vector z has not been initialized, sample a latent vector z from the standard normal distribution N(0, 1); otherwise, continue to use the existing latent vector z. Use the generator G in the pre-trained generative adversarial network to map the latent vector z to the image space and generate a reconstructed image G(z). Next, denote the current local model set, and use all the local models in the current local model set as the common attack target. By inputting the reconstructed image G(z) into each local model in the local model set , the output result of each local model can be obtained, and the identity loss L iden (z) can be calculated using this result. The identity loss L iden (z) can reflect the matching degree between the reconstructed image G(z) and the target class c, that is: where ω i represents the loss weight of the local model f i , ω i ≥ 0 and k represents the number of local models, and L i (z) refers to the loss of the i-th local model f i in the local model set, which is expressed as: L i (z) = L(f i (G(z)), c) Where L(·,·) is the cross-entropy or other suitable loss function; Local model f i The loss L i (z) can characterize the capture of the features that the reconstructed image focuses on by the local model f i The larger the L i (z), the worse the capture of the features that the reconstructed image focuses on by the local model f i The adaptive weight adjustment strategy makes the attack process pay more attention to these local models with larger losses by assigning higher loss weights to the local models with larger losses during the attack process, so as to guide the attack in a more accurate direction, speed up the capture of the features that the reconstructed image focuses on by these local models, and improve the transferability of the reconstructed image. The adaptive weight adjustment strategy also uses softmax to provide smooth and flexible weight adjustment for each local model. The weight obtained by the local model f i through adaptive weight adjustment can be expressed as: where α is a hyperparameter used to control the degree of weight smoothing, and L j (z) represents the loss of the j-th local model in the local model ensemble; To ensure that the reconstructed image is realistic enough and close to the real image in terms of distribution, the model inversion attack also needs to introduce a prior loss \(L_D(z)\) from the discriminator \(D\) in the generative adversarial network: prior (z): L prior (z) = -D(G(z)) The weights of the identity loss and the prior loss are adjusted by the hyperparameter λ, and the total loss L total (z) is expressed as: L total ψ(z) = L prior ψ(z)+λL iden ψ(z) Using the total loss L total The latent vector z can be optimized to obtain the latent vector z that minimizes the total loss * , and the optimization objective is expressed as: z * = argmin z L total (z).
4. The method for highly transferable image reconstruction based on an integration method according to claim 3, wherein In step 3), after a preset number of attack iterations, a dynamic model update strategy needs to be adopted to construct and replace the local model set, enabling the reconstructed image to capture features that more local models in the local model pool F pool focus on, thereby improving the transferability of the reconstructed image; Given a current set of local models, the dynamic model update strategy first calculates the classification loss of the reconstructed images generated by each local model in the current set of local models for the current latent vector; then, it selects the local model f with the largest classification loss max as part of the new set of local models Retaining this local model helps the reconstructed images capture more features and ensures that the subsequent attack direction is more accurate; Next, the dynamic model update strategy randomly samples k local models from the local model pool F pool to obtain a set of local models F samp , which together with f max forms a new local model set; The new local model obtained through the dynamic model update strategy is determined based on the current local model set under the attacked situation, and the new local model obtained in this way can adapt to the changing attack requirements. The new local model set can be expressed as:
5. The high-migrability image reconstruction method based on the integration method according to claim 4, characterized in that In step 4), by performing an integrated model inversion attack on the local model set with the adaptive weight adjustment strategy and constructing and replacing the local model set with the dynamic model update strategy every preset number of attack iterations, the latent vector z can be gradually optimized; after sequentially attacking N local model sets, the final latent vector can be obtained; inputting the final latent vector into the generator G can generate a reconstructed image with high transferability.