Remote sensing image safety protection method and device based on model fusion and storage medium
Through a model fusion method, an adversarial sample deception deep learning model is generated, which solves the problem that the existing technology cannot effectively resist the malicious behavior of the black box model on remote sensing images, and realizes the security protection of remote sensing images in a deep learning environment.
Patent Information
- Application Number
- CN202411188783.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-31
AI Technical Summary
Existing remote sensing image protection technology has failed to effectively resist deep learning models, especially black box models, and malicious behaviors on sensitive remote sensing images, resulting in limited protection scope and inability to meet the security needs in deep learning environments.
Using a model fusion method, by obtaining remote sensing target images, using the logits fusion strategy to form a fusion model, compute the logits output of the fusion model, construct the loss function, and use dynamic sampling non-local gradient descent method to generate adversarial samples to deceive the neural network classification model .
Effectively resist the attackers' malicious extraction and detection of remote sensing images, meet the security protection needs in the deep learning environment, while maintaining the visual authenticity and shared circulation of images, and is suitable for diverse deep learning scenarios.
Smart Images
Figure CN119229174A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing image processing, and in particular to a remote sensing image security protection method, device and storage medium based on model fusion. Background Art
[0002] Although the existing remote sensing image protection technology can meet the requirements of ensuring the security of remote sensing images to a certain extent, with the rapid development of artificial intelligence technologies such as deep learning, the use scenarios of remote sensing images have become diversified and complicated. At the same time, in real scenarios, the neural network classification model used by attackers is a black box in most cases, and the information of the model is unknown. How to resist the malicious recognition behavior of black box models on sensitive remote sensing images is also an issue that needs to be considered urgently. Therefore, it is necessary to closely combine the essential characteristics of remote sensing images with the needs of open sharing in the Internet environment, analyze the current application status of deep learning models in remote sensing images, and study how to protect the security of sensitive content of remote sensing images under the current trend of great development of artificial intelligence. Summary of the invention
[0003] The purpose of the present invention is to propose a remote sensing image security protection method, device and storage medium based on model fusion, so as to solve the technical problem that traditional remote sensing image protection does not take deep learning into consideration, resulting in a limited protection scope and cannot meet current needs.
[0004] The present invention provides a remote sensing image security protection method based on model fusion, which comprises the following steps:
[0005] S1: Acquire remote sensing target images;
[0006] S2: Using the logits fusion strategy, multiple neural network classification sub-models are combined into a fusion model;
[0007] S3: Calculate the logits output of the fusion model for the input remote sensing target image sample based on the logits output of the sub-model and the fusion weight;
[0008] S4: Use the softmax function and the intermediate output value logits of the fusion model to construct the loss function of the fusion model;
[0009] S5: According to the loss function of the fusion model, the dynamic sampling non-local gradient descent method is used to create adversarial samples on the fusion model;
[0010] S6: Solve the constrained loss function maximization problem to obtain adversarial samples that can deceive the fusion model.
[0011] A storage medium stores instructions and data for implementing a remote sensing image security protection method based on model fusion.
[0012] A remote sensing image security protection device based on model fusion comprises: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a remote sensing image security protection method based on model fusion.
[0013] The beneficial effects provided by the present invention are: the present invention can deceive the neural network classification model, resist the attacker from using the neural network model to maliciously extract and detect important geographical targets in remote sensing images, meet the security protection requirements of remote sensing images in a deep learning environment, and at the same time ensure that the visual authenticity and sharing and circulation requirements of remote sensing images are not significantly affected, and effectively target the malicious identification and information extraction behaviors of black box models on sensitive remote sensing images. The present invention can be applied to provide security protection for remote sensing images in a variety of deep learning scenarios, filling the gap in the field of remote sensing image security protection in the context of deep learning in China today. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a schematic flow chart of the method of the present invention;
[0015] Figure 2 Graph of the process of generating adversarial examples;
[0016] Figure 3 Generate renderings for adversarial examples;
[0017] Figure 4 This is the effect diagram of the logits fusion strategy;
[0018] Figure 5 It is a working schematic diagram of the hardware device of the present invention. DETAILED DESCRIPTION
[0019] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0020] Before formally describing the present invention, the scheme of the present invention is first generally described for easy understanding.
[0021] Please refer to Figure 1 , Figure 1 It is a schematic flow diagram of the method of the present invention;
[0022] The present invention provides a remote sensing image security protection method based on model fusion, comprising the following steps:
[0023] S1: Acquire remote sensing target images;
[0024] S2: Using the logits fusion strategy, multiple neural network classification sub-models are combined into a fusion model;
[0025] It should be noted that the neural network classification model described in step S2 includes Inception v3, Inceptionv4, Inception Resnetv2 models and Ens3-adv-inception-v3, Ens4-adv-inception-v3 and Ens-adv-inception-resnet-v2 adversarial training models.
[0026] In step S2, each sub-model receives the submitted remote sensing target image as input, and each sub-model generates an intermediate output value logits.
[0027] S3: Calculate the logits output of the fusion model for the input remote sensing target image sample based on the logits output of the sub-model and the fusion weight;
[0028] It should be noted that step S3 is specifically as follows:
[0029] S31: The prediction probability of the sub-model for the category of the input sample is used as the fusion weight of the sub-model, which is calculated as follows:
[0030] w k =p k (x t ,y true )
[0031] Among them, w k is the fusion weight of the kth sub-model, x t is the input sample of the current iteration, y true is the true category label of the original sample, p k (x t ,y true ) represents the sample x in the kth sub-model t Classified as label y true The classification probability of
[0032] S32: The logits of the fusion model are obtained by calculating the weighted average of the logits of each sub-model. The calculation method is:
[0033]
[0034] Among them, x t is the input sample of the current iteration, K is the number of models included in the model fusion framework, and after normalization, the sum of the fusion weights of each model is still 1, l k () represents the logits of the kth sub-model, l DEM () represents the logits of the fusion model.
[0035] Preferably, as an embodiment of step S3, please refer to Figure 2 ;like Figure 2 As shown in the figure, DEM represents a dynamic fusion strategy. In each iteration, the predicted probability output by each model reflects the attack effect of the model on the input sample. The method uses the probability of the sub-model classifying the input sample as the true category label of the original sample as the weight of each model, weighing the impact of different sub-models on the disturbance, so as to construct adversarial samples with better transferability and adversarial resistance. The calculation method of the fusion process is:
[0036]
[0037] in represents the adversarial sample in the current iteration, y true is the true category label of the original sample x, Represents the sample in the kth sub-model Classified as label y true The classification probability of the fusion model in the tth iteration, about the sample The loss function is the softmax cross entropy loss function, given the true label y true And the logits of each sub-model are calculated as input:
[0038]
[0039] After obtaining the loss function of the fusion model, the DSN-I-FGSM algorithm uses the dynamic sampling gradient descent method to guide the generation process of adversarial samples. The perturbation is continuously updated according to the customized number of iterations until the loss function of the adversarial sample reaches maximum convergence. Figure 3 Generate effect maps for adversarial samples of remote sensing images.
[0040] S4: Use the softmax function and the intermediate output value logits of the fusion model to construct the loss function of the fusion model;
[0041] It should be noted that step S4 uses the intermediate output value logits of the fusion model as the input of the softmax function to construct the loss function of the fusion model, which is calculated as follows:
[0042] J DEM (x t ,y true )=-1 ytrue log(softmax(l DEM (x t )))
[0043] Among them, x t is the input sample of the current iteration, ytrue is the true category label of the original sample, -1 ytrue Represents the true label y true One-hot encoding of J DEM (x t ,y true ) is the fusion model about x t The loss function of .
[0044] S5: According to the loss function of the fusion model, the dynamic sampling non-local gradient descent method is used to create adversarial samples on the fusion model;
[0045] It should be noted that step S5 is specifically as follows:
[0046] S51: Calculate the dynamic sampling non-local gradient of the fusion model loss function, which is calculated as follows:
[0047]
[0048] Among them, x t is the input sample of the current iteration, F θ is the neural network classification sub-model, and the corresponding parameter is θ, is the neural network sub-model F θ The loss function, y true is the true category label of the original sample, represents the historical sampling points of the i-th iteration sampling operation, w i is related to the loss function The weight of the gradient of ;
[0049] S52: Make a perturbation based on the dynamic sampling non-local gradient, which is calculated as follows:
[0050]
[0051] Among them, δ represents the disturbance, x t is the input sample of the current iteration, y true is the true category label of the original sample, It is the dynamically sampled non-local gradient of the loss function of the fusion model with respect to the current input sample;
[0052] S53: Add the perturbation to the input sample to obtain the adversarial sample of this iteration, which is calculated as follows:
[0053]
[0054] Where δ represents the disturbance, It means that the input sample x in the current iteration cycle tThe adversarial sample generated after adding the perturbation is also the input sample of the next iteration cycle; Clip(·) is the clipping function, which ensures that the perturbation added to the sample in each iteration is constrained to the ε-neighborhood.
[0055] Preferably, as an embodiment of step S5, still refer to Figure 2 ;like Figure 2 As shown in the figure, the DSN-I-FGSM algorithm calculates the gradient through a dynamic sampling mechanism, stably updates the perturbation in each iteration, and accelerates the convergence of adversarial samples. In each iteration, after obtaining the loss function of the fusion model, the DSN-I-FGSM algorithm first searches for sampling points in the neighborhood of the current input sample. The calculation process can be expressed as:
[0056]
[0057] in, represents the sampling point of the i-th iteration, is the random variable used to find the sampling point for the i-th iteration, x i Represents the input sample of the i-th iteration.
[0058] After completing the sampling operation of this iteration, it is necessary to update the gradient of the current iteration sampling point and the gradient of the historical sampling point to weigh the contribution of the gradients of different sampling points to the gradient of the current iteration input sample. The weight update process of the sampling point is:
[0059]
[0060] Among them, x t is the input sample of the current iteration, is the sampling point in the i-th attack iteration. If the current sample x t and a sampling point in the sampling history If the distance is greater than σ, we should reduce The importance of σ in gradient calculation. On the contrary, sampling points that are still within the sample neighborhood defined by σ should be given a larger weight. β is a hyperparameter that dynamically updates the gradient weight of the sampling point.
[0061] After the weights are updated, the dynamic sampling non-local gradient of the loss function is calculated. The calculation process is:
[0062]
[0063] Among them, F θ is a neural network classification model with parameters θ, is the neural network model F θ The loss function, x t is the sample given in the current iteration, y trueis the correct category of the original sample, and t represents the current attack iteration.
[0064] The adversarial sample of this iteration is made using dynamic sampling non-local gradient. The calculation process is:
[0065]
[0066] Among them, x represents the original sample, y true is the correct category of x, is the loss function of the target model with respect to the current input sample Dynamically sampled nonlocal gradients, It means that the input sample in the current iteration cycle The adversarial sample generated after adding the perturbation is also the input sample of the next iteration cycle. Clip(·) is a clipping function, which ensures that the perturbation added to the sample in each iteration is constrained to the ε-neighborhood. For images, the maximum pixel value cannot exceed 255.
[0067] S6: Solve the constrained loss function maximization problem to obtain adversarial samples that can deceive the fusion model.
[0068] The calculation formula for maximizing the loss function in step S6 is:
[0069]
[0070] ∈ represents adding to the adversarial sample x * , y is the true category label of the original sample x, and the subscript 2 indicates the norm type;
[0071] Calculate the loss function value of the model for the adversarial sample and check whether the loss function value reaches the maximum convergence condition. If so, the training ends and goes to step S6; otherwise, return to step S2 and continue iteration. If the loss function of the model for the adversarial sample reaches the maximum, it means that the process of making adversarial samples by the model has reached convergence, the iteration ends, and the adversarial sample obtained can deceive the classification model and make it misclassify.
[0072] The adversarial samples of remote sensing images produced by the present invention are very similar to the original images in human vision, but can cause the neural network classification model to recognize errors with high confidence. The perturbations in the adversarial samples themselves have two characteristics: concealment and confusion. The concealment of the perturbation means that the perturbation is very small, and after being added to the original sample, it has an effect that is imperceptible to the human eye; the confusion of the perturbation means that for the deep learning model, after the perturbation is added to specific pixels of the original image, the new image generated can induce the target model to misclassify. Therefore, this specific adversarial perturbation is added to the image to obtain an adversarial image. It is difficult for the naked eye to find the difference between the two, but the deep learning model gives different classification results.
[0073] The above method is used to realize the application of adversarial samples produced by the model fusion scheme on remote sensing images. Figure 4 This is the effect diagram of the logits fusion strategy.
[0074] See also Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a remote sensing image security protection device 401 based on model fusion, a processor 402 and a storage medium 403.
[0075] A remote sensing image security protection device 401 based on model fusion: The remote sensing image security protection device 401 based on model fusion implements the remote sensing image security protection method based on model fusion.
[0076] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the remote sensing image security protection method based on model fusion.
[0077] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the remote sensing image security protection method based on model fusion.
[0078] The beneficial effects of the present invention are: the present invention can deceive the neural network classification model, resist the attacker from using the neural network model to maliciously extract and detect important geographical targets in remote sensing images, meet the security protection requirements of remote sensing images in a deep learning environment, and at the same time ensure that the visual authenticity and sharing and circulation requirements of remote sensing images are not significantly affected, and effectively target the malicious identification and information extraction behaviors of sensitive remote sensing images implemented by black box models. The present invention can be applied to provide security protection for remote sensing images in a variety of deep learning scenarios, filling the gap in the field of remote sensing image security protection in the context of deep learning in China today.
[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A remote sensing image security protection method based on model fusion, characterized by: The method comprises the following steps: S1: Acquire remote sensing target images; S2: Using the logits fusion strategy, multiple neural network classification sub-models are combined into a fusion model; S3: Calculate the logits output of the fusion model for the input remote sensing target image sample based on the logits output of the sub-model and the fusion weight; S4: Use the softmax function and the intermediate output value logits of the fusion model to construct the loss function of the fusion model; S5: According to the loss function of the fusion model, the dynamic sampling non-local gradient descent method is used to create adversarial samples on the fusion model; S6: Solve the constrained loss function maximization problem to obtain adversarial samples that can deceive the fusion model.
2. The remote sensing image security protection method based on model fusion according to claim 1, characterized in that: The neural network classification models described in step S2 include Inception v3, Inceptionv4, Inception Resnetv2 models and Ens3-adv-inception-v3, Ens4-adv-inception-v3 and Ens-adv-inception-resnet-v2 adversarial training models.
3. The remote sensing image security protection method based on model fusion according to claim 1, characterized in that: In step S2, each sub-model receives the submitted remote sensing target image as input, and each sub-model generates an intermediate output value logits.
4. The remote sensing image security protection method based on model fusion as claimed in claim 3, characterized in that: Step S3 is specifically as follows: S31: The prediction probability of the sub-model for the category of the input sample is used as the fusion weight of the sub-model, which is calculated as follows: w k =p k (x t ,y true ) Among them, w k is the fusion weight of the kth sub-model, x t is the input sample of the current iteration, y true is the true category label of the original sample, p k (x t ,y true ) represents the sample x in the kth sub-model t Classified as label y true The classification probability of S32: The logits of the fusion model are obtained by calculating the weighted average of the logits of each sub-model. The calculation method is: Among them, x t is the input sample of the current iteration, K is the number of models included in the model fusion framework, and after normalization, the sum of the fusion weights of each model is still 1, l k (x) represents the logits of the kth sub-model, l DEM (x) represents the logits of the fusion model.
5. A remote sensing image security protection method based on model fusion as claimed in claim 4, characterized in that: Step S4 uses the intermediate output value logits of the fusion model as the input of the softmax function to construct the loss function of the fusion model, which is calculated as follows: Among them, x t is the input sample of the current iteration, y true is the true category label of the original sample, Represents the true label y true One-hot encoding of J DEM (x t ,y true ) is the fusion model about x t The loss function of .
6. A remote sensing image security protection method based on model fusion as claimed in claim 5, characterized in that: Step S5 is specifically as follows: S51: Calculate the dynamic sampling non-local gradient of the fusion model loss function, which is calculated as follows: Among them, x t is the input sample of the current iteration, F θ is the neural network classification sub-model, and the corresponding parameter is θ, is the neural network sub-model F θ The loss function, y true is the true category label of the original sample, represents the historical sampling points of the i-th iteration sampling operation, w i is related to the loss function The weight of the gradient of ; S52: Make a perturbation based on the dynamic sampling non-local gradient, which is calculated as follows: Among them, δ represents the disturbance, x t is the input sample of the current iteration, y true is the true category label of the original sample, It is the dynamically sampled non-local gradient of the loss function of the fusion model with respect to the current input sample; S53: Add the perturbation to the input sample to obtain the adversarial sample of this iteration, which is calculated as follows: Where δ represents the disturbance, It means that the input sample x in the current iteration cycle t The adversarial sample generated after adding the perturbation is also the input sample of the next iteration cycle; Clip(·) is the clipping function, which ensures that the perturbation added to the sample in each iteration is constrained to the ε-neighborhood.
7. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a remote sensing image security protection method based on model fusion as described in any one of claims 1 to 6.
8. A remote sensing image security protection device based on model fusion, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a remote sensing image security protection method based on model fusion as described in any one of claims 1 to 6.