A model security migration method based on iterative screening and random mask learning

Abstract

The application discloses a model security migration method based on iterative screening and random mask learning, and the steps include: 1) fixing the weight value of an existing unknown channel teacher model and initializing the weight of a student model; 2) screening the channels output by each level of the teacher model using an iterative reinforcement screening algorithm, strengthening the high semantic information channels to improve the transmission information accuracy, and weakening and deleting the channels that cause significant changes in the output after being disturbed to avoid triggering the backdoor attack mode; 3) inputting a target training data set, shielding the student model output using different random masks in each round of training to regularize the learning of the teacher network channel output, avoiding the continuous learning of untrustworthy channels and learning the global strong semantic channel until the student model converges. The method can be applied to the migration of a pre-training model trained by an unsafe source to a student model, can solve the potential harm of the backdoor migrated to the student model and improve the accuracy of the student model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and particularly relates to a model security migration method based on iterative screening and random mask learning. BACKGROUND

[0002] With the deepening of research in the field of deep learning, the artificial intelligence ecology of the application end has brought strong vitality, and it is a significant trend to use and research deep learning models in various industries. With the continuous development of large models, the mode of training a large model by large companies on large-scale data sets with a large amount of time, manpower and material resources and then optimizing and fine-tuning it has gradually become mainstream (Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners [J]. OpenAI blog, 2019, 1(8): 9.). However, due to commercial reasons, large companies often do not open source the specific parameters, data sets and even training method details of the trained model (OpenAI. "GPT-4 Technical Report." ArXiv abs / 2303.08774 (2023): n. pag.), making it difficult for other researchers to follow up on the research.

[0003] Compared with large companies, ordinary scientific researchers and small research units do not have powerful computing device resources, do not have a large number of research and development personnel to support optimization, and do not have large-scale data sets, so their research can usually only be carried out on a relatively small scale. In order to improve the performance of small-scale models, knowledge distillation (Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network [J]. arXiv preprint arXiv:1503.02531, 2015.) and migration technology based on pre-trained models can effectively improve the accuracy and generalization of small models, and at the same time, can weaken the ability of large-scale models and migrate them to small-scale models for research and application work.

[0004] However, the pre-trained model published on the Internet has the possibility of being maliciously attacked due to the opacity of the training data set, training method, model parameters and other information. Based on the particularity of artificial intelligence model, the main attack mode executable through the pre-trained model is backdoor attack, that is, by injecting backdoor in the wide-area data set, modifying the training process or modifying the weight, the normal model weight is injected with the response weight of the backdoor attack trigger mode (Yao Y, Li H, Zheng H, et al. Latent backdoor attacks on deep neural networks [C] / / Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS 2019). 2019: 2041-2055.). The pre-trained model injected with the backdoor trigger will produce the behavior predetermined by the attacker, such as performance degradation, purposeful damage or wrong decision, when encountering the same or similar input mode as the trigger.

[0005] The process of knowledge distillation or transfer of deep learning models can also migrate some trigger patterns to the trained model, posing a serious threat to small-scale research. To resist backdoor attacks in pre-trained models, the academic community has improved the knowledge distillation paradigm to deal with backdoors (Li Y, Lyu X, Koren N, et al. Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks [C] / / International Conference on Learning Representations.), which uses a small amount of clean data set to fine-tune the model as a teacher model, and then distills the original model again, so as to use the teacher model purification effect brought by fine-tuning to adjust the original model to remove more backdoor triggers. This kind of solution in the research is mainly for relatively small models, that is, the purpose is to remove the backdoor of the model itself, which is not suitable for the solution of migrating a large model to a small model. The large model itself is difficult to reason multiple times, and this solution is not thorough in removing backdoors because it is directly imitating learning of pixel points (Shen L, Ji S, Zhang X, et al. Backdoor Pre-trained Models Can Transfer to All [C] / / Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (CCS 2021). 2021: 3141-3158.), and the process of distilling again will cause the accuracy of the original model to decrease.

[0006] Due to the natural sparsity of deep learning weights, the distribution of semantic information among different channels is not uniform (Han S, Pool J, Tran J, et al. Learning both weights and connections for efficient neural network [J]. Advances in Neural Information Processing Systems (NIPS), 2015, 28.), so using channels with strong semantic information for information transmission can maximize the preservation of semantic information while reducing the possibility of malicious information transmission in other channels. Through the partial screening mechanism, this effect can be further enhanced.

[0007] Since the teacher model has a overwhelming advantage in the number of parameters over the student model, the possibility of overfitting is greater, and in the process of model learning, a regular process can help reduce such phenomena (Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks [J]. Communications of the ACM, 2017, 60 (6): 84-90.). Learning by matching the teacher network with the masked part of the random mask can play a regular role, and switching the channels covered by the random mask during different training processes can play a role in learning strong semantic channels in full channels, thereby improving accuracy. SUMMARY

[0008] In view of the problems in the prior art, the purpose of the present application is to provide a model security migration method based on iterative screening and random mask learning, which can efficiently obtain a secure sub-model with strong anti-disturbance and resistance to migration overfitting problems. The specific solution idea is:

[0009] 1. For the natural semantic sparsity problem of neural network models, the present application proposes a channel semantic reinforcement method, which analyzes the feature maps of different channels in different layers of the pre-trained model, determines the semantic information strength and contribution ability to the final output of each channel, and thus strengthens the channels with effective semantics and weakens the channels with invalid or negative semantics, thereby improving the efficiency of the distillation migration process;

[0010] 2. For the problem of poor resistance to backdoor attack disturbance between different channel feature maps of neural network models, the present application proposes a feature map channel stability analysis and screening algorithm based on disturbance parameters, which can obtain feature map channels with strong anti-interference ability, and use the strong stability channels to perform learning process to obtain a distillation migration model with strong security;

[0011] 3. For the natural overfitting problem in large pre-trained models, the present application proposes to use random mask to randomly mask the channels for learning. This random masking process can play a regular role, and can also combine the aforementioned screening and reinforcement measures for the feature channels of the pre-trained model to make the trained model always learn strong semantic information and avoid continuous learning of unsafe information, and replace the channels after a certain number of training times to completely train each channel of the trained model to obtain a distillation migration model with strong ability.

[0012] To achieve the above objectives, this invention provides the following technical solution: a secure transfer method for neural network models based on iterative filtering reinforcement and random mask regularization learning, comprising the following steps:

[0013] (1) Select and initialize the large model corresponding to the parent application task of the target application task as the teacher model M. tea Download and use the pre-trained model weights stored on the public platform to cover the teacher model M. tea After the parameter weights are fixed after coverage is completed, the output level and M are selected and initialized. tea Consistent small models or with M tea Consistent model as student model M stu Initialization for M stu After preprocessing the target dataset of the scenario that does not contain malicious samples, a dataset loader is created.

[0014] For example, when the target application task is a dense small object detection task or a multi-instance small object segmentation task, the parent application task is generally an image classification task. In this case, the large model can refer to image classification neural network models such as BASIC-L and ViT-e, which are trained using large-scale datasets and have a large number of parameters.

[0015] (2) Take a specified number of target dataset input samples and input them into M. tea In this context, the output feature map set Feats at each level is used. tea Each layer uses a self-attention mechanism to calculate the correlation weights within the feature map at its output, and uses Global Average Pooling to obtain the mean of the activation feature values ​​of different channels in each feature map, labeled H. ilk The mean activation feature value of channel k of the feature map corresponding to intermediate level l of training sample i is used. Then, for each intermediate level l, a sample-level average is performed based on the mean activation feature value of each channel k of the feature map corresponding to that intermediate level l of the multiple training samples to obtain the global average response intensity H of each channel k of the feature map corresponding to that intermediate level l. lk If H ilk Below H lk Then, label the channel k of the feature map corresponding to the intermediate layer l of training sample i as Channel. low Otherwise, mark it as a Channel high ;

[0016] (3) Initialize the disturbance parameter set σ, which is used to adjust M tea The output value of each intermediate layer l is perturbed, and this perturbation value is a trainable parameter, where the perturbation parameter σ of each layer l is...l The distribution of the perturbation features in the current layer l will be fitted using the KL divergence on the perturbation parameter σ. l The distribution is fitted to the feature map of the corresponding layer, and the fitting loss is labeled as... Initialize the Gaussian distribution variable ε, then randomly sample ε and multiply it by the perturbation parameter σ. l With sampling perturbation;

[0017] (4) For M tea Feats, the feature map of level l tea [l] Add the perturbation parameter σ to the corresponding layer sampling. l Put the perturbed feature map back into M tea The original hierarchical reasoning continues; among which, the feature map Feats after perturbation of the l-th layer is utilized. tea [l] Complete the target application task inference to obtain the final attacked model output. The loss of the original target task is obtained by comparing the true labels of the samples in the target application dataset. The purpose of calculating this loss is to keep the output of the model unchanged after the perturbation by reducing this loss, combined with the loss from the fitted data distribution. Obtain the total loss value for the intermediate layer l and backpropagate the gradient to the perturbation parameter σ of layer l. l Training stops after a specified number of iterations for M. tea The above training is performed on the output of each layer to obtain the perturbation parameters σ of each layer after training. l The set of perturbation parameters σ is composed of the following: where the total loss of the l-th layer is... Using the perturbation parameter σ of the intermediate level l l The feature map Feats corresponding to the intermediate level l tea [l] Perform a disturbance and calculate the disturbance loss before and after the disturbance. Teacher Model M tea Feats using the perturbed feature map corresponding to training sample i tea [l] Complete the target application task reasoning and obtain the output. according to The task loss corresponding to the intermediate layer l is calculated based on the label corresponding to the training sample i. Based on the perturbation loss corresponding to the training sample i Mission loss Calculate the loss value corresponding to the intermediate level l with respect to the disturbance parameter σ. l Optimize;

[0018] (5) The perturbation parameters σ trained at each layer l This can be considered as the maximum disturbance that different channels can withstand while maintaining the goal of constant output, and the hierarchical disturbance parameter σ. lIf the variance of the mean of the activation feature values ​​of channel k is greater than the variance of the activation values ​​of the corresponding channel in the data samples used during training, it is considered a safe and stable channel and is marked as a Channel. robust Otherwise, channel k is considered an insecure and unstable channel and is marked as a Channel. non-robust ;

[0019] (6) Combine the outputs of steps (2) and (5) to filter and strengthen the channels, whereby channels that simultaneously satisfy the Channel... high With Channel robust The channel is assigned twice the computational weight of a regular channel, and this weight will have a greater impact on the student model M during subsequent loss calculations. stu This is used to strengthen the system and simultaneously satisfy the Channel requirements. low With Channel non-robust The channels are masked to be removed during subsequent learning, and the remaining channels use the channel mean of the response intensity generated in step (2) as the channel weighting parameter. weight ;

[0020] (7) Input the target dataset into the teacher model and student model respectively, and obtain feature maps at the intermediate levels of each same stage to generate the teacher feature map set Feats. tea With student feature map set Feats stu And the probability distribution Logits of the final output. tea With Logits stu ;

[0021] (8) Feats tea Each channel is processed according to the channel processing parameters obtained in step (6), that is, based on the Channel... high With Channel robust Enhancement parameters, based on Channel low With Channel non-robust The mask parameters and the weighted parameters of each channel (Channel) weight For Feats stu The output of each level is based on the removed Feats. tea Given the total number of channels d in [l], use a random mask to block the corresponding number of channels d. Then, for the Feats after removing the masked channels... tea With Feats stu The mean squared error loss is calculated for the remaining unmasked channels. distill Combined with Logits stu Calculate the target task loss Lossdistill With The loss gradient returned after weighted summation is used to train the student model M stu Training is performed, and after a specified number of training is completed, the channel mask is randomly re-masked until M stu converges or training is completed, and the model M stu obtained after training is the large model M tea safe migration.

[0022] Further, the download platform of the pre-trained model of the specified model in step (1) selects an open source or commercial pre-trained model weight providing platform, but for the training data source, training method, and training parameters, the model weight is considered as an untrusted source for subsequent processing.

[0023] The student model M stu is equal to or weaker than the teacher model M tea in model capability; the target data set refers to the data set of the target application task to which the pre-trained model needs to be migrated; and the preprocessing step refers to the operation of preprocessing the data set before training, including standardization, denoising, data enhancement, and the like.

[0024] Further, the feature map set Feats tea is a more obvious reusable paradigm in the deep learning model processing flow, and the final output of each reusable paradigm is extracted as the output of a layer to form the set.

[0025] The internal correlation weight refers to the correlation between the data in the feature map, and its calculation process is to multiply the feature map transpose to calculate the probability distribution, and then multiply the feature map itself to obtain the weighted feature map, and its calculation formula is as follows:

[0026]

[0027] where Feat is the feature map participating in the calculation of the weighted weight, the superscript T represents performing a matrix transpose operation on the parameters it contains, K is the number of channels of the feature map, and Softmax is a function that normalizes the feature values in the specified channel to obtain the probability ratio of each parameter in the channel.

[0028] After obtaining the weighted feature map, GlobalAveragePooling is used to obtain the activation mean of each channel, which refers to taking the mean of all data in each channel of each sample output, and can represent the concentration degree of semantic information from the channel level. Taking the mean of the result with respect to the sample can obtain the semantic understanding distribution of the model on the data set, and the semantic intensity can be judged from the response intensity of the obtained mean.

[0029] Further, the perturbation parameter σ in the step (3) is initialized as 0 value, and the perturbed data itself will be fitted in the training process, and the fitting objective function formula is as follows:

[0030]

[0031] Where x is the input data, F l () is the neural network layer operator before l layer, F l (x) is the Feats tea [l], p and q are respectively the symbols representing the distribution of feature map and the distribution of perturbation parameter, D KL is the Kullback-Leible divergence mutual entropy objective function, and its formula is as follows:

[0032]

[0033] The Gaussian distribution variable ε is a variable subject to the distribution , which is randomly sampled and used for calculation each time.

[0034] Further, in the step (4), after adding the perturbation, the process of M tea is continued, and the perturbation information is given by σ l and ε, and its formula is as follows:

[0035]

[0036] Where L is the total number of M tea feature maps, F l→output is all neural network operators from l layer to the final output.

[0037] The original target task loss refers to the loss loss custom of the target task of user demand migration, and the combined loss refers to the weighted sum, and its formula is as follows:

[0038]

[0039] Where α l , β l are parameters for controlling the loss weight of the lth layer.

[0040] Further, in the step (5), since the perturbation parameter σ l is required to increase its own information while reducing the influence on the final output in the training process, after the training is completed, the perturbation parameter σ l represents the maximum tolerable perturbation of its perturbation corresponding position, Channel robustThe index calculation formula of is as follows:

[0041]

[0042] Wherein var() is variance, max() is maximum, 1() indicates that 1 is set if the condition is met, k indicates the index of the channel, The square value of the kth channel of the disturbance parameter of the lth layer is identified, which can be regarded as variance since the disturbance parameter has no horizontal and vertical variables. The relative one that is not set to 1 is the unstable channel, which is marked as Channel non-robust .

[0043] Further, the strengthening operation in step (6) refers to assigning a higher weight value when calculating the loss related to the strengthened channel.

[0044] The channel weighting parameter calculation is completed using softmax based on the channel response intensity, and the calculation formula is as follows:

[0045]

[0046] Wherein z represents the channel response intensity, j represents the jth channel, J represents the remaining channels that need to be calculated, and the summary of softmax, wherein z j represents the mean of the activation feature value of channel j, the numerator represents the mapping of the channel response intensity in the exponential domain, and the denominator represents the sum of the exponential domain mappings of all channels that need to be calculated. The ratio of the intensity of the jth channel in all channels can be obtained by comparing the two.

[0047] Further, the feature map set Feats tea and Feats stu have a more obvious reusable paradigm in the deep learning model processing flow. The final output of each reusable paradigm is extracted as the output of a layer to form a set.

[0048] The final output probability distribution Logits stu and Logits tea are the estimated values of the correct probability of each class before the neural network model outputs the final result.

[0049] Further, the random mask masking measure in step (8) refers to multiplying the channels that need to be masked by a very small weight value during the loss calculation process of the masked channels to mask the loss calculation process. The mean square error calculation formula is as follows, wherein the weight in step (6) is used for weighted loss calculation during the loss calculation process of the channel:

[0050]

[0051] wherein J represents the total number of all channels after removing the shielded channels, L represents the total number of the aforementioned reusable paradigm feature layers, N represents the number of training data, and λ l is an optional parameter, which can set the proportion of the loss of different layer features in the total loss, represents the channel weight of the jth channel of the lth layer, Feats tea [i][l][j] and Feats stu [i][l][j] respectively represent the feature maps of the jth channel in the lth layer of the ith sample respectively after processing in the model tea and M stu and M tea and M stu the weighted square sum of the difference of all pixels at each position of each channel feature map of each layer output by all samples.

[0052] the Logits stu The calculation of the original objective function refers to the objective function applied to the target field required by the user, i.e., Custom loss , which is defined and calculated by the user.

[0053] The weighting parameters include the layer weight parameters λ l and the loss α that controls the proportion of the distillation part and the original target loss, and the formula is as follows:

[0054] Total loss =α×Distill loss +Custom loss

[0055] The backpropagation training student model refers to the method of optimizing the model by using the error backpropagation optimization algorithm, and the selected optimization function can be defined by the user.

[0056] The advantages of the present application are as follows:

[0057] 1. The teacher network channel is enhanced by screening from the perspective of the influence of all channels on the final output. First, the channels that can significantly affect the final output by disturbing within the channel can be screened out. Such channels are most likely to contain backdoor trigger patterns, and are removed to avoid backdoor transmission.

[0058] 2. The possibility of learning useless information by the student network can be reduced by learning through the screened channels, thereby reducing the possibility of error information being transmitted from an untrusted model to a trained model.

[0059] 3. The above process can be completed at the same time in the process of improving the precision, which helps to reduce the transfer of pre-training model overfitting information and ensures the refinement of the trained model learning;

[0060] 4. Using random mask to shield part of the channel during student model learning process and switching training between different rounds helps the student model to learn more accurate semantic information from all channels, thereby improving the precision. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is the flowchart of the present application.

[0062] Figure 2 is the channel reinforcement screening process described in the present application.

[0063] Figure 3 is the training flowchart of the disturbance parameter information described in the present application.

[0064] Figure 4 is the training flowchart of the weighted channel rain random mask regular learning described in the present application. DETAILED DESCRIPTION

[0065] The present application will be further described in detail below in conjunction with the accompanying drawings, and the examples are only used to explain the present application, and are not used to limit the scope of the present application.

[0066] The embodiment of the present application provides a model security migration method based on iterative screening and random mask learning, and the flowchart is as shown in Figure 1 , which comprises the following steps:

[0067] (1) Select and initialize a large model in the upper application task to which the target application task belongs as a teacher model M tea , download and use the pre-training model weight stored in the public platform to cover the parameter weight of the teacher model M tea , fix the parameter weight after covering, select and initialize a small model with the same output stage number as M tea or a model consistent with M tea as a student model M stu , initialize the safe target data set without malicious samples for M stu scene, create a data set loader after necessary preprocessing; the specific description is as follows:

[0068] (1a) The download platform of the pre-training model of the model M tea selects an open source or commercial pre-training model weight providing platform, but for its training data source, training method and training parameter, the model weight is regarded as an untrusted source for subsequent processing;

[0069] (1b) the model Mstu equal to or weaker than the teacher model M tea , and the total number of output stages is equal to M tea , for example, when the target application task is a dense small target detection task or a multi-instance small target segmentation task, the upper-level application task is generally an image classification task, and at this time, the large model can refer to a large image classification neural network model such as BASIC-L, ViT- e , etc. trained using a large-scale dataset and having a large number of parameters.

[0070] (1c) The target dataset refers to the dataset of the target task to which the pre-trained model needs to be migrated in this embodiment.

[0071] (1d) The preprocessing step refers to the operation of pre-processing the dataset before training, including standardization, denoising, data enhancement, etc.

[0072] (2) Take 200 target dataset input samples and input them into M tea , and calculate the output feature map set Feats tea at each stage, respectively use self-attention mechanism to calculate the output internal correlation weight, and use global average pooling GlobalAveragePooling to obtain the activation mean value of each channel, marked as H ilk is the activation feature value mean of channel k of the feature map corresponding to the middle level l of the training sample i, and then for each middle level l, the global average response intensity H lk of each channel k of the feature map corresponding to the middle level l is obtained by averaging the activation feature value mean of each channel k of the feature map corresponding to the middle level l of the plurality of training samples. ilk If H lk is lower than H low , the channel k of the feature map corresponding to the middle level l of the training sample i is marked as Channel high , otherwise it is marked as Channel tea ; the process is shown in Figure 2 , and the specific description is as follows:

[0073] (2a) The feature map set Feats tea is a deep learning model processing flow with a more obvious reusable paradigm, and the final output of each reusable paradigm is extracted as the output of a layer to form a set.

[0074] (2b) The internal correlation weight refers to the calculation of the correlation within the data of the feature map, and the calculation process is to multiply the feature map transpose to calculate the probability distribution, and then multiply the feature map itself to obtain the weighted feature map. The calculation formula is as follows:

[0075]

[0076] wherein Feats is the feature map participating in the calculation of the weighted weight, the superscript T represents performing a matrix transposition operation on the parameters contained therein, K is the number of channels of the feature map, and Softmax is a function for obtaining the probability proportion of each parameter within the channel by normalizing the feature value on the specified channel.

[0077] (2c) After obtaining the weighted feature map, GlobalAveragePooling is used to obtain the activation mean of each channel, which means that the mean of all data in each channel of each sample output is calculated, and the concentration degree of semantic information can be represented from the channel level. The semantic understanding distribution of the data set by the model can be obtained by calculating the mean of the result for each sample, and the semantic intensity can be judged from the response intensity of the calculated mean.

[0078] (3) Initialize the perturbation parameter set σ, which is used to perturb M tea Each stage output value is perturbed, and the perturbation value is a trainable parameter, wherein the perturbation parameter σ l of the current layer l is fitted to the distribution of the perturbed feature, and the fitting method is to use the KL divergence to fit the perturbation parameter σ l to the distribution of the corresponding layer feature map, and the fitting loss is marked as Initialize the Gaussian distribution variable ε, which is used to randomly sample the perturbation parameter σ; the specific description is as follows:

[0079] (3a) The perturbation parameter σ is initialized to 0 value, which will be fitted to the perturbed data itself in the training process, and the fitting target function formula is as follows:

[0080]

[0081] wherein x is the input data, F l is the operator of the previous neural network layer up to layer l, F l (x) is the Feats tea [l], and p and q are the notations representing the distribution of the feature map and the distribution of the perturbation parameter, respectively, D KL is the Kullback-Leible divergence mutual entropy objective function, and its formula is as follows:

[0082]

[0083] (3b) The Gaussian distribution variable ε is a variable subject to the distribution , and the variable randomly obtains sampling data for calculation each time it is used.

[0084] (4) M teaFeats, the feature map of level l tea [l] Add the perturbation parameter σ to the corresponding layer sampling. l The perturbated feature map is then placed back into M. tea The original hierarchy continues to reason to obtain the final output of the attacked entity. The loss of the original target task is obtained by comparing the true labels of the samples in the target application dataset. The purpose of calculating this loss is to keep the output of the model unchanged after the perturbation by reducing this loss, combined with the loss from the fitted data distribution. Obtain the total loss value and backpropagate the gradient to the perturbation parameter σ of layer l. l Training stops after a specified number of iterations for M. tea The above training is performed on the output of each layer to obtain the perturbation parameters σ of each layer after training. l The set of perturbation parameters σ is composed of the following; its training process is as follows: Figure 3 As shown, the specific explanation is as follows:

[0085] (4a) During the process of adding perturbation information to the output feature map, the perturbation information is changed by σ. l Given together with ε, its formula is as follows:

[0086]

[0087] Where L is M tea Total number of feature map stages, F l→output It refers to all neural network operators from layer l onwards to the final output;

[0088] (4b) Calculating the original target task loss refers to the loss of the target task that needs to be migrated in this embodiment. custom The combined loss refers to the weighted summation, and its formula is as follows:

[0089]

[0090] Where α l ,β l It is a parameter that controls the loss weights of the l-th layer.

[0091] (5) The perturbation parameters σ trained at each layer l This can be considered as the maximum disturbance that different channels can withstand while maintaining the goal of constant output, and the hierarchical disturbance parameter σ. l Channels whose variance is greater than the variance of the corresponding channel activation values ​​in the data samples used during training are considered safe and stable and are marked as Channels. robust The remaining channels are then labeled as Channel. non-robust The specific details are as follows:

[0092] (5a)Channel robust The index calculation formula of the channel is as follows:

[0093]

[0094] wherein var() is the variance, max() is the maximum value, 1() indicates that 1 is set if the condition is met, k represents the index of the channel, The square value of the kth channel of the disturbance parameter of the lth layer is identified, since the disturbance parameter has no horizontal and vertical variable, it can be regarded as a variance, and the one that is not set to 1 is the unstable channel, which is marked as Channel non-robust .

[0095] (6) Combine steps (2) and (5) to produce a screening and strengthening of the channel, wherein the channel that meets Channel high and Channel robust is given a high calculation weight, which will have a greater impact on the student model M stu in the subsequent loss calculation process, thereby strengthening, and the channel that meets Channel low and Channel non-robust is masked using a mask to delete it in subsequent learning, and the remaining channels use the channel mean of the response strength produced in step (2) as the channel weighting parameter Channel weight ; The specific description is as follows:

[0096] (6a) The strengthening operation means that the weight is doubled when calculating the loss related to the strengthened channel; the channel weighting parameter is calculated using softmax based on the channel response strength, and the calculation formula is as follows:

[0097]

[0098] wherein z represents the channel response strength, j represents the jth channel, J represents the total number of channels that need to be calculated using softmax, wherein z j represents the activation feature value mean of channel j, the numerator represents the mapping of the corresponding strength of the channel in the exponential domain, and the denominator represents the sum of the exponential domain mappings of all channels, and the ratio of the two can obtain the proportion of the strength of the jth channel in all channels.

[0099] (7) The target domain data set is input into the teacher model and the student model, respectively, to obtain the feature map in the middle layer at the same stage, to produce the teacher feature map set Feats tea and the student feature map set Feats stu , and the final output probability distribution Logits teaWith Logits stu ;

[0100] (8) Feats tea Each channel is processed according to the channel processing parameters obtained in step (6), that is, based on the Channel... high With Channel robust Enhancement parameters, based on Channel low With Channel non-robust The mask parameters and the weighted parameters of each channel. weight For Feats stu The output of each level is based on the removed Feats. tea Given the total number of channels d in [l], use a random mask to block the corresponding number of channels d. Then, for the Feats after removing the masked channels... tea With Feats stu The mean squared error loss is calculated for the remaining unmasked channels. distill Combined with Logits stu Calculate the target task loss Loss distill and The weighted summation and backpropagation of the loss gradient to the student model M stu Training is performed, and after a specified number of training iterations, the channel mask is re-randomized until M is reached. stu Convergence or training completion; the model M obtained after training. stu That is, to use a large model M tea A secure model is generated after a secure migration. The process is as follows: Figure 4 As shown, the specific explanation is as follows:

[0101] (8a) Random masking refers to multiplying the channel to be masked by a minimum weight during the loss calculation process of the masked channel to mask its loss calculation process. The formula for calculating the mean square error is shown below. In the process of calculating the loss of the channel, the weighted loss calculation is performed using the weights defined in step (6):

[0102]

[0103] Where J represents the total number of channels after removing the masked channels, L represents the total number of reusable paradigm feature layers mentioned above, N represents the number of training data sets, and λ l This is an optional parameter that allows you to set the proportion of feature loss from different layers in the total loss. Feats represents the channel weight of the j-th channel in the l-th layer. tea [i][l][j] and Feats stu[i][l][j] respectively represent the i-th sample respectively in the M tea with M stu the feature map of the j-th channel in the l-th layer after processing within the model, the above formula represents the teacher model M tea and the student model M stu the weighted sum of squares of the difference of all pixels in each position of each channel feature map of each layer output by the student model;

[0104] (8b) Logits stu The calculation of the original objective function refers to the implementation of the objective function in the field required for example transfer, i.e. Custom loss , which is defined and calculated by the user;

[0105] (8c) The weighting parameters in the calculation include the level weight parameters λ l and the loss α that controls the proportion of the distillation part and the original target loss, whose formula is as follows:

[0106] Total loss = α × Distill loss + Custom loss

[0107] (8d) The backpropagation training of the student model refers to the method of optimizing the model using the error backpropagation optimization algorithm, and the optimization function selected is defined by the embodiment.

[0108] Although the specific embodiments of the present application are disclosed for the purpose of illustrating the present application and helping to understand and implement the present application, those skilled in the art can understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed content of the best embodiment, and the scope of the present application is defined by the scope of the claims.

Claims

1. A model security migration method based on iterative screening and random mask learning, comprising the steps of: 1) selecting and initializing a large model corresponding to a superior application task of a target application task as a teacher model M tea , obtaining the pre-training model weight of the teacher model M tea , covering the parameter weight of the teacher model M tea ; selecting and initializing a model as a student model M stu ; the output stage number of the student model M stu is consistent with the teacher model M tea ; 2) inputting a plurality of training samples in the selected target dataset to the teacher model M tea ; for each training sample input, obtaining a feature map set Feats tea consisting of the intermediate level outputs of the teacher model M tea , and for each feature map in the feature map set Feats tea , using a self-attention mechanism to calculate the internal correlation weight of the feature map, and the average value of the activation feature values of different channels of the feature map is obtained by using global average pooling, H ilk is the average value of the activation feature values of the channel k of the feature map corresponding to the intermediate level l in the training sample i. Then for each intermediate level l, the global average response strength H of each channel k of the feature map corresponding to the intermediate level l is obtained by sample-level averaging the average of the activation feature values of each channel k of the feature map corresponding to the intermediate level l according to the plurality of training samples lk If H ilk is lower than H lk , the channel k of the feature map corresponding to the intermediate level l in the training sample i is marked as Channel low , otherwise, it is marked as Channel high ; 3) initialize a set of perturbation parameters σ for perturbing the teacher model M tea each intermediate level output value is perturbed; 4) using the perturbation parameter σ of the intermediate level l l the feature map Feats corresponding to the intermediate level l tea [l] is perturbed, and the perturbation loss before and after the perturbation is calculated the teacher model M tea the perturbed feature map Feats corresponding to the training sample i tea [l] completes the target application task inference to obtain the output According to the task loss corresponding to the intermediate level l is calculated according to the label corresponding to the training sample i According to the perturbation loss corresponding to the training sample i the task loss the loss value corresponding to the intermediate level l is calculated according to the perturbation parameter σ l optimization; 5) If the optimized perturbation parameter σ l If the variance value of the activation feature value mean of the middle channel k is greater than the variance value of the activation feature value mean of the middle level l corresponding feature map of the middle channel k during training, it is determined that the channel k is a safe stable channel and is marked as Channel robust , otherwise it is determined that the channel k is an unsafe stable channel and is marked as Channel non-robust ; 6) to meet Channel high and Channel robust , a high weight is set for the corresponding channel to strengthen the corresponding channel, to meet Channel low and Channel non-robust , a mask is set for the corresponding channel to delete the corresponding channel, and the remaining channels use the mean value of the activation feature values of the corresponding channels as the weight of the corresponding channel; the high weight of the channel is greater than the mean value of the activation feature values of the channel. 7) inputting a sample of the selected target dataset into the teacher model M tea and the student model M stu , obtaining the feature map set Feats tea composed of the feature maps output by each intermediate layer of the teacher model M tea , and the probability distribution Logits tea output by the last layer of the teacher model M tea , obtaining the feature map set Feats stu composed of the feature maps output by each intermediate layer of the student model M stu , and the probability distribution Logits stu output by the last layer of the student model M stu ; 8) Use the channel settings determined in step 6) to configure the Feats tea Each channel of each feature map is processed accordingly, and then Feats are calculated. tea With Feats stu The mean squared error loss between feature maps output from the same intermediate layer is used to obtain the loss value Loss by weighted summation. distill Combined with Logits stu Calculate the loss value for the target application task. Then regarding Loss distill and Total loss value obtained by weighted summation loss Optimize the student model M stu 9) sequentially repeating steps 7)~8) using samples within the target dataset until the student model M stu converges, and the student model M stu as a security model generated after the security migration of the large model.

2. The method of claim 1, wherein, Disturbance loss where x is the input data, F l (x) is the feature map of the intermediate level l of the teacher model M tea corresponding to the intermediate level l of the teacher model M tea [l], p is the distribution of the feature map, q is the distribution of the disturbance parameter, D KL is the Kullback-Leible divergence cross-entropy objective function.

3. The method of claim 1, wherein, The fitting mode is to use KL divergence to disturb the parameter σ l Distributed fitting is performed on the feature map output by the intermediate level l; Gaussian distribution variable ε is initialized, and ε multiplied by the disturbance parameter σ is randomly sampled l The sampling disturbance is performed.

4. The method of claim 3, wherein, Using KL divergence on the perturbation parameter s l Distribution fitting on the feature maps output by the intermediate level l to get the perturbed feature maps where F l (x) is the feature map Feats tea [l] corresponding to the intermediate level l of the teacher model M tea [l] of the teacher model M 5. The method of claim 1, wherein, Step 6) the weight of the remaining channel j where z j denotes the activation feature value mean of channel j, J denotes the total number of remaining channels.

6. The method of claim 1, wherein, The student model M stu is equal or weaker in model capacity than the teacher model M tea .

7. The method of claim 1, wherein, The target data set refers to the data set that the user needs to migrate the pre-trained model to the target application task.

8. A server, characterized by A computer program is stored in a memory and executed by a processor, and the computer program comprises instructions for executing the steps of any one of the methods of claims 1 to 7.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of any one of the methods of claims 1 to 7.

Images