Task execution method based on robust distillation, electronic equipment and medium

Abstract

The invention discloses a task execution method based on robust distillation, electronic equipment and a medium, and the method comprises the steps: inputting a clean data sample into a teacher model, and carrying out the detoxification to obtain a detoxification feature corresponding to each distillation layer; the first loss function is the variance of the detoxification feature corresponding to the last distillation layer of the teacher model; the second loss function is the distance between the detoxification feature corresponding to each distillation layer in the teacher model and the feature representation output by the corresponding distillation layer in the student model; the third loss function is cross entropy loss calculated based on the detoxication feature corresponding to the last distillation layer of the teacher model and the label corresponding to the clean data sample; taking the sum of the first, second and third loss functions as a total loss function; training a teacher model and a student model by using the total loss function; taking the trained student model as a task execution model; wherein the task execution model is used for executing a downstream target task.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a task execution method, electronic equipment, and medium based on robust distillation. Background Art

[0002] At present, deep learning has achieved great success in academia and industry, and the fundamental reason lies in its scalability and ability to encode large-scale data. However, the main challenge of deep learning is that due to the limitation of resource capacity, deep neural models are difficult to deploy on resource-constrained devices. Such as embedded devices and mobile devices. Therefore, a large number of model compression and acceleration technologies have emerged, and knowledge distillation is one of them. Knowledge distillation can effectively learn a small student model from a large teacher model. Recent work has shown that backdoors in the teacher model are transferred to the student model during the knowledge distillation process. Although many knowledge distillation methods have been proposed, most of them focus on the distillation of high-performance student models without considering robustness. In addition, some studies have adopted knowledge distillation technology as an effective backdoor mitigation tool, but they fail to perform model compression at the same time. In practice, trained teacher models can be easily downloaded from third-party websites, such as Hugging Face. However, shared models on third-party websites cannot guarantee the security check of the system, which brings potential vulnerability risks to users. For example, models with backdoors are released and downloaded for use, and attackers may steal sensitive data such as user personal information, trade secrets or confidential information, thereby causing serious data leakage incidents. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a task execution method, electronic device, and medium based on robust distillation.

[0004] In a first aspect, an embodiment of the present invention provides a task execution method based on robust distillation, the method comprising:

[0005] Input the clean data samples into the teacher model, and use the output features corresponding to each clean data sample in each distillation layer of the teacher model as the original features; sample the original features corresponding to each distillation layer, and use the average value of the original features as the initialization feature mask of the distillation layer;

[0006] Set the mask threshold, detoxify the initialized feature mask corresponding to each distillation layer, and obtain the detoxified feature corresponding to the distillation layer;

[0007] Setting a first loss function, where the first loss function is the variance of the detoxified features corresponding to the last distillation layer of the teacher model;

[0008] Setting a second loss function, where the second loss function is the distance between the detoxified features corresponding to each distillation layer of the teacher model and the feature representation output by the corresponding distillation layer in the student model;

[0009] Setting a third loss function, wherein the third loss function is a cross entropy loss calculated based on the detoxification features corresponding to the last distillation layer of the teacher model and the labels corresponding to the clean data samples;

[0010] The sum of the first loss function, the second loss function, and the third loss function is used as the total loss function; the total loss function is used to train the teacher model and the student model;

[0011] The trained student model is used as a task execution model; wherein the task execution model is used to execute downstream target tasks.

[0012] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned task execution method based on robust distillation.

[0013] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned task execution method based on robust distillation.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned task execution method based on robust distillation.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention provides a task execution method based on robust distillation, which sets a loss function according to the variance of the detoxified features corresponding to the last distillation layer of a teacher model, and sets a loss function according to the distance between the detoxified features corresponding to each distillation layer of the teacher model and the feature representation output by the corresponding distillation layer in the student model. The teacher model and the student model are trained by the loss function to obtain a robust student model with potential backdoors removed. The trained student model is used as a task execution model, thereby protecting privacy data and preventing data leakage during the execution of downstream target tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 A flowchart of a task execution method based on robust distillation provided by an embodiment of the present invention;

[0019] Figure 2 A schematic block diagram of a task execution method based on robust distillation provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0023] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a task execution method based on robust distillation, the method comprising:

[0024] Step S1, input the clean data samples into the teacher model, and use the output features corresponding to each clean data sample in each distillation layer of the teacher model as the original features; sample the original features corresponding to each distillation layer, and use the average value of the original features as the initialization feature mask of the distillation layer; set the mask threshold, detoxify the initialization feature mask corresponding to each distillation layer, and obtain the detoxified features corresponding to the distillation layer.

[0025] Specifically, the step S1 comprises the following steps:

[0026] Step S101, select some clean data samples from the clean dataset; input the clean data samples into the teacher model, obtain the output features of each clean data sample in each distillation layer of the teacher model, and use them as the original features.

[0027] Step S102, sampling the original features corresponding to each distillation layer, and using the average value of the original features as the initialization feature mask of the distillation layer.

[0028] Step S103, setting a mask threshold, detoxifying the initialized feature mask corresponding to each distillation layer, and obtaining the detoxified feature corresponding to the distillation layer.

[0029] Specifically, the expression of the detoxification process is as follows:

[0030] F t '=F t +p*m

[0031] Where p*m represents the detoxification feature mask, and p is the initialization feature mask F t A tensor of the same dimension, m is the mask threshold, F t ' represents the detoxification features corresponding to each distillation layer of the teacher model, F t Represents the initialized feature mask corresponding to each distillation layer of the teacher model.

[0032] It should be noted that when the student model learns the features of the teacher model, it will lead to the propagation of backdoors. In this example, the features of the teacher model are carefully detoxified to ensure the accuracy of the teacher model on the clean data sample by eliminating the potential knowledge caused by the potential backdoor in the features, laying the foundation for the subsequent feature distillation to use the purified detoxified features to ensure the performance of the student model.

[0033] Step S2, setting a first loss function, the first loss function is the variance of the detoxified features corresponding to the last distillation layer of the teacher model; setting a second loss function, the second loss function is the distance between the detoxified features corresponding to each distillation layer of the teacher model and the feature representation output by the corresponding distillation layer in the student model; setting a third loss function, the third loss function is the cross entropy loss calculated based on the detoxified features corresponding to the last distillation layer of the teacher model and the labels corresponding to the clean data samples; taking the sum of the first loss function, the second loss function, and the third loss function as the total loss function; and using the total loss function to train the teacher model and the student model.

[0034] The expression of the first loss function is as follows:

[0035] L f =Var(F T ')

[0036] In the formula, Var(·) represents the variance operation, F T ' represents the detoxified features corresponding to the last distillation layer of the teacher model.

[0037] It should be noted that after being attacked by a backdoor, the variance of feature values ​​within the model increases significantly. This deviates significantly from the variance observed for the same sample when it is not subjected to such an attack. A possible explanation is that the backdoor model tends to pay special attention to the backdoor region in the sample, which is different from the more diffuse attention shown by the clean model. This selective attention to the backdoor region is believed to be a factor leading to the observed escalation in feature variance, indicating that the model is more sensitive to and attaches more importance to the components in the data that affect the backdoor. Therefore, in order to reduce potential backdoors in the teacher model while ensuring that the distillation process enables the student model to learn entirely from clean features, a first loss is designed in this example to optimize the detoxified features.

[0038] The expression of the second loss function is as follows:

[0039] L distill =d p (δ mC (F t '),r(F s ))

[0040] Where, d p (.) represents the distance function, represents the teacher transformation, r(·) represents the student transformation, and F t ' represents the detoxification features corresponding to each distillation layer of the teacher model, F s Represents the feature representation of the corresponding distillation layer output in the student model.

[0041] Among them, the distance function d p The definition of (.) is as follows:

[0042]

[0043] In the formula, T represents the teacher feature, S represents the student feature, i represents the i-th component, W represents the width, H represents the height, C represents the number of channels, and T i represents the i-th component of the teacher’s characteristics, S i represents the i-th component of the student characteristics, S i ∈R.

[0044] The teacher changes Use marginal ReLU function;

[0045]

[0046] In the formula, m c represents the channel-by-channel expected value of the negative response;

[0047] For channel C and the detoxified features F corresponding to each distillation layer of the teacher modelt i The i-th element, channel m C The edge values ​​are set to:

[0048]

[0049] The student transform r(·) adopts a 1×1 convolutional layer.

[0050] It should be noted that in order to match the dimensions between the teacher features and the student features, the teacher features and the student features need to be transformed respectively, and the distance between the transformed teacher features and the transformed student features is used as the second loss function for robust distillation.

[0051] Furthermore, the teacher transformation converts the teacher features into a form that is easy to transmit, which will lead to information loss in distillation; since the features include positive information and negative information, it is very important to distinguish them and avoid missing favorable information; through the marginal ReLU proposed in the present invention, the positive information can be used without any conversion, while the negative information is suppressed, and distillation can be performed without losing the positive information. A 1×1 convolutional layer is used as the student transformation to match the dimension of the teacher features, so that the student feature size will not decrease, but increase, so there will be no information loss.

[0052] Furthermore, the location where distillation occurs is also critical. FitNets uses the end of any intermediate layer as the distillation point, which has been shown to perform poorly. In the present invention, a group of layers with the same spatial size is called a layer group. ReLU allows beneficial information to pass through and filters out unfavorable information. Therefore, knowledge distillation must be designed with this information dissipation in mind. This example brings the features before the ReLU function, called pre-ReLU. Positive and negative values ​​are retained in the position before ReLU without any deformation.

[0053] Further, the present invention designs an appropriate distance function based on the teacher transformation and the distillation point in the pre-ReLU position. The pre-ReLU information is transferred from the teacher model to the student model, but the negative values ​​of the pre-ReLU features contain unfavorable information. The negative values ​​of the Pre-ReLU features are blocked by the ReLU activation and are not used by the teacher network. The transfer of all knowledge may have an adverse effect on the student network. The distance function provided by the present invention can skip the information distillation on the negative area, and in the features of the teacher model, the positive response is actually used for the network, which means that the positive response of the teacher model should be passed through its exact value. However, this is not the case for negative responses. For negative teacher responses, if the response of the student model is higher than the target value, the value should be reduced, but if the response of the student model is lower than the target value, there is no need to increase it.

[0054] The expression of the third loss function is as follows:

[0055]

[0056] In the formula, i represents the serial number of the clean data sample, N represents the number of clean data samples, and y i represents the label corresponding to the i-th clean data sample, B(·) represents the output of the fully connected layer, and F T ' represents the detoxified features corresponding to the last distillation layer of the teacher model.

[0057] The expression of the total loss function is as follows:

[0058] L t =L distill +L ce +L f

[0059] Where, L f represents the first loss function, which is used to eliminate the potential backdoor “dark knowledge” of the teacher model; L distill represents the second loss function, which is used to achieve robust distillation; L ce Represents the third loss function, which is used to ensure the main task performance of the teacher model, thereby ensuring the performance of the student model.

[0060] Step S3, using the trained student model as a task execution model; wherein the task execution model is used to execute downstream target tasks.

[0061] Example 1

[0062] In this example, taking the downstream target task as an image classification task as an example, the effectiveness of a task execution method based on robust distillation provided by the present invention is explained.

[0063] The present invention uses a CIFAR-100 image dataset as a training set. The first group uses a WRN 28-4 model as a teacher model and a WRN 16-2 model as a student model. The second group uses a WRN 28-4 model as a teacher model and a ResNet 56 model as a student model. The third group uses a Pyramid-200 model as a teacher model and a WRN 28-4 model as a student model. The fourth group uses a Pyramid-200 model as a teacher model and a Pyramid-110 model as a student model to implement the task execution method based on robust distillation provided by the present invention.

[0064] The comparison results of the accuracy (ACC) and backdoor attack success rate (Attack success rate, ASR) of the student model after robust feature distillation of the method of the present invention and the accuracy and backdoor attack success rate of the student model after feature distillation of the existing method are shown in the following Table 1:

[0065] Table 1: Results of the student model’s accuracy and backdoor attack success rate

[0066]

[0067] As shown in Table 1, the student models using robust distillation generally achieved at least 80% detoxification effect, and the highest detoxification effect was 90%. Although there are still a few backdoors in the student models, these backdoors are not enough to pose a serious security threat to the model.

[0068] In summary, the present invention provides a task execution method based on robust distillation, which sets a loss function according to the variance of the detoxified features corresponding to the last distillation layer of the teacher model, and sets a loss function according to the distance between the detoxified features corresponding to each distillation layer of the teacher model and the feature representation output by the corresponding distillation layer in the student model. The teacher model and the student model are trained by the loss function to obtain a robust student model with potential backdoors removed; the trained student model is used as a task execution model, thereby protecting privacy data and preventing data leakage during the execution of downstream target tasks. The present invention helps to improve the security of the entire software ecosystem and protect developers and end users from attacks.

[0069] like Figure 3 As shown, an embodiment of the present application provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method in the first aspect described above is implemented.

[0070] The memory 101, the processor 102 and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0071] Among them, the memory 101 can be, but is not limited to, a random access memory 101 (Random Access Memory, RAM), a read only memory 101 (Read Only Memory, ROM), a programmable read-only memory 101 (Programmable Read-Only Memory, PROM), an erasable read-only memory 101 (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable read-only memory 101 (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0072] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor 102, including a central processing unit 102 (CPU), a network processor 102 (NP), etc.; it may also be a digital signal processor 102 (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0073] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0074] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0075] On the other hand, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 102, a method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory 101 (ROM, Read-Only Memory), a random access memory 101 (RAM, RandomAccess Memory), a disk or an optical disk.

[0076] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A task execution method based on robust distillation, characterized in that: The method comprises: Input the clean data samples into the teacher model, and use the output features corresponding to each clean data sample in each distillation layer of the teacher model as the original features; sample the original features corresponding to each distillation layer, and use the average value of the original features as the initialization feature mask of the distillation layer; Set the mask threshold, detoxify the initialized feature mask corresponding to each distillation layer, and obtain the detoxified feature corresponding to the distillation layer; Setting a first loss function, where the first loss function is the variance of the detoxified features corresponding to the last distillation layer of the teacher model; Setting a second loss function, where the second loss function is the distance between the detoxified features corresponding to each distillation layer of the teacher model and the feature representation output by the corresponding distillation layer in the student model; Setting a third loss function, wherein the third loss function is a cross entropy loss calculated based on the detoxification features corresponding to the last distillation layer of the teacher model and the labels corresponding to the clean data samples; The sum of the first loss function, the second loss function, and the third loss function is used as the total loss function; the total loss function is used to train the teacher model and the student model; The trained student model is used as a task execution model; wherein the task execution model is used to execute downstream target tasks.

2. A task execution method based on robust distillation according to claim 1, characterized in that: Detoxification of the initialized feature mask corresponding to each distillation layer includes: F t '=F t +p*m Where p*m represents the detoxification feature mask, and p is the initialization feature mask F t A tensor of the same dimension, m is the mask threshold, F t ' represents the detoxification features corresponding to each distillation layer of the teacher model, F t Represents the initialized feature mask corresponding to each distillation layer of the teacher model.

3. The task execution method based on robust distillation according to claim 1, characterized in that: The expression of the first loss function is as follows: L f =There is(F T ') In the formula, Var(·) represents the variance operation, F T ' represents the detoxified features corresponding to the last distillation layer of the teacher model.

4. The task execution method based on robust distillation according to claim 1, characterized in that: The expression of the second loss function is as follows: Where, d p (.) represents the distance function, represents the teacher transformation, r(·) represents the student transformation, and F t ' represents the detoxification features corresponding to each distillation layer of the teacher model, F s Represents the feature representation of the corresponding distillation layer output in the student model.

5. A task execution method based on robust distillation according to claim 4, characterized in that: Distance function d p The definition of (.) is as follows: In the formula, T represents the teacher feature, S represents the student feature, i represents the i-th component, W represents the width, H represents the height, C represents the number of channels, and T i represents the i-th component of the teacher’s characteristics, S i represents the i-th component of the student characteristics, S i ∈R.

6. A task execution method based on robust distillation according to claim 4, characterized in that: The teacher changes Use marginal ReLU function; In the formula, m c represents the channel-by-channel expected value of the negative response; For channel C and the detoxified features F corresponding to each distillation layer of the teacher model t i The i-th element, channel m C The edge values ​​are set to: m C =E[F t i |F t i <0,i∈C] The student transform r(·) adopts a 1×1 convolutional layer.

7. The task execution method based on robust distillation according to claim 1, characterized in that: The expression of the third loss function is as follows: In the formula, i represents the serial number of the clean data sample, N represents the number of clean data samples, and y i represents the label corresponding to the i-th clean data sample, B(·) represents the output of the fully connected layer, and F T ' represents the detoxified features corresponding to the last distillation layer of the teacher model.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the task execution method based on robust distillation as described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the task execution method based on robust distillation as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the task execution method based on robust distillation described in any one of claims 1-7 is implemented.

Images