Model distillation method, device, electronic equipment and readable storage medium

Abstract

The present disclosure relates to a model distillation method, device, electronic equipment and readable storage medium, determining at least one sample image, and a first classification model comprising a first processing layer and a second classification model comprising a same number of second processing layers. And determine the adversarial optimization function of the first classification model and the second classification model. The input parameters of the adversarial optimization function include the first classification result and the second classification result corresponding to the first classification model and the second classification model respectively, and the first feature vector and the second feature vector corresponding to each processing layer. According to at least one sample image, the parameters of the second classification model are adjusted until the adversarial optimization function meets the convergence condition. The present disclosure trains the second classification model through the pre-trained first classification model, and introduces the defense characteristics of the adversarial sample in the training process through the setting of the adversarial optimization function to assist the training, thereby improving the robustness of the second classification model obtained by model distillation in response to the adversarial sample.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a model distillation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Deep learning models used for data classification are vulnerable to adversarial attacks during learning and application. This means that even small perturbations introduced during model learning and application can easily cause classification errors, severely impacting the model's performance. Therefore, it is necessary to defend against adversarial attacks during the use of deep learning models. Summary of the Invention

[0003] In view of this, this disclosure proposes a model distillation method, apparatus, electronic device and readable storage medium, aimed at increasing the stability of the model in the face of adversarial examples.

[0004] According to a first aspect of this disclosure, a model distillation method is provided, the method comprising:

[0005] Identify at least one sample image;

[0006] Determine a first classification model and a second classification model, wherein the first classification model includes the same number of first processing layers as the second classification model includes the same number of second processing layers.

[0007] Determine the adversarial optimization function for the first classification model and the second classification model. The input parameters of the adversarial optimization function include the first classification result and the second classification result obtained by inputting any image into the first classification model and the second classification model, as well as the first feature vector output by each of the first processing layers and the second feature vector output by each of the second processing layers.

[0008] The second classification model is adjusted at least once based on the at least one sample image until the adversarial optimization function meets the convergence condition.

[0009] In one possible implementation, the parameter adjustment process includes:

[0010] The at least one sample image is input into the first classification model, and the corresponding first classification result and the first feature vector output by each of the first processing layers are output.

[0011] For each sample image, when the first classification result and first feature vector corresponding to the sample image, and the model parameters of the second classification model are fixed, the second classification result obtained by inputting different adversarial images into the second classification model, and the second feature vector output by each of the second processing layers are used as the input of the adversarial optimization function, the adversarial image corresponding to the largest minimum adversarial optimization function value is taken as the target adversarial image corresponding to the sample image.

[0012] The weighted sum of the first classification result and first feature vector corresponding to each sample image, the second classification result obtained by inputting the target adversarial image corresponding to the sample image into the second classification model, and the second feature vector output by each of the second processing layers is determined as the input adversarial optimization function;

[0013] Adjust the model parameters of the second classification model to minimize the weighted sum.

[0014] In one possible implementation, the method further includes:

[0015] Determine the noise disturbance threshold;

[0016] The constraint condition for determining the adversarial image is that the difference between the adversarial image and the corresponding sample image is not greater than the noise perturbation threshold.

[0017] In one possible implementation, the difference is determined based on the norm between the adversarial image and the corresponding sample image.

[0018] In one possible implementation, determining the adversarial optimization functions for the first classification model and the second classification model includes:

[0019] Determine a first optimization term to characterize the similarity between the first classification result and the second classification result;

[0020] Determine the transformation function between each first processing layer in the first classification model and the second processing layer in the second classification model corresponding to each first processing layer;

[0021] Each second feature vector is converted into a third feature vector according to the corresponding transformation function;

[0022] Determine a second optimization term to characterize the mean square error between each of the third eigenvectors and the corresponding first eigenvector;

[0023] The weighted sum of the first optimization term and the second optimization term is determined as the adversarial optimization function.

[0024] In one possible implementation, determining the transformation function between each of the first processing layers in the first classification model and the second processing layer in the second classification model corresponding to each of the first processing layers includes:

[0025] The first processing layer and the second processing layer, whose positions in the first classification model are the same as those in the second classification model, are identified as processing layer pairs;

[0026] Obtain the target sample image from the at least one sample image;

[0027] The target sample image is input into the first classification model and the second classification model respectively, and the first feature vector and the second feature vector output by each pair of the first processing layer and the corresponding second processing layer are obtained respectively.

[0028] The corresponding transformation function is determined based on the first feature vector and the second feature vector of each processing layer pair.

[0029] In one possible implementation, determining the corresponding transformation function based on the first feature vector and the second feature vector of each processing layer pair includes:

[0030] Determine the correlation matrix between the first and second feature vectors of each processing layer pair;

[0031] The transformation function of the processing layer pair is determined to be the product of the correlation matrix and the second eigenvector.

[0032] According to a second aspect of this disclosure, a model distillation apparatus is provided, the apparatus comprising:

[0033] Image determination module, used to determine at least one sample image;

[0034] The model determination module is used to determine a first classification model and a second classification model, wherein the first classification model includes the same number of first processing layers as the second classification model includes the same number of second processing layers.

[0035] The function determination module is used to determine the adversarial optimization function of the first classification model and the second classification model. The input parameters of the adversarial optimization function include the first classification result and the second classification result obtained by inputting any image into the first classification model and the second classification model, as well as the first feature vector output by each of the first processing layers and the second feature vector output by each of the second processing layers.

[0036] The model distillation module is used to adjust the parameters of the second classification model at least once based on the at least one sample image until the adversarial optimization function satisfies the convergence condition.

[0037] In one possible implementation, the parameter adjustment process includes:

[0038] The at least one sample image is input into the first classification model, and the corresponding first classification result and the first feature vector output by each of the first processing layers are output.

[0039] For each sample image, when the first classification result and first feature vector corresponding to the sample image, and the model parameters of the second classification model are fixed, the second classification result obtained by inputting different adversarial images into the second classification model, and the second feature vector output by each of the second processing layers are used as the input of the adversarial optimization function, the adversarial image corresponding to the largest minimum adversarial optimization function value is taken as the target adversarial image corresponding to the sample image.

[0040] The weighted sum of the first classification result and first feature vector corresponding to each sample image, the second classification result obtained by inputting the target adversarial image corresponding to the sample image into the second classification model, and the second feature vector output by each of the second processing layers is determined as the input adversarial optimization function;

[0041] Adjust the model parameters of the second classification model to minimize the weighted sum.

[0042] In one possible implementation, the device further includes:

[0043] The threshold determination module is used to determine the noise disturbance threshold.

[0044] The constraint addition module is used to determine the adversarial image by using the condition that the difference between the adversarial image and the corresponding sample image is no greater than the noise perturbation threshold.

[0045] In one possible implementation, the difference is determined based on the norm between the adversarial image and the corresponding sample image.

[0046] In one possible implementation, the function determining module includes:

[0047] The first item determines the submodule, which is used to determine the first optimization term used to characterize the similarity between the first classification result and the second classification result;

[0048] The transformation function determination submodule is used to determine the transformation function between each first processing layer in the first classification model and the second processing layer in the second classification model corresponding to each first processing layer.

[0049] The vector transformation submodule is used to transform each second feature vector into a third feature vector according to the corresponding transformation function;

[0050] The second determination submodule is used to determine a second optimization term for characterizing the mean square error between each of the third feature vectors and the corresponding first feature vectors;

[0051] The function determination submodule is used to determine the weighted sum of the first optimization term and the second optimization term as the adversarial optimization function.

[0052] In one possible implementation, the transformation function determines the submodule, including:

[0053] The processing layer pair determination unit is used to determine the first processing layer and the second processing layer, which have the same position in the first classification model and the second classification model, as a processing layer pair;

[0054] A sample image determination unit is used to obtain a target sample image from the at least one sample image;

[0055] An image input unit is used to input the target sample image into the first classification model and the second classification model respectively, and obtain the first feature vector and the second feature vector output by each pair of the first processing layer and the corresponding second processing layer respectively;

[0056] The function determination unit is used to determine the corresponding transformation function based on the first feature vector and the second feature vector of each processing layer pair.

[0057] In one possible implementation, the function determining unit includes:

[0058] A matrix determination subunit is used to determine the correlation matrix between the first and second feature vectors of each processing layer pair;

[0059] The function determines the sub-unit, which is used to determine that the transformation function of the processing layer pair is the product of the correlation matrix and the second eigenvector.

[0060] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0061] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0062] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0063] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0064] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0065] Figure 1 A flowchart of a model distillation method according to an embodiment of the present disclosure is shown;

[0066] Figure 2 A schematic diagram of a classification model structure according to an embodiment of the present disclosure is shown;

[0067] Figure 3 A schematic diagram of a model distillation process according to an embodiment of the present disclosure is shown;

[0068] Figure 4 A schematic diagram of a model distillation apparatus according to an embodiment of the present disclosure is shown;

[0069] Figure 5 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0070] Figure 6 A schematic diagram of another electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0071] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0072] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0073] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0074] In one possible implementation, the model distillation method of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. The electronic device can implement the model distillation method of this disclosure by having a processor call computer-readable instructions stored in memory.

[0075] Figure 1 A flowchart of a model distillation method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the model distillation method of this disclosure embodiment may include the following steps S10-S40.

[0076] Step S10: Determine at least one sample image.

[0077] In one possible implementation, the electronic device can determine at least one sample image in any manner to perform the model distillation process using the sample image. Optionally, the electronic device can directly receive at least one sample image from other devices, acquire at least one sample image from local memory, or obtain at least one sample image through at least one image acquisition operation performed by an image acquisition device built into or connected to the electronic device. The sample image is an image with corresponding labeled classification results, which can be determined according to the application scenario. For example, in an application scenario requiring the classification of lesions in medical images, the sample image can be any format of medical image obtained from acquiring CT images, MRI images, and X-ray images of any lesion. In an application scenario requiring the classification of animal categories included in an image, the sample image can be an animal image obtained from acquiring images of any animal such as cats, dogs, birds, and fish.

[0078] Optionally, the sample images in this embodiment of the present disclosure may also be feature images obtained by feature extraction of information in formats such as text information, semantic information, and video information, which are used to characterize the features of the corresponding information.

[0079] Step S20: Determine the first classification model and the second classification model.

[0080] In one possible implementation, the electronic device can determine a first classification model as a teacher model and a second classification model as a student model during the model distillation process. The first classification model is a pre-trained model, and the second classification model is a model to be trained. Both the first and second classification models include multiple sequentially connected processing layers, with the output of each layer serving as the output of the next layer, until the final output layer outputs the model result. The training images for the first classification model can be at least one sample image determined in step S10, or other images.

[0081] Optionally, the first classification model includes multiple first processing layers, and the second classification model includes multiple second processing layers, with the number of first processing layers in the first classification model being the same as the number of second processing layers in the second classification model. The first and second processing layers in the first and second classification models each have a corresponding processing order, and the first and second processing layers with the same processing order can be processing layers with a corresponding relationship.

[0082] Optionally, both the first and second classification models are used to classify the input image. During the classification process, each first processing layer of the first classification model can input a corresponding first feature vector, which serves as the input to the next sequential first processing layer, until the first classification result is output through the first output layer. Similarly, each second processing layer of the second classification model can input a corresponding second feature vector, which serves as the input to the next sequential second processing layer, until the second classification result is output through the second output layer. Both the first and second classification results can be vector-based classification results, where each element corresponds to a category, representing the probability that the input image belongs to that category.

[0083] Figure 2 A schematic diagram of a classification model structure according to an embodiment of the present disclosure is shown. Figure 2 As shown, both the first and second classification models include M sequentially ordered processing layers. After the model receives an input image, each processing layer processes the input content in turn and outputs a corresponding feature vector. The feature vector output by an earlier processing layer serves as the input for the next processing layer, until the feature vector M output by the last processing layer passes through the output layer to output the classification result of the entire classification model.

[0084] Step S30: Determine the adversarial optimization functions for the first classification model and the second classification model.

[0085] In one possible implementation, the electronic device can determine an adversarial optimization function based on a first classification model and a second classification model. The adversarial optimization function is used to optimize the second classification model based on a first classification model with fixed parameters, achieving model distillation. It can be determined based on the difference between the outputs of the first and second classification models, and the difference between the output feature vectors of the corresponding first and second processing layers. Optionally, the input parameters of the adversarial optimization function may include the first classification result obtained by inputting any image into the first and second classification models, the second classification result, the first feature vector output by each first processing layer, and the second feature vector output by each second processing layer; that is, the adversarial optimization function includes four input parameters.

[0086] Optionally, the adversarial optimization function may include two optimization terms, one representing the difference between the first classification result output by the first classification model and the other representing the difference between the second classification result output by the second classification model, and the other representing the first feature vector output by each first processing layer in the first classification model and the other representing the second feature vector output by each second processing layer in the second classification model. That is, the process of determining the adversarial optimization function by the electronic device may include determining a first optimization term to characterize the similarity between the first and second classification results; determining a transformation function between each first processing layer in the first classification model and the corresponding second processing layer in the second classification model; converting each second feature vector into a third feature vector according to the corresponding transformation function; determining a second optimization term to characterize the mean squared error between each third feature vector and the corresponding first feature vector; and determining the weighted sum of the first and second optimization terms as the adversarial optimization function.

[0087] Optionally, the first classification model, the second classification model, and the first and second processing layers therein can all be represented by formulas. The first classification result is represented by function T, the second classification result by function f, the first feature vector output by the i-th first processing layer in the first classification model by function Ti, and the second feature vector output by the i-th second processing layer in the second classification model by function fi. That is, when the input image of the first classification model is x and the input image of the second classification model is y, the first classification result can be T(x), the second classification result can be f(y), the first feature vector output by the i-th first processing layer is Ti(x), and the second feature vector output by the i-th second processing layer is fi(y). Furthermore, the first and second classification models also have corresponding model parameters α and θ, respectively. The classification result and feature vector output by each classification model are related to the corresponding model parameters; that is, the first classification result and the first feature vector can be represented as T(x; α) and Ti(x; α), and the second classification result and the second feature vector can be represented as f(y; θ) and fi(y; θ). Since the first classification model is a pre-trained model, the model parameters can be omitted, and the first classification result and the first feature vector can be directly represented as T(x) and Ti(x).

[0088] Furthermore, the electronic device can determine the similarity between the first classification result T(x) and the second classification result f(y; θ) based on any similarity function, thus obtaining the first optimization term. For example, if the similarity function is the Kullback–Leibler divergence (KL, relative entropy) function used to measure the similarity between two probability distributions, then the electronic device can determine the first optimization term in the adversarial optimization function as KL(T(x), f(y; θ)).

[0089] In one possible implementation, the processing functions of the first processing layer in the first classification model and the corresponding second processing layer in the second classification model are different, resulting in different output feature vectors for the same input image. Therefore, a transformation function can be determined between each first processing layer in the first classification model and the corresponding second processing layer in the second classification model. This transformation function characterizes the relationship between the corresponding first and second processing layers and can be represented as φ. i (F;w i Wherein, the input F is a tensor with the same dimensions as the second feature vector fi(y; θ), and w i The parameters of the transformation function are given, and the output is a tensor with the same dimension as the first eigenvector Ti(x), which is used to adjust the dimensions of the first and second eigenvectors to be consistent.

[0090] Optionally, the electronic device can determine the transformation function by identifying the first and second processing layers, which occupy the same positions in the first and second classification models respectively, as a pair of processing layers. A target sample image is acquired from at least one sample image, and the target sample image is input into both the first and second classification models to obtain the first and second feature vectors output by each pair of first and second processing layers, respectively. The corresponding transformation function is determined based on the first and second feature vectors of each pair of processing layers. For example, the electronic device can identify the i-th first processing layer in the first classification model and the i-th second processing layer in the second classification model as a pair of processing layers, and randomly acquire an image x from at least one sample image as the target sample image. The target sample image x is input into both the first and second classification models, and the transformation function corresponding to the i-th pair of processing layers in the first and second classification models is determined based on the first feature vector Ti(x) output by the i-th first processing layer in the first classification model and the second feature vector fi(x; θ) output by the i-th second processing layer in the second classification model. Optionally, in this embodiment, the transformation function corresponding to each pair of processing layers can be re-determined based on the adjusted parameters of the second classification model after each adjustment of the second classification model.

[0091] Furthermore, the electronic device can determine the transformation function corresponding to the i-th processing layer pair in any way. For example, it can solve the transformation function using a stochastic gradient descent algorithm and the first and second eigenvectors of the i-th processing layer pair. The number of iterations of the stochastic gradient descent algorithm can be preset. Alternatively, the correlation matrix between the first and second eigenvectors of each processing layer pair can be determined first, and then the transformation function of the processing layer pair can be determined as the product of the correlation matrix and the second eigenvector. For example, in determining the first eigenvector Ti(x) output by the i-th first processing layer in the first classification model and the second eigenvector fi(x; θ) output by the i-th second processing layer in the second classification model, the correlation matrix w that converts the second eigenvector fi(x; θ) into the first eigenvector Ti(x) can be determined by quotient or other methods. i The transformation function for the i-th processing layer pair in the first and second classification models is determined to be φ. i (f i (y;θ);w i ) = T i (x), the specific functional form of the transformation function can be the product w of the correlation matrix and the second eigenvector. i ·f i (y;θ;)

[0092] In one possible temporal approach, given that the input image for the first classification model is x, the input image for the second classification model is y, and there are M processing layer pairs in both models, the electronic device, after determining the second feature vector fi(y; θ) output by the i-th second processing layer, inputs it into the transformation function φ of the i-th processing layer pair. i We obtain a third eigenvector φ with the same dimension as the first eigenvector. i (f i (y;θ);w i Then, calculate the third feature vector φ corresponding to the second feature vector fi(y; θ) output by the second processing layer in each processing layer pair. i (f i (y;θ);w i ), and the first feature vector Ti(x) output by the first processing layer in the processing layer pair. The mean square error yields the second optimization term.

[0093] Optionally, after determining the first and second optimization terms, the electronic device can obtain the following formula as an adversarial optimization function by calculating the weighted sum of the two optimization terms.

[0094]

[0095] In the adversarial optimization function, the weights of the first and second optimization terms are λ1 and λ2, respectively, which can be preset according to the importance of the first and second optimization terms in the adversarial optimization process.

[0096] Step S40: Adjust the parameters of the second classification model at least once based on the at least one sample image until the adversarial optimization function meets the convergence condition.

[0097] In one possible implementation, after determining a first classification model and a second classification model, the electronic device iteratively adjusts the parameters of the second classification model at least once based on at least one sample image and an adversarial optimization function. After each adjustment, it is determined whether the adversarial optimization function meets the convergence condition. If the convergence condition is met, the iterative process ends and model distillation is completed. Optionally, during each parameter adjustment, a sample image can be input into the first classification model, and then an adversarial image similar to the input sample image into the first classification model can be determined and input into the second classification model. Whether the adversarial image interferes with the stability of the second classification model is determined based on whether the adversarial optimization function meets the constraints, thereby enhancing the stability of the second classification model in the face of adversarial images during model distillation.

[0098] Optionally, during each parameter adjustment process, at least one sample image can be input into the first classification model to output the corresponding first classification result and the first feature vector output by each first processing layer. Then, with the first classification result and first feature vector corresponding to each sample image, and the second classification result obtained by inputting different adversarial images into the second classification model while keeping the model parameters of the second classification model fixed, and the second feature vector output by each second processing layer, are used as inputs to the adversarial optimization function to obtain the adversarial optimization function value. The adversarial image input to the second classification model when the adversarial optimization function value is maximized is determined as the target adversarial image corresponding to the sample image. The weighted sum of the adversarial optimization function input is determined by inputting the first classification result and first feature vector corresponding to each sample image, the target adversarial image corresponding to the sample image into the second classification model, and the second feature vector output by each second processing layer. The model parameters of the second classification model are adjusted to minimize the weighted sum. Optionally, after each adjustment of the model parameters, the transformation function between each pair of processing layers is redefined, and the weighted sum is calculated based on the redefined transformation function until the weighted sum of the adversarial optimization function reaches its minimum value.

[0099] Furthermore, in the process of determining the adversarial image, the electronic device can acquire a sample image x from a set D consisting of at least one sample image. j To determine each sample image x j Corresponding target adversarial image Optionally, the sample image can be input into the first classification model to obtain the corresponding first classification result T(x). j ) and the first feature vector T output by the i-th first processing layer i (x j The first classification result T(x) corresponding to each sample image is then assigned to the image. j ) and the first eigenvector T i (x j ), and with the model parameters θ of the second classification model fixed, different adversarial images The second classification result obtained by inputting the second classification model and the second feature vector output by the i-th second processing layer As input to the adversarial optimization function, the value of the adversarial optimization function is obtained. By comparing different adversarial images When the second classification result and the second feature vector are used as inputs to the adversarial optimization function, the adversarial image corresponding to the minimum adversarial optimization function value is determined. The sample image x is input to the first classification model. jThe corresponding target adversarial image. Alternatively, with the first classification result and the first feature vector fixed in the adversarial optimization function value, and the model parameters in the second classification result and the second feature vector fixed, the adversarial image corresponding to the minimum value of the adversarial optimization function can be directly solved. The sample image x is input to the first classification model. j The corresponding adversarial image can be solved using the stochastic gradient descent method. Furthermore, during the first parameter tuning, the model parameters of the second classification model are the preset initial model parameters. In each subsequent parameter tuning, the model parameters of the second classification model are the parameters after the previous tuning.

[0100] Optionally, to prevent excessive differences between the sample image and the target adversarial image from affecting the model distillation effect, the electronic device can first set a noise perturbation threshold for determining the target adversarial image corresponding to each sample image, and then use the constraint that the difference between the target adversarial image and the corresponding sample image is no greater than the noise perturbation threshold as the condition for determining the adversarial image. The difference between the adversarial image and the corresponding sample image can be determined based on the norm between the adversarial image and the corresponding sample image, which can be any norm such as L1 norm or L2 norm. For example, the noise perturbation threshold can be set to ∈, and ||x| can be determined based on the noise perturbation threshold. * j -x j || p The constraint ≤∈ serves as a condition for determining the target adversarial image corresponding to the sample image, indicating that the LP norm of both the sample image and the target adversarial image is less than the noise perturbation threshold. In other words, the target adversarial image can be solved using projection gradient descent adjustment according to the following formula.

[0101]

[0102] ||x * -x|| p ≤∈

[0103] Where M is the number of the first processing layer or the second processing layer, T(x j ), T i (x j With the values ​​of θ and θ fixed, the solution is obtained using the projective gradient descent algorithm.

[0104] In one possible implementation, after determining the target adversarial image corresponding to each sample image during the current parameter adjustment process, the electronic device can input the first classification result and first feature vector corresponding to each sample image, the second classification result obtained by inputting the target adversarial image corresponding to the sample image into the second classification model, and the second feature vector output by each second processing layer as a weighted sum of the input adversarial optimization function. Then, the model parameters of the second classification model are adjusted to minimize the weighted sum. That is, the model parameters of the second classification model can be adjusted using the following formula:

[0105]

[0106] Where n is the number of sample images, M is the number of the first or second processing layers, and T(x) j ), T i (x j )and The value of θ is fixed and can be solved using the stochastic gradient descent algorithm.

[0107] Optionally, after each parameter adjustment, the electronic device determines whether the adversarial optimization function meets the convergence condition. If the adversarial optimization function does not meet the convergence condition, the model parameters of the second classification model after this adjustment are used as the parameters of the second classification model for the next parameter adjustment process. The parameter adjustment and model distillation process ends when the adversarial optimization function meets the convergence condition, resulting in the trained second classification model.

[0108] Figure 3 A schematic diagram of a model distillation process according to an embodiment of the present disclosure is shown. Figure 3 As shown, the electronic device can determine the target adversarial image of the sample image during each parameter adjustment process, input the sample image into the first classification model, and input the corresponding target adversarial image into the second classification model. Further, based on the similarity between the first classification result output by the first classification model and the second classification result of the second classification model, and the mean square error between the third feature vector obtained by transforming the second feature vector output by the second processing layer and the first feature vector output by the first processing layer in each processing layer pair, the value of the adversarial optimization function is determined. The value of the adversarial optimization function is then iteratively solved to adjust the model parameters of the second classification model.

[0109] Based on the aforementioned technical features, embodiments of this disclosure can train a second classification model using a pre-trained first classification model. Furthermore, by setting an adversarial optimization function, defensive characteristics of adversarial examples are introduced during the training process to assist in training, thereby improving the robustness of the second classification model obtained through model distillation against adversarial examples. Specifically, the adversarial optimization function considers the overall output difference between the first and second classification models, as well as the output difference of each layer. This enables the training of the second classification model based on intermediate layer distillation using a first classification model with different intermediate structures, thus improving robustness and increasing the universality of model distillation scenarios. Simultaneously, the electronic device first determines the adversarial image by creating an adversarial optimization function, and then performs model distillation based on the sample image and the adversarial image using the adversarial optimization function. The defensive characteristics of adversarial examples are introduced during the model distillation process to assist in training, balancing the computational efficiency of the trained second classification model with its security against adversarial examples.

[0110] Figure 4 A schematic diagram of a model distillation apparatus according to an embodiment of the present disclosure is shown. Figure 4 As shown, the model distillation apparatus of this disclosure embodiment may include:

[0111] Image determination module 40 is used to determine at least one sample image;

[0112] The model determination module 41 is used to determine a first classification model and a second classification model, wherein the first classification model includes the same number of first processing layers as the second classification model includes the same number of second processing layers.

[0113] The function determination module 42 is used to determine the adversarial optimization function of the first classification model and the second classification model. The input parameters of the adversarial optimization function include the first classification result and the second classification result obtained by inputting any image into the first classification model and the second classification model, as well as the first feature vector output by each of the first processing layers and the second feature vector output by each of the second processing layers.

[0114] The model distillation module 43 is used to adjust the parameters of the second classification model at least once based on the at least one sample image until the adversarial optimization function meets the convergence condition.

[0115] In one possible implementation, the parameter adjustment process includes:

[0116] The at least one sample image is input into the first classification model, and the corresponding first classification result and the first feature vector output by each of the first processing layers are output.

[0117] For each sample image, when the first classification result and first feature vector corresponding to the sample image, and the model parameters of the second classification model are fixed, the second classification result obtained by inputting different adversarial images into the second classification model, and the second feature vector output by each of the second processing layers are used as the input of the adversarial optimization function, the adversarial image corresponding to the largest minimum adversarial optimization function value is taken as the target adversarial image corresponding to the sample image.

[0118] The weighted sum of the first classification result and first feature vector corresponding to each sample image, the second classification result obtained by inputting the target adversarial image corresponding to the sample image into the second classification model, and the second feature vector output by each of the second processing layers is determined as the input adversarial optimization function;

[0119] Adjust the model parameters of the second classification model to minimize the weighted sum.

[0120] In one possible implementation, the device further includes:

[0121] The threshold determination module is used to determine the noise disturbance threshold.

[0122] The constraint addition module is used to determine the adversarial image by using the condition that the difference between the adversarial image and the corresponding sample image is no greater than the noise perturbation threshold.

[0123] In one possible implementation, the difference is determined based on the norm between the adversarial image and the corresponding sample image.

[0124] In one possible implementation, the function determining module 42 includes:

[0125] The first item determines the submodule, which is used to determine the first optimization term used to characterize the similarity between the first classification result and the second classification result;

[0126] The transformation function determination submodule is used to determine the transformation function between each first processing layer in the first classification model and the second processing layer in the second classification model corresponding to each first processing layer.

[0127] The vector transformation submodule is used to transform each second feature vector into a third feature vector according to the corresponding transformation function;

[0128] The second determination submodule is used to determine a second optimization term for characterizing the mean square error between each of the third feature vectors and the corresponding first feature vectors;

[0129] The function determination submodule is used to determine the weighted sum of the first optimization term and the second optimization term as the adversarial optimization function.

[0130] In one possible implementation, the transformation function determines the submodule, including:

[0131] The processing layer pair determination unit is used to determine the first processing layer and the second processing layer, which have the same position in the first classification model and the second classification model, as a processing layer pair;

[0132] A sample image determination unit is used to obtain a target sample image from the at least one sample image;

[0133] An image input unit is used to input the target sample image into the first classification model and the second classification model respectively, and obtain the first feature vector and the second feature vector output by each pair of the first processing layer and the corresponding second processing layer respectively;

[0134] The function determination unit is used to determine the corresponding transformation function based on the first feature vector and the second feature vector of each processing layer pair.

[0135] In one possible implementation, the function determining unit includes:

[0136] A matrix determination subunit is used to determine the correlation matrix between the first and second feature vectors of each processing layer pair;

[0137] The function determines the sub-unit, which is used to determine that the transformation function of the processing layer pair is the product of the correlation matrix and the second eigenvector.

[0138] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0139] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0140] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0141] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0142] Figure 5 A schematic diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0143] Reference Figure 5 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0144] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0145] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0146] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0147] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0148] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0149] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0150] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0151] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0152] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0153] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0154] Figure 6 A schematic diagram of another electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0155] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0156] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0157] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0158] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0159] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0160] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0161] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0162] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0163] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0165] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A model distillation method, characterized in that, The method includes: Identify at least one sample image; Determine a first classification model and a second classification model, wherein the first classification model includes the same number of first processing layers as the second classification model includes the same number of second processing layers. Determine the adversarial optimization function for the first classification model and the second classification model. The input parameters of the adversarial optimization function include the first classification result and the second classification result obtained by inputting any image into the first classification model and the second classification model, as well as the first feature vector output by each of the first processing layers and the second feature vector output by each of the second processing layers. The second classification model is adjusted at least once based on the at least one sample image until the adversarial optimization function meets the convergence condition. During each parameter adjustment, the sample image is input into the first classification model, and an adversarial image similar to the input sample image into the first classification model is determined. Whether the adversarial image interferes with the stability of the second classification model is determined based on whether the adversarial optimization function meets the convergence condition.

2. The method according to claim 1, characterized in that, The parameter adjustment process also includes: The at least one sample image is input into the first classification model, and the corresponding first classification result and the first feature vector output by each of the first processing layers are output. For each sample image, when the first classification result and first feature vector corresponding to the sample image, and the model parameters of the second classification model are fixed, the second classification result obtained by inputting different adversarial images into the second classification model, and the second feature vector output by each of the second processing layers are used as the input of the adversarial optimization function, the adversarial image corresponding to the largest minimum adversarial optimization function value is taken as the target adversarial image corresponding to the sample image. The weighted sum of the first classification result and first feature vector corresponding to each sample image, the second classification result obtained by inputting the target adversarial image corresponding to the sample image into the second classification model, and the second feature vector output by each of the second processing layers is determined as the input adversarial optimization function; Adjust the model parameters of the second classification model to minimize the weighted sum.

3. The method according to claim 2, characterized in that, The method further includes: Determine the noise disturbance threshold; The constraint condition for determining the adversarial image is that the difference between the adversarial image and the corresponding sample image is not greater than the noise perturbation threshold.

4. The method according to claim 3, characterized in that, The difference is determined based on the norm between the adversarial image and the corresponding sample image.

5. The method according to any one of claims 1-4, characterized in that, Determining the adversarial optimization functions for the first classification model and the second classification model includes: Determine a first optimization term to characterize the similarity between the first classification result and the second classification result; Determine the transformation function between each first processing layer in the first classification model and the second processing layer in the second classification model corresponding to each first processing layer; Each second feature vector is converted into a third feature vector according to the corresponding transformation function; Determine a second optimization term to characterize the mean square error between each of the third eigenvectors and the corresponding first eigenvector; The weighted sum of the first optimization term and the second optimization term is determined as the adversarial optimization function.

6. The method according to claim 5, characterized in that, Determining the transformation function between each first processing layer in the first classification model and the second processing layer in the second classification model corresponding to each first processing layer includes: The first processing layer and the second processing layer, whose positions in the first classification model are the same as those in the second classification model, are identified as processing layer pairs; Obtain the target sample image from the at least one sample image; The target sample image is input into the first classification model and the second classification model respectively, and the first feature vector and the second feature vector output by each pair of the first processing layer and the corresponding second processing layer are obtained respectively. The corresponding transformation function is determined based on the first feature vector and the second feature vector of each processing layer pair.

7. The method according to claim 6, characterized in that, The step of determining the corresponding transformation function based on the first feature vector and the second feature vector of each processing layer pair includes: Determine the correlation matrix between the first and second feature vectors of each processing layer pair; The transformation function of the processing layer pair is determined to be the product of the correlation matrix and the second eigenvector.

8. A model distillation apparatus, characterized in that, The device includes: Image determination module, used to determine at least one sample image; The model determination module is used to determine a first classification model and a second classification model, wherein the first classification model includes the same number of first processing layers as the second classification model includes the same number of second processing layers. The function determination module is used to determine the adversarial optimization function of the first classification model and the second classification model. The input parameters of the adversarial optimization function include the first classification result and the second classification result obtained by inputting any image into the first classification model and the second classification model, as well as the first feature vector output by each of the first processing layers and the second feature vector output by each of the second processing layers. The model distillation module is used to adjust the parameters of the second classification model at least once based on the at least one sample image until the adversarial optimization function meets the convergence condition. In each parameter adjustment process, the sample image is input into the first classification model, and then an adversarial image similar to the input sample image into the first classification model is determined. Whether the adversarial image interferes with the stability of the second classification model is determined based on whether the adversarial optimization function meets the convergence condition.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images