Adversarial knowledge distillation method and system based on feature semantic guidance and feature orthogonalization

Abstract

The application discloses an adversarial knowledge distillation method and system based on feature semantic guidance and feature orthogonalization, which firstly acquires an image classification dataset, extracts text features, and constructs a semantic feature space.Secondly, a knowledge distillation framework including a teacher model, a student model and a projection head is constructed, the input of the teacher model and the student model is an image in the dataset, and the output is an image category prediction result. Then, the student model parameters are frozen, the projection head is updated through momentum tracking, in the feature level semantic adversarial sample generation stage, all network parameters are frozen, and normalized disturbance is constructed to obtain adversarial features. Finally, in the student model multi-objective joint optimization stage, the teacher model and the two projection heads are fixed, the student model parameters are updated, the classification loss, the distillation loss, the consistency loss and the orthogonalization loss are introduced, and optimization is completed. The application improves the fine-grained discrimination and generalization ability of the student model image classification, and does not introduce additional reasoning overhead.

Description

Technical Field

[0001] This invention relates to the field of knowledge distillation based on deep learning, and in particular to an adversarial knowledge distillation method and system based on feature semantic guidance and feature orthogonalization. Background Technology

[0002] In deep learning-based image classification tasks, while deep neural networks (DNNs) perform exceptionally well on training data, high-performance models suffer from high complexity, high computational cost, high latency, and stringent deployment requirements. To achieve lightweight models—that is, to provide powerful generalization performance while maintaining low computational resource consumption—researchers have proposed various methods. Among them, knowledge distillation (KD) has attracted considerable attention for its ability to efficiently simplify models while preserving performance. In this method, the teacher model is typically a large and complex deep neural network, trained on a large dataset and possessing high pattern recognition capabilities; the student model, on the other hand, is lightweight and aims to learn the knowledge and generalization capabilities of the teacher model. This balances high performance and high efficiency in practical applications, thereby improving the flexibility of model deployment, particularly providing technical support for the needs of edge computing.

[0003] The core objective of knowledge distillation is to efficiently transfer complex knowledge from teacher models to lightweight student models. However, in fine-grained classification scenarios, even if the student model completes output or local feature imitation, it may still retain a significant amount of confusion in high-level semantic organization. In summary, existing methods have shortcomings in the following three main aspects: First, feature alignment lacks global semantic guidance. Conventional feature distillation often uses Euclidean space measures such as mean squared error to numerically align teacher and student features. While these methods can reduce feature differences, they struggle to explicitly maintain the semantic topological relationships between categories. Second, pixel-level perturbations cannot directly correspond to deep semantic confusion. Existing techniques attempt to introduce adversarial mechanisms into the distillation process, constructing perturbation samples in the input space to enhance learning of difficult samples. However, pixel-level perturbations primarily affect noise sensitivity and cannot directly characterize the high-level semantic shifts between fine-grained categories. Third, the high coupling of feature components limits discriminative power. Lightweight student models have limited capacity and tend to encode discriminative and confusion information together in the same feature. Without explicit decoupling constraints, the model struggles to reduce redundant components in the confusion direction.

[0004] Therefore, identifying the main confusion direction under stable semantic reference and organizing perturbations and constraints around this direction to improve the fine-grained discrimination ability of student models has become a key issue in improving the knowledge distillation effect. Summary of the Invention

[0005] To address the shortcomings of existing knowledge distillation methods in feature semantic modeling and confusion decoupling, this invention proposes an adversarial knowledge distillation method (SGAKD) and system based on feature semantic guidance and feature orthogonalization. This method utilizes a contrastive language-image pre-training (CLIP) model to construct external semantic anchors, characterizing easily confused semantic directions in the feature space, and combines feature orthogonalization constraints to optimize the discriminative representation of the student model. SGAKD employs three complementary mechanisms to form a semantic adversarial distillation framework: First, it introduces CLIP as a cross-modal semantic anchor, establishing a unified semantic feature space. This allows the student model to obtain additional global semantic guidance during training and enables different categories to have comparable relative positions within the same semantic coordinate system. Second, it constructs semantic adversarial perturbations in the feature space. Instead of using traditional pixel-level noise, it calculates the similarity between the current sample features and all non-real categories in the CLIP semantic space, identifies the most easily confused semantic direction, and performs adversarial shifts along this direction, allowing the model to access more difficult samples closer to the category boundaries at the feature level. Third, it designs an orthogonalization-based feature decoupling mechanism, introducing feature orthogonalization loss. This suppresses redundant semantic information by constraining the projection of student features onto the confusion direction, while simultaneously introducing consistency loss to constrain the output stability of the model before and after semantic perturbation. Furthermore, this invention introduces a dual-projection head and momentum update mechanism to suppress frequent jumps in adversarial targets and provide a stable reference for semantic perturbation generation and orthogonalization constraints. It also adopts a three-stage alternating optimization strategy, which ensures that semantic mapping, adversarial perturbation, and distillation learning are completed sequentially in each iteration. The two stages involving parameter updates (projection head update stage and student model joint optimization stage) use independent optimizers and freeze the corresponding model parameters, so that the parameter updates in each stage do not interfere with each other and avoid different objective functions directly competing for the same set of gradients in the same stage.

[0006] In one aspect, this invention provides an adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization, the specific steps of which include: S1. Obtain an image classification dataset consisting of three-channel RGB format images, and pre-compute the entire dataset using the text encoder of the Contrastive Language-Image Pre-trained (CLIP) model. Text features of each category are used to construct a semantic feature space. The specific construction process is as follows: For each category label in the image classification dataset, text input is constructed using cue word templates, and then extracted by the text encoder of the CLIP model. The text features are analyzed and L2 normalized to obtain the semantic feature space. ,in For the first Text features corresponding to each category, and each text feature All are unit vectors. This represents the embedding dimension of the CLIP model.

[0007] S2. Construct a knowledge distillation framework, including a teacher model, a student model, and a projection head. Both the teacher and student models take images from an image classification dataset as input and output confidence vectors for image category predictions. The student model is divided into a shallow feature extractor and a deep feature extractor. The shallow feature extractor maps the input image to visual features, while the deep feature extractor maps the visual features to classification outputs. The projection head includes an online projection head. and target projection head It is used to map the visual features of student models to the CLIP semantic feature space. This is to apply cross-modal semantic guidance to the intermediate feature layer of the student model. The specific design and construction are as follows: S21. Construct two independent neural network models as the teacher model and the student model, respectively. The teacher model has a larger number of parameters and higher accuracy than the student model, while the student model has a smaller inference speed and lower memory usage than the teacher model.

[0008] S22. Divide the student model into shallow feature extractors along the network depth. and deep feature extractor The two parts are connected, with the boundary being before the global pooling layer, where the shallow feature extractor is located. The deep feature extractor is responsible for mapping the input image to visual features. It is responsible for mapping visual features to classification output.

[0009] S23. Construct the projection head The shallow feature extractor of the student model will be used. For the input image The clean features obtained by extraction and flattening Mapped to dimension CLIP semantic space, i.e. : ,in The dimensions of the student model features; the projection head includes two independent projection heads, namely the online projection head. and target projection head .

[0010] S3. During the projection head update and momentum tracking phase, the student model parameters remain frozen, and only the online projection head is updated. Optimization is performed to progressively map and align visual features to the CLIP semantic feature space. Simultaneously, a target projection head is constructed by combining the exponential moving average (EMA) mechanism. The specific steps are as follows: S31. Freeze the student model parameters for the input image. ( The number of channels in the input image. and (Height and width of the input image, respectively), by a shallow feature extractor Its cleaning characteristics were obtained. Cleaning features Input online projector head Mapping and L2 normalization are then performed to obtain normalized semantic features. .

[0011] S32. Using normalized semantic features Its true category text features The cosine distance is used as the projection head loss. ,in For the input image Real category labels In semantic feature space The corresponding text features, .

[0012] S33. Using Projector Head Loss Perform backpropagation, updating only the online projector head. Online projection head The gradient descent algorithm is used to quickly update and capture the latest feature representation of the current batch of data.

[0013] S34. By using the exponential moving average (EMA), the target projection head is adjusted. parameters With momentum coefficient Slow tracking online projector head parameters The evolutionary trajectory enables momentum renewal.

[0014] S4. In the feature-level semantic adversarial example generation stage, freeze all network parameters and target clean features. Using the target projection head The most easily confused category direction is identified, and a normalized perturbation is constructed using feature gradients to shift the samples in the semantic space towards the confusion direction, thus obtaining adversarial features. The specific steps are as follows: S41. Freeze all network parameters, including student model and online projector. and target projection head Cleaning features Input target projection head Mapping and L2 normalization are performed to obtain stable normalized semantic features. .

[0015] S42. In semantic feature space In the middle, from all Excluding true category labels from each category Then the candidate category label set is obtained. ; Calculate stable normalized semantic features Text features of all candidate categories similarity And select the category with the highest similarity as the semantic adversarial target. Its corresponding text features are denoted as .

[0016] S43. Define cleaning features for pre-projection Semantic adversarial loss Minimizing this loss can drive the unnormalized projected features Text features for semantic adversarial targets The direction is closer, and the loss is calculated with respect to the cleaning feature. gradient .

[0017] S44. Along the normalized gradient Direction to cleaning characteristics Apply perturbations to generate feature-level semantic adversarial examples, i.e., adversarial features. .

[0018] S5. In the multi-objective joint optimization phase of the student model, the teacher model and the two projection heads remain fixed, and only the parameters of the student model are updated, based on the clean features obtained in step S31. Adversarial features obtained in step S44 By comprehensively introducing classification loss, distillation loss, consistency loss and orthogonalization loss, the student model can suppress the interference of confusing components while maintaining basic classification performance, thereby improving the stability of the discriminative representation.

[0019] In another aspect, the present invention also provides an adversarial knowledge distillation system based on feature semantic guidance and feature orthogonalization, comprising the following modules: The semantic feature space module is used to acquire an image classification dataset consisting of three-channel RGB format images, extract text features, and construct a semantic feature space.

[0020] The knowledge distillation framework module is used to construct a knowledge distillation framework that includes a teacher model, a student model, and a projector head. The input to the teacher model and the student model is images from the image classification dataset, and the output is the image category prediction result.

[0021] The projection head update module is used to keep the student model parameters frozen and update the projection head through momentum tracing.

[0022] The adversarial feature module is used to freeze all network parameters and construct normalized perturbations through feature gradients to obtain adversarial features after the projection head is updated, during the feature-level semantic adversarial sample generation stage.

[0023] The multi-objective joint optimization module is used to update the parameters of the student model while keeping the teacher model and the two projectors fixed during the multi-objective joint optimization phase of the student model. It incorporates adversarial features and introduces classification loss, distillation loss, consistency loss, and orthogonalization loss to complete the optimization.

[0024] Beneficial effects of the invention The Adversarial Knowledge Distillation (SGAKD) method, based on feature semantic guidance and feature orthogonalization, shifts the adversarial action space from the pixel space of the input layer to the feature space of the intermediate layer. It introduces CLIP cross-modal semantic knowledge as guidance and constructs a collaborative optimization mechanism around the fine-grained semantic confusion problem: teacher knowledge provides reference, semantic perturbation exposes the problem, orthogonal constraints suppress confusion, and output consistency stabilizes the boundary. This invention offers the following advantages: 1. Using CLIP as a semantic anchor provides an external semantic reference system relatively independent of the current distillation process, ensuring that adversarial target selection and orthogonalization constraints are established on a unified and stable semantic coordinate system; 2. Compared to pixel space perturbation, feature space perturbation has a clearer semantic orientation, pushing feature shifts along the confusion category direction, while its generation process only relies on feature gradients, resulting in relatively low training overhead; 3. The design of feature orthogonalization loss compresses the projection of features onto the confusion direction, making the feature representation more focused on the discriminative subspace, compressing redundant information related to category confusion, and achieving effective solutions for both confusion and discriminative components. 4. Introducing consistency loss encourages the model to maintain a stable prediction distribution when facing semantic perturbations, helping the model form a relatively smooth and stable classification boundary in the feature space; 5. Employing a dual-projection head and momentum update design suppresses frequent jumps in adversarial targets and provides a stable reference for semantic perturbation generation and orthogonalization constraints, making the supervision signal during training smoother; 6. Using a three-stage alternating optimization strategy avoids different objective functions directly competing for the same set of gradients in the same stage, and the separation of module responsibilities makes the entire training process easier to converge; 7. The projection head is only used as a training auxiliary module and can be removed after training, thus not introducing additional inference overhead. In summary, this invention enhances the discrimination and generalization capabilities of student models in fine-grained scenarios, provides a practical and efficient solution for resource-constrained scenarios, and demonstrates certain application prospects and value. Attached Figure Description

[0025] Figure 1This is an architecture diagram of an adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization; Figure 2 This is a diagram illustrating the process of projector head updates and momentum tracking. Figure 3 This is a diagram illustrating the process of generating feature-level semantic adversarial samples. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0027] As a preferred embodiment of the present invention, an adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization is provided, the architecture of which is as follows: Figure 1 As shown, it includes the following steps: S1. The knowledge distillation method of this invention is applied to an image classification dataset consisting of three-channel RGB format images. This embodiment uses the CIFAR-100 dataset as an example, where the input image has 3 channels, a length and width of 32, and a number of categories. =100. The text encoder of the CLIP model pre-computes the text features for all 100 categories of the dataset to construct a semantic feature space. The specific construction process is as follows: For each category label in the image classification dataset, text input is constructed using cue word templates. The CLIP model's text encoder extracts 512-dimensional text features, which are then L2 normalized to obtain the semantic feature space. ,in For the first Text features corresponding to each category, and each text feature All are unit vectors, and 512 is the embedding dimension of the CLIP model, i.e. The purpose of this prompting method is to embed category names into a unified visual-linguistic semantic space through natural language description, so that text features can stably represent category semantic attributes.

[0028] S2. Construct a knowledge distillation framework, including a teacher model, a student model, and a projection head. Both the teacher and student models take images from an image classification dataset as input and output confidence vectors for image category predictions. The student model is divided into a shallow feature extractor and a deep feature extractor. The shallow feature extractor maps the input image to visual features, while the deep feature extractor maps the visual features to classification outputs. The projection head includes an online projection head. and target projection head It is used to map the visual features of student models to the CLIP semantic feature space. This is to apply cross-modal semantic guidance to the intermediate feature layer of the student model. The specific design and construction are as follows: S21. Construct two independent neural network models, one as the teacher model and the other as the student model. The teacher model has a larger number of parameters and higher accuracy than the student model, while the student model has a smaller inference speed and lower memory usage. For example, models such as ResNet32x4, WRN-40-2, and VGG13 can be used as teacher models, while models such as ResNet8x4, ResNet20, WRN-40-1, ShuffleNetV2, MobileNetV2, and VGG8 can be used as student models. This embodiment uses ResNet32x4 as the teacher model and ResNet8x4 as the student model as an example.

[0029] S22. Taking ResNet8x4 as an example, the student model is divided into shallow feature extractors along the network depth. and deep feature extractor The two parts are connected, with the boundary being before the global average pooling layer, where the shallow feature extractor is located. The deep feature extractor is responsible for mapping the input image to visual features. It is responsible for mapping visual features to classification output.

[0030] S23. Construct the projection head The shallow feature extractor of the student model will be used. For the input image The clean features obtained by extraction and flattening Mapped to a CLIP semantic space of dimension 512, i.e. : ,in This represents the dimension of the student model features. The projection head employs a lightweight architecture, consisting of fully connected layers ( ), batch normalization layer, Activation function ( , (for the input of this layer) and fully connected layer ( The projection head comprises two independent projection heads, namely an online projection head. and target projection head Target projection head Initialize as an online projection head A copy.

[0031] S3. During the projection head update and momentum tracking phase, such as Figure 2 As shown, the student model parameters remain frozen, only for the online projection head. Optimization is performed to progressively map and align visual features to the CLIP semantic feature space. Simultaneously, a more stable and smoother target projection head is constructed by combining the EMA mechanism. This provides a reliable reference for subsequent semantic perturbation generation. The specific steps are as follows: S31. Freeze the student model parameters for the input image. ( The number of channels in the input image. and (Height and width of the input image, respectively), by a shallow feature extractor Its cleaning characteristics were obtained. Cleaning features Input online projector head Mapping and L2 normalization are then performed to obtain normalized semantic features. During this stage, the student's core skills are fixed, and only the online projector is trained. This prevents cross-modal alignment errors from being propagated back to the student, thus avoiding disruption of the visual representation already learned by the student during the alignment process. The formula is expressed as: S32. Using normalized semantic features Its true category text features The cosine distance is used as the projection head loss. ,in For the input image Real category labels In semantic feature space The corresponding text features, ;because and All are L2-normalized unit vectors, and the projection head loss is... The value range of is [0,2], and the formula is expressed as: S33. Using Projector Head Loss Perform backpropagation, updating only the online projector head. Online projection head The gradient descent algorithm is used for rapid updates to capture the latest feature representation of the current batch of data; in terms of optimizer selection, an online projection head is used. An Adaptive Moment Estimation (Adam) optimizer is employed to adapt more quickly to cross-modal alignment tasks.

[0032] S34. By using the exponential moving average (EMA), the target projection head is adjusted. parameters With momentum coefficient Slow tracking online projector head parameters The evolution trajectory of momentum is updated, and the formula is expressed as: Among them, superscript Indicates the number of training steps and momentum coefficient. =0.996, this mechanism makes the target projection head The parameter changes are much slower than those of online projection heads. This provides a stable semantic reference for subsequent adversarial target selection, semantic perturbation generation, and orthogonalization constraints.

[0033] S4. In the feature-level semantic adversarial sample generation stage, such as Figure 3 As shown, all network parameters are frozen, and the clean features obtained in step S31 are considered. Using the target projection head The most easily confused category direction is identified, and a normalized perturbation is constructed using feature gradients to shift the samples in the semantic space towards the confusion direction, thus obtaining adversarial features. The specific steps are as follows: S41. Freeze all network parameters, including student model and online projector. and target projection head The cleaning feature obtained in step S31 Input target projection head Mapping and L2 normalization are performed to obtain stable normalized semantic features. : S42. In semantic feature space In the middle, the true category labels are excluded from all 100 categories. Then the candidate category label set is obtained. ; Calculate stable normalized semantic features Text features of all candidate categories similarity And select the category with the highest similarity as the semantic adversarial target. This approach aims to approximate adversarial features to the most easily confused category boundaries, thereby maximizing the exposure of weaknesses in fine-grained discrimination by the student model. Semantic adversarial objective. The corresponding text features are denoted as The formula is expressed as: in, semantic feature space Medium category Corresponding text features; due to and All are L2-normalized unit vectors, and the similarity is... The cosine similarity value ranges from -1 to 1.

[0034] S43. Define cleaning features for pre-projection Semantic adversarial loss Minimizing this loss can drive the unnormalized projected features Text features for semantic adversarial targets The direction is closer, and the loss is calculated with respect to the cleaning feature. gradient The formula is expressed as: in, Represents cosine similarity. The unnormalized projection features output by the target projection head. Indicates to Inside Find the gradient.

[0035] S44. Cleanliness features along the normalization gradient direction Apply perturbations to generate feature-level semantic adversarial examples, i.e., adversarial features. The formula is expressed as: in, The perturbation strength controls the degree of shift of adversarial features along the semantic direction; L2 normalization ensures the consistency of the perturbation direction and avoids the influence of gradient magnitude. The decay occurs gradually during training, meaning that as training rounds progress, the value decreases from the initial value. =0.2 linear decays to the lower limit =0.1, to avoid excessive interference to the student model when the semantic space is still unstable in the early stages of training; then adversarial features will be added. Reconstruct the dimensions of the original feature map before flattening so that it can be subsequently input into the deep feature extractor of the student model. .

[0036] S5. In the multi-objective joint optimization phase of the student model, the teacher model and the two projection heads remain fixed, and only the parameters of the student model are updated, based on the clean features obtained in step S31. Adversarial features obtained in step S44 By comprehensively incorporating classification loss, distillation loss, consistency loss, and orthogonalization loss, the student model can suppress interference from confusing components while maintaining basic classification performance, thereby improving the stability of the discriminative representation. The specific steps are as follows: S51. Freeze Online Projector Head and target projection head The parameters are updated only for the student model; the teacher model updates the parameters for the input image. Output features The adversarial features obtained in step S44 Input student model deep feature extractor To obtain the feature output of the adversarial branch And category prediction, while also including cleanliness features Input deep feature extractor This yields the category prediction for the clean branch. Simultaneously, the forward clean branch and the adversarial branch allow subsequent classification, distillation, and consistency constraints to work together under both feature conditions, balancing basic discriminative power and perturbation robustness.

[0037] S52. Deep Feature Extractor In adversarial features With cleaning characteristics The category predictions on the above are compared with the true labels. Calculate cross-entropy loss The classification loss is obtained by averaging the two values. The formula is expressed as: Cross-entropy loss simultaneously constrains the classification output of both the clean branch and the adversarial branch, ensuring that the model maintains basic classification ability under both feature conditions.

[0038] S53. Output the features of the student model Feature output of the teacher model Alignment, Calculation of Distillation Losses This embodiment employs logits-based temperature distillation to enhance the student model's adversarial features. logits output With the teacher model on the input image logits output Alignment, the formula is expressed as: in, This represents the Softmax normalized exponential function. Represents relative entropy. The temperature coefficient is used to smooth the probability distribution output by the teacher model, enabling the student model to learn the relative relationships between categories more fully. In this embodiment... =4.0.

[0039] S54. Deep Feature Extractor for Student Models In cleaning features and adversarial features normalized exponential function The obtained output probabilities (i.e., the predicted probability distributions of the clean branch and the adversarial branch) are respectively and The formula is expressed as: Using relative entropy (KL divergence) defines the consistency loss. The student model is constrained to maintain a consistent prediction distribution before and after semantic perturbation, as expressed by the formula: in, and They are respectively and The Each component.

[0040] S55. Calculating Feature Orthogonalization Loss Based on Semantic Confusion Vector , where semantic confusion vector Defined as the difference between the text features of the adversarial target category and the real category in the CLIP semantic space, expressed by the formula: in, This is the normalized semantic confusion vector. The stable normalized semantic features obtained in step S41, The text features corresponding to the semantic adversarial target selected in step S42, For real category labels Corresponding text features; due to Both are L2-normalized unit vectors, and the orthogonality loss is... The value of is in the range of [0,1]. Minimizing this loss aims to compress the projection of features onto the confusion direction, making the feature representation more focused on the discriminative subspace.

[0041] S56. Total loss in the multi-objective joint optimization phase of the student model The sum of classification loss, distillation loss, consistency loss, and orthogonalization loss, where distillation loss is the weighted sum of these losses. Consistency loss Orthogonalization loss The weight hyperparameters are respectively Orthogonalization loss weights A linear preheating strategy is adopted, denoted as dynamic weight. The formula is expressed as: Among them, distillation weight =0.5, Consistency Weight =0.5, This indicates the current training round and the number of warm-up rounds. =72, which represents the first 30% of the total 240 training rounds, and is the target weight cap. =0.5; in the early stages of training, a smaller value is needed. Used to reduce the interference of orthogonalization constraints on student feature learning, as training progresses, Gradually increase and eventually reach This allows for more thorough compression of redundant components in the confusion direction after the semantic space has stabilized relatively. The total loss is then used. Backpropagation is performed, and the Stochastic Gradient Descent (SGD) optimizer is used to update the parameters only on the student model.

[0042] During the inference phase, the projection head is used only as a training aid module and can be removed after training, so it does not introduce additional inference overhead.

[0043] In another aspect, the present invention also provides an adversarial knowledge distillation system based on feature semantic guidance and feature orthogonalization, for implementing the aforementioned adversarial knowledge distillation method, comprising a semantic feature space module, a knowledge distillation framework module, a projection head update module, an adversarial feature module, and a multi-objective joint optimization module: The semantic feature space module is used to acquire an image classification dataset consisting of three-channel RGB format images, extract text features, and construct a semantic feature space. The knowledge distillation framework module is used to construct a knowledge distillation framework including a teacher model, a student model, and a projection head. The input of the teacher model and the student model is the image in the image classification dataset, and the output is the image category prediction result.

[0044] The projection head update module is used to keep the student model parameters frozen and update the projection head through momentum tracking.

[0045] The adversarial feature module is used to freeze all network parameters and construct normalized perturbations through feature gradients to obtain adversarial features after the projection head is updated, during the feature-level semantic adversarial sample generation stage.

[0046] The multi-objective joint optimization module is used to update the student model parameters while keeping the teacher model and two projectors fixed during the multi-objective joint optimization phase of the student model. It also incorporates adversarial features and introduces classification loss, distillation loss, consistency loss, and orthogonalization loss to complete the optimization.

[0047] Example: The steps in this embodiment are the same as those in the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0048] This invention's technique was implemented on the CIFAR-100 (Canadian Institute for Advanced Study 100-class dataset), Tiny-ImageNet, and CUB-200 datasets. The CIFAR-100 dataset contains 50,000 training images and 10,000 test images, with 100 classes and an image resolution of 32×32. This dataset has a large number of classes, and some classes exhibit visual similarity, making it suitable for evaluating the model's multi-class discrimination ability. The Tiny-ImageNet dataset is a subset of ImageNet, containing 200 classes, with 500 training images and 50 test images per class, and an image resolution of 64×64. This dataset has a higher number of classes and image complexity than CIFAR-100, placing higher demands on the model's generalization ability. The CUB-200 dataset contains 11,788 images representing 200 bird species, with 5,994 images in the training set and 5,794 in the test set. This dataset is characterized by finely labeled samples and high background complexity, providing a typical challenge for fine-grained classification tasks in intra-class discriminative scenarios. During training, random cropping and random horizontal flipping were used as data augmentation strategies, and the input images were normalized. During testing, only image resizing and normalization were performed to ensure consistent evaluation results.

[0049] For the model training process, a total of 240 training epochs were set, with a batch size of 128 per epoch. The student model used a stochastic gradient descent (SGD) optimizer, with its learning rate reduced to one-tenth at epochs 150, 180, and 210; the online projector head used a separate Adam optimizer. Hyperparameters such as loss weights can be adjusted according to specific task requirements.

[0050] To evaluate the performance of the SGAKD method in different knowledge distillation frameworks, KD (a neural network-based knowledge distillation method) and DKD (a goal-decoupling-based knowledge distillation method), the experimental setup included two main schemes: homogeneous network comparison experiments and heterogeneous network comparison experiments. In the homogeneous network, the teacher and student models used the same or similar network architectures, while in the heterogeneous network, the teacher and student models had different network structures. Simultaneously, comparisons were made with existing knowledge distillation methods such as FitNets, AT (Attention Transfer), OFD, ReviewKD, CTKD, CRD, MSFF, and MFAKD. All comparison methods adopted the training strategies recommended in their original papers or public implementations and were evaluated under the same dataset, network structure, and training epoch settings to ensure the fairness of the experimental comparisons. The network architectures included: ResNet (residual network), WRN (wide residual network), VGG (visual geometric perception neural network), ShuffleNet (lightweight neural network based on channel shuffling), and MobileNet (lightweight neural network for mobile devices).

[0051] The experimental results are shown in Tables 1, 2, 3, and 4. In Table 1, the teacher and student models use a homogeneous architecture combination, while in Table 2, they use a heterogeneous architecture combination. Tables 1 and 2 are implemented on the CIFAR-100 dataset, Table 3 on the Tiny-ImageNet dataset, and Table 4 on the CUB-200 dataset. The experimental metric used is Top-1 accuracy (%), which is the ratio of the class with the highest predicted probability in the model's predictions to the true label. The values ​​in the teacher and student model rows represent the accuracy of each model when trained individually. The feature semantic guidance and feature orthogonalization strategies of this invention are applied to existing KD and DKD methods, respectively. In the tables, KD+this invention and DKD+this invention represent the improved results after incorporating this invention, and are expressed as... The rows show the performance gain relative to the corresponding baseline method. The best results in each column are highlighted in bold.

[0052] Table 1. Comparative Experiments on Knowledge Distillation of Isomorphic Model Combinations on the CIFAR-100 Dataset

[0053] Table 2. Comparative Experiments on Knowledge Distillation of Heterogeneous Model Combinations on the CIFAR-100 Dataset

[0054] Table 3. Comparison of knowledge distillation experiments for different model combinations on the Tiny-ImageNet dataset

[0055] Table 4. Comparative experiment of knowledge distillation of different model combinations on the CUB-200 dataset

[0056] Experimental results show that after introducing the feature semantic guidance and feature orthogonalization strategy of this invention into the KD and DKD methods, the student model's accuracy on the test set is improved relative to the corresponding baseline methods in both homogeneous network combinations and heterogeneous network combinations: On the CIFAR-100 dataset, the gain relative to the KD baseline is +0.38 to +1.25 percentage points, and the gain relative to the DKD baseline is +0.23 to +0.72 percentage points; on the Tiny-ImageNet dataset with more categories and more complex image distribution, the gain relative to the KD baseline is +0.48 to +1.15 percentage points, and the gain relative to the DKD baseline is +0.56 to +0.72 percentage points; on the fine-grained classification dataset CUB-200, the gain relative to the KD baseline is +0.46 to +0.76 percentage points, and the gain relative to the DKD baseline is +0.45 to +0.81 percentage points. Furthermore, the DKD+invention combination achieves the best results among all compared methods in most combinations. The benefits of this invention do not primarily depend on the absolute accuracy of the teacher model itself, but are more related to the existence of a significant semantic confusion space in the current distillation task: in fine-grained scenarios like CUB-200, where category similarity is higher and confusion directions are more concentrated, adversarial modeling and orthogonalization constraints in the semantic space are more likely to play a role, resulting in a more significant improvement. This demonstrates that this invention provides an efficient improvement scheme for knowledge distillation technology.

Claims

1. An adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization, characterized in that, Includes the following steps: S1. Obtain an image classification dataset consisting of three-channel RGB format images, extract text features, and construct a semantic feature space. ; S2. Construct a knowledge distillation framework that includes a teacher model, a student model, and a projection head. The input to the teacher model and the student model is the image in the image classification dataset, and the output is the image category prediction result. S3. The student model parameters are kept frozen, and the projection head is updated via momentum tracking; S4. After the projection head is updated, during the feature-level semantic adversarial example generation stage, all network parameters are frozen, and normalized perturbations are constructed through feature gradients to obtain adversarial features. S5. In the multi-objective joint optimization stage of the student model, the teacher model and the two projectors are kept fixed. The parameters of the student model are updated. Combining adversarial features, classification loss, distillation loss, consistency loss and orthogonalization loss are introduced to complete the optimization.

2. The adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: For each category label in the image classification dataset, text input is constructed using cue word templates, and then extracted by the text encoder of the contrastive language-image pre-trained CLIP model. The text features are analyzed and L2 normalized to obtain the semantic feature space. ,in For the first Text features corresponding to each category, and each text feature All are unit vectors. This represents the embedding dimension of the CLIP model.

3. The adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization according to claim 2, characterized in that, In step S2, the knowledge distillation framework is specifically implemented as follows: S21. Construct two independent neural network models as the teacher model and the student model, respectively. The teacher model has a larger number of parameters and higher accuracy than the student model, while the student model has a smaller inference speed and lower memory usage than the teacher model. S22. Divide the student model into shallow feature extractors along the network depth. and deep feature extractor The two interconnected parts include a shallow feature extractor. The deep feature extractor is responsible for mapping the input image to visual features. Responsible for mapping visual features to classification output; S23. Construct the projection head The shallow feature extractor of the student model will be used. For the input image The clean features obtained by extraction and flattening Mapped to dimension CLIP semantic space, i.e. : ,in The dimensions of the student model features; the projection head includes two independent projection heads, namely the online projection head. and target projection head .

4. The adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization according to claim 3, characterized in that, The specific implementation process of step S3 is as follows: S31. Freeze the student model parameters for the input image. , The number of channels in the input image. and The height and width of the input image are respectively determined by a shallow feature extractor. Obtain cleaning characteristics Cleaning features Input online projector head Mapping and L2 normalization are then performed to obtain normalized semantic features. ; S32. Using normalized semantic features Its category text features The cosine distance is used as the projection head loss. ,in For the input image Real category labels In semantic feature space The corresponding text features, ;because and All are L2-normalized unit vectors, and the projection head loss is... Represented as: S33. Using Projector Head Loss Perform backpropagation, updating only the online projector head. Online projection head The gradient descent algorithm is used to update and capture the latest feature representation of the current batch of data; S34. By using the exponential moving average (EMA), the target projection head is adjusted. parameters With momentum coefficient Tracking online projectors parameters The evolution trajectory of momentum is updated, and the formula is expressed as: Among them, superscript This indicates the number of training steps.

5. The adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization according to claim 4, characterized in that, The specific implementation process of step S4 is as follows: S41. Freeze all network parameters, including student model and online projector. and target projection head ; The cleaning feature obtained in step S31 Input target projection head Mapping and L2 normalization are performed to obtain stable normalized semantic features. ; S42. In semantic feature space In the middle, from all Excluding true category labels from each category Then the candidate category label set is obtained. ; Calculate stable normalized semantic features Text features of all candidate categories similarity And select the category with the highest similarity as the semantic adversarial target. Its corresponding text features are denoted as ; S43. Based on text features The unnormalized projection features output by the target projection head are used to define cleaning features before projection. Semantic adversarial loss And calculate the loss with respect to the cleaning feature. gradient The formula is expressed as: in, Indicates to Inside Find the gradient; S44. Along the normalized gradient Direction to cleaning characteristics Apply perturbations to generate feature-level semantic adversarial examples, i.e., adversarial features. .

6. The adversarial knowledge distillation method based on feature semantic guidance and feature orthogonalization according to claim 5, characterized in that, The specific implementation process of step S5 is as follows: S51. Freeze Online Projector Head and target projection head The parameters are updated only for the student model; the teacher model updates the parameters for the input image. Output features ; The adversarial features obtained in step S44 Input student model deep feature extractor To obtain the feature output of the adversarial branch And category prediction, while also including cleanliness features Input deep feature extractor This yields a category prediction for the clean branch; S52. Deep Feature Extractor In adversarial features With cleaning characteristics The category predictions on the above are compared with the true labels. Calculate cross-entropy loss The classification loss is obtained by averaging the two values. ; S53. Output the features of the student model Feature output of the teacher model Alignment, Calculation of Distillation Losses ; S54. Deep Feature Extractor for Student Models In cleaning features and adversarial features normalized exponential function The obtained output probabilities are respectively and ; against and The consistency loss is defined using KL divergence. This constrains the student model to maintain a consistent prediction distribution before and after semantic perturbation; S55. Calculating Feature Orthogonalization Loss Based on Semantic Confusion Vector , where semantic confusion vector Defined as the difference between the text features of the adversarial target category and the real category in the CLIP semantic space; S56. Total loss in the multi-objective joint optimization phase of the student model The weighted sum of classification loss, distillation loss, consistency loss, and orthogonalization loss; the orthogonalization loss weights. A linear preheating strategy is adopted; total loss is used. Backpropagation is performed, and parameters are updated only on the student model.

7. An adversarial knowledge distillation system based on feature semantic guidance and feature orthogonalization, used to implement the adversarial knowledge distillation method according to any one of claims 1 to 6, characterized in that, Includes the following modules: The semantic feature space module is used to acquire an image classification dataset consisting of three-channel RGB format images, extract text features, and construct a semantic feature space. The knowledge distillation framework module is used to construct a knowledge distillation framework that includes a teacher model, a student model, and a projection head. The input to the teacher model and the student model is an image from the image classification dataset, and the output is the image category prediction result. The projection head update module is used to keep the parameters of the student model frozen and update the projection head through momentum tracking. The adversarial feature module is used to freeze all network parameters and construct normalized perturbations through feature gradients to obtain adversarial features after the projection head is updated, during the feature-level semantic adversarial sample generation stage. The multi-objective joint optimization module is used to update the parameters of the student model while keeping the teacher model and the two projectors fixed during the multi-objective joint optimization phase of the student model. It incorporates adversarial features and introduces classification loss, distillation loss, consistency loss, and orthogonalization loss to complete the optimization.

Images