Layered contrast distillation method and system based on self-supervised enhancement task

Abstract

The invention discloses a layered contrast distillation method and system based on a self-supervised enhancement task. The method comprises the following steps: defining a plurality of image transformations in an original data set to perform self-supervised enhancement on data; an auxiliary classifier is added in the teacher model, the teacher model is trained to output joint probability distribution of an intermediate layer in a joint self-supervision enhancement task and an original classification task and final layer probability distribution, and feature information of the intermediate layer of the model is stored; knowledge information of the teacher model is distilled to the student model and trained, intermediate layer probability distribution loss and final layer probability distribution loss of the student model are calculated, intermediate layer feature information of the teacher model and the student model is processed by a hierarchical projection head, features are constrained to a hypersphere of a contrast learning unit for feature zooming, and feature loss is calculated; and finally, carrying out weighted calculation on the probability distribution loss, the feature loss and the final layer logit loss of the student model to obtain total loss.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and machine learning technology, and in particular relates to a hierarchical comparative distillation method and system based on self-supervised augmentation tasks. Background Technology

[0002] Knowledge distillation (KD), as an efficient model compression paradigm, transfers the "hidden knowledge" in complex teacher models to lightweight student models, demonstrating excellent generalization ability in fields such as computer vision and natural language processing, and has become a key technology for balancing model performance and efficiency.

[0003] Traditional KD methods primarily rely on output layer logits matching, and existing research suggests using intermediate feature maps and refined information for transfer between teacher and student models. However, current methods struggle to fully exploit the rich intermediate representations in the teacher model, leading to limitations in the student model's fine-grained feature understanding and generalization capabilities. Two key challenges exist: first, the hierarchical semantic representation differences in deep neural networks—shallow features focus on local textures, while deep features encode global semantics, making single-level knowledge transfer insufficient to fully capture the model's representational capabilities; second, insufficient discriminative optimization of the feature space, where traditional L2 or KL divergence alignment can lead to over-smoothing of the student model, making it difficult to maintain sensitivity to class boundaries in a compact architecture.

[0004] In recent years, the fusion of self-supervised learning (SSL) and contrastive learning has injected new vitality into knowledge distillation. SSL, by generating supervisory signals from the data itself, significantly reduces reliance on manual annotation and has become an important way to improve model representation capabilities in recent years. Its core idea is to guide the model to learn the intrinsic structure of the data by designing surrogate tasks (such as rotation prediction, jigsaw puzzle reconstruction, mask modeling, etc.), thereby capturing generalizable feature representations. This characteristic naturally aligns with the goals of knowledge distillation: on the one hand, self-supervised tasks can enhance the robustness of intermediate layer features in teacher-student models, mitigating knowledge transfer bias caused by capacity differences; on the other hand, self-supervised techniques such as contrastive learning can explicitly model sample relationships in the feature space, providing fine-grained similarity supervision for distillation. Based on this, this invention proposes a hierarchical contrastive distillation method and system based on self-supervised enhancement tasks to solve the aforementioned problems existing in the prior art. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a hierarchical contrastive distillation method and system based on self-supervised augmentation tasks. This invention achieves collaborative optimization of multi-level semantic and contrastive discriminative learning, transmits intra-class correlations between corresponding levels, improves the model's sensitivity to deep feature pairs, and enhances the performance of student models.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A hierarchical comparative distillation method based on a self-supervised augmentation task includes the following steps:

[0008] Step 1: In the input dataset of the image classification task, self-supervised augmentation is performed on the data by defining various image transformations, and a joint label space is generated together with the original image. The joint label space is then input into the larger model (teacher model) and the smaller model (student model) respectively.

[0009] Step 2: Add an auxiliary classifier to the original teacher model, train the enhanced teacher model (i.e., add the auxiliary classifier) ​​to generate the joint probability distribution on the joint label space of Step 1 (original classification image task label space and self-supervised enhancement label space), and extract the intermediate layer feature information of the teacher model during training.

[0010] Step 3: Training the student model: First, while learning the joint probability distribution of the teacher model and calculating the probability distribution loss, input the self-supervised augmentation image generated in Step 1 for image classification task training; second, pass the intermediate layer features of the augmented teacher model and the student model through an independent hierarchical projection head to project the feature information onto a unit sphere contrast space to bring the features closer together and calculate the feature loss; finally, add the two losses with the final layer logit loss of the student model in a weighted sum to obtain the final loss.

[0011] This invention unifies multi-level semantic alignment and contrastive discriminative learning, conveys intra-class correlations between corresponding levels, improves the model's sensitivity to deep feature alignment, and significantly enhances the performance of the student model.

[0012] Preferably, in step 1, both the teacher model and the student model adopt a deep neural network structure, such as ResNet or Wide ResNet (WRN).

[0013] Preferably, in step 1, the original tag space is defined. And through various image transformations Data augmentation forms define self-supervised augmentation label space By using Cartesian product, a joint label space is generated from the two label spaces. The radix represents the tag space, and * indicates element-wise multiplication.

[0014] Preferably, in step 2, an auxiliary classifier is added to the teacher model to extract richer task-aware information. Specifically, the auxiliary classifier includes a feature extraction module, a global average pooling layer, and a linear classifier, wherein the dimension of the linear classifier matches the auxiliary task used; for example, for a self-supervised augmentation task, the dimension is [dimensionality missing]. The feature extraction module consists of several building blocks that are identical to those in the original backbone network (i.e., the incomplete blocks in ResNets).

[0015] Preferably, in step 2, the input to the network f(·) is defined as x. The model f(t) represents j (x) The feature information in the Lth convolutional stage, respectively in the Lth stage Transmit to the corresponding auxiliary branch The learning assistance task is performed. The final layer output is in the original category space. The original task probability distribution is as follows:

[0016]

[0017] The probability distribution of the original categories in the intermediate layer is represented as follows:

[0018]

[0019] Input in tag space In the process of converting an image to t j (x), where And t1(x) = x, the intermediate layer will obtain a self-supervised classification probability distribution:

[0020]

[0021] In the joint space The self-supervised augmentation probability distribution can be obtained from the above:

[0022]

[0023] Therefore, after adding the auxiliary branch, a teacher model with a self-supervised augmentation probability distribution needs to be retrained. In the loss calculation of the teacher model, unlike the conventional approach, cross-entropy needs to be calculated between the output and its corresponding label in different label spaces. The specific implementation is as follows:

[0024]

[0025] Where N represents the original task label space The calculation is performed above, where L represents the total number of auxiliary classifiers, l represents the l-th stage, τ is the temperature coefficient, and y is the true label in the original task label.

[0026]

[0027] in, To add label spaces with various image variations, i.e., to achieve self-supervised augmentation tasks, let j be the ground truth label in the label space M. In the joint label space... Up, use q l (t j Let (x) and τ) represent the probability distribution of the model output, and the cross-entropy loss be:

[0028]

[0029] Preferably, in step 3, when training the student model, loss calculations need to be performed on the probability distribution and feature distribution output by the student model respectively. For the logit information of the intermediate layer (this part of the logit information is generated by the student model during training under self-supervised augmentation input), the student model loss is narrowed down using the KL divergence calculation method to approximate the probability distribution generated by the teacher model through the auxiliary classifier. The specific calculation formula is as follows:

[0030]

[0031] Where τ is the distillation temperature coefficient. The logit information of the final layer is represented by minimizing the Kullback-Leibler divergence distance between the logits generated by the teacher model and the student model:

[0032]

[0033] Preferably, in step 3, the feature list (features_list:[feature1,feature2,…featureN]) of the intermediate layer between the teacher model and the student model is input into the hierarchical projection head, specifically including:

[0034] Step 3.1: Feed the features in the feature list into the global average pooling module (this global average pooling layer is the global average pooling layer in the hierarchical projection head) and perform spatial dimension compression. This significantly reduces the number of parameters while preserving channel information. The formula is as follows:

[0035]

[0036] Step 3.2: The compressed features are fed into independent projection heads, which assign independent linear transformation matrices to each layer. Map the features to a uniform dimension D and perform L2 normalization:

[0037]

[0038] Step 3.3: Constrain the features to a unit hypersphere and calculate the cosine similarity:

[0039]

[0040] Preferably, in step 3, after the transformation by the hierarchical projection head, the intermediate layer feature outputs of the teacher model and the student model can be aligned and input into the contrastive loss function to bring the features closer. By calculating the cosine similarity of the normalized features of the corresponding layers of the teacher model and the student model, the contrastive loss is used to maximize the similarity of positive sample pairs.

[0041]

[0042] Where τ is the contrast learning temperature coefficient.

[0043] Preferably, in step 3, the final loss is obtained by using the logit distillation loss of the intermediate layers of the model, the feature contrast learning loss, and the logit loss of the final layer of the model. Where α and β are weighting parameters.

[0044] This invention also discloses a hierarchical comparative distillation method based on a self-supervised enhancement task, used to perform the above method, which includes the following modules:

[0045] The self-supervised enhancement module is used to perform self-supervised enhancement on the input dataset of the image classification task by defining various image transformations, and generate a joint label space together with the original image. The joint label space is then input into the teacher model and the student model.

[0046] Auxiliary classifier module: used to add an auxiliary classifier to the original teacher model, train the enhanced teacher model to generate a joint probability distribution in the joint label space, and extract the intermediate layer feature information of the teacher model during training;

[0047] The student model training module is used to train the image classification task by inputting a self-supervised augmented image when the student model learns the joint probability distribution of the teacher model and calculates the probability distribution loss. Then, the intermediate layer features of the augmented teacher model and the student model are passed through an independent hierarchical projection head to project the feature information onto a unit sphere contrast space to bring the features closer together and calculate the feature loss. Finally, the two losses are weighted and added to the final layer logit loss of the student model to obtain the final loss.

[0048] This invention combines the technical advantages of self-supervised learning and contrastive learning in knowledge distillation, and adopts the idea of ​​synergistic optimization of multi-level semantic and contrastive discriminative learning. This enables the multi-level semantic knowledge information of the teacher model to be effectively distilled into the student model, which transmits the intra-class correlation between corresponding levels, improves the model's sensitivity to deep feature pairs, and enhances the performance of the student model. Attached Figure Description

[0049] To more clearly illustrate the hierarchical comparative distillation method based on self-supervised augmentation tasks described in this invention, the various parts involved in the entire process are illustrated below with accompanying drawings.

[0050] Figure 1 This is a flowchart of a preferred embodiment of the present invention, which describes a hierarchical comparative distillation method based on a self-supervised augmentation task.

[0051] Figure 2 This is a schematic diagram of the auxiliary classifier added in this invention;

[0052] Figure 3 This is a detailed flowchart of the hierarchical projection head designed to achieve layered projection in this invention;

[0053] Figure 4 This is a flowchart illustrating the feature shape transformation process of the feature through the hierarchical projection head in this invention.

[0054] Figure 5 This is a schematic diagram illustrating the feature convergence through contrastive learning used in this invention;

[0055] Figure 6 This is a block diagram of a hierarchical comparative distillation system based on a self-supervised enhancement task, according to a preferred embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0058] A preferred embodiment of the present invention discloses a hierarchical comparative distillation method based on a self-supervised augmentation task, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0059] Step 1: In the input dataset of the image classification task, self-supervised augmentation is performed on the data by defining various image transformations, and a joint label space is generated together with the original image. The joint label space is then input into the larger model (teacher model) and the smaller model (student model) respectively.

[0060] Step 2: Add an auxiliary classifier to the original teacher model, such as... Figure 2 As shown, the enhanced teacher model is trained to generate a joint probability distribution in the joint label space of the original classification image task label space and the self-supervised enhanced label space, and the intermediate layer feature information of the teacher model is extracted during the training process.

[0061] Step 3: Training the Student Model: First, while the student model learns the joint probability distribution of the teacher model and calculates the probability distribution loss, it is trained on the self-supervised augmented image generated in Step 1 for an image classification task. During this process, it learns the joint probability distribution knowledge of the teacher model. Second, the intermediate layer features of the augmented teacher model and the student model are passed through an independent hierarchical projection head, such as... Figure 3 As shown, the feature information is projected onto a unit sphere contrast space to bring the features closer together, such as... Figure 5 As shown, the feature loss is calculated; finally, the two losses are weighted and added to the logit loss of the final layer of the student model to obtain the final loss.

[0062] This embodiment achieves performance enhancement for image classification by using a hierarchical contrastive distillation method based on a self-supervised augmentation task. The student model enhances knowledge distillation performance by learning deeper semantic information from the intermediate layers of the teacher model.

[0063] Specifically, in step 1 of this embodiment, both the teacher model and the student model employ a deep neural network structure. The original label space is defined. Through various image transformations Data augmentation forms define self-supervised augmentation label space By using Cartesian product, a joint label space is generated from the two label spaces. The radix represents the tag space, and * indicates element-wise multiplication.

[0064] Specifically, in step 2, an auxiliary classifier is added to both the teacher and student models. This auxiliary classifier includes a feature extraction module, a global average pooling layer, and a linear classifier to extract richer task-aware information. The dimension of the linear classifier matches the auxiliary task used; for example, for a self-supervised augmentation task, the dimension is [dimensionality missing]. The feature extraction module consists of several building blocks that are identical to those in the original backbone network (i.e., the incomplete blocks in ResNets).

[0065] Define the input of network f(·) as x, The model f(t) represents j (x) The feature information in the Lth convolutional stage, respectively in the Lth stage Transmit to the corresponding auxiliary branch The learning assistance task is performed. The final layer output is in the original category space. The original task probability distribution is as follows:

[0066]

[0067] The probability distribution of the original categories in the intermediate layer is represented as follows:

[0068]

[0069] Input in tag space In the process of converting an image to t j (x), where And t1(x) = x, the intermediate layer will obtain a self-supervised classification probability distribution:

[0070]

[0071] In the joint space The self-supervised augmentation probability distribution can be obtained from the above.

[0072]

[0073] After adding the auxiliary branch, a teacher model with a self-supervised augmentation probability distribution needs to be retrained. Unlike the conventional approach, the loss calculation for the teacher model requires calculating the cross-entropy between the output and its corresponding label in different label spaces. The specific implementation is as follows:

[0074]

[0075] Where N represents the original task label space The calculation is performed above, where L represents the total number of auxiliary classifiers, l represents the l-th stage, τ is the temperature coefficient, and y is the true label in the original task label.

[0076]

[0077] in, To add label spaces with various image variations, i.e., to achieve self-supervised augmentation tasks, let j be the ground truth label in the label space M. In the joint label space... Up, use ql (t j Let (x) and τ) represent the probability distribution of the model output. The cross-entropy loss, which is the computational loss for training the teacher model, is:

[0078]

[0079] Specifically, in step 3, training the student model requires calculating the loss for both the probability distribution and feature distribution output by the student model. For the logit information of the intermediate layers, the student model loss is approximated using the KL divergence method and compared to the probability distribution generated by the teacher model through the auxiliary classifier. The specific calculation formula is as follows:

[0080]

[0081] Where τ is the distillation temperature coefficient. The logit information of the final layer is represented by calculating the KL divergence distance between the logits generated by the teacher model and the student model:

[0082]

[0083] In step 3, the feature lists (features_list: [feature1, feature2, ..., featureN]) of the intermediate layers of the teacher model and student model are input into the hierarchical projection head, and the projection-normalized feature list (proj_list: [proj_feature1, proj_feature2, ..., proj_featureN]) is output. The feature relationships are one-to-one, and the calculation method specifically includes:

[0084] The features from each intermediate layer of the teacher and student models in the feature list are fed into the global average pooling module and subjected to spatial dimension compression. This significantly reduces the number of parameters while preserving channel information. The formula is as follows:

[0085]

[0086] The compressed features are fed into independent projection heads, which assign independent linear transformation matrices to each layer. Map the features to a uniform dimension D and perform L2 normalization:

[0087]

[0088] The specific transformation process of the feature shape is as follows: Figure 4 As shown, the features are constrained to a unit hypersphere, and cosine similarity is calculated:

[0089]

[0090] After transformation by the hierarchical projection head, the intermediate layer feature outputs of the teacher model and the student model can be aligned and input into the contrastive loss function to bring the features closer. By calculating the cosine similarity of the normalized features of corresponding layers in the teacher model and the student model, the contrastive loss is then used to maximize the similarity of positive sample pairs.

[0091]

[0092] Where τ is the contrast learning temperature coefficient.

[0093] The final loss is obtained by using the logit distillation loss of the intermediate layers, the feature contrastive learning loss, and the logit loss of the final layer. Where α and β are weighting parameters.

[0094] The method of the present invention will be compared with the prior art in the following experiments:

[0095] Hierarchical Contrastive Distillation (HCD) represents the method described in this invention. "-" indicates experimental results lacking in existing technologies. "*" indicates methods based on self-supervised augmentation tasks. Image classification tasks were performed using the CIFAR-100 dataset, as shown in Table 1. KD, FitNet, SP, RKD, CRD, SSKD, CRCD, PACKD, CTKD, CAT-KD, LSKD, WTTM, SKD, CRC, and HSAKD are existing technologies.

[0096] Table 1: Image classification accuracy results (%) on the CIFAR-100 dataset

[0097]

[0098] The superiority of the invention method can be demonstrated from the above experimental results. In order to prove the effectiveness of the invention from the perspective of the distillation framework, the experimental results in bold are the best performing methods among all the comparison methods, and the underlined methods are the distillation results without applying the self-supervised reinforcement task. It can be seen from the results that the method of the invention outperforms all existing technologies in the case marked with *, and even without using the self-supervised reinforcement task, the method of the invention is the best method in the case of half teacher-student model pairs.

[0099] like Figure 6 As shown, this embodiment discloses a hierarchical comparative distillation system based on a self-supervised augmentation task, used to perform the above method, including the following modules:

[0100] The self-supervised enhancement module is used to perform self-supervised enhancement on the input dataset of the image classification task by defining various image transformations, and generate a joint label space together with the original image. The joint label space is then input into the teacher model and the student model.

[0101] Auxiliary classification module: used to add an auxiliary classifier to the original teacher model, train the enhanced teacher model, generate a joint probability distribution in the joint label space, and extract the intermediate layer feature information of the teacher model during training;

[0102] The student model training module is used to train the image classification task by inputting the generated self-supervised augmented image when the student model learns the joint probability distribution of the teacher model and calculates the probability distribution loss. Then, the intermediate layer features of the augmented teacher model and the student model are passed through an independent hierarchical projection head to project the feature information onto a unit sphere contrast space to bring the features closer together and calculate the feature loss. Finally, the two losses are weighted and added to the final layer logit loss of the student model to obtain the final loss.

[0103] Other aspects of this embodiment can be found in the above method embodiments.

[0104] In summary, this invention discloses a hierarchical contrastive distillation method and system based on a self-supervised enhancement task. This invention, by combining the probability distributions of multi-level self-supervised tasks and the original task, deeply mines semantic information at different levels of the model. During the distillation process, richer hidden knowledge from the teacher model can be transferred to the student model. By inputting the intermediate layer features of the teacher and student models into a hierarchical projection head, the corresponding intermediate layer outputs of the teacher and student models are projected onto a contrastive learning unit sphere space to bring features closer together, transferring intra-class correlations between corresponding levels and improving the model's sensitivity to deep feature pairings. In conclusion, this invention achieves synergistic optimization of multi-level semantics and contrastive discriminative learning, significantly improving the performance and versatility of knowledge distillation in image classification tasks.

[0105] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0106] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A hierarchical comparative distillation method based on a self-supervised augmentation task, characterized in that, Includes the following steps: Step 1: In the input dataset of the image classification task, self-supervised augmentation is performed on the data by defining various image transformations, and a joint label space is generated together with the original image. The joint label space is then input into the teacher model and the student model. Step 2: Add an auxiliary classifier to the original teacher model, train the teacher model with the auxiliary classifier added, generate the joint probability distribution in the joint label space of Step 1, and extract the intermediate layer feature information of the teacher model during training. Step 3: When the student model learns the joint probability distribution of the teacher model and calculates the probability distribution loss, it inputs the self-supervised augmentation image generated in Step 1 to train the image classification task. The intermediate layer features of the enhanced teacher model and the student model are then passed through an independent hierarchical projection head to project the feature information onto a unit sphere contrast space for feature aggregation, and the feature loss is calculated. Finally, the two losses are weighted and added to the final layer logit loss of the student model to obtain the final loss.

2. The hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 1, characterized in that, In step 1, both the teacher model and the student model adopt a deep neural network structure.

3. The hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 1, characterized in that, In step 1, the label space of the original image is defined. Through various image transformations Data augmentation forms define self-supervised augmentation label space By using Cartesian product, a joint label space is generated from the two label spaces. The radix represents the tag space, and * indicates element-wise multiplication.

4. The hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 1, characterized in that, In step 2, the auxiliary classifier includes a feature extraction module, a global average pooling layer, and a linear classifier, wherein the dimension of the linear classifier matches the auxiliary task used; the feature extraction module consists of several building blocks identical to those of the original backbone network.

5. A hierarchical comparative distillation method based on a self-supervised augmentation task according to any one of claims 1-4, characterized in that, In step 2, the input to the network f(·) is defined as x. The model f(t) represents j (x) The feature information in the Lth convolutional stage, respectively in the Lth stage Transmit to the corresponding auxiliary branch The learning assistance task is performed in the middle layer; the final layer output is in the original category space. The original task probability distribution is as follows: The probability distribution of the original categories in the intermediate layer is represented as follows: Input in tag space In the process of converting an image to t j (x), where, And t1(x) = x, the intermediate layer obtains a self-supervised classification probability distribution: In the joint space The self-supervised enhancement probability distribution is obtained above: In the loss calculation of the teacher model, it is necessary to calculate the cross-entropy between the output and the corresponding label in different label spaces. The specific implementation is as follows: Where N represents the original task label space The calculation is performed above, where L represents the total number of auxiliary classifiers, l represents the l-th stage, τ is the temperature coefficient, and y is the true label in the original task label; in, To add a label space with various image variations, where j is the true label in label space M; in the joint label space... Up, use q l (t j Let (x) and τ) represent the probability distribution of the model output, and the cross-entropy loss be:

6. The hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 5, characterized in that, In step 3, for the logit information of the intermediate layer, the student model loss is approximated by the probability distribution generated by the teacher model through the auxiliary classifier using the KL divergence calculation method. The specific calculation formula is as follows: Where τ is the distillation temperature coefficient; The final layer logit information is represented by calculating the KL divergence distance between the logits generated by the teacher model and the student model:

7. The hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 6, characterized in that, In step 3, the intermediate layer feature lists of the teacher model and the student model are input into the hierarchical projection head, specifically including: Step 3.1: Feed the features in the feature list into the global average pooling module and perform spatial dimension compression, expressed by the formula: Step 3.2: The compressed features are fed into independent projection heads, which assign independent linear transformation matrices to each layer. Map the features to a uniform dimension D and perform L2 normalization: Step 3.3: Constrain the features to a unit hypersphere and calculate the cosine similarity:

8. The hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 7, characterized in that, In step 3, after calculating the cosine similarity of the normalized features of corresponding layers in the teacher model and student model, contrastive loss is used to maximize the similarity of positive sample pairs: Where τ is the contrast learning temperature coefficient.

9. A hierarchical comparative distillation method based on a self-supervised augmentation task according to claim 8, characterized in that, In step 3, the final loss is obtained by using the logit distillation loss of the intermediate layers, the feature contrastive learning loss, and the logit loss of the final layer: Where α and β are weighting parameters.

10. A hierarchical comparative distillation system based on a self-supervised augmentation task, used to perform the method as described in any one of claims 1-9, characterized in that, Includes the following modules: The self-supervised enhancement module is used to perform self-supervised enhancement on the input dataset of the image classification task by defining various image transformations, and generate a joint label space together with the original image. The joint label space is then input into the teacher model and the student model. Auxiliary classification module: used to add an auxiliary classifier to the original teacher model, train the teacher model with the auxiliary classifier to generate a joint probability distribution in the joint label space, and extract the intermediate layer feature information of the teacher model during training; Student Model Training Module: Used to train image classification tasks by inputting the self-supervised augmented images generated by the self-supervised augmentation module when the student model learns the joint probability distribution of the teacher model and calculates the probability distribution loss; The intermediate layer features of the enhanced teacher model and the student model are then passed through an independent hierarchical projection head to project the feature information onto a unit sphere contrast space for feature aggregation, and the feature loss is calculated. Finally, the two losses are weighted and added to the final layer logit loss of the student model to obtain the final loss.

Images