Knowledge distillation method and device based on self-supervised learning and storage medium
By introducing instance feature queues in knowledge distillation and optimizing similarity distribution, the performance degradation caused by feature alignment among different network architectures is solved, and better student model performance and adaptability are achieved.
Patent Information
- Application Number
- CN202411994933.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-29
AI Technical Summary
When existing knowledge distillation methods directly align feature distributions between different network architectures (such as Transformer and CNN), it is easy to lead to information loss or negative migration, which leads to poor student model performance.
Using a knowledge distillation method based on self-supervised learning, the knowledge distillation of the teacher model to the student model is realized by introducing a queue of instance features that dynamically maintain the teacher model to the extracted features in the model pre-training stage by optimizing the instance similarity distribution.
It effectively avoids information loss or negative transfer caused by direct alignment feature space, enhances the model performance of the student model, and improves the model adaptability and generalization ability of the model.
Smart Images
Figure CN119904713A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of machine vision technology, and in particular, relates to a knowledge distillation method, device and storage medium based on self-supervised learning. Background Art
[0002] Knowledge Distillation (KD) is a method of guiding the training of student models through a teacher model with excellent performance. The purpose is to compress the model without significantly reducing the performance and pass the capabilities and feature advantages of the teacher model to the student model.
[0003] Knowledge distillation technology can be mainly divided into two types: logit-based knowledge distillation and feature-based knowledge distillation. The logit-based knowledge distillation method relies on labeled data, and it requires the logit output in the supervised task scenario, which cannot be applied in the self-supervised learning scenario. At present, feature-based knowledge distillation guides student learning by bringing the distribution of the intermediate layer features of the teacher model and the student model closer. However, when the teacher and the student use different network architectures, such as Transformer and CNN, the differences in model structure and inductive bias will lead to information loss or negative transfer when directly aligning feature distributions, which in turn leads to poor performance of the student model obtained by distillation, and the knowledge distillation effect is poor. Summary of the invention
[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a knowledge distillation method, device and storage medium based on self-supervised learning, which can better realize the knowledge distillation from the teacher model to the student model in the pre-training stage, ensuring that the student model has good performance.
[0005] In a first aspect, the present application provides a knowledge distillation method based on self-supervised learning, comprising:
[0006] Inputting the current sample image into the teacher model and the student model respectively to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model;
[0007] updating a currently maintained instance feature queue according to the first image feature, wherein the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length;
[0008] respectively determining a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue;
[0009] Pre-training distillation of the teacher model to the student model is performed according to the first similarity distribution and the second similarity distribution.
[0010] In some embodiments, respectively determining a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the instance feature queue comprises:
[0011] Projecting the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature;
[0012] Respectively calculating the similarity between the first projection feature and the second projection feature and each instance feature in the instance feature queue to obtain a first similarity and a second similarity;
[0013] The first similarity and the second similarity are converted into a first similarity distribution and a second similarity distribution respectively.
[0014] In some embodiments, projecting the first image feature and the second image feature into the same feature space to obtain the first projected feature and the second projected feature includes:
[0015] Performing average pooling on the second image features to obtain pooled features;
[0016] The first image feature is input into a first projection layer to obtain a first projection feature, and the pooled feature is input into a second projection layer to obtain a second projection feature, wherein the first projection layer and the second projection layer have the same network structure, and during the pre-training distillation process, the parameters of the first projection layer are frozen.
[0017] In some embodiments, the instance feature queue is a first-in-first-out queue, and updating the currently maintained instance feature queue according to the first image feature includes:
[0018] The instance feature at the head of the instance feature queue is removed, and the first image feature is added as an instance feature to the tail of the instance feature queue.
[0019] In some embodiments, performing pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution includes:
[0020] Calculating a KL divergence between the first similarity distribution and the second similarity distribution;
[0021] According to the KL divergence corresponding to all sample images in the current batch, the parameters of the student model are adjusted.
[0022] In some embodiments, the knowledge distillation method based on self-supervised learning further includes:
[0023] calculating a mean square error between the first image feature and the second image feature;
[0024] The adjusting the parameters of the student model according to the KL divergence corresponding to all sample images of the current batch includes: adjusting the parameters of the student model according to the KL divergence and the mean square error corresponding to all sample images of the current batch.
[0025] In a second aspect, the present application provides a knowledge distillation device based on self-supervised learning, comprising:
[0026] An input module, used for inputting the current sample image into the teacher model and the student model respectively, so as to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model;
[0027] An updating module, used for updating a currently maintained instance feature queue according to the first image feature, wherein the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length;
[0028] A determination module, configured to respectively determine a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue;
[0029] A pre-training module performs pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution.
[0030] In some embodiments, the determining module is specifically used to:
[0031] Projecting the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature;
[0032] Respectively calculating the similarity between the first projection feature and the second projection feature and each instance feature in the instance feature queue to obtain a first similarity and a second similarity;
[0033] The first similarity and the second similarity are converted into a first similarity distribution and a second similarity distribution respectively.
[0034] In some embodiments, the determining module is specifically used to:
[0035] Performing average pooling on the second image features to obtain pooled features;
[0036] The first image feature is input into a first projection layer to obtain a first projection feature, and the pooled feature is input into a second projection layer to obtain a second projection feature, wherein the first projection layer and the second projection layer have the same network structure, and during the pre-training distillation process, the parameters of the first projection layer are frozen.
[0037] In some embodiments, the instance feature queue is a first-in-first-out queue, and the update module is specifically used to:
[0038] The instance feature at the head of the instance feature queue is removed, and the first image feature is added as an instance feature to the tail of the instance feature queue.
[0039] In some embodiments, the pre-training module is specifically used for:
[0040] Calculating a KL divergence between the first similarity distribution and the second similarity distribution;
[0041] According to the KL divergence corresponding to all sample images in the current batch, the parameters of the student model are adjusted.
[0042] In some embodiments, the determining module is further configured to:
[0043] calculating a mean square error between the first image feature and the second image feature;
[0044] The pre-training module is specifically used to adjust the parameters of the student model according to the KL divergence and the mean square error corresponding to all sample images in the current batch.
[0045] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned knowledge distillation methods based on self-supervised learning.
[0046] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the knowledge distillation method based on self-supervised learning as described in any one of the above items is implemented.
[0047] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned knowledge distillation methods based on self-supervised learning.
[0048] The knowledge distillation method, device, storage medium, electronic device and computer program product based on self-supervised learning provided by the embodiments of the present application respectively input the current sample image into the teacher model and the student model to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model; update the currently maintained instance feature queue according to the first image feature, the instance feature in the instance feature queue is obtained by feature extraction by the teacher model and has a fixed queue length; respectively determine the first similarity distribution and the second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue; perform pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution, that is, introduce an instance feature queue that dynamically maintains the features extracted by the teacher model, and realize knowledge distillation from the teacher model to the student model by optimizing the instance similarity distribution in the model pre-training stage, thereby solving the problem of model performance degradation caused by direct alignment of features of different architecture models in the existing distillation method, avoiding information loss or negative transfer caused by direct alignment of feature space, effectively enhancing the model performance of the student model, and improving the model adaptability and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0050] Figure 1 Schematic diagram of a process of a knowledge distillation method based on self-supervised learning provided in an embodiment of the present application;
[0051] Figure 2 It is a schematic diagram of the framework structure for performing knowledge distillation during the pre-training process provided in the embodiment of the present application;
[0052] Figure 3 is another flow chart of the knowledge distillation method based on self-supervised learning provided in an embodiment of the present application;
[0053] Figure 4 is a schematic diagram of the structure of a knowledge distillation device based on self-supervised learning provided in an embodiment of the present application;
[0054] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0055] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.
[0057] At present, knowledge distillation that transfers the feature advantages of large basic visual models (such as Transformer) to small models (such as CNN) can be achieved in the model pre-training stage or the model fine-tuning stage. In the fine-tuning stage, the combination of specific task loss and distillation loss can effectively improve the performance of downstream tasks of the model. However, in actual application, this requires additional calculation and storage of large models, which will greatly reduce the training efficiency of small models and consume more storage space. If distillation is performed in the pre-training stage, it will not bring any additional calculation and storage. The large model can effectively provide high-quality initial features for the small model in the pre-training stage, thereby accelerating the training efficiency of the small model in the fine-tuning stage and improving the model accuracy.
[0058] In some embodiments, a feature-based knowledge distillation method is used to achieve the distillation of the basic visual large model to a small model. This distillation method guides student learning by bringing the distribution of intermediate layer features of the teacher model and the student model closer. However, when the network architectures of the teacher and the student are different (such as Transformer and CNN), resulting in different feature dimensions and inductive biases, directly aligning the feature distribution may lead to information loss or negative transfer, which in turn leads to poor performance of the student model obtained by distillation, and the knowledge distillation effect is poor.
[0059] In order to solve at least one of the above technical problems, an embodiment of the present application provides a knowledge distillation method, device, storage medium, electronic device and computer program product based on self-supervised learning.
[0060] See also Figure 1 , Figure 1 1 is a flow chart of a knowledge distillation method based on self-supervised learning provided in an embodiment of the present application. The knowledge distillation method based on self-supervised learning is applied to an electronic device, and the knowledge distillation method based on self-supervised learning can be performed by a functional module in the electronic device. The knowledge distillation method based on self-supervised learning includes the following steps 101-104, wherein:
[0061] 101. Input the current sample image into the teacher model and the student model respectively to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model.
[0062] The teacher model is usually a model that has been pre-trained on a large-scale dataset and has rich feature representation capabilities. The student model is a smaller model whose architecture can be different from the teacher model. For example, see Figure 2 , Figure 2 This is a schematic diagram of the framework structure for performing knowledge distillation during the pre-training process provided by the embodiment of the present application. The teacher model is a large visual model based on the Transformer architecture, and the student model is a convolutional neural network (CNN). For two models with different architectures, the extracted features usually have different dimensions. For example, the Transformer model usually captures global and local information through a self-attention mechanism, while the CNN gradually extracts features from local to global through convolution operations.
[0063] Of course, in other embodiments, the student model and the teacher model may also use the same architecture, which is not limited here. In addition, before the sample image is input into the model, it may be preprocessed to ensure that the input image meets the input requirements of the model, and the preprocessing includes but is not limited to data cleaning, normalization, scaling and other operations.
[0064] 102. Update a currently maintained instance feature queue according to the first image feature, where the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length.
[0065] If the queue length of the instance feature queue is fixed to m, it includes m instance features, which are image features extracted by the teacher model and are continuously updated according to the features extracted by the teacher model of the same batch of sample images. The sample images of the same batch can be obtained by augmenting a small number of sample images with different data, which are unlabeled data and used for self-supervised learning.
[0066] In some embodiments, the example feature queue is a first-in-first-out (FIFO) queue, and the above step 102 specifically includes:
[0067] The instance feature at the head of the instance feature queue is removed, and the first image feature is added as an instance feature to the tail of the instance feature queue.
[0068] For example, see Figure 2 , before pre-training distillation, initialize an instance feature queue D with a queue length of m. For the sample image i of the current batch, when the teacher model and the student model are input, the teacher model extracts the first image feature and the student model extracts the second image feature. At this time, the first image feature needs to be added to the instance feature queue D based on the first-in-first-out rule.
[0069] 103. Determine a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue, respectively.
[0070] In some embodiments, see Figure 3 , Figure 3 1 is another flow chart of the knowledge distillation method based on self-supervised learning provided in an embodiment of the present application. The above step 103 specifically includes the following steps 1031-1033, wherein:
[0071] 1031. Project the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature;
[0072] In some embodiments, the above step 1031 specifically includes:
[0073] Performing average pooling on the second image feature to obtain a pooled feature;
[0074] The first image feature is input into the first projection layer to obtain a first projection feature, and the pooled feature is input into the second projection layer to obtain a second projection feature. The first projection layer and the second projection layer have the same network structure, and during the pre-training distillation process, the parameters of the first projection layer are frozen.
[0075] Among them, since the student model and the teacher model often have different feature dimensions, feature dimension alignment is required. In order to retain the feature advantages of the teacher to the greatest extent and prevent information loss, it is necessary to use the projector layer to align the output features of the student network with the feature dimensions of the teacher network. The projector refers to a network structure used to map high-dimensional features to low-dimensional space for visualization or further analysis.
[0076] The teacher model branch and the student model branch use the same projector structure, which is the projector used in the pre-training stage of the large model. During the pre-training process, the teacher model branch uses a frozen projector, while the projector used by the student model branch participates in the training, so as to map the features of the student model to the same dimension as the teacher model, close the feature representation of the two networks, and complete the feature channel alignment to the teacher model.
[0077] 1032. Calculate the similarity between the first projection feature and the second projection feature and each instance feature in the instance feature queue respectively to obtain a first similarity and a second similarity.
[0078] Among them, for the current processing sample, the features output by the student and teacher models are respectively calculated similarity with the instance features in the queue, such as by Figure 2The similarity is calculated by the inner product (dot product) method shown. The similarity can also be calculated by cosine similarity or other measurement methods, which are not limited here.
[0079] 1033. Convert the first similarity and the second similarity into a first similarity distribution and a second similarity distribution, respectively.
[0080] Among them, the corresponding probability distribution can be obtained through softmax. By introducing the instance feature queue, the features extracted by the teacher model are dynamically maintained in the instance feature queue and a reference benchmark is provided for the student model, so that the goal of directly optimizing the feature distribution is transformed into optimizing the instance similarity distribution.
[0081] 104. Perform pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution.
[0082] Among them, by converting the goal of optimizing feature distribution into optimizing instance similarity distribution during the pre-training distillation process, the direct impact of feature differences caused by different model architectures can be effectively weakened, and the student model can effectively enhance the instance discrimination ability of the student model by learning the similarity distribution with the instance feature queue.
[0083] In some embodiments, see Figure 2 and Figure 3 , the above step 104 specifically includes:
[0084] 1041. Calculate the KL divergence between the first similarity distribution and the second similarity distribution;
[0085] 1042. According to the KL divergence corresponding to all sample images in the current batch, the parameters of the student model are adjusted.
[0086] Among them, KL divergence (Kullback-Leibler Divergence) is an asymmetric measure used to measure the difference between two probability distributions. During the pre-training distillation process, the network parameters of the teacher model are usually kept frozen and not adjusted. The network parameters of the student model need to be adjusted. This adjustment is achieved by minimizing the loss value, which includes KL divergence. By minimizing KL divergence, the similarity distribution of the student model can be made as close as possible to the similarity distribution of the teacher model, thereby achieving effective knowledge transfer.
[0087] In some embodiments, the knowledge distillation method based on self-supervised learning may further include:
[0088] A mean square error between the first image feature and the second image feature is calculated.
[0089] At this time, the above step 1042 specifically includes: adjusting the parameters of the student model according to the KL divergence and mean square error corresponding to all sample images in the current batch.
[0090] That is, when determining the overall loss, in addition to calculating the KL divergence, the mean square error (MSE) can also be calculated, and the parameters of the student model can be adjusted by combining the KL divergence loss and the MSE loss. In other words, in addition to optimizing the instance similarity distribution, the feature distance is directly optimized as an auxiliary means to constrain the feature space and further improve the distillation effect of the model. MSE is a loss function that measures the difference between two numerical distributions.
[0091] It should be pointed out that in the embodiment of the present application, by performing knowledge distillation from the teacher model to the student model in the pre-training stage, it can ensure that the teacher model effectively provides high-quality initial features for the student model in the pre-training stage without any additional calculations and storage, thereby accelerating the training efficiency of the student model in the fine-tuning stage and improving the model accuracy. At the same time, the pre-training distillation of the self-supervised paradigm is suitable for large-scale unlabeled data sets, without relying on label information or logit output of specific tasks.
[0092] Furthermore, in view of the difference in feature dimensions between the Transformer architecture and the CNN architecture, the feature advantages of the teacher model can be retained by using a feature projector to align feature channels. By introducing an instance feature queue that dynamically maintains the features extracted by the teacher model and using a distillation method that optimizes the instance similarity distribution instead of directly aligning the feature distribution, the problem of performance degradation caused by direct alignment of features between the Transformer and CNN architectures in existing distillation methods can be solved, and information loss or negative transfer caused by direct alignment of feature spaces can be avoided, effectively enhancing the model performance of the student model, improving the model adaptability and generalization ability, and providing better feature expression for downstream tasks.
[0093] As can be seen from the above, the knowledge distillation method based on self-supervised learning provided in the embodiment of the present application inputs the current sample image into the teacher model and the student model respectively to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model; updates the currently maintained instance feature queue according to the first image feature, and the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length; determines the first similarity distribution and the second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue respectively; performs pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution, that is, introduces an instance feature queue that dynamically maintains the features extracted by the teacher model, and realizes knowledge distillation from the teacher model to the student model by optimizing the instance similarity distribution in the model pre-training stage, thereby solving the problem of model performance degradation caused by direct alignment of features of models of different architectures in the existing distillation method, avoiding information loss or negative transfer caused by direct alignment of feature spaces, effectively enhancing the model performance of the student model, and improving the model adaptability and generalization ability.
[0094] According to the knowledge distillation method based on self-supervised learning described in the above embodiment, the present application embodiment also provides a knowledge distillation device based on self-supervised learning. Figure 4 , Figure 4 Schematic diagram of the structure of the knowledge distillation device based on self-supervised learning provided in the embodiment of the present application. The knowledge distillation device based on self-supervised learning is applied to electronic devices, and the knowledge distillation method based on self-supervised learning described in the above embodiment can be executed by the functional modules in the electronic devices.
[0095] The knowledge distillation device based on self-supervised learning includes an input module 201, an update module 202, a determination module 203 and a pre-training module 204, wherein:
[0096] An input module 201 is used to input the current sample image into the teacher model and the student model respectively to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model;
[0097] An updating module 202, configured to update a currently maintained instance feature queue according to the first image feature, wherein the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length;
[0098] A determination module 203, configured to respectively determine a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue;
[0099] The pre-training module 204 performs pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution.
[0100] In some embodiments, the determination module 203 is specifically used to:
[0101] Projecting the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature;
[0102] Respectively calculating the similarity between the first projection feature and the second projection feature and each instance feature in the instance feature queue to obtain a first similarity and a second similarity;
[0103] The first similarity and the second similarity are converted into a first similarity distribution and a second similarity distribution, respectively.
[0104] In some embodiments, the determination module 203 is specifically used to:
[0105] Performing average pooling on the second image feature to obtain a pooled feature;
[0106] The first image feature is input into the first projection layer to obtain a first projection feature, and the pooled feature is input into the second projection layer to obtain a second projection feature. The first projection layer and the second projection layer have the same network structure, and during the pre-training distillation process, the parameters of the first projection layer are frozen.
[0107] In some embodiments, the instance feature queue is a first-in-first-out queue, and the update module 202 is specifically used to:
[0108] The instance feature at the head of the instance feature queue is removed, and the first image feature is added as an instance feature to the tail of the instance feature queue.
[0109] In some embodiments, the pre-training module 204 is specifically used to:
[0110] Calculating a KL divergence between the first similarity distribution and the second similarity distribution;
[0111] According to the KL divergence corresponding to all sample images in the current batch, the parameters of the student model are adjusted.
[0112] In some embodiments, the determination module 203 is further configured to:
[0113] Calculating a mean square error between the first image feature and the second image feature;
[0114] The pre-training module 204 is specifically used to adjust the parameters of the student model according to the KL divergence and the mean square error corresponding to all sample images in the current batch.
[0115] It should be noted that the specific details of each module unit in the above-mentioned knowledge distillation device 200 based on self-supervised learning have been described in detail in the embodiment of the above-mentioned knowledge distillation method based on self-supervised learning, and will not be repeated here.
[0116] As can be seen from the above, the knowledge distillation device 200 based on self-supervised learning provided in the embodiment of the present application inputs the current sample image into the teacher model and the student model respectively through the input module 201 to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model; the updating module 202 updates the currently maintained instance feature queue according to the first image feature, and the instance feature in the instance feature queue is obtained by feature extraction by the teacher model and has a fixed queue length; the determination module 203 determines the first similarity distribution and the second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue respectively; the pre-training module 204 performs pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution, that is, introduces an instance feature queue that dynamically maintains the features extracted by the teacher model, and realizes the knowledge distillation from the teacher model to the student model by optimizing the instance similarity distribution in the model pre-training stage, thereby solving the problem that the existing distillation method directly aligns features of different architecture models, resulting in a decrease in model performance, avoiding information loss or negative transfer caused by directly aligning feature spaces, effectively enhancing the model performance of the student model, and improving the model adaptability and generalization ability.
[0117] In some embodiments, Figure 5 As shown, an embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, each process of the above-mentioned knowledge distillation method embodiment based on self-supervised learning is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0118] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
[0119] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0120] The electronic device 400 includes but is not limited to: a radio frequency module 401, a network module 402, an audio output module 403, an item input module 404, a sensor 405, a display module 406, a user item input module 407, an interface module 408, a memory 409 and a processor 410 and other components.
[0121] In some embodiments, the processor 410 implements the following steps when executing the program:
[0122] Inputting the current sample image into the teacher model and the student model respectively to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model;
[0123] updating a currently maintained instance feature queue according to the first image feature, wherein the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length;
[0124] Determine a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue, respectively;
[0125] According to the first similarity distribution and the second similarity distribution, pre-training distillation of the teacher model to the student model is performed.
[0126] Furthermore, the processor 410 implements the following steps:
[0127] Projecting the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature;
[0128] Respectively calculating the similarity between the first projection feature and the second projection feature and each instance feature in the instance feature queue to obtain a first similarity and a second similarity;
[0129] The first similarity and the second similarity are converted into a first similarity distribution and a second similarity distribution, respectively.
[0130] Furthermore, the processor 410 implements the following steps:
[0131] Performing average pooling on the second image feature to obtain a pooled feature;
[0132] The first image feature is input into the first projection layer to obtain a first projection feature, and the pooled feature is input into the second projection layer to obtain a second projection feature. The first projection layer and the second projection layer have the same network structure, and during the pre-training distillation process, the parameters of the first projection layer are frozen.
[0133] Further, the example feature queue is a first-in-first-out queue, and the processor 410 implements the following steps:
[0134] The instance feature at the head of the instance feature queue is removed, and the first image feature is added as an instance feature to the tail of the instance feature queue.
[0135] Furthermore, the processor 410 implements the following steps:
[0136] Calculating a KL divergence between the first similarity distribution and the second similarity distribution;
[0137] According to the KL divergence corresponding to all sample images in the current batch, the parameters of the student model are adjusted.
[0138] Furthermore, the processor 410 implements the following steps:
[0139] Calculating a mean square error between the first image feature and the second image feature;
[0140] According to the KL divergence corresponding to all sample images in the current batch and the mean square error, the parameters of the student model are adjusted.
[0141] Those skilled in the art will appreciate that the electronic device 400 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 410 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0142] It should be understood that in the embodiment of the present application, the project input module 404 may include a graphics processor (Graphics Processing Unit, GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display module 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user project input module 407 includes a touch panel 4071 and at least one of other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include two parts: a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0143] The memory 409 can be used to store software programs and various data. The memory 409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 409 may include a volatile memory or a non-volatile memory, or the memory 409 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0144] The processor 410 may include one or more processing modules; the processor 410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 410.
[0145] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned knowledge distillation method embodiment based on self-supervised learning are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0146] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0147] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned knowledge distillation method based on self-supervised learning.
[0148] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0149] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0151] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0152] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0153] In the description of the present application, “plurality” means two or more.
[0154] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0155] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A knowledge distillation method based on self-supervised learning, characterized in that: include: Inputting the current sample image into the teacher model and the student model respectively to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model; updating a currently maintained instance feature queue according to the first image feature, wherein the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length; respectively determining a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue; Pre-training distillation of the teacher model to the student model is performed according to the first similarity distribution and the second similarity distribution.
2. The knowledge distillation method based on self-supervised learning according to claim 1, characterized in that: The determining respectively a first similarity distribution and a second similarity distribution of the first image feature and the second image feature with respect to the instance feature queue comprises: Projecting the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature; Respectively calculating the similarity between the first projection feature and the second projection feature and each instance feature in the instance feature queue to obtain a first similarity and a second similarity; The first similarity and the second similarity are converted into a first similarity distribution and a second similarity distribution respectively.
3. The knowledge distillation method based on self-supervised learning according to claim 2, characterized in that: The step of projecting the first image feature and the second image feature into the same feature space to obtain a first projected feature and a second projected feature includes: Performing average pooling on the second image features to obtain pooled features; The first image feature is input into a first projection layer to obtain a first projection feature, and the pooled feature is input into a second projection layer to obtain a second projection feature, wherein the first projection layer and the second projection layer have the same network structure, and during the pre-training distillation process, the parameters of the first projection layer are frozen.
4. The knowledge distillation method based on self-supervised learning according to claim 1, characterized in that: The instance feature queue is a first-in-first-out queue, and updating the currently maintained instance feature queue according to the first image feature includes: The instance feature at the head of the instance feature queue is removed, and the first image feature is added as an instance feature to the tail of the instance feature queue.
5. The knowledge distillation method based on self-supervised learning according to claim 1, characterized in that: The pre-training distillation of the teacher model to the student model according to the first similarity distribution and the second similarity distribution includes: Calculating a KL divergence between the first similarity distribution and the second similarity distribution; According to the KL divergence corresponding to all sample images in the current batch, the parameters of the student model are adjusted.
6. The knowledge distillation method based on self-supervised learning according to claim 5, characterized in that: Also includes: calculating a mean square error between the first image feature and the second image feature; The adjusting the parameters of the student model according to the KL divergence corresponding to all sample images of the current batch includes: adjusting the parameters of the student model according to the KL divergence and the mean square error corresponding to all sample images of the current batch.
7. A knowledge distillation device based on self-supervised learning, characterized in that: include: An input module, used for inputting the current sample image into the teacher model and the student model respectively, so as to obtain the first image feature extracted by the teacher model and the second image feature extracted by the student model; An updating module, used for updating a currently maintained instance feature queue according to the first image feature, wherein the instance features in the instance feature queue are obtained by feature extraction by the teacher model and have a fixed queue length; A determination module, configured to respectively determine a first similarity distribution and a second similarity distribution of the first image feature and the second image feature relative to the updated instance feature queue; A pre-training module performs pre-training distillation from the teacher model to the student model according to the first similarity distribution and the second similarity distribution.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the knowledge distillation method based on self-supervised learning is implemented as described in any one of claims 1-6.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the knowledge distillation method based on self-supervised learning is implemented as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the knowledge distillation method based on self-supervised learning is implemented as described in any one of claims 1-6.