Knowledge migration method and system based on cross-view consistency regularization

Abstract

The invention provides a knowledge migration method and system based on cross-view consistency regularization, and the method comprises the steps: generating a multi-view training sample, transmitting the multi-view training sample into a teacher network and a student network for forward propagation, and obtaining a prediction logic value of each view sample; establishing a consistency regularization constraint for the teacher network and the student network; the prediction logic value of the student network is supervised by using a consistency regularization constraint and a truth value label together, and a total training loss value is calculated; and according to the current total training loss value, updating parameters of the student network, and continuously circulating the optimization process until a preset maximum training period is reached, thereby completing knowledge migration. According to the invention, common problems such as confirmation error, too high confidence of a teacher network and overfitting of a student network in knowledge distillation are effectively relieved; the negative effect of an unreliable teacher soft label in guiding students to learn on the network is effectively reduced, so that the problem of confirmation error in knowledge distillation is further relieved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation technology, and in particular, to a knowledge migration method and system based on cross-view consistency regularization, as well as a corresponding computer terminal and computer-readable storage medium. Background Art

[0002] With the development of deep learning technology, the performance of deep neural network models on various downstream visual tasks has been continuously improved. However, the performance improvement is inevitably accompanied by a significant increase in the number of model parameters and complexity, which faces challenges in the deployment of practical applications, especially in edge devices with limited computing resources. In order to reduce the number of network parameters and complexity while maintaining model performance as much as possible, Hinton et al. first proposed a model compression method, namely knowledge distillation, by migrating the task capabilities of a large high-performance network (teacher network) to a small lightweight network (student network).

[0003] To date, knowledge distillation algorithms can be roughly divided into methods based on feature map transfer, methods based on prediction value transfer, and methods based on implicit relationship transfer. Methods based on feature map transfer enable the student network to learn to imitate and generate the same intermediate feature maps as those generated by the teacher network, thereby indirectly obtaining the task capabilities of the teacher network. Representative algorithms of this type include FitNets, VID, and OFD. Similarly, methods based on prediction value transfer enable the end of the student network to imitate and output the prediction values ​​output by the teacher network, which are logical values ​​(logits) in image classification tasks, thereby directly obtaining its task capabilities. Typical methods include NKD, DKD, and MLLD. Methods based on implicit relationship transfer mine implicit relationships between samples and feature levels contained in the teacher network, and enable the student network to learn to replicate such relationships in its output. Representative algorithms include RKD, CRD, and ICKD.

[0004] Although the performance of knowledge distillation algorithms has made great progress in recent years, they are still limited by two basic problems: 1. Overconfident teacher; 2. Confirmation bias. When the teacher network produces an output with extremely high confidence, the knowledge contained in its predicted output will be dominated by the knowledge with the highest confidence, and a large amount of dark knowledge that is crucial to the performance of knowledge distillation will be significantly suppressed, so that the student network can only learn specific dominant knowledge, but cannot effectively acquire a large amount of beneficial but diminishing dark knowledge. When the teacher network outputs incorrect prediction information, the student may overfit to the incorrect soft label and thus learn incorrect knowledge. This phenomenon is confirmation bias. Both the overconfident teacher and confirmation bias are widely present in existing algorithms and fundamentally limit the performance of existing knowledge distillation methods. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a knowledge transfer method and system based on cross-view consistency regularization, and also provides a corresponding computer terminal and computer-readable storage medium.

[0006] According to one aspect of the present invention, a knowledge transfer method based on cross-view consistency regularization is provided, comprising:

[0007] Generate multi-view training samples, send the multi-view training samples to the teacher network and the student network for forward propagation, and obtain the predicted logical values ​​of the teacher network and the student network for each view sample;

[0008] Establishing a consistency regularization constraint on the teacher network and the student network;

[0009] Using the consistency regularization constraint to supervise the predicted logical value of the student network, and using the true value label to supervise the predicted logical value of the student network, and calculating the total training loss value;

[0010] According to the current total training loss value, the parameters of the student network are updated, and the optimization process is continuously cycled until the preset maximum training cycle is reached to complete the knowledge transfer.

[0011] Preferably, generating multi-view training samples includes:

[0012] For any original training sample image x, we apply random cropping and random horizontal flipping operations to it, and the resulting image is the weak-view image sample x of the original training sample. w ;

[0013] For the original training sample image x, randomly apply any two image transformation operations, random cropping, random horizontal flipping and random Cutout algorithm operations in the RandAugment data enhancement method to it, and the resulting image is the strong perspective image sample x of the original training sample s .

[0014] Preferably, the multi-view training samples are sent to the teacher network and the student network for forward propagation to obtain the predicted logical values ​​of each view sample of the teacher network and the student network, including:

[0015] For the original training sample image x, its weak perspective image sample x w The results are sent to the teacher network and the student network for forward propagation respectively, and the teacher network logical value prediction output is obtained. And the student network logical value prediction output

[0016] For the original training sample image x, its strong perspective image sample x s The data are sent to the teacher network and the student network for forward propagation respectively to obtain the logical value prediction output of the teacher network. Logical value prediction output with student network

[0017] Preferably, establishing a consistency regularization constraint on the teacher network and the student network includes:

[0018] Establish an intra-view regularization optimization objective, including:

[0019] The distance between the student network’s weak-perspective predicted logical value and the teacher network’s weak-perspective predicted logical value;

[0020] The distance between the strong-perspective predicted logical value of the student network and the strong-perspective predicted logical value of the teacher network;

[0021] Establish a cross-view regularization optimization objective, including:

[0022] The distance between the logical value predicted by the weak perspective of the student network and the logical value predicted by the strong perspective of the teacher network;

[0023] The distance between the logical value predicted by the strong perspective of the student network and the logical value predicted by the weak perspective of the teacher network.

[0024] Preferably, the predicted logical value of the student network is supervised using the consistency regularization constraint, comprising:

[0025] For each training sample image x, the KL divergence loss is used to calculate the intra-view consistency regularization loss and and cross-view consistency regularization loss and

[0026] According to the preset threshold τ under weak viewing angle and strong viewing angle w and τ s And the corresponding teacher network output, obtain the binary mask M that identifies the reliable samples w and M s ; The binary mask M w Applied to consistency regularization loss and M s Applied to consistency regularization loss and The updated intra-view and cross-view consistency regularization losses are obtained to optimize the predicted logical values ​​of the weak-view and strong-view image samples of the student network.

[0027] Preferably, the using of the true value label to supervise the predicted logical value of the student network comprises:

[0028] Using cross entropy loss, the true value of the training sample is used to optimize the predicted logical value of the student network weak-view and strong-view image samples; where:

[0029] For each training sample image x and its true value label y, calculate the cross entropy loss and

[0030]

[0031] Preferably, the total training loss value is a weighted sum of the cross entropy loss and the consistency regularization loss.

[0032] Preferably, updating the parameters of the student network according to the current total training loss value includes:

[0033] According to the current total training loss, the parameters of the student network are updated through back propagation;

[0034] Randomly sample new training samples and repeat the training and parameter updating process until the preset maximum training cycle is reached to complete the knowledge transfer.

[0035] According to another aspect of the present invention, a knowledge transfer system based on cross-view consistency regularization is provided, comprising:

[0036] A training sample acquisition module, which is used to generate multi-view training samples;

[0037] A network training module, which is used to send the multi-view training samples to the teacher network and the student network for forward propagation to obtain the predicted logical values ​​of each view sample by the teacher network and the student network respectively;

[0038] A regularization constraint establishment module, which is used to establish consistent regularization constraints on the teacher network and the student network;

[0039] A total loss calculation module, which uses the consistency regularization constraint to supervise the predicted logical value of the student network, and uses the true value label to supervise the predicted logical value of the student network, and calculates the total training loss value;

[0040] The student network optimization module continuously updates the parameters of the student network according to the current total training loss value until the preset maximum training cycle is reached to complete the knowledge transfer.

[0041] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute any one of the methods described above in the present invention, or to run the system described above in the present invention.

[0042] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above in the present invention, or to run the system described above in the present invention.

[0043] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0044] The present invention proposes a brand-new knowledge distillation technology based on logical values. By using perspective transformation operations to generate weak-perspective images and strong-perspective images from input images, the paper creatively designs intra-perspective consistency regularization constraints and cross-perspective consistency regularization constraints, thereby effectively alleviating common problems in knowledge distillation, such as confirmation bias, over-confidence of the teacher network, and overfitting of the student network.

[0045] The present invention designs a reliable pseudo-label mining technology based on the confidence of each sample teacher. By filtering out teacher predictions with low confidence, the negative effects of unreliable teacher soft labels in guiding students' online learning are effectively reduced, thereby further alleviating the confirmation bias problem in knowledge distillation.

[0046] The present invention is significantly superior to all existing knowledge distillation methods based on feature migration or logical value migration, and does not require any additional computational or parameter costs during testing. In addition, the present invention has excellent generalization performance, is compatible with existing knowledge distillation methods based on logical values, significantly improves their performance, and is applicable to various classic convolutional neural networks and visual attention mechanisms (Transformer) models. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0048] Figure 1 The present invention is a flowchart of a knowledge transfer method based on cross-view consistency regularization in a preferred embodiment of the present invention.

[0049] Figure 2 The figure is a schematic diagram of the component modules of a knowledge transfer system based on cross-view consistency regularization in a preferred embodiment of the present invention.

[0050] Figure 3 Schematic diagrams of the working principles of three knowledge distillation algorithms; among them, (a) is a schematic diagram of the working principle of the existing single-view point-to-point knowledge transfer method, (b) is a schematic diagram of the working principle of the existing relationship-based knowledge transfer method, and (c) is a schematic diagram of the working principle of the method in a preferred embodiment of the present invention.

[0051] Figure 4 Schematic diagram of the network structure and data flow of a knowledge transfer method based on cross-view consistency regularization in a preferred embodiment of the present invention.

[0052] Figure 5 The graphs are a curve diagram showing the changes in training loss and verification loss of a method in a preferred embodiment of the present invention and an existing method in network training; wherein, (a) is a curve diagram showing the changes in training loss of a preferred embodiment of the present invention (CRLD) and an existing method, and (b) is a curve diagram showing the changes in verification loss of a preferred embodiment of the present invention (CRLD) and an existing method. DETAILED DESCRIPTION

[0053] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0054] Existing knowledge distillation methods usually have defects such as excessive teacher confidence and confirmation bias. To address this problem, an embodiment of the present invention provides a knowledge transfer method based on cross-perspective consistency regularization, which innovatively uses image transformation operations to generate weak-perspective and strong-perspective samples from original samples, constructs the predictions of the teacher network and the student network for weak-perspective and strong-perspective samples into multiple sets of intra-perspective and cross-perspective consistency regularization constraints, and uses the confidence of the teacher network prediction to filter out the teacher soft labels with low confidence. Without introducing additional test computational overhead, it effectively alleviates the problems of excessive teacher confidence and confirmation bias in knowledge distillation, and greatly improves the knowledge distillation performance under various tasks and settings compared to existing methods.

[0055] Specifically, Figure 1 As shown, the knowledge transfer method based on cross-view consistency regularization provided by this embodiment may include the following operations:

[0056] S1, generate multi-view training samples, send the multi-view training samples to the pre-trained teacher network and the student network to be trained for forward propagation, and obtain the predicted logical values ​​of each view sample by the teacher network and the student network respectively;

[0057] S2, establishes consistency regularization constraints on the teacher network and the student network;

[0058] S3, supervise the predicted logical value of the student through the teacher network using the consistency regularization constraint, and supervise the predicted logical value of the student network using the true value label, and calculate the total training loss value;

[0059] S4, update the parameters of the student network according to the current total training loss value;

[0060] S5, continuously looping the optimization process of S1 to S4 above until the training end condition (such as the preset maximum training cycle) is reached, and the knowledge transfer is completed.

[0061] In some preferred implementations, the above S1, generating multi-view training samples, may further include the following operations:

[0062] Through different image transformation operations, weak-view image samples and strong-view image samples are generated from the same input image sample; wherein:

[0063] S11, for any original training sample image x, perform random cropping and random horizontal flipping operations on it, and the resulting image is the weak-view image sample x of the original training sample w ;

[0064] S12: for the original training sample image x, randomly apply any two image transformation operations, random cropping, random horizontal flipping and random Cutout algorithm operations in the RandAugment data enhancement method to it in sequence, and the resulting image is the strong perspective image sample x of the original training sample. s .

[0065] In some preferred implementations, the above S1, sending the multi-view training samples to the pre-trained teacher network and the student network to be trained for forward propagation to obtain the predicted logical values ​​of each view sample by the teacher network and the student network, may further include the following operations:

[0066] The multi-view samples are forward propagated through the teacher network and the student network separately to obtain the predicted logical values ​​of each view sample by the teacher network and the student network respectively; where:

[0067] S13, for the original training sample image x, its weak perspective image sample x w The data are sent to the teacher network and the student network for forward propagation respectively to obtain the logical value prediction output of the teacher network. And the student network logical value prediction output

[0068]

[0069] S14, for the original training sample image x, its strong perspective image sample x s The data are sent to the teacher network and the student network for forward propagation respectively to obtain the logical value prediction output of the teacher network. Logical value prediction output with student network

[0070] In some preferred implementations, the above S2, establishing a consistency regularization constraint on the teacher network and the student network, may further include the following operations:

[0071] Establish an intra-view regularization optimization objective, including:

[0072] The distance between the student network’s weak-perspective predicted logical value and the teacher network’s weak-perspective predicted logical value;

[0073] The distance between the strong-perspective predicted logical value of the student network and the strong-perspective predicted logical value of the teacher network;

[0074] Establish a cross-view regularization optimization objective, including:

[0075] The distance between the logical value predicted by the weak perspective of the student network and the logical value predicted by the strong perspective of the teacher network;

[0076] The distance between the logical value predicted by the strong perspective of the student network and the logical value predicted by the weak perspective of the teacher network.

[0077] In some preferred implementations, the above S3, using a consistency regularization constraint to supervise the predicted logical value of the student network, may further include the following operations:

[0078] S31, using KL divergence loss to optimize the optimization objective of the consistency regularization constraint; where:

[0079] S311, for each training sample image x, use KL divergence loss to calculate the intra-view consistency regularization loss and and cross-view consistency regularization loss and

[0080]

[0081] S312, based on the preset thresholds τ under weak viewing angle and strong viewing angle w and τ s And the corresponding teacher network output, obtain the binary mask M that identifies the reliable samples w and M s ; The binary mask M w Applied to consistency regularization loss and M s Applied to consistency regularization loss and The updated intra-view and cross-view consistency regularization losses are obtained to optimize the predicted logical values ​​of the weak-view and strong-view image samples of the student network.

[0082] For each training sample x (and its corresponding strong-view and weak-view samples), A total of 4 loss values ​​are calculated. Therefore, for a training batch containing B samples, four corresponding loss vectors will be obtained, each of which contains B loss values, that is, the loss value of each sample. On the other hand, the pre-trained teacher network can predict the logical value for both weak and strong samples of each training sample x, and the largest logical value among all categories in each prediction is taken as the confidence of the teacher network for the sample. For the teacher prediction of a weak-view sample, when the confidence is greater than τ w When the sample is in the binary mask M w The value in is 1, otherwise it is 0; for a strong sample teacher prediction, when the confidence is greater than τ s When the sample is in the binary mask M s The value in is 1, otherwise it is 0. Therefore, for a training batch containing B samples, a binary mask M is generated w and a binary mask M s, both masks contain B values ​​of 0 or 1, 0 means the confidence is lower than the corresponding threshold, and 1 means the confidence is higher than or equal to the corresponding threshold. w Multiply When , the loss value of each sample will be multiplied by its corresponding binary value. For samples with low confidence, the binary value is 0, that is, the loss value of the sample is set to 0, so that the loss of the low confidence sample will not affect the total loss. s The same applies to other losses.

[0083] In some preferred implementations, the above S3, using the true value label to supervise the predicted logical value of the student network, may further include the following operations:

[0084] S32, using the cross entropy loss, using the true value of the training sample to optimize the predicted logical value of the weak-view and strong-view image samples of the student network; wherein:

[0085] For each training sample image x and its true value label y, calculate the cross entropy loss and

[0087] In some preferred implementations, the above S3, calculating the total training loss value, may further include the following operations:

[0088] S33, the total training loss value is the weighted sum of the cross entropy loss and the consistency regularization loss; where:

[0089] The cross entropy loss is and sum;

[0090] Consistency regularization loss, including: intra-view consistency regularization loss and cross-view consistency regularization loss.

[0091] In some preferred implementations, the above S4, updating the parameters of the student network according to the current total training loss value, may further include the following operations:

[0092] According to the current total training loss, the parameters of the student network are updated through back propagation.

[0093] In some preferred implementations, the above S5, repeating the above process until the training end condition is reached, may further include the following operations:

[0094] Randomly sample new training samples and repeat the above process until the preset maximum training cycle is reached, and the student network and its parameters after knowledge distillation can be obtained.

[0095] In some preferred embodiments, the training sample set can use CIFAR-100 and Tiny-ImageNet datasets. For this dataset, the training cycle can be 240, the initial value of the learning rate can be 0.05 (preferably 0.025 when the student network is a lightweight model such as MobileNet or ShuffleNet), and can be reduced by 10 times after the 150th, 180th, and 210th cycles respectively; the training can use the SGD optimizer, the momentum value can be 0.9, and the batch size can be 64.

[0096] In some preferred embodiments, the training sample set can also use the ImageNet dataset. For this dataset, the training cycle can be 100, the initial value of the learning rate can be 0.2, and it can be reduced by 10 times after the 30th, 60th, and 90th cycles respectively; the training can use the SGD optimizer, the momentum value can be 0.9, and the batch size can be 64.

[0097] Based on the same inventive concept, an embodiment of the present invention further provides a knowledge transfer system based on cross-view consistency regularization.

[0098] Specifically, Figure 2 As shown, the knowledge transfer system based on cross-view consistency regularization provided by this embodiment may include the following modules:

[0099] A training sample acquisition module, which is used to generate multi-view training samples;

[0100] A network training module, which is used to send the multi-view training samples to the teacher network and the student network for forward propagation to obtain the predicted logical values ​​of each view sample by the teacher network and the student network respectively;

[0101] A regularization constraint establishment module, which is used to establish consistent regularization constraints on the teacher network and the student network;

[0102] A total loss calculation module, which uses the consistency regularization constraint to supervise the predicted logical value of the student network, and uses the true value label to supervise the predicted logical value of the student network, and calculates the total training loss value;

[0103] The student network optimization module continuously updates the parameters of the student network according to the current total training loss value until the preset maximum training cycle is reached to complete the knowledge transfer.

[0104] The following further describes the working content of the knowledge transfer system based on cross-view consistency regularization provided by the above embodiment of the present invention.

[0105] The training sample acquisition module generates weak-view and strong-view images from the same input image through different image transformation operations to obtain multi-view training samples; wherein:

[0106] For any original training sample image x, we apply random cropping and random horizontal flipping operations to it, and the resulting image is the weak-view image sample x of the original sample. w .

[0107] For the original training sample image x, randomly apply any two image transformation operations, random cropping, random horizontal flipping and random Cutout operations in the RandAugment method to it in sequence, and the resulting image is the strong perspective image sample x of the original sample s .

[0108] The network training module forward-propagates the multi-view training samples through the teacher network and the student network separately to obtain the predicted logical values ​​of each view sample by the teacher network and the student network respectively; where:

[0109] For the original training sample x, its weak view sample x w Send them to the teacher network and student network respectively to get the teacher network logical value prediction output Logical value prediction output with student network

[0110] For the original training sample x, its strong perspective sample x s Send them to the teacher network and student network respectively to get the teacher network logical value prediction output Logical value prediction output with student network

[0111] Regularization constraint establishment module, establish the following consistency regularization constraints:

[0112] The intra-perspective regularization optimization objectives include: ① the distance from the weak-perspective prediction value of the student network to the weak-perspective prediction value of the teacher network, ② the distance from the strong-perspective prediction value of the student network to the strong-perspective prediction value of the teacher network; the cross-perspective regularization optimization objectives include: ③ the distance from the weak-perspective prediction value of the student network to the strong-perspective prediction value of the teacher network, ④ the distance from the strong-perspective prediction value of the student network to the weak-perspective prediction value of the teacher network.

[0113] The total loss calculation module uses KL divergence loss to optimize the four sets of optimization objectives of consistency regularization. At the same time, the prediction of weak-view and strong-view samples of the student network is optimized by cross-entropy loss using the true value of the training sample. The total loss is the weighted sum of the cross-entropy loss and all consistency regularization losses. The confidence of the teacher network for each sample is used to filter out the loss of unreliable samples from the total loss; where:

[0114] Calculate the loss and perform network parameter optimization, which includes:

[0115] For each training sample x, the KL divergence loss is used to calculate the intra-view consistency loss And cross-view consistency loss

[0116] According to the preset threshold τ w , τ s With the teacher network output τ w , and obtain the binary mask M that identifies reliable samples w 、M s .

[0117] M w Application M s Application Get the updated intra-view and cross-view consistency losses.

[0118] For each training sample x and its true value label y, calculate the cross entropy loss and

[0119] Calculate the total loss, that is, calculate the weighted sum of the cross entropy loss, the intra-view consistency loss, and the cross-view consistency loss.

[0120] The student network optimization module uses the back propagation algorithm to update the student network parameters according to the current total training loss. Randomly sample new training samples and repeat the above training steps until the preset maximum training cycle is reached to obtain the student network and its parameters after knowledge distillation.

[0121] The performance of the knowledge transfer method and system provided in the above embodiment is further explained below in conjunction with specific verification examples.

[0122] like Figure 3 (a) to (c) are schematic diagrams of the principle relationship of the three knowledge distillation algorithms.

[0123] like Figure 4 , which is a schematic diagram of the network structure and data flow of the knowledge transfer method based on cross-view consistency regularization provided in the above preferred embodiment of the present invention.

[0124] Verification Example 1:

[0125] The method provided by the above embodiment of the present invention (hereinafter referred to as the present method) is experimentally verified on image recognition datasets such as CIFAR-100, Tiny-ImageNet, and ImageNet for knowledge transfer between the same network architectures to verify the performance of the present method in achieving knowledge distillation between the same network architectures.

[0126] (1) Quantitative experimental results of knowledge distillation on the CIFAR-100 dataset with the same network architecture:

[0127] Table 1 shows the performance comparison of knowledge distillation between multiple groups of teacher-student networks with the same architecture on the CIFAR-100 dataset between this method and existing methods.

[0128] Table 1

[0129]

[0130] According to the experimental results in Table 1, the performance of this method significantly exceeds that of existing methods when performing knowledge distillation between the same network models on the CIFAR-100 dataset. For example, when this method is used to distill the knowledge of the ResNet32×4 pre-trained model to the lightweight student network of ResNet8×4, the ResNet8×4 network can achieve a Top-1 classification accuracy of 78.31%, which is about 2 percentage points higher than the best existing method. * indicates the use of a longer training strategy proposed by MLLD. It can be seen that this method is also applicable to different training strategies and can further improve performance.

[0131] (2) Quantitative experimental results of knowledge distillation on the ImageNet dataset with the same network architecture:

[0132] Table 2 (left) shows the performance comparison of knowledge distillation between the teacher-student network with the same architecture on the ImageNet dataset between this method and existing methods.

[0133] Table 2

[0134]

[0135] According to the experimental results in Table 2 (left), this method still has excellent performance advantages on the ImageNet large-scale image recognition dataset. Compared with other existing knowledge distillation methods, when this method is used to distill the knowledge of the pre-trained ResNet34 network, ResNet18 can achieve the highest classification accuracy, with the Top-1 accuracy reaching more than 72% for the first time and the Top-5 accuracy exceeding 90.7% for the first time. This set of experimental results verifies the effectiveness and versatility of this method on large datasets.

[0136] Verification Example 2:

[0137] This method is experimentally verified on the CIFAR-100 and ImageNet image recognition datasets for knowledge transfer between different network models to verify the performance of this method in achieving knowledge distillation between different network models.

[0138] (1) Quantitative experimental results of knowledge distillation of heterogeneous network models on the CIFAR-100 dataset:

[0139] Table 3 shows the performance comparison of knowledge distillation between different model teacher-student networks on the CIFAR-100 dataset between this method and existing methods.

[0140] Table 3

[0141]

[0142] According to the experimental results in Table 3, the knowledge distillation performance of this method on multiple groups of heterogeneous model teacher-student networks significantly exceeds that of all existing knowledge distillation methods.

[0143] (2) Quantitative experimental results of knowledge distillation of heterogeneous network models on the ImageNet dataset:

[0144] Table 2 (right) shows the performance comparison of knowledge distillation between heterogeneous teacher-student networks on the ImageNet dataset between this method and existing methods.

[0145] According to the experimental results in Table 2 (right), when this method performs knowledge distillation of heterogeneous teacher-student networks on large-scale datasets, its performance is still better than the existing knowledge distillation methods. This set of experimental results verifies the effectiveness and generalization of this method in knowledge distillation between networks of different models.

[0146] Verification Example 3:

[0147] This method is experimentally verified by transferring knowledge between different network architectures on the Tiny-ImageNet dataset to verify the performance of this method in achieving knowledge distillation between different network architectures.

[0148] Table 4 shows the performance comparison of knowledge distillation between different-architecture teacher-student networks of this method and existing methods on the Tiny-ImageNet dataset.

[0149] Table 4

[0150]

[0151] According to the experimental results in Table 4, this method can effectively transfer the knowledge of the ViT pre-trained network based on the visual attention (Transformer) architecture to the ResNet-18 network based on the convolutional neural network, significantly improving the performance of the latter, and this method shows obvious performance advantages compared with existing methods. This set of experimental results verifies the effectiveness and generalization of this method in knowledge distillation between models with different network architectures.

[0152] Verification Example 4:

[0153] This method is applied to the experimental verification of the existing knowledge distillation method based on logical values ​​to verify the compatibility of this method with the existing method.

[0154] Table 5 shows the performance comparison of knowledge distillation on the CIFAR-100 dataset when this method is applied to the existing knowledge distillation method based on logical values.

[0155] Table 5

[0156]

[0157] According to the experimental results in Table 5, this method is well compatible with existing knowledge distillation methods based on logical value transfer and significantly improves the performance of these methods. This set of experimental results verifies the compatibility and generalization of this method with different baseline methods.

[0158] The training loss and verification loss variation curves of the method provided in the above preferred embodiment of the present invention and the existing method in network training are as follows: Figure 5 As shown in (a) and (b) (CRLD is the method of the present invention). Through experimental verification on benchmark image classification datasets such as CIFAR-100, Tiny-ImageNet and ImageNet, it can be seen that the knowledge transfer technology provided by the above embodiment of the present invention is significantly superior to all existing knowledge distillation methods based on feature transfer or logical value transfer, and does not require any additional calculation or parameter cost during testing.

[0159] A specific application example in image classification:

[0160] A possible specific application example is a lightweight face recognition system for mobile devices (such as mobile phones and tablets). First, a large, bloated, high-performance face classification system is pre-trained locally, and the categories can be user 1, user 2, user 3 and others. Then, a lightweight, efficient, and low-energy student model suitable for deployment on the mobile terminal is selected, such as MobileNet. The method proposed in the present invention can then be used to transfer the capabilities of the teacher network to the lightweight student network as much as possible. The student network trained by distillation can be deployed on a mobile device and realize the face recognition unlocking function, such as unlocking the device only when the face classification result is user 1.

[0161] An embodiment of the present invention further provides a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0162] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0163] The knowledge transfer method and system based on cross-perspective consistency regularization provided by the above-mentioned embodiments of the present invention adopt a new knowledge distillation technology based on logical values, and generate weak-perspective images and strong-perspective images from input images by using perspective transformation operations, and creatively design consistency regularization constraints within the perspective and consistency regularization constraints across perspectives, thereby effectively alleviating common problems in knowledge distillation, such as confirmation bias, excessive confidence of the teacher network, and overfitting of the student network; reliable pseudo-label mining technology based on the confidence of each sample teacher effectively reduces the negative effects of unreliable teacher soft labels in guiding student network learning by filtering out teacher predictions with lower confidence, thereby further alleviating the confirmation bias problem in knowledge distillation.

[0164] All matters not covered in the above embodiments of the present invention are well known in the art.

[0165] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A knowledge transfer method based on cross-view consistency regularization, characterized in that: include: Generate multi-view training samples, send the multi-view training samples to the teacher network and the student network for forward propagation, and obtain the predicted logical values ​​of the teacher network and the student network for each view sample; Establishing a consistency regularization constraint on the teacher network and the student network; Supervising the predicted logical value of the student network using the consistency regularization constraint through the teacher network, and supervising the predicted logical value of the student network using the true value label, and calculating the total training loss value; According to the current total training loss value, the parameters of the student network are updated, and the optimization process is continuously cycled until the preset maximum training cycle is reached to complete the knowledge transfer.

2. The knowledge transfer method based on cross-view consistency regularization according to claim 1, characterized in that: The generating of multi-view training samples comprises: For any original training sample image x, we apply random cropping and random horizontal flipping operations to it, and the resulting image is the weak-view image sample x of the original training sample. w ; For the original training sample image x, randomly apply any two image transformation operations, random cropping, random horizontal flipping and random Cutout algorithm operations in the RandAugment data enhancement method to it, and the resulting image is the strong perspective image sample x of the original training sample s .

3. The knowledge transfer method based on cross-view consistency regularization according to claim 1, characterized in that: The multi-view training samples are sent to the teacher network and the student network for forward propagation to obtain the predicted logical values ​​of each view sample of the teacher network and the student network, including: For the original training sample image x, its weak perspective image sample x w The data are sent to the teacher network and the student network for forward propagation respectively to obtain the logical value prediction output of the teacher network. And the student network logical value prediction output For the original training sample image x, its strong perspective image sample x s The data are sent to the teacher network and the student network for forward propagation respectively to obtain the logical value prediction output of the teacher network. Logical value prediction output with student network 4. The knowledge transfer method based on cross-view consistency regularization according to claim 1, characterized in that: Establishing consistency regularization constraints on the teacher network and the student network, including: Establish an intra-view regularization optimization objective, including: The distance between the student network’s weak-perspective predicted logical value and the teacher network’s weak-perspective predicted logical value; The distance between the strong-perspective predicted logical value of the student network and the strong-perspective predicted logical value of the teacher network; Establish a cross-view regularization optimization objective, including: The distance between the logical value predicted by the weak perspective of the student network and the logical value predicted by the strong perspective of the teacher network; The distance between the logical value predicted by the strong perspective of the student network and the logical value predicted by the weak perspective of the teacher network.

5. The knowledge transfer method based on cross-view consistency regularization according to claim 1, characterized in that: The use of consistency regularization constraints to supervise the predicted logical values ​​of the student network includes: For each training sample image x, the KL divergence loss is used to calculate the intra-view consistency regularization loss and and cross-view consistency regularization loss and According to the preset threshold τ under weak viewing angle and strong viewing angle w and τ s And the corresponding teacher network output, obtain the binary mask M that identifies the reliable samples w and M s ; The binary mask M w Applied to consistency regularization loss and M s Applied to consistency regularization loss and Get the updated intra-view and cross-view consistency regularization losses to optimize the predicted logistic values ​​of the student network's weak-view and strong-view image samples; The method of using the true value label to supervise the predicted logical value of the student network includes: Using cross entropy loss, the true value of the training sample is used to optimize the predicted logical value of the student network weak-view and strong-view image samples; where: For each training sample image x and its true value label y, calculate the cross entropy loss and 6. The knowledge transfer method based on cross-view consistency regularization according to claim 5, characterized in that: The total training loss value is the weighted sum of the cross entropy loss and the consistency regularization loss.

7. The knowledge transfer method based on cross-view consistency regularization according to claim 1, characterized in that: The updating of the parameters of the student network according to the current total training loss value includes: According to the current total training loss, the parameters of the student network are updated through back propagation; Randomly sample new training samples and repeat the training and parameter updating process until the preset maximum training cycle is reached to complete the knowledge transfer.

8. A knowledge transfer system based on cross-view consistency regularization, characterized in that: include: A training sample acquisition module, which is used to generate multi-view training samples; A network training module, which is used to send the multi-view training samples to the teacher network and the student network for forward propagation to obtain the predicted logical values ​​of each view sample by the teacher network and the student network respectively; A regularization constraint establishment module, which is used to establish consistent regularization constraints on the teacher network and the student network; A total loss calculation module, which uses the consistency regularization constraint to supervise the predicted logical value of the student network through the teacher network, and uses the true value label to supervise the predicted logical value of the student network, and calculates the total training loss value; The student network optimization module continuously updates the parameters of the student network according to the current total training loss value until the preset maximum training cycle is reached to complete the knowledge transfer.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images