Large model knowledge distillation integrated calibration method and system based on data enhancement strategy
By constructing a multi-scenario data augmentation module and a multi-teacher model system, and combining knowledge distillation and confidence-based decision-making, the problem of insufficient generalization ability of student models in existing technologies is solved, and efficient detection and resource optimization in complex environments are achieved.
Patent Information
- Application Number
- CN202511152109.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-18
- Publication Date
- 2025-12-16
AI Technical Summary
Existing knowledge distillation methods have significant limitations in terms of adaptability to complex scenarios, efficiency of knowledge integration, and accuracy of model compression. Traditional single teacher models and fixed data augmentation strategies are difficult to cover features of multiple abnormal scenarios, resulting in insufficient generalization ability of student models.
A multi-scenario data augmentation module is constructed. A training subset is generated through a hybrid augmentation strategy to train a multi-teacher model to learn scenario-specific features. Knowledge is then integrated and calibrated into the student model through knowledge distillation. Finally, confidence is calculated using a discriminator to make the final decision.
It significantly improves the recognition accuracy and generalization ability of the student model in multi-anomaly scenarios, reduces the model size and computational cost, making it suitable for running on resource-constrained edge devices, and improves detection performance and robustness.
Smart Images
Figure CN121145985A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of remote sensing data processing, and in particular to a large model knowledge distillation integrated calibration method and system based on a data enhancement strategy. BACKGROUND
[0002] With the wide application of deep learning technology in the field of computer vision, large models have achieved significant performance improvement in image recognition, target detection and other tasks due to their powerful feature extraction capability. However, in the actual scenarios of power inspection, industrial defect detection and other tasks, large models face challenges such as limited computing resources and difficult deployment on edge devices. As a key means of model compression, knowledge distillation technology, which transfers knowledge from a teacher model to a student model, becomes a key path to solve the above problems. Existing knowledge distillation methods mainly include logits-based and feature-based methods. The former, such as KD and DIS, guides student learning by softening the output of the teacher model, and the latter, such as FitNet and OFD, improves the distillation effect by aligning intermediate features, but still has significant limitations in adaptability to complex scenarios, knowledge integration efficiency and model compression accuracy.
[0003] In summary, for the field of image recognition technology, there are two challenges, first, for common target detection tasks such as recognizing specific objects, traditional methods are limited by limited data sets and cannot fully cover various target shapes and complex and variable background environments, resulting in limited model generalization performance. Second, existing knowledge distillation technology mainly focuses on the transmission of single feature knowledge, ignoring the diversity of feature structures and knowledge information in different application scenarios, which poses a challenge to the robustness and accuracy of the model on resource-limited devices such as edge devices. SUMMARY
[0004] The purpose of the present application is to overcome the defects of the prior art and provide a large model knowledge distillation integrated calibration method and system based on a data enhancement strategy to solve or partially solve the problem that traditional single teacher models and fixed data enhancement strategies cannot cover multiple abnormal scene features, resulting in insufficient generalization ability of student models.
[0005] The purpose of the present application can be achieved by the following technical solutions:
[0006] According to one aspect of the present application, a large model knowledge distillation integrated calibration method based on a data enhancement strategy is provided, comprising the following steps:
[0007] A multi-scene data enhancement module is constructed to perform data enhancement strategy combination on the original data set through hybrid enhancement to generate training subsets covering different abnormal scenes;
[0008] Training a multi-teacher model using the training subset, each teacher model corresponding to at least one data enhancement strategy to learn scene-specific features;
[0009] Based on the constructed multiple teacher models, through knowledge distillation, the knowledge corresponding to the learning scene is integrated and calibrated, and a lightweight student model is obtained through fine-tuning;
[0010] The student model and the preset deep learning model are combined, the confidence of the inference result of the deep learning model is calculated by the discriminator, if the confidence is greater than the preset value, the result of the target detection model is taken as the final result, otherwise the result of the student model is taken as the final result.
[0011] As a preferred technical solution, the mixed enhanced data enhancement strategy combination includes applying one of mixup, cutout, cutmix and augmix to the same original data set respectively.
[0012] As a preferred technical solution, under the mixup strategy, the loss function of the teacher model training is:
[0013]
[0014] x mu =λx1+(1-λ)x2
[0015] y mu =λy1+(1-λ)y2
[0016] Wherein, is the loss function of the teacher model under the mixup strategy, λ is a parameter, f1() represents the teacher model network under the mixup strategy, x1 and x2 are two randomly sampled input points, y1 and y2 are their corresponding one-hot encoding labels, is the cross-entropy loss function.
[0017] As a preferred technical solution, under the cutout strategy, the loss function of the teacher model training is:
[0018]
[0019] x co =M⊙x+(1-M)⊙Z
[0020] Wherein, is the loss function of the teacher model training under the cutout strategy, f2() represents the teacher model network under the cutout strategy, x is the original image, x co is the cropped image, Z is a zero matrix with the same size as the input image x, M is a binary mask, and represents pixel-wise multiplication.
[0021] As a preferred technical solution, in the cutmix strategy:
[0022] x cm =Mx1+(1-M)x2
[0023] y cm =ly1+(1-l)y2
[0024] r x ~U(0,W),
[0025] r y ~U(0,H),
[0026] Wherein, x cm , y cm is the sample and corresponding label processed by the cutmix strategy, x1, x2 are two samples in the dataset, y1 and y2 are their corresponding labels, M is a binary mask representing the cutting and filling area of the two images drawn at random, l e [0,1], the coordinates of the bounding box are B=(r x ,r y ,r w ,r h ), represents the cutting and filling area of the image, W is the width of the image, and H is the height of the image.
[0027] As a preferred technical solution, under the augmix strategy, the loss function of the teacher model training is:
[0028]
[0029] Wherein, is the cross entropy loss, is the Jensen Shannon loss, k is the weighted hyperparameter, p a , p b , p c is the probability of the teacher model predicting the original image x, the probability distribution of the two augmented images x b , x c , M is the average value of p a , p b , p c , KL[] represents the KL divergence operator.
[0030] As a preferred technical solution, by knowledge distillation, the knowledge corresponding to the learning scene is integrated and calibrated, and the process of fine-tuning the lightweight student model is obtained. The loss function is:
[0031]
[0032] z i =P(y|f i ,x)i=1,2,3,4
[0033] z s =P(y|f s ,x)
[0034] wherein, is a total loss function, and a is a parameter, respectively are divergence loss and loss of a specific task, f s is a student model, f1, f2, f3, f4 are teacher models trained using four data augmentation strategies respectively, z i , z s respectively represent probability distributions of teacher models and student models, x, y respectively represent samples and corresponding labels, and KL[] represents a KL divergence operator.
[0035] In another aspect of the present application, a large model knowledge distillation integrated calibration system based on a data augmentation strategy is provided for implementing the aforementioned large model knowledge distillation integrated calibration method based on a data augmentation strategy. The large model knowledge distillation integrated calibration system comprises:
[0036] A multi-scene data augmentation module is configured to perform a combination of data augmentation strategies on an original data set through hybrid augmentation to generate training subsets covering different abnormal scenes.
[0037] A multi-teacher model training module is configured to train multiple teacher models using the training subsets, each teacher model corresponding to at least one data augmentation strategy to learn scene-specific features, and assigning guidance weights to each teacher model based on the similarity between the data set and the target application scene to form a teacher supervision framework.
[0038] A lightweight student model generation module is configured to integrate and calibrate the knowledge corresponding to the learned scenes through knowledge distillation based on the constructed multiple teacher models to fine-tune a lightweight student model.
[0039] A joint inference module is configured to jointly use the student model and a preset deep learning model to calculate the confidence of the inference result of the deep learning model using a discriminator. If the confidence is greater than a preset value, the result of the target detection model is taken as the final result, otherwise the result of the student model is taken as the final result.
[0040] In another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory having stored therein one or more programs, the one or more programs including instructions for performing the aforementioned large model knowledge distillation integrated calibration method based on a data augmentation strategy.
[0041] In another aspect of the present application, a computer-readable storage medium is provided, comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for performing the aforementioned large model knowledge distillation integrated calibration method based on data enhancement strategy.
[0042] Compared with the prior art, the present application has at least one of the following beneficial effects:
[0043] (1) Training data covers multiple abnormal scenarios: The present application constructs a multi-scene data augmentation module, generates diversified training subsets by mixing augmentation strategies such as mixup and cutout, and each teacher model learns scene-specific features corresponding to at least one enhancement strategy, overcoming the problem of insufficient training data coverage and weak generalization ability of student models caused by single data enhancement strategy in the prior art. The training data of the present application covers multiple abnormal scenarios, and the recognition accuracy of the student model for abnormalities is improved.
[0044] (2) Transfer the knowledge corresponding to the learning scene to the student model: The present application uses an integrated architecture to distill the knowledge in multiple teacher models into a student model, each teacher model is trained using different enhancement techniques, and a loss function predefined by itself is used during training, thereby improving the classification performance and model generalization of the student model. In the distillation process, the target function maximizes the loss between the temperature-scaled class probability distributions of multiple teachers, which helps the student model to learn hidden knowledge from the teacher models.
[0045] (3) Accelerate inference speed: In view of the problem that the size of the student model after distillation is large and the time-consuming is long for simple inference tasks, the present application combines the student model and the preset deep learning model, uses the discriminator to calculate the confidence of the deep learning model inference result, and if the confidence is greater than the preset value, the result of the target detection model is taken as the final result, otherwise the result of the student model is taken as the final result. In simple tasks, the deep learning model is used for inference to improve the inference speed, and in complex tasks, the knowledge in the student model is used for accurate inference to ensure the inference efficiency. BRIEF DESCRIPTION OF DRAWINGS
[0046] Figure 1 Flowchart of the large model knowledge distillation integrated calibration method based on data enhancement strategy in the embodiment;
[0047] Figure 2 Schematic diagram of the large model knowledge distillation integrated calibration system based on data enhancement strategy in the embodiment;
[0048] Figure 3 Schematic diagram of the electronic device in the embodiment. DETAILED DESCRIPTION
[0049] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the protection scope of the present application.
[0050] Embodiment 1
[0051] To solve the problems in the prior art, the embodiment provides a large model knowledge distillation integrated calibration method based on a data enhancement strategy. For common target detection tasks, such as recognizing specific objects, the traditional method is limited by a limited data set and is difficult to comprehensively cover various target shapes and complex and changeable background environments, resulting in limited model generalization performance. Therefore, the method uses LabelImg and other tools to construct a large-scale labeled data set, and uses four data enhancement technologies, such as mixup, cutout, cutmix, and augmix, to effectively expand the data set and significantly improve the model's detection generalization ability for various targets. Secondly, existing knowledge distillation technologies mainly focus on the transmission of single feature knowledge, ignoring the diversity of feature structures and knowledge information in different application scenarios, which poses a challenge to the robustness and accuracy of the model on resource-constrained devices such as edge devices. The method constructs a multi-teacher model architecture, integrates and calibrates the diversified knowledge transmitted by each teacher model, so that the student model can comprehensively learn and integrate feature information in various application scenarios, thereby exhibiting higher detection performance in complex and changeable environments, and providing solid technical support for the intelligentization and precision of image recognition.
[0052] Reference is made to Figure 1 , the method comprises the following steps:
[0053] Step S1, acquiring image abnormal sample data.
[0054] LabelImg is used to make abnormal image labeled data set, which contains four abnormal scenes and a total of 1300 images.
[0055] Step S2, data enhancement stage: construct a multi-scene data enhancement module, perform data enhancement strategy combination on the original data set through hybrid enhancement, and generate training subsets covering different abnormal scenes.
[0056] Specifically, it includes hybrid enhancement, such as sample mixing, region cropping, content replacement, etc., and generates diversified training data for different abnormal scenes. It supports dynamic configuration of enhancement strategy combination to adapt to specific scene requirements. Finally, the original data set is enhanced by multiple strategies to generate training subsets covering different abnormal scenes.
[0057] Four data augmentation strategies are used to enhance the abnormal image labeled dataset, namely mixup, cutout, cutmix, and augmix. The teacher model is trained using different data augmentation strategies, so that each teacher model has different generalization ability for different anomaly detection scenarios. The student model focuses on the single knowledge distillation of feature knowledge when learning the knowledge transferred by the teacher model, but can also focus on the feature structure and knowledge information of different scenarios. The knowledge transferred by different teacher models is integrated and calibrated, thereby enhancing the robustness and accuracy of the student model when running on edge devices.
[0058] The four data augmentation strategies are as follows:
[0059] mixup: mix two images at a certain ratio to generate new images, thereby increasing data diversity and improving model generalization ability
[0060] cutout: randomly crop a part of the image during training to simulate the occlusion situation that may be encountered in actual application and enhance the robustness of the model to occlusion scenarios.
[0061] cutmix: cut and exchange a part of two randomly selected images to generate new training data, thereby increasing the robustness of the model to image position and content.
[0062] augmix: combine multiple different image transformations and perform weighted averaging, then minimize the Jensen-Shannon divergence to make the augmented image maintain a similar distribution as the original image.
[0063] Step S3, teacher model training phase: train multiple teacher models using the training subset, each corresponding to a data augmentation strategy to learn scene-specific features.
[0064] Specifically, each teacher model is associated with a specific data augmentation strategy to learn scene-specific features, and generates differentiated knowledge expression through parallel training mechanism.
[0065] Training the teacher model using the mixup data augmentation strategy: during the training of the teacher model, the samples generated by the mixup data augmentation strategy and their predicted labels can be expressed as (1) and (2). Let f1 be the teacher model trained using mixup. During training, y mu Instead of using y1, y2, the cross-entropy loss is used as shown in equation (3):
[0066] x mu = λx1+(1-λ)x2 (1)
[0067] y mu = λy1 + (1 - λ)y2 (2)
[0068]
[0069] where x1 and x2 are two randomly sampled input points, y1 and y2 are their corresponding one-hot encoded labels, f1 is the teacher model trained with mixup, λ ∈ [0, 1] is drawn from a beta (a, a) distribution, and a is a hyperparameter.
[0070] Training the teacher model with cutout data augmentation strategy: first, randomly crop the image using formula (4).
[0071] x co = M 0 x + (1 - M) 0 Z (4)
[0072] where x is the original image, x co is the cropped image, Z is a zero matrix (black pixels) with the same size as the input image x, and M is a binary mask indicating which pixels are replaced with black pixels. During training, the cross-entropy loss function is used, as shown in formula (5).
[0073]
[0074] where f2 is the teacher model trained with cutout.
[0075] Training the teacher model with cutmix data augmentation strategy: the mathematical formula of cutmix is shown in formula (6) and (7).
[0076] x cm = M 0 x1 + (1 - M) 0 x2 (6)
[0077] y cm = λy1 + (1 - λ)y2 (7)
[0078] where x1 and x2 are two samples in the dataset, y1 and y2 are their corresponding labels, M is a binary mask representing the cutting and filling area of the two randomly drawn images, λ ∈ [0, 1] is drawn from a beta distribution. The coordinates of the bounding box are B = (r x , r y , r w , r h ), representing the cutting and filling area of the image, and the bounding box sampling is represented by formula (8) and (9):
[0079]
[0080] where W is the width of the image and H is the height of the image. The loss function used during training is consistent with mixup.
[0081] Training the teacher model with augmix data augmentation strategy
[0082] AugMix uses the input image itself to perform data mixing. It first transforms (shift, cutout, rotate, etc.) the input image and then mixes it with the original image. During training, we use the Jensen-Shannon divergence loss and the standard loss as shown in equation (10):
[0083]
[0084] For cross-entropy loss, Jensen-Shannon loss, k is a weighted hyperparameter.
[0085]
[0086] where p a , p b , p c is the probability distribution of the original image x predicted by the teacher model, and the two augmented images x b , x c , M is the average of p a , p b , p c .
[0087] Step S4, knowledge fusion stage: based on the constructed multiple teacher models, the knowledge corresponding to the learning scene is integrated and calibrated through knowledge distillation, and the student model is integrated and calibrated through knowledge distillation.
[0088] In the large model knowledge distillation integration calibration process, the embodiment provides a large model knowledge distillation integration calibration framework, which distills the knowledge in multiple teacher models into a student model through the use of an integration architecture, thereby improving the classification performance and model generalization of the student model. Each teacher model is trained using a different augmentation technique and uses its own pre-defined (augmentation-based) loss during training.
[0089] Assuming (x, y) is a sample of the original dataset D, f s is the student model, f1, f2, f3, f4 are teacher models trained using mixup, cutout, cutmix, and augmix respectively. The objective function of our method maximizes the loss between the temperature-scaled class probability distributions of multiple teachers (z1, z2, z3, z4), where z i = P(y|fi , the probability distribution z of the student model s s , the divergence loss helps the student model learn the implicit knowledge from the teacher model, and the objective function is as follows:
[0090]
[0091] wherein is the divergence loss, in addition to using the divergence loss, we also use the loss of a specific task, for example:
[0092]
[0093] The total loss is calculated as follows:
[0094]
[0095] Step S5, jointly implementing target detection with the student model and the preset target detection deep learning model.
[0096] The student model obtained in step S4 is combined with the pre-trained target detection deep learning model, and the discriminator is used to calculate the confidence of the target detection result of the deep learning model. If the confidence is greater than a first preset threshold, the result of the target detection model is taken as the final result, otherwise the result of the student model is taken as the final result.
[0097] As a preferred technical solution, the confidence of the student model result is calculated using the discriminator, and if the confidence is lower than a second threshold, a prompt is issued for manual assistance.
[0098] To demonstrate the effectiveness of the method, an experiment was conducted using a real image anomaly dataset covering a variety of scene anomaly image samples. The image resolution of the dataset is 800x1300 pixels, the total number of data samples is 1300, and the image state classification (i.e. label) has 4 types of anomalies, with 160, 166, 108, and 866 samples for each type of anomaly.
[0099] (1) Data augmentation phase.
[0100] Data augmentation is performed using mixup, the dataset is augmented by 20%, and the augmented dataset increases by 260, which is used to train the teacher model f1.
[0101] Data augmentation is performed using cutout, the dataset is augmented by 20%, and the augmented dataset increases by 260, which is used to train the teacher model f2.
[0102] Data augmentation is performed using cutmix, the dataset is augmented by 20%, and the augmented dataset increases by 260, which is used to train the teacher model f3.
[0103] Data augmentation with augmix, data set augmentation 20%, augmented data set increased by 260, used to train the teacher model f4.
[0104] The four data augmentation strategies together increased 1040, plus the original data set 1300, the data set expanded to 2340, divided into training set and test set according to the ratio of 8:2, training set 1872, test set 468, training set for training student model f s , test set for testing the recognition effect of student model.
[0105] (2) Teacher model training phase.
[0106] Performance evaluation of teacher model after mixup data augmentation training, evaluate the accuracy of teacher model after mixup data augmentation training. 156 test set samples were verified, and the following evaluation indexes were obtained. Accuracy before data augmentation: 88%, accuracy after data augmentation: 92%, recall rate before data augmentation: 80%, recall rate after data augmentation: 83%.
[0107] Performance evaluation of teacher model after cutout data augmentation training, evaluate the accuracy of teacher model after cutout data augmentation training. 156 test set samples were verified, and the following evaluation indexes were obtained. Accuracy before data augmentation: 82%, accuracy after data augmentation: 83%, recall rate before data augmentation: 83%, recall rate after data augmentation: 85%.
[0108] Performance evaluation of teacher model after cutmix data augmentation training, evaluate the accuracy of teacher model after cutmix data augmentation training. 156 test set samples were verified, and the following evaluation indexes were obtained. Accuracy before data augmentation: 84%, accuracy after data augmentation: 84%, recall rate before data augmentation: 87%, recall rate after data augmentation: 89%.
[0109] Performance evaluation of teacher model after augmix data augmentation training, evaluate the accuracy of teacher model after augmix data augmentation training. 156 test set samples were verified, and the following evaluation indexes were obtained. Accuracy before data augmentation: 79%, accuracy after data augmentation: 82%, recall rate before data augmentation: 84%, recall rate after data augmentation: 85%.
[0110] These results show that data augmentation techniques can significantly improve the recognition accuracy of teacher model in image anomaly detection.
[0111] (3) Knowledge fusion phase.
[0112] The four data enhancement strategies increase a total of 1040 images. Adding the original data set of 1300 images, the data set is expanded to 2340 images. The training set and test set are divided in an 8:2 ratio, with 1872 images in the training set and 468 images in the test set. The training set is used to train the student model, and the test set is used to test the recognition effect of the student model.
[0113] The recognition effect of the student model before knowledge distillation and integration calibration is as follows:
[0114] The recognition accuracy of anomaly 1 is 66%, the recognition recall rate is 70%, the recognition accuracy of anomaly 2 is 60%, the recognition recall rate is 80%, the recognition accuracy of anomaly 3 is 55%, the recognition recall rate is 58%, the recognition accuracy of anomaly 4 is 88%, and the recognition recall rate is 90%;
[0115] The recognition effect of the student model after knowledge distillation and integration calibration is as follows:
[0116] The recognition accuracy of anomaly 1 is 72%, the recognition recall rate is 80%, the recognition accuracy of anomaly 2 is 65%, the recognition recall rate is 87%, the recognition accuracy of anomaly 3 is 70%, the recognition recall rate is 65%, the recognition accuracy of anomaly 4 is 92%, and the recognition recall rate is 95%;
[0117] These experimental results show that the method can effectively improve the recognition accuracy and generalization ability of the student model in the image anomaly detection scene, reduce the energy consumption and cost of the device, and make the model more suitable for running on resource-limited edge devices.
[0118] The method improves knowledge distillation to adapt to resource-limited edge devices in the image anomaly detection scene. The method constructs a multi-teacher model system and uses multiple data enhancement strategies such as mixup, cutout, cutmix, and augmix to expand the data set, so that each teacher model can learn rich feature representations and generalization ability for specific detection scenarios. Then, by integrating and calibrating the knowledge transferred by different teacher models, the student model can learn feature information in multiple scenarios, thereby significantly improving its detection performance in complex environments. The precision and robustness of the student model in the image anomaly detection task are significantly improved, making it better adapt to complex and variable detection scenarios, effectively reducing the volume and computational complexity of the model, making it more suitable for running on resource-limited edge devices, thereby improving real-time performance and reducing device energy consumption and cost. In summary, the method provides strong technical support for the intelligent and precise application of image anomaly detection, and has important practical application value.
[0119] Embodiment 2
[0120] Based on Embodiment 1, the present embodiment provides a large model knowledge distillation and integration calibration system based on data enhancement strategies, as shown in Figure 2 The system comprises:
[0121] (1) Multi-scene data augmentation module, for performing data augmentation strategy combination on the original dataset by hybrid enhancement to generate training subsets covering different abnormal scenes;
[0122] (2) Multi-teacher model training module, for training multi-teacher models using the training subsets, each teacher model corresponding to at least one data augmentation strategy to learn scene-specific features, and assigning guidance weights to each teacher model based on the similarity of the dataset to the target application scenario;
[0123] (3) Lightweight student model generation module, for integrating and calibrating the knowledge corresponding to the learned scenes based on the constructed multiple teacher models through knowledge distillation, and fine-tuning to obtain a lightweight student model;
[0124] (4) Joint inference module, for jointly using the student model and a preset deep learning model to calculate the confidence of the deep learning model inference result using a discriminator, and if the confidence is greater than a preset value, using the target detection model result as the final result, otherwise using the student model result as the final result.
[0125] The large model knowledge distillation integration calibration system includes multi-teacher model collaborative training, as well as data augmentation strategy dynamic configuration, knowledge fusion and calibration mechanism. By using the large model knowledge distillation integration calibration technology based on data augmentation strategy, through the application of diversified data augmentation technology, multiple teacher models can be trained respectively, each model is specially designed for a specific image anomaly recognition environment, thus having different generalization performance. In the student model learning process, it not only absorbs the feature knowledge from the teacher model, but also considers the feature structure and knowledge information under different scenes. The student model comprehensively and calibrates the knowledge transmitted by multiple teacher models, significantly reduces the model size while maintaining similar behavior and performance, which helps to improve its robustness and recognition accuracy on edge computing devices.
[0126] The scene-adaptive knowledge integration method covers feature alignment, weight allocation and consistency constraint processes. Through the large model knowledge distillation integration calibration technology based on data augmentation strategy, we can train multiple teacher models with specific domain knowledge in various image recognition or target detection tasks using diversified data augmentation means. These teacher models are optimized for different detection environments or target categories, thus exhibiting unique generalization ability. In the learning process of the student model, by comprehensively and calibrating the knowledge from different teacher models, the student model can obtain more comprehensive and rich feature representation, which not only helps to improve its recognition accuracy in specific tasks, but also significantly enhances its adaptability in new environments or unknown scenes.
[0127] Therefore, whether it is vehicle recognition in the field of intelligent transportation, abnormal behavior detection in security monitoring, or lesion recognition in the field of medical health, and other general scenarios, the technical solution provided by the present application can effectively reduce the model size while maintaining or improving the performance and robustness of the model, making it more suitable for running on resource-constrained edge computing devices. This innovative technical breakthrough not only provides strong support for the intelligentization of large models, but also lays a solid foundation for promoting the application and development of image recognition and target detection technology in a wider range of fields.
[0128] Embodiment 3
[0129] The embodiment provides an electronic device, including one or more processors and memories, the memories store one or more programs, the one or more programs include instructions for executing the large model knowledge distillation integrated calibration method based on the data enhancement strategy as described in embodiment 1.
[0130] As Figure 3 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course, it can also include other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above Figure 1 described method. Of course, in addition to the software implementation, the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0131] The memory can include non-permanent memory in the computer readable medium, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer readable medium.
[0132] Embodiment 4
[0133] The embodiment provides a computer readable storage medium, characterized in that it includes one or more programs for one or more processors of an electronic device to execute, the one or more programs include instructions for executing the large model knowledge distillation integrated calibration method based on the data enhancement strategy as described in embodiment 1.
[0134] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.
[0135] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A large-scale model knowledge distillation integrated calibration method based on data augmentation strategy, characterized in that, Includes the following steps: A multi-scenario data augmentation module is constructed, which performs a combination of data augmentation strategies on the original dataset through hybrid augmentation to generate training subsets covering different abnormal scenarios; A multi-teacher model is trained using the training subset, with each teacher model corresponding to at least one data augmentation strategy to learn scene-specific features. Based on the constructed multiple teacher models, knowledge distillation is used to integrate and calibrate the knowledge corresponding to the learning scenario, and fine-tuning is performed to obtain a lightweight student model; By combining the student model and the preset deep learning model, the confidence level of the inference result of the deep learning model is calculated using a discriminator. If the confidence level is greater than the preset value, the result of the object detection model is used as the final result; otherwise, the result of the student model is used as the final result.
2. The large model knowledge distillation integration calibration method based on data augmentation strategy according to claim 1, characterized in that, The aforementioned combination of data augmentation strategies includes applying one of the following strategies to the same original dataset: mixup, cutout, cutmix, and augmix.
3. The large model knowledge distillation integration calibration method based on data augmentation strategy according to claim 2, characterized in that, Under the mixup strategy, the loss function for training the teacher model is: x mu =λx1+(1-λ)x2 y mu =λy1+(1-λ)y2 in, Let f1() be the loss function of the teacher model under the mixup strategy, λ be the parameter, f1() represent the teacher model network under the mixup strategy, x1 and x2 be two randomly sampled input points, and y1 and y2 be their corresponding one-hot encoded labels. This is the cross-entropy loss function.
4. The large model knowledge distillation integration calibration method based on data augmentation strategy according to claim 2, characterized in that, Under the cutout strategy, the loss function for training the teacher model is: x co =M⊙x+(1-M)⊙Z in, Let f2() be the loss function for training the teacher model under the cutout strategy, and let x be the original image. co Z is the cropped image, Z is a zero matrix with the same size as the input image x, M is a binary mask, and ⊙ represents pixel-by-pixel multiplication.
5. The large model knowledge distillation integration calibration method based on data augmentation strategy according to claim 2, characterized in that, The Cutmix strategy is as follows: x cm =M⊙x1+(1-M)⊙x2 y cm =λy1+(1-λ)y2 Where, x cm y cm Here are the samples and their corresponding labels after processing by the Cutmix strategy. x1 and x2 are two samples in the dataset, y1 and y2 are their corresponding labels, M is a binary mask representing the cropped and filled regions of two randomly drawn images, λ∈[0,1], and the coordinates of the bounding box are B=(r x ,r y ,r w ,r h ), representing the cut and fill areas of the image, where W is the width of the image and H is the height of the image.
6. The large model knowledge distillation integration calibration method based on data augmentation strategy according to claim 2, characterized in that, Under the augmix strategy, the loss function for training the teacher model is: in, For cross-entropy loss, The loss is the Jensen-Shannon loss, k is the weighted hyperparameter, and p is the weighted hyperparameter. a p b p c To predict the probability of the original image x for the teacher model, two augmented images x b x c The probability distribution, M is p a p b p c The average value of KL[] represents the KL divergence operator.
7. The large model knowledge distillation integration calibration method based on data augmentation strategy according to claim 1, characterized in that, The process of integrating and calibrating knowledge relevant to the learning scenario through knowledge distillation, and then fine-tuning it to obtain a lightweight student model, uses the following loss function: z i =P(y|f i ,x)i=1,2,3,4 z s =P(y|f s ,x) in, Let α be the total loss function, and α be the parameter. These are the divergence loss and the task-specific loss, respectively, f. s For the student model, f1, f2, f3, and f4 are the teacher models trained using four data augmentation strategies, respectively. i z s Let x and y represent the probability distributions of the teacher model and the student model, respectively, and let KL[] represent the KL divergence operator.
8. A large-scale model knowledge distillation integrated calibration system based on data augmentation strategies, characterized in that, For implementing the large model knowledge distillation integrated calibration method based on data augmentation strategies as described in any one of claims 1-7, the large model knowledge distillation integrated calibration system comprises: The multi-scenario data augmentation module is used to perform data augmentation strategy combinations on the original dataset through hybrid augmentation to generate training subsets covering different abnormal scenarios; A multi-teacher model training module is used to train a multi-teacher model using the training subset. Each teacher model corresponds to at least one data augmentation strategy to learn scenario-specific features. Guidance weights are assigned to each teacher model based on the similarity between the dataset and the target application scenario to form a teacher supervision framework. The lightweight student model generation module is used to integrate and calibrate the knowledge of the corresponding learning scenario based on multiple constructed teacher models through knowledge distillation, and fine-tune it to obtain a lightweight student model. The joint inference module is used to combine the student model and the preset deep learning model, and use a discriminator to calculate the confidence of the inference result of the deep learning model. If the confidence is greater than the preset value, the result of the target detection model is used as the final result; otherwise, the result of the student model is used as the final result.
9. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the large model knowledge distillation integrated calibration method based on any one of the data augmentation strategies as described in claims 1-7.
10. A computer-readable storage medium, characterized in that, Includes one or more programs that are executed by one or more processors of an electronic device, said one or more programs including instructions for performing the large model knowledge distillation integrated calibration method based on any one of claims 1-7.