Distillation system based on training and pushing integrated scene knowledge

Abstract

The invention provides a training and pushing integrated scene knowledge-based distillation system, and belongs to the technical field of cloud computing. The system comprises a teacher model and a student model. The teacher model and the student model perform distillation in the same model, the student model learns from a soft label generated by the teacher model, and the soft label is used for representing probability distribution of flow peak values in different time periods; the student model is deployed at an edge computing node, localized flow prediction is realized through a self-distillation technology, and the student model autonomously performs knowledge transmission by generating a soft label in different training stages. According to the method, data enhancement and a mixed training strategy are combined, so that the student model not only can learn richer feature information in diversified data distribution, but also can effectively avoid the problems of information loss, over-fitting and the like. According to the method, the model can be ensured to quickly respond and provide an accurate prediction result under the condition of limited resources.

Description

Technical Field

[0001] This invention belongs to the field of cloud computing technology, specifically relating to a scenario-based knowledge distillation system based on integrated training and push, which is mainly applied to scenarios that require compression of deep learning models, especially on devices with limited computing resources, such as mobile devices, embedded systems and IoT devices. Background Technology

[0002] Knowledge distillation (KD) originated from the need to optimize and compress deep learning models. Initially, it was proposed as a model compression technique, with the core idea of ​​transferring knowledge from a large-scale, computationally expensive teacher model to a small-scale student model. As research progressed, the application of knowledge distillation expanded from simple model compression to other fields such as transfer learning, multimodal learning, semi-supervised learning, and unsupervised learning. With the development of deep learning technology, knowledge distillation has also been applied to a wider range of tasks, especially in scenarios involving efficient inference and real-time computation. On hardware with limited computing resources, such as embedded devices, smartphones, and IoT devices, knowledge distillation has become a key means of compressing large models and improving inference efficiency. Because knowledge distillation can significantly reduce the computational complexity and memory footprint of models without significantly sacrificing accuracy, it plays a crucial role in modern artificial intelligence applications.

[0003] While knowledge distillation has achieved significant results in various fields, some challenges and problems remain. How to better preserve the knowledge of the teacher model in the student model, how to design an effective distillation loss function, and how to handle potential information loss during the distillation process are still important issues that urgently need to be addressed. With the continuous emergence of new distillation algorithms and optimization methods, the application prospects of knowledge distillation will further expand, especially in large-scale data processing and smart terminal applications, where it will play an even more crucial role.

[0004] The drawback of knowledge distillation lies in its strong dependence on the student model architecture. When the student model has a small capacity or an unreasonable architecture, it may not be able to fully learn the knowledge from the teacher model, resulting in poor distillation performance. This is especially true when the teacher model is highly complex, while the student model is overly simplified; the student model may fail to acquire the core knowledge of the teacher model, exhibiting lower accuracy and generalization ability. During the learning process, the conversion of the teacher model's knowledge into soft labels may lead to information loss, further affecting the training effect of the student model. Therefore, designing a reasonable hybrid training architecture that can effectively learn and transfer the teacher model's knowledge is a major challenge in knowledge distillation.

[0005] In cloud computing scenarios within the telecommunications sector, intelligent network traffic prediction and optimization systems have extremely high requirements for real-time performance, accuracy, and resource utilization. Traditional methods rely on external teacher models to predict traffic peaks in telecommunications networks, but these methods have significant drawbacks: teacher models are typically highly complex, have significantly different architectures, and deviate from lightweight student models in terms of data distribution and task objectives. This makes it difficult for student models to accurately learn the prediction logic, resulting in delays or resource waste during real-time traffic scheduling. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a knowledge distillation system based on integrated training and inference scenarios. By performing distillation within the same model, this invention enables the model to autonomously generate soft labels and transfer knowledge at different training stages, without relying on complex external teacher models. This approach reduces the capability differences between teacher and student models, promoting self-optimization of student models within the same structure. Furthermore, combining data augmentation and hybrid training strategies enhances the adaptability of student models across different data distributions, thereby improving their generalization performance. In addition, by incorporating multi-task learning or ensemble learning methods, multiple student models can learn collaboratively and jointly optimize performance, thus alleviating the bottleneck problems that may arise with a single student model.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A knowledge distillation system based on integrated training and push scenarios includes: a teacher model and a student model; the teacher model and the student model perform distillation within the same model, and the student model learns from soft labels generated by the teacher model, the soft labels being used to represent the probability distribution of traffic peaks in different time periods; the student model is deployed on edge computing nodes and achieves localized traffic prediction through self-distillation technology, and the student model autonomously transfers knowledge by generating soft labels at different training stages.

[0008] Optionally, the teacher model is a combination of a convolutional neural network and a Transformer, and the student model is a lightweight convolutional neural network.

[0009] Optionally, the student model uses the separated convolutions and a lightweight network architecture, MobileNet.

[0010] Optionally, during training, the student model dynamically adjusts the generation method and hyperparameters of the soft labels, wherein the smoothness of the soft labels is controlled by the temperature parameter.

[0011] Optionally, in the early stages of training, the student model is trained using a supervised learning method to generate initial soft labels.

[0012] Optionally, in the later stages of training, the student model generates predicted values ​​as soft labels through forward propagation at each training phase.

[0013] Optionally, in the later stages of training, the student model learns by minimizing the difference between the soft labels generated in the current training phase and the soft labels generated in the previous training phase.

[0014] Optionally, the student model uses supervised learning cross-entropy loss for learning in the early stages of training; when the performance of the student model approaches that of the teacher model, distillation loss is introduced to optimize the model by minimizing the difference between the probability distributions output by the student model and the teacher model.

[0015] Optionally, the student model uses low-rank matrix factorization to remove redundant parameters in the network and pruning techniques to remove unnecessary network connections.

[0016] Optionally, the system includes multiple student models that are optimized collaboratively, with each student model sharing intermediate feature representations.

[0017] The beneficial effects of this invention are: 1. This invention, by combining data augmentation and hybrid training strategies, enables student models to learn richer feature information from diverse data distributions while effectively avoiding problems such as information loss and overfitting. Especially in integrated training and inference scenarios, the combination of these technologies can effectively improve training efficiency and ensure efficient deployment of the model during the inference phase.

[0018] 2. Data augmentation enables models to maintain good generalization ability when faced with different types of data; while hybrid training strategies, through multi-task learning and self-supervised learning, enable student models to be optimized from multiple levels, improving the comprehensiveness and accuracy of their learning.

[0019] 3. This invention addresses the problems of incomplete learning, knowledge loss, and low reasoning efficiency that may be encountered in the traditional distillation process. It not only improves the capabilities of the student model but also enhances the flexibility and adaptability of the entire training process. Especially in reasoning scenarios, it can ensure that the model responds quickly and provides accurate prediction results even with limited resources.

[0020] 4. By rationally designing the student model architecture, combining multi-task learning and ensemble learning, and employing adaptive adjustment strategies, this invention demonstrates strong adaptability in the integrated training and inference platform. It not only improves the student model's ability to learn complex tasks but also significantly enhances the model's efficiency and performance during the inference phase. Attached Figure Description

[0021] Figure 1This is a schematic diagram of the working principle of the integrated training and promotion scenario knowledge distillation system of the present invention. Detailed Implementation

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0023] This invention proposes a knowledge distillation system based on integrated training and push scenarios, comprising a teacher model and a student model. The student model is deployed on edge computing nodes and achieves localized traffic prediction through self-distillation technology. The core of self-distillation technology lies in generating soft labels, i.e., the probability distribution of historical traffic data, within the model to achieve knowledge transfer. Specifically, in the initial stage, the model uses historical traffic data to generate soft labels. These labels not only contain category information but also reflect the inherent probability distribution of the data. For example, in a traffic prediction task, soft labels may represent the probability distribution of traffic peaks in different time periods, rather than simple binary classification results. Through internal knowledge transfer, model parameters are gradually optimized, improving prediction accuracy. As training progresses, the model iteratively generates more accurate soft labels at different stages. This process is similar to "self-teaching," enabling the model to gradually adapt to complex and ever-changing network environments. To further improve generalization performance, the system combines data augmentation and hybrid training strategies. Data augmentation technology generates synthetic data by simulating traffic fluctuations in different time periods: sudden traffic spikes and network failure scenarios, and mixes this synthetic data with real data for training. In 5G network scenarios, data augmentation can simulate traffic surges in densely populated areas or traffic interruptions caused by network equipment failures, enabling the model to better adapt to unexpected situations in real-world applications. Hybrid training strategies, by fusing real and synthetic data, balance data distribution and prevent model overfitting. Furthermore, the system employs multi-task learning or ensemble learning, with multiple student models collaboratively optimizing. For example, one model focuses on traffic prediction, while another handles resource allocation; both share intermediate feature representations, collectively improving overall efficiency. This collaborative mechanism not only reduces computational redundancy but also enhances the model's robustness through feature sharing.

[0024] The implementation of self-distillation technology involves multiple stages, encompassing model initialization, soft label generation, parameter optimization, and dynamic resource scheduling. In the initial stage, the model is trained based on historical traffic data, generating initial soft labels. For example, in edge computing scenarios, the model utilizes locally collected traffic data—user request frequency, data transmission volume—to generate soft labels that reflect the temporal and spatial distribution characteristics of the traffic. Through internal knowledge transfer, model parameters are progressively optimized, improving prediction accuracy. As training progresses, the model iteratively generates more accurate soft labels at different stages. This process is achieved through backpropagation, ensuring the model can gradually adapt to complex and ever-changing network environments.

[0025] During model training, the system employs a hybrid training strategy, integrating real and synthetic data. For example, in 5G network scenarios, real data may come from traffic information reported by user devices, while synthetic data is generated through simulation tools to simulate traffic fluctuations under different network conditions. This hybrid training strategy not only enriches the data sources but also enhances the model's generalization ability through data augmentation techniques. Furthermore, the system utilizes multi-task learning or ensemble learning, with multiple student models collaboratively optimizing. For instance, one model focuses on traffic prediction, while another handles resource allocation; both share intermediate feature representations, collectively improving overall efficiency. This collaborative mechanism reduces computational redundancy and enhances model robustness through feature sharing and task coupling. This solution enables telecom operators to dynamically adjust cloud computing resource allocation, reducing operating costs while ensuring user service quality. For example, in 5G network scenarios, the system dynamically adjusts the resource allocation of edge computing nodes through real-time traffic prediction, avoiding resource waste. When a traffic surge is predicted, the system automatically increases computing resources to ensure timely processing of user requests; when traffic decreases, the system reduces resource usage, lowering operating costs. This dynamic resource scheduling mechanism not only improves resource utilization but also enhances user satisfaction by reducing video buffering or call interruptions.

[0026] Self-distillation technology has broad application prospects and significant commercial value in cloud computing scenarios within the telecommunications sector. In 5G networks and edge computing scenarios, the system effectively addresses network congestion and equipment failures through real-time traffic prediction and dynamic resource scheduling. For example, deploying lightweight student models on edge computing nodes and using self-distillation technology to achieve localized traffic prediction reduces data transmission latency and improves user experience. Furthermore, the system further enhances overall efficiency through multi-task learning or ensemble learning, where multiple student models collaboratively optimize. For instance, one model focuses on traffic prediction, while another handles resource allocation; both share intermediate feature representations, jointly improving overall efficiency. From a commercial perspective, self-distillation technology brings significant economic benefits to telecom operators by reducing operating costs, minimizing cloud resource waste, and improving user service quality, such as reducing video buffering or call interruptions. For example, in 5G network scenarios, the system avoids resource waste and reduces operating costs through dynamic resource scheduling; simultaneously, real-time traffic prediction ensures timely processing of user requests, improving user satisfaction. This technical solution is not only applicable to the telecommunications sector but can also be extended to other scenarios requiring real-time prediction and resource scheduling, such as the Internet of Things and intelligent transportation, demonstrating broad application prospects.

[0027] The working principle of the integrated training and promotion scenario knowledge distillation system is as follows: Figure 1 As shown, it specifically includes the following parts.

[0028] 1. Construction of the Self-Distillation Method By performing distillation within the same model, the dependency on external teacher models is eliminated. The model autonomously transfers knowledge by generating soft labels at different training stages, enabling student models to self-optimize within the same architecture. This reduces the capability differences between student and teacher models, thus avoiding the poor distillation results caused by architectural mismatch between teacher and student models in traditional methods.

[0029] In the field of deep learning, as model size increases and computational resource demands rise, compressing models without sacrificing performance has become a critical issue. Knowledge distillation, which transfers knowledge from a large, complex teacher model to a smaller student model, has become a common solution. However, traditional knowledge distillation methods rely on an external teacher model, which can lead to suboptimal distillation results when there is an architectural mismatch between the teacher and student models. To address this problem, self-distillation has emerged. By performing distillation within the same model, it eliminates the dependence on an external teacher model, enabling knowledge transfer and self-optimization within the same model architecture.

[0030] Self-distillation essentially transfers knowledge within the same model by using soft labels generated at different training stages, rather than relying on an external teacher model. Traditional knowledge distillation techniques typically employ a more complex teacher model to guide a simpler student model. The teacher model outputs soft labels (i.e., the probability distribution predicted by the model), and the student model learns by minimizing the difference between its predictions and those of the teacher model. In self-distillation, the entire process occurs within the same model: the student model not only learns from external sources but also optimizes itself through its own knowledge acquisition.

[0031] During training, the model generates predictions through forward propagation at each training stage. These predictions (i.e., probability distributions) are used as soft labels. Unlike traditional methods where the output of the teacher model serves as the soft label, in self-distillation, the model itself generates soft labels at different training stages to guide subsequent training. These soft labels provide the student model with additional information, not only limited to hard labels (i.e., true labels), but also including finer-grained classification information and relative relationships between samples.

[0032] During training, the student model learns by minimizing the difference between its output and the soft labels generated in the previous stage. This process allows the student model to continuously improve itself, absorbing knowledge generated at different stages, and ultimately achieving self-optimization within the same architecture.

[0033] In many practical applications, especially those requiring real-time inference on devices with limited computing resources, the efficiency of training and inference is crucial. Traditional knowledge distillation methods rely on teacher models, typically requiring separate operations during the training and inference phases. This separation complicates the entire process, particularly in scenarios demanding efficient inference. However, in integrated training and inference platforms, the training and inference processes need to be unified as much as possible to reduce latency in the inference phase and lower the computational burden.

[0034] Self-distillation is well-suited for integrated training and inference scenarios because it allows for the direct generation and self-optimization of soft labels during training. This means the model is optimized not only during training but also effectively applied during inference. Self-distillation in integrated training and inference scenarios can be achieved through the following steps: Training Phase: In the initial training stage, the student model is trained using traditional supervised learning methods and generates preliminary soft labels. As training progresses, the model gradually learns more complex features and is able to generate new soft labels at different training stages. Each generated soft label contains more refined classification information and can guide the model to continuously optimize in subsequent training.

[0035] Self-optimization: In subsequent training phases, the student model uses previously generated soft labels to perform self-optimization. The soft labels for each training phase not only guide the model's learning in the current phase but also help the model better adjust parameters in the next phase. This self-optimization process does not rely on an external teacher model, thus effectively avoiding problems caused by architectural mismatch between the teacher and student models.

[0036] Inference Phase: During the inference phase, the self-distilled model can directly utilize the knowledge learned during training for efficient inference. Since the student model has already been optimized through self-distillation during training, the inference process is more streamlined and can run efficiently on computationally limited devices.

[0037] In traditional knowledge distillation, soft labels are provided by a teacher model, typically representing the teacher model's predicted probability distribution for each input sample. In self-distillation, however, soft labels are generated by the model itself. This process usually involves the following steps: In the early stages of training, the model generates predictions through regular supervised learning. In subsequent stages, the model utilizes not only the true labels but also the soft labels generated in the previous training stage as targets for more refined adjustments and optimizations. Through multiple iterations, the soft labels generated by the model gradually become more stable and accurate, guiding the model to continuously adjust itself.

[0038] Designing a suitable loss function is crucial for enabling student models to effectively learn soft labels. In self-distillation, the traditional supervised learning loss measures the difference between the model's predictions and the true labels. The distillation loss term measures the difference between the model's predictions and the soft labels, using KL divergence as a metric to encourage model predictions to align with the soft labels generated in the previous stage. The self-distillation training process involves multiple stages, each in which the model generates new soft labels and optimizes. Specific training strategies include using standard supervised learning in the initial stages to help the model build preliminary knowledge. As training progresses, self-distillation is gradually introduced to guide the model for more refined optimization using previously generated soft labels. Model performance is periodically evaluated during training to ensure the effectiveness of soft label generation and the stability of the training.

[0039] To further improve the efficiency of the model in integrated training and inference scenarios, the training process can be optimized by dynamically adjusting the soft label generation strategy and training parameters (such as the learning rate). In the early stages of training, the soft labels generated by the model may be relatively coarse, affecting the distillation effect. At this time, the weight of the distillation loss can be appropriately reduced, and supervised loss can be used first. In the later stages of training, when the model has a strong self-learning ability, the weight of the distillation loss can be increased to strengthen the role of soft labels in model optimization.

[0040] For integrated training and inference scenarios, self-distillation technology tightly integrates the training and inference processes. During training, it generates soft labels for self-optimization, and the inference process becomes more efficient, making it particularly suitable for devices with limited computing resources. Furthermore, self-distillation further improves the model's generalization ability and inference efficiency through optimized training strategies and adaptive adjustments, enabling the model to achieve excellent performance in complex real-time applications.

[0041] Self-distillation technology can not only compress models and improve inference efficiency without relying on external teacher models, but also provide a more flexible and efficient solution for optimizing deep learning models.

[0042] 2. Configure the hybrid training strategy based on data augmentation. By adapting to an integrated training and extension platform and combining data augmentation and hybrid training strategies, the student model's adaptability to different data distributions is enhanced, reducing information loss. Through diverse training data, the student model can better understand the knowledge conveyed by the teacher model, thereby improving its generalization ability and overcoming the problems of incomplete learning or knowledge loss that may occur in the traditional distillation process.

[0043] Traditional knowledge distillation methods often rely on fixed training data and a single training strategy, which can easily lead to information loss or incomplete learning, especially when faced with diverse data distributions. To overcome these problems, data augmentation and hybrid training strategies have been introduced to enhance the adaptability of student models to different data distributions, thereby improving their generalization performance. Especially in the application scenario of integrated training and inference platforms, a reasonable combination of these two approaches not only helps improve training efficiency but also facilitates rapid deployment during the inference phase.

[0044] Data augmentation is a technique that transforms training data to generate new samples. It is typically used to increase the diversity of datasets, thereby helping models generalize better. In the context of knowledge distillation, data augmentation not only helps improve the adaptability of student models to various inputs but also effectively enhances the accuracy and robustness of student models in learning knowledge from teacher models.

[0045] Data augmentation typically involves processing different types of data, such as images, text, or audio. For image tasks, common augmentation methods include rotation, cropping, scaling, flipping, and color dithering; for text data, operations such as synonym replacement, random insertion, and random deletion can be performed; and for audio data, enhancement can be achieved through time delays and pitch shifts. These augmentation methods allow student models to encounter more variant data during training, thereby improving the model's generalization ability.

[0046] In traditional knowledge distillation, the soft labels generated by the teacher model are usually produced on a specific dataset, which means that the student model can only learn knowledge from these data samples. However, with the help of data augmentation, the student model can be trained on an augmented dataset. It can not only learn from the knowledge conveyed by the teacher model, but also obtain a wider range of patterns through the augmented data, avoiding the problem of overfitting the model on a specific data distribution.

[0047] While data augmentation can significantly improve a model's generalization ability, its application also faces certain challenges. Over-augmentation may introduce noise, preventing student models from effectively extracting knowledge. Therefore, balancing the intensity of data augmentation with the model's training performance is a key issue in knowledge distillation.

[0048] Hybrid training strategies combine different training methods or objectives to optimize the learning process of a student model. These strategies typically combine traditional supervised learning, knowledge distillation, and self-supervised learning, allowing the model to be optimized at different levels and thus enhancing its learning ability.

[0049] Design of hybrid training strategies: In the context of knowledge distillation, hybrid training strategies include: Combining Standard Supervised Learning with Distillation Learning: In the early stages of training, standard supervised learning loss (cross-entropy loss) can be used to help the student model learn basic task information. Later, as the student model gradually approaches the performance of the teacher model, distillation loss can be introduced to further optimize the model's details and feature representation.

[0050] Combining self-supervised learning with knowledge distillation: Self-supervised learning methods can generate pseudo-labels from unlabeled data, further enriching the learning information of the student model. When combined with self-supervised learning and knowledge distillation, the model can not only learn from the soft labels of the teacher model, but also learn and optimize itself by generating its own labels, thereby further improving its generalization ability.

[0051] Multi-task learning designs multiple task objectives, allowing the model to simultaneously optimize the loss functions of several related tasks during training. This multi-task training guides the student model to learn from multiple aspects simultaneously, thereby accelerating its knowledge acquisition process and avoiding overfitting of the model to a single task.

[0052] An image classification task is implemented on a training and push-integrated platform. The student model is a lightweight convolutional neural network (CNN), and the teacher model is a large-scale deep residual network (ResNet). In the initial training phase, supervised learning is used to optimize the student model, and the loss function is cross-entropy loss. This phase focuses on training the model's ability to recognize labels.

[0053] As the student model's training progresses and it approaches the performance of the teacher model, distillation loss can be introduced. Distillation loss further optimizes the student model's feature representation by minimizing the difference between the student model's output probability distribution and the teacher model's output probability distribution (typically using KL divergence or MSE loss). This gradual transition strategy allows the student model to gain basic task understanding through supervised learning in the early stages, and then improve its accuracy and generalization ability through distillation learning in the later stages.

[0054] In the integrated training and inference platform, the combination of standard supervised learning and distillation learning can effectively improve the performance of student models during the inference phase. This strategy allows student models to learn from the teacher model's knowledge within a relatively simplified architecture, significantly improving inference efficiency while ensuring enhanced model performance.

[0055] In the initial training phase, the student model is trained using only supervised learning to quickly build basic task knowledge. As the student model approaches the level of the teacher model, distillation learning is introduced to gradually improve the model's generalization ability. Distillation learning not only enhances the student model's ability to absorb knowledge from the teacher model but also makes it more stable and efficient by adjusting the student model's feature representation. Because the student model employs a lightweight structure, it achieves a lower computational burden during the inference phase, which is significant for applications with high real-time inference requirements (such as smart terminals).

[0056] In practical applications, the teacher model is a large deep learning network, the Transformer, while the student model is a lightweight RNN (Recurrent Neural Network). Due to the scarcity of labeled data, traditional supervised learning and distillation methods may not fully unlock the potential of the student model. In this case, self-supervised learning can be introduced.

[0057] Specifically, self-supervised learning methods can be used to generate pseudo-labels through contrastive learning. This allows the student model to not only learn the soft labels from the teacher model during training but also optimize its feature representation using the self-generated pseudo-labels. Self-supervised learning helps student models learn more knowledge from unlabeled data without manually labeled data, thus enhancing their generalization ability.

[0058] Using contrastive learning techniques, the student model can learn the underlying structure and features of unlabeled speech data by comparing different slices, thus providing more valuable auxiliary signals for the distillation process. In the integrated training and promotion platform, the student model can not only learn from the teacher model's knowledge on labeled data, but also further optimize the model's learning process through self-supervised tasks.

[0059] Knowledge distillation is used in an image recognition and image captioning generation task. The teacher model is a large-scale combination of a convolutional neural network (CNN) and a Transformer, capable of performing both image classification and image captioning generation tasks simultaneously. The student model is a lightweight CNN that learns knowledge from the teacher model through distillation. Within this multi-task learning framework, the student model learns not only image classification knowledge from the teacher model but also image captioning generation knowledge. Multi-task learning, by sharing the underlying convolutional feature extraction module, allows the student model to simultaneously optimize the loss function for multiple tasks during training. Through this shared module, the student model learns jointly in both tasks, effectively improving its understanding of image content.

[0060] In the early stages of training, the student model is trained on an image classification task using standard supervised learning. Later, distillation loss is introduced, and multi-task learning comes into play, allowing the student model to simultaneously learn from the teacher model's knowledge in image classification and description generation. By sharing the underlying feature extraction network, the student model can learn across multiple tasks, thus improving learning efficiency. Multi-task learning avoids overfitting to a single task, improving the model's generalization ability. Through multi-task learning, the student model can acquire knowledge from the teacher model across multiple tasks, accelerating its knowledge acquisition and optimization process.

[0061] By combining hybrid training strategies such as standard supervised learning with distillation, self-supervised learning with distillation, and multi-task learning within an integrated training and inference platform, student models can achieve more comprehensive learning across different tasks and data distributions, thereby improving their performance and generalization ability. The combination of standard supervised learning and distillation provides an effective transition mechanism, helping student models quickly learn basic task information in the early stages and optimize model details later. The combination of self-supervised learning and distillation addresses the problem of data scarcity, further enhancing the learning ability of student models. Multi-task learning, through parameter sharing and task collaboration, improves the overall performance of student models and avoids overfitting. Ultimately, these strategies enable efficient model training and inference within the integrated training and inference platform, improving the model's performance in practical applications.

[0062] Hybrid training strategies enable student models to optimize across different task objectives, promoting the synergistic development of different model capabilities. In knowledge distillation scenarios, hybrid training not only helps student models learn the soft labels of teacher models but also allows them to acquire more comprehensive knowledge from diverse training objectives, thereby improving their performance in the inference phase.

[0063] While hybrid training strategies can significantly improve the overall performance of models, their application requires overcoming several key issues: The loss functions for different tasks may differ numerically. How to reasonably allocate the weights of each task in multi-task learning and ensure the balance between different tasks is an important challenge in hybrid training strategies.

[0064] Hybrid training strategies can complicate the training process, requiring more hyperparameter tuning and training cycles. Therefore, ensuring both performance and the efficiency and operability of the training process is a problem that needs to be addressed.

[0065] In integrated training and inference platforms, the training and inference processes typically need to be closely integrated, especially when facing real-time inference tasks on resource-constrained devices. To enable self-distillation and hybrid training strategies to operate efficiently in such scenarios, the platform usually needs to be specifically adapted and optimized.

[0066] In the integrated training and inference platform, the inference phase requires efficient computation with limited resources. Therefore, the student model needs to be optimized to reduce its computational burden. Through data augmentation and hybrid training strategies, the student model can accumulate richer knowledge during training, avoiding the need for excessive computational resources for inference during the inference phase.

[0067] Integrated training and inference platforms typically allocate computational resources appropriately between the training and inference phases. In hybrid training, some computational tasks may be completed during the training phase, while inference is handled in real-time using a pre-optimized student model. This optimized resource allocation maximizes the efficiency of both training and inference, ensuring the model can respond in real-time even in complex application scenarios.

[0068] The integrated training and inference platform allows for dynamic adjustments to training and inference strategies. During training, data augmentation methods can be dynamically adjusted based on real-time feedback to ensure rapid and efficient inference results during the inference phase. This dynamic adjustment mechanism helps the platform flexibly optimize training and inference according to different task requirements.

[0069] By combining data augmentation and hybrid training strategies, student models can not only learn richer feature information from diverse data distributions, but also effectively avoid problems such as information loss and overfitting. Especially in integrated training and inference scenarios, the combination of these technologies can effectively improve training efficiency and ensure efficient deployment of the model during the inference phase.

[0070] 3. Student model architecture adapted to the integrated training and promotion platform To adapt to the integrated training and promotion platform, avoid overly simplified student model architectures and insufficient capacity, and ensure that student models can effectively learn the core knowledge of teacher models, a reasonable student model architecture needs to be designed based on the specific task. A well-designed model can improve the performance of student models, enabling them to fully absorb the knowledge from teacher models and improve distillation efficiency.

[0071] In integrated training and promotion scenarios, student models need to run efficiently on resource-constrained hardware platforms, while also possessing sufficient capacity and complexity to support the core knowledge of teacher models. Designing a reasonable student model architecture is a crucial step for successful knowledge distillation.

[0072] In knowledge distillation, the design of the student model architecture faces multiple challenges. First, the student model must be large enough to accommodate the knowledge transferred from the teacher model. Especially in complex tasks (such as image recognition and natural language processing), the teacher model typically possesses higher complexity and deeper network layers, capable of capturing rich feature representations and abstract high-order knowledge. Therefore, the student model must have sufficient capacity to learn these complex features.

[0073] The student model should not be designed to be overly complex, otherwise it will lead to excessive model size and computational load, negating the advantages of an integrated training and inference platform. Especially on edge devices and smart terminals, an overly large model not only consumes more computational resources but also results in slower inference speeds, impacting system response time. Therefore, designing a reasonable student model architecture that balances model capacity and computational resources is crucial for the successful application of knowledge distillation technology in an integrated training and inference platform.

[0074] The design of the student model architecture must determine its capacity based on the task requirements. Smaller models are prone to overfitting or underlearning during training, while larger models may not be able to fully adapt to the resource constraints of an integrated training and inference platform. Therefore, a reasonable model architecture design needs to be optimized in the following aspects: Network depth and width are two crucial factors determining the capacity of a student model. In deep learning models, deeper networks can capture more complex features but lead to higher computational complexity. Width, on the other hand, relates to the model's expressive power; overly narrow networks may result in information loss, while overly wide networks increase unnecessary computation. During training, shallower networks can serve as the initial architecture for the student model. Subsequently, the network structure can be expanded and deepened to further improve the model's capacity, balancing model complexity and computational resources.

[0075] In task design, feature extraction modules (such as convolutional layers and attention mechanisms) are crucial components of the model. For smaller student models, redundant computation can be reduced by designing more efficient feature extraction networks, such as using separable convolutions and the lightweight network architecture MobileNet. This ensures sufficient feature information is extracted while maintaining computational efficiency.

[0076] While ensuring model capacity, minimizing redundant computations is a crucial consideration when designing the student model architecture. Low-rank matrix factorization can effectively reduce redundant parameters in the network, lowering computational complexity; pruning techniques can remove unnecessary network connections, further compressing the student model's size. These methods can effectively improve the efficiency of the student model during the inference phase, ensuring efficient computation under limited resources.

[0077] Designing a reasonable student model architecture is not only about ensuring computational efficiency, but also about guaranteeing that the student model can effectively learn and transfer knowledge from the teacher model. The teacher model typically generates multi-layered feature information, which is crucial for the student model's learning. The student model's task in the distillation process is to mimic the teacher model's feature representations, and the student model architecture must be designed to ensure it has sufficient expressive power to learn these complex features.

[0078] In the knowledge distillation process, the student model typically learns not only the final output of the teacher model, but also the feature representations of the teacher model at various intermediate layers. To effectively perform this knowledge transfer, the student model architecture should match the hierarchical structure of the teacher model to a certain extent, especially in lower network layers (such as convolutional layers) and higher fully connected layers, to ensure that the student model can capture multi-level feature information.

[0079] In traditional knowledge distillation methods, student models typically learn using soft labels generated by the teacher model. Soft labels provide probability distributions between categories, helping the student model learn richer information. However, soft labels can also lead to information loss, especially at higher task complexity levels. To address this issue, a well-designed student model architecture should combine hard and soft label learning methods. Hard labels help the student model perform basic task recognition, while soft labels help it learn deeper levels of knowledge.

[0080] During training, the architecture design of the student model not only needs to address network layer matching and computational efficiency issues, but also needs to consider how to further optimize the student model in practical applications. By adopting the following optimization strategies, the performance of the student model can be further improved: By employing a fine-grained distillation strategy, student models can acquire knowledge from different levels and modules of the teacher model. This strategy allows for the integration of feature representations at different levels into the design of the student model, ensuring that the model can transfer knowledge across multiple levels.

[0081] Through adaptive training strategies, student models can dynamically adjust their network architecture and parameters based on task requirements and data characteristics. This approach allows student models to gradually adapt to task changes during training, thereby optimizing their structure and performance.

[0082] A well-designed student model architecture can not only effectively improve distillation results but also achieve efficient computation during the inference phase. On an integrated training and inference platform, the design of the student model needs to balance computational resources and performance requirements, avoiding overly simplified or complex models that could hinder practical applications. By introducing optimization methods for deep learning models, such as low-rank matrix factorization, network pruning, and multi-level feature matching, the student model can effectively learn the core knowledge of the teacher model and achieve high efficiency and accuracy during inference.

[0083] A well-designed student model architecture can also improve the model's generalization ability and avoid performance degradation due to overfitting. In practical applications, especially in smart devices and edge computing platforms, optimizing the student model can significantly reduce computational overhead and improve response speed, thereby meeting real-time requirements.

[0084] In an integrated training and inference platform for knowledge distillation, designing a reasonable student model architecture is one of the key factors ensuring the success of the distillation process. By balancing the capacity, computational complexity, feature extraction capability, and learning strategy of the student model, it is possible to enable the student model to fully absorb the knowledge of the teacher model and maintain high efficiency and accuracy during the inference phase. A reasonable model design not only improves distillation results but also adapts to the platform's resource limitations, ensuring good performance of the student model in practical applications.

[0085] Student models should strike a balance between computational complexity and learning ability. Overly simplified models may lack sufficient learning capacity and fail to effectively absorb knowledge from the teacher model; conversely, overly complex models may impose excessive computational burden on the inference phase, violating the efficiency requirement of integrated training and inference. Using modular attention mechanisms can improve the adaptability of student models to complex tasks while facilitating efficient execution during the inference phase. Based on the specific task (e.g., image classification, natural language processing, or multimodal learning), design lightweight model architectures suitable for that task. Convolutional Neural Networks (CNNs) are suitable for image tasks, while lightweight Transformers are suitable for natural language processing tasks.

[0086] The model is made more compact by removing redundant neurons or weights without sacrificing key functionality. Weights and activation functions are compressed from 32-bit floating-point numbers to 8 bits or lower, reducing computational complexity. Objectives are designed for different layers of the student model, with shallow layers responsible for learning low-level features and deeper layers responsible for learning high-level semantic knowledge, ensuring efficient learning at each layer. During distillation, the outputs of corresponding layers in the student and teacher models are matched to enhance the knowledge absorption capacity of the student model.

[0087] In integrated training and inference scenarios, a well-designed student model architecture can significantly reduce the computational burden during the inference phase while ensuring efficient knowledge transfer during the training phase. Based on a lightweight Transformer architecture, it is possible to simultaneously meet the requirements for efficient training and rapid inference in language modeling tasks.

[0088] 4. Combination of multi-task learning and ensemble learning To adapt to an integrated training and extension platform, multi-task learning or ensemble learning strategies are introduced to enable multiple student models to collaboratively optimize and improve the overall performance of the model. These student models can complement each other by sharing information, further enhancing their performance and avoiding the bottleneck problems that may occur when a single model faces complex tasks. By integrating knowledge from multiple tasks or models, the overall performance of student models can be improved within the integrated training and extension platform. This strategy compensates for the shortcomings of a single model in complex tasks by allowing multiple student models to work collaboratively and share information.

[0089] In multi-task learning, different tasks share some model parameters (such as sharing the underlying feature extraction module), while a specific head module, the classification layer, is designed for each task. By analyzing the correlation between different tasks, complementary loss functions are designed. For example, in image classification and object detection tasks, sharing the feature extraction part can significantly improve the performance of both. A weighted loss function is used, summing the loss functions of multiple tasks with weights dynamically adjusted according to the importance of each task. The overall prediction accuracy is improved by combining the prediction results of multiple student models (e.g., using averaging or weighted voting).

[0090] During training, different student models share intermediate layer information or soft labels, forming a knowledge-sharing mechanism that enhances the learning ability of each student model. In the inference phase, the most suitable student model is dynamically selected based on the characteristics of the input data, thereby optimizing inference performance.

[0091] Through multi-task learning, student models can enhance their adaptability to specific tasks based on shared knowledge, while ensemble learning improves the overall performance of the model through collaborative optimization. In integrated training and inference scenarios, this combined strategy not only reduces the training and inference overhead of single-task models but also ensures that the model achieves better results in complex tasks.

[0092] 5. Adaptive adjustment Adapting to an integrated training and push platform, this approach continuously optimizes the distillation strategy of the student model during training. For example, by dynamically adjusting hyperparameters such as soft label generation and learning rate, the student model can constantly adapt to changes in task and data characteristics, thus achieving an adaptive optimization process. This method enables the student model to flexibly respond to different training stages and task requirements, maximizing distillation effectiveness.

[0093] The adaptive adjustment strategy aims to dynamically optimize the distillation process of the student model based on task changes and data characteristics. This strategy enables continuous optimization of the student model by adjusting training parameters and distillation strategies in real time, flexibly adapting to different training stages and task requirements.

[0094] In knowledge distillation, the smoothness of soft labels is controlled by a temperature parameter. By dynamically adjusting the temperature parameter, smoother labels can be provided in the early stages of model training, while more accurate labels can be provided in the later stages, thus improving distillation efficiency. An adaptive learning rate adjustment algorithm is used to dynamically adjust the learning rate based on the model's convergence, ensuring stable training of the model at different stages.

[0095] For multi-task learning scenarios, the weights of the loss function are dynamically adjusted based on the real-time requirements and importance of the tasks, ensuring that the student model prioritizes optimizing key tasks. During training, the model adjusts the distillation strategy in real time based on performance evaluation metrics (such as validation set loss or accuracy). For example, when the performance of the student model is close to that of the teacher model, the weight of the supervised loss can be gradually reduced to enhance the effect of the distillation loss. The enhancement methods are dynamically adjusted based on the distribution characteristics of the training data. For example, in scenarios with imbalanced data distribution, sampling weights are increased for long-tail categories to ensure that the student model can effectively learn knowledge from each category.

[0096] The adaptive adjustment strategy enables the student model to maintain high learning efficiency in dynamically changing task and data environments. In the integrated training and inference platform, this strategy ensures that the training and inference processes of the student model can be flexibly adjusted according to real-time needs, thereby maximizing distillation effectiveness.

[0097] In summary, the main features of this invention are as follows: 1. Traditional methods rely on complex external teacher models for knowledge transfer, where the teacher model guides the training of student models by outputting soft labels. However, in traditional methods, when the architectures of the teacher and student models differ significantly, the student model often struggles to effectively learn the knowledge from the teacher model, resulting in poor distillation performance. This is especially true when the teacher model is highly complex, as the student model cannot fully grasp its core information. This invention utilizes self-distillation technology, performing distillation within the same model. The model autonomously generates soft labels for knowledge transfer at different training stages, thus eliminating the need for an external teacher model. This method effectively reduces the capability differences between the teacher and student models, enabling the student model to self-optimize within the same architecture and improving distillation performance.

[0098] 2. Traditional methods, when passing soft labels to student models, may result in information loss, causing the student model to fail to accurately acquire key knowledge from the teacher model. This invention combines data augmentation and hybrid training strategies to further improve the adaptability of student models under different data distributions, mitigate the information loss problem, and thus enhance the generalization performance of student models. In this way, student models can learn and transfer knowledge from the teacher model, ensuring that training effectiveness is not affected by structural differences.

[0099] 3. Traditional methods typically rely on a single student model for distillation, which can lead to performance bottlenecks. This invention combines multi-task learning or ensemble learning strategies, allowing multiple student models to learn collaboratively and jointly optimize performance. This effectively solves the performance bottleneck problem of a single student model and improves the overall model performance. Therefore, this invention overcomes several limitations of traditional methods through self-distillation and diversified learning strategies, providing a more efficient and flexible model optimization scheme.

[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A training and inference integrated scene knowledge distillation system, characterized in that, Comprise: Teacher model and student model; the teacher model and student model are distilled within the same model, the student model learns from the soft label generated by the teacher model, and the soft label is used to represent the probability distribution of traffic peak in different time periods; the student model is deployed on the edge computing node, and the self-distillation technology is used to realize the localized traffic prediction, and the student model autonomously transfers knowledge in different training stages by generating soft labels.

2. The training and inference integrated scene knowledge distillation system of claim 1, wherein: The teacher model is a combined model of convolutional neural network and Transformer, and the student model is a lightweight convolutional neural network.

3. The training and inference integrated scene knowledge distillation system of claim 2, wherein: The student model uses separated convolution and lightweight network structure MobileNet.

4. The training and inference integrated scene knowledge distillation system of claim 1, wherein: During the training process, the student model dynamically adjusts the generation mode and hyperparameters of the soft label, wherein the smoothness of the soft label is controlled by a temperature parameter.

5. The training and inference integrated scenario knowledge distillation system of claim 4, wherein: In the early stage of training, the student model is trained using a supervised learning method to generate preliminary soft labels.

6. The training and inference integrated scenario knowledge distillation system of claim 5, wherein: In the later stage of training, the student model generates a prediction value as a soft label through forward propagation in each training stage.

7. The training and inference integrated scene knowledge distillation system of claim 6, wherein: In the later stage of training, the student model learns by minimizing the difference between the soft label generated in the current training stage and the soft label generated in the previous training stage.

8. The training and inference integrated scene knowledge distillation system of claim 1, wherein: The student model uses the cross-entropy loss of supervised learning in the early stage of training; when the performance of the student model approaches that of the teacher model, a distillation loss is introduced to optimize by minimizing the difference between the probability distribution output by the student model and the probability distribution output by the teacher model.

9. The system according to claim 1, wherein the system is based on the integration of training and inference scenario knowledge distillation. The student model uses low-rank matrix decomposition technology to remove redundant parameters in the network and uses pruning technology to remove unnecessary network connections.

10. The training and inference integrated scene knowledge distillation system of claim 1, wherein: The system comprises a plurality of student models optimized in collaboration, and each student model shares intermediate feature representations.

Images