Visual large model optimization deployment method based on knowledge distillation

Abstract

The invention provides a visual large model optimization deployment method based on knowledge distillation, and relates to the technical field of artificial intelligence, and the method comprises the steps: firstly constructing a reference model which comprises a backbone network and a basic reference network, training the reference model, replacing the backbone network in the trained reference model with a pre-trained Conformer network, and carrying out the optimization deployment of a visual large model; obtaining a Conformer-reference model after replacement, and performing fine tuning optimization on the Conformer-reference model to obtain a Conformer-reference model after first optimization; taking the Conformer-reference model after the first optimization as a teacher model, taking the reference model as a student model, and performing joint optimization on the teacher model and the student model based on knowledge distillation to obtain a Conformer-reference model after the second optimization; and compressing the Conformer-reference model after the second optimization, and performing deployment optimization on the Conformer-reference model to realize the balance between the model performance and the deployment efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for optimizing the deployment of large visual models based on knowledge distillation. Background Technology

[0002] With the rapid development of artificial intelligence technology in the field of large-scale models, the ever-expanding model size has brought about higher demands for computing resources and increased deployment difficulty. In the field of intelligent transportation, such as object detection and action recognition tasks, traditional benchmark models, limited by their simple structure, are unable to meet the requirements of complex recognition and detection tasks. While high-performance large models can improve the accuracy of tasks such as object detection and action recognition, they are difficult to run in real time on edge devices with limited computing power due to their high computational resource consumption and high inference latency. This contradiction is mainly reflected in three aspects: First, edge devices lack sufficient memory and computing power to support the high parameter scale of large models. Large models typically require high computing power and large memory, while edge devices (such as roadside units and vehicle terminals) have limited hardware resources. Second, model compression needs to balance accuracy and efficiency; over-optimization may lead to the loss of key features. Third, traffic scenarios have extremely high real-time requirements (such as millisecond-level decisions in autonomous driving), but the complex computation of large models on edge devices may not meet the low-latency requirements. For example, when an object detection model is deployed at the edge, if the inference speed is not optimized, it may lead to lag in video stream processing, increasing the risk of traffic accidents. Summary of the Invention

[0003] To address the mismatch between the performance requirements of large-scale models and the performance of the edge devices on which large-scale model deployment relies, this invention proposes a knowledge distillation-based method for optimizing the deployment of large-scale visual models. Through joint optimization, a balance between model performance and deployment efficiency is achieved.

[0004] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A method for optimizing the deployment of large visual models based on knowledge distillation includes the following steps: S1: Obtain a traffic dataset, use the traffic dataset to train the constructed benchmark model, and obtain the trained benchmark model; the benchmark model includes a backbone network and a basic benchmark network; S2: Replace the backbone network in the trained baseline model with the pre-trained Conformer network to obtain the Conformer-baseline model. Then, fine-tune and optimize the Conformer-baseline model to obtain the first optimized Conformer-baseline model. S3: Using the first optimized Conformer-benchmark model as the teacher model and the benchmark model as the student model, the teacher model and student model are jointly optimized based on knowledge distillation to obtain the second optimized Conformer-benchmark model. S4: Compress the second optimized Conformer-baseline model and optimize its deployment.

[0005] Preferably, after acquiring the traffic dataset, the traffic dataset is preprocessed, and the preprocessing includes: transforming, data augmenting, and class balancing the images in the traffic dataset.

[0006] Preferably, the benchmark model is trained using the preprocessed traffic dataset. The process is as follows: set the batch size and learning rate of the traffic dataset, combine self-supervised learning with a loss function to guide the benchmark model to gradually learn the key features in the traffic dataset, record the training loss in real time during the training process, and provide feedback on the model performance.

[0007] Preferably, the Conformer network includes several layers, several layers, and a task head network, and the Conformer network is pre-trained on the ImageNet-21K dataset.

[0008] Preferably, the process of fine-tuning and optimizing the Conformer-benchmark model is as follows: The first few layers of the Conformer network in the Conformer-based model are frozen, while the last few layers of the Conformer network and the task network are fine-tuned. A traffic dataset for fine-tuning is loaded, and the batch size, learning rate, and training iteration period of the traffic dataset are set. The Conformer-based model is trained using cosine decay scheduling. During training, gradient clipping is performed, and Dropout rate and BatchNorm are added. After training is completed, the first optimized Conformer-based model is obtained.

[0009] Preferably, in fine-tuning the Conformer-benchmark model, a fine-tuning loss is constructed, and the expression for the fine-tuning loss is:

[0010] in, For Focal or cross-entropy, This is an L2 regularization term.

[0011] Preferably, the process of jointly optimizing the teacher model and student model based on knowledge distillation is as follows: The knowledge distillation loss is expressed as follows:

[0012] in, For hard-label cross-entropy, For the teacher model, soft labeling The soft label for the student model, T is the temperature parameter used to amplify the distributional differences in the teacher model's output, and KL is the Kullback-Leibler divergence.

[0013] By minimizing KL, the probability distribution of the student model learning the teacher model is achieved. Build The core loss function for the feature alignment strategy in knowledge distillation is expressed as:

[0014] in, These are intermediate features of the teacher model. These are intermediate features of the student model; Construct the total loss function, expressed as:

[0015] The teacher model and student model are trained based on the total loss function. The student model is guided to gradually approach the teacher model in terms of hard label classification accuracy, soft label distribution consistency and intermediate feature representation effectiveness. The trained teacher model and student model are then used as the second optimized Conformer-benchmark model.

[0016] In this technical solution, the total loss function is a weighted combination of knowledge distillation loss and feature alignment loss. Knowledge distillation loss combines soft and hard labels; hard labels supervise the student model's learning of basic classification knowledge, while soft labels supervise the student model's learning of the teacher model's generalization knowledge. Feature alignment loss supervises the student model's learning of the teacher model's intermediate feature extraction knowledge. Preferably, in S4, the second optimized Conformer-baseline model is pruned to compress the model.

[0017] Preferably, in S4, the second optimized Conformer-benchmark model is quantized to compress the model.

[0018] Preferably, TensorRT is used to accelerate inference, and the deployment optimization is performed on the second optimized Conformer-baseline model.

[0019] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a knowledge distillation-based method for optimizing and deploying large-scale visual models. A baseline model, comprising a backbone network and a basic baseline network, is constructed and trained on a traffic dataset. The backbone network of the trained baseline model is replaced with a pre-trained Conformer network, resulting in a replaced Conformer-based model. This Conformer-based model is then fine-tuned to obtain a first-optimized Conformer-based model. Using this first-optimized Conformer-based model as the teacher model and the baseline model as the student model, knowledge distillation is used to jointly optimize both models, resulting in a second-optimized Conformer-based model, further reducing model complexity. Finally, the second-optimized Conformer-based model is compressed and optimized for deployment, improving model performance in intelligent transportation tasks and achieving a balance between model performance and deployment efficiency. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the knowledge distillation-based visual large model optimization deployment method proposed in this embodiment of the invention; Figure 2 This is a flowchart illustrating the fine-tuning and optimization of the Conformer-benchmark model proposed in this embodiment of the invention. Figure 3 This is a flowchart illustrating the joint optimization of the teacher and student models based on knowledge distillation proposed in this embodiment of the invention. Detailed Implementation

[0021] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a method for optimizing the deployment of large visual models based on knowledge distillation. (See also...) Figure 1 This includes the following steps: S1: Obtain the traffic dataset, use the traffic dataset to train the constructed benchmark model, and obtain the trained benchmark model; the benchmark model includes the backbone network and the basic benchmark network; S2: Replace the backbone network in the trained baseline model with the pre-trained Conformer network to obtain the Conformer-baseline model. Then, fine-tune and optimize the Conformer-baseline model to obtain the first optimized Conformer-baseline model. S3: Using the first optimized Conformer-benchmark model as the teacher model and the benchmark model as the student model, the teacher model and student model are jointly optimized based on knowledge distillation to obtain the second optimized Conformer-benchmark model. S4: Compress the second optimized Conformer-baseline model and optimize its deployment.

[0024] In this embodiment, a baseline model is constructed, comprising a backbone network and a basic baseline network. The baseline model is trained on a traffic dataset. The backbone network in the trained baseline model is replaced with a pre-trained Conformer network to obtain a replaced Conformer-based baseline model. This Conformer-based baseline model is then fine-tuned to obtain a first-optimized Conformer-based baseline model. Using the first-optimized Conformer-based baseline model as the teacher model and the baseline model as the student model, the teacher and student models are jointly optimized based on knowledge distillation to obtain a second-optimized Conformer-based baseline model, further reducing model complexity. The second-optimized Conformer-based baseline model is then compressed and deployed to improve the model's performance in intelligent transportation tasks, achieving a balance between model performance and deployment efficiency.

[0025] Example 2 This embodiment proposes a visual large model optimization deployment method based on knowledge distillation. After acquiring the traffic dataset, the traffic dataset is preprocessed, including: transforming the images in the traffic dataset, data augmentation, and class balance.

[0026] In this embodiment, the traffic dataset includes a public dataset and a private traffic dataset. First, the public dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio. The private dataset is labeled manually or automatically, with an initial size of 5 million image pairs or 1 million video frames. The images in the traffic dataset are standardized to 640x640, and the video frame rate is fixed at 30fps. Data augmentation is applied, such as random flipping and brightness adjustment, to increase diversity. Multimodal alignment is ensured with an error of <10ms, and synchronization between video and sensor data is ensured. Statistical distribution is applied, and minority class samples are oversampled to ensure an average of 650 samples per class.

[0027] In some preferred embodiments, the benchmark model is trained using the preprocessed traffic dataset. The process involves setting the batch size and learning rate of the traffic dataset, combining self-supervised learning with a loss function to guide the benchmark model to gradually learn the key features in the traffic dataset, recording the training loss in real time during the training process, and providing feedback on the model performance.

[0028] In this embodiment, for the object detection task, the RetinaNet model is selected as the baseline model. The feature extractor uses a ResNet-50 backbone network, combined with a Feature Pyramid Network (FPN) to generate multi-scale features. The classification head uses Focal loss to handle class imbalance, making it suitable for multi-object detection tasks in intelligent transportation. The DETR model is selected, using a ResNet backbone combined with a Transformer decoder to directly predict bounding boxes and classes, making it suitable for object detection in complex scenes. For the action recognition task, a 3D convolutional network is selected: the input video frame sequence is processed through 3D convolutional layers to extract spatiotemporal features, and the convolution kernel size is 3x3x3 to capture dynamic information. The TimeSformer model processes long video sequences through a spatiotemporal attention mechanism, focusing on the global context of actions, making it suitable for complex action recognition. It has 12 self-attention heads to ensure global spatiotemporal dependency modeling.

[0029] The formula for Focal loss is:

[0030] in, The model's predicted probability for the true class t. γ is a balancing factor used to adjust the weights of positive and negative classes; the gradient is calculated using the chain rule to ensure that the focus is on the difficult samples.

[0031] The expression for the bounding box regression loss, Smooth L1, is:

[0032] Here, x represents the error between the predicted and the true value. The L1 loss |x| is non-differentiable at x=0, while the L2 loss x2 is sensitive to outliers. Smooth L1 uses L2 gradient smoothing when |x|<1 and L1 linear growth outside of it. By minimizing SL1, the bounding box coordinate difference is optimized, improving the localization accuracy.

[0033] The expression for cross-entropy loss is:

[0034] Where y is the one-hot label and y is the softmax output. Through gradient... Optimize the probability distribution.

[0035] During the data loading phase, the batch size of the traffic dataset was set to 128, the learning rate to 0.001, and the Adam optimizer was used. The Xavier method was employed for weight initialization to avoid gradient explosion. For the object detection task, Focal loss combined with bounding box regression loss was used with a weight ratio of 0.7:0.3. For the action recognition task, cross-entropy loss was used, and L2 regularization was added to prevent overfitting. A training period of 200 epochs was set to improve the stability of gradient estimation. An early stopping mechanism was implemented, pausing for 5 epochs based on validation set accuracy to avoid overfitting and meet the requirements for baseline model stability. The learning rate was halved every 50 epochs using a step decay method. After training, mean precision (mAP) and recall were calculated on the test set to ensure stable baseline performance. Error cases, such as missed detections of small targets, were analyzed to guide subsequent optimization. TensorBoard was used to visualize the curves.

[0036] In some preferred embodiments, the Conformer network includes several initial layers, several subsequent layers, and a task head network. The Conformer network is pre-trained on the ImageNet-21K dataset.

[0037] In this embodiment, the Conformer network is pre-trained on ImageNet-21K with approximately 100 million parameters, possessing richer general features and being more adaptable than the ResNet network when transferred to intelligent transportation scenarios.

[0038] In some preferred embodiments, see Figure 2 The process of fine-tuning and optimizing the Conformer-benchmark model is as follows: The first few layers of the Conformer network in the Conformer-based model are frozen, while the last few layers of the Conformer network and the task network are fine-tuned. A traffic dataset for fine-tuning is loaded, and the batch size, learning rate, and training iteration period of the traffic dataset are set. The Conformer-based model is trained using cosine decay scheduling. During training, gradient clipping is performed, and Dropout rate and BatchNorm are added. After training is completed, the first optimized Conformer-based model is obtained.

[0039] In this embodiment, the first 80% of the Conformer layers, mainly convolutional layers and the bottom Transformer layers, contain general features. Freezing these layers avoids losing pre-trained general knowledge and prevents the model from being unable to adapt to the specific tasks of intelligent transportation. Only the later layers and the task head are fine-tuned. The last 20% of the layers are high-level Transformer layers responsible for global context integration, and the task head is responsible for task-specific output. Fine-tuning these layers allows the model to adapt general features to intelligent transportation scenarios. The batch size is set to 64 to reduce gradient fluctuations and maintain the stability of fine-tuning. The learning rate is 1e-4, using cosine decay scheduling. The initial learning rate is 0.001, periodically decaying to 1e-6, resulting in rapid convergence in the early stages and fine-tuning in the later stages, suitable for fine-tuning tasks. 100 iteration cycles are set, and the validation set loss is monitored. Performance is evaluated every 10 cycles, and gradient clipping (norm<1) is used to prevent explosion. A Dropout rate of 0.1 and BatchNorm are added to prevent overfitting and ensure generalization ability.

[0040] The expression for the Conformer attention mechanism is:

[0041] Where Q, K, and V are generated by linear projection, dk is the dimension of Q and K, and QKT is used to calculate similarity; Combined convolutional layers The overall expression is:

[0042] Feature fusion has been improved.

[0043] In some preferred embodiments, in fine-tuning the Conformer-benchmark model, a fine-tuning loss is constructed, and the expression for the fine-tuning loss is:

[0044] in, For Focal or cross-entropy, This is an L2 regularization term.

[0045] In this embodiment, λ = 1e-5. Decaying weights, gradient By updating This ensures that the weighting remains stable.

[0046] Example 3 This embodiment proposes a method for optimizing the deployment of large visual models based on knowledge distillation. (See also...) Figure 3 The process of jointly optimizing the teacher and student models based on knowledge distillation is as follows: The knowledge distillation loss is expressed as follows:

[0047] in, For hard-label cross-entropy, For the teacher model, soft labeling The soft label for the student model, T is the temperature parameter used to amplify the distributional differences in the teacher model's output, and KL is the Kullback-Leibler divergence.

[0048] By minimizing KL, the probability distribution of the student model learning the teacher model is achieved. Build The core loss function for the feature alignment strategy in knowledge distillation is expressed as:

[0049] in, These are intermediate features of the teacher model. These are intermediate features of the student model; Construct the total loss function, expressed as:

[0050] The teacher model and student model are trained based on the total loss function. The student model is guided to gradually approach the teacher model in terms of hard label classification accuracy, soft label distribution consistency and intermediate feature representation effectiveness. The trained teacher model and student model are then used as the second optimized Conformer-benchmark model.

[0051] In this embodiment, the total loss function is a weighted combination of knowledge distillation loss and feature alignment loss. Knowledge distillation loss combines soft and hard labels; hard labels supervise the student model's learning of basic classification knowledge, while soft labels supervise the student model's learning of the teacher model's generalization knowledge. Feature alignment loss supervises the student model's learning of the teacher model's intermediate feature extraction knowledge. A temperature parameter T=4 amplifies the distributional differences in the teacher model's output, allowing the student model to learn not only the correct category but also the similarity between categories. The iteration is performed for 50 cycles with a learning rate of 1e-5 to monitor the convergence of the distillation loss. Weight parameters α=0.9 and β=0.5 balance the contributions of different supervision signals. The expression for minimizing feature differences in the L2 norm is:

[0052] The feature extraction layer of the student model is optimized using the chain rule.

[0053] In some preferred embodiments, in S4, the second optimized Conformer-baseline model is pruned to compress the model.

[0054] In this embodiment, low-weight connections are removed with a threshold of 0.01 and an iterative pruning rate of 20%. Low weights are gradually removed, and performance is restored after each pruning. The formula for the pruning rate is:

[0055] Where θ=0.01, target P=50%, and removal is achieved through iteration. Weights ≤ θ maintain a performance degradation of less than 5%.

[0056] In some preferred embodiments, in S4, the second optimized Conformer-benchmark model is quantized to compress the model.

[0057] In this embodiment, 8-bit integer quantization is used, and the scaling factor is dynamically calculated to reduce floating-point operations. The quantization formula is as follows:

[0058] Where s is the scaling factor and z is the zero point. To ensure minimal quantization error, rounding is performed.

[0059] In some preferred embodiments, TensorRT is used to accelerate inference and to optimize the deployment of the second optimized Conformer-baseline model.

[0060] In this embodiment, the quantization reduces the model size by 70% and increases inference speed by 3 times. The deployment optimization phase ensures the model runs efficiently on edge devices, and analyzes the performance before and after compression.

[0061] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing the deployment of large visual models based on knowledge distillation, characterized in that, Includes the following steps: S1: Obtain a traffic dataset, use the traffic dataset to train the constructed benchmark model, and obtain the trained benchmark model; the benchmark model includes a backbone network and a basic benchmark network; S2: Replace the backbone network in the trained baseline model with the pre-trained Conformer network to obtain the Conformer-baseline model. Then, fine-tune and optimize the Conformer-baseline model to obtain the first optimized Conformer-baseline model. S3: Using the first optimized Conformer-benchmark model as the teacher model and the benchmark model as the student model, the teacher model and student model are jointly optimized based on knowledge distillation to obtain the second optimized Conformer-benchmark model. S4: Compress the second optimized Conformer-baseline model and optimize its deployment.

2. The method for optimizing and deploying a large visual model based on knowledge distillation according to claim 1, characterized in that, After obtaining the traffic dataset, the traffic dataset is preprocessed, including transforming the images in the traffic dataset, data augmentation, and class balance.

3. The method for optimizing and deploying a large visual model based on knowledge distillation according to claim 2, characterized in that, The baseline model is trained using the preprocessed traffic dataset. The process is as follows: set the batch size and learning rate of the traffic dataset, combine self-supervised learning with a loss function to guide the baseline model to learn the key features in the traffic dataset step by step, record the training loss in real time during the training process, and provide feedback on the model performance.

4. The method for optimizing and deploying a large visual model based on knowledge distillation according to claim 1, characterized in that, The Conformer network comprises several layers, several layers, and a task head network, and is pre-trained on the ImageNet-21K dataset.

5. The method for optimizing and deploying a large visual model based on knowledge distillation according to claim 4, characterized in that, The process of fine-tuning and optimizing the Conformer-benchmark model is as follows: The first few layers of the Conformer network in the Conformer-based model are frozen, while the last few layers of the Conformer network and the task network are fine-tuned. A traffic dataset for fine-tuning is loaded, and the batch size, learning rate, and training iteration period of the traffic dataset are set. The Conformer-based model is trained using cosine decay scheduling. During training, gradient clipping is performed, and Dropout rate and BatchNorm are added. After training is completed, the first optimized Conformer-based model is obtained.

6. The method for optimizing the deployment of a large visual model based on knowledge distillation according to claim 5, characterized in that, In fine-tuning the Conformer-baseline model, a fine-tuning loss is constructed, and the expression for the fine-tuning loss is as follows: in, For Focal or cross-entropy, This is an L2 regularization term.

7. The method for optimizing the deployment of a large visual model based on knowledge distillation according to claim 1, characterized in that, The process of jointly optimizing the teacher and student models based on knowledge distillation is as follows: The knowledge distillation loss is expressed as follows: in, For hard-label cross-entropy, For the teacher model, soft labeling The soft label for the student model, T is the temperature parameter used to amplify the distributional differences in the teacher model's output, and KL is the Kullback-Leibler divergence. By minimizing KL, the probability distribution of the student model learning the teacher model is achieved. Build The core loss function for the feature alignment strategy in knowledge distillation is expressed as: in, These are intermediate features of the teacher model. These are intermediate features of the student model; Construct the total loss function, expressed as: The teacher model and student model are trained based on the total loss function. The student model is guided to gradually approach the teacher model in terms of hard label classification accuracy, soft label distribution consistency and intermediate feature representation effectiveness. The trained teacher model and student model are then used as the second optimized Conformer-benchmark model.

8. The method for optimizing the deployment of a large visual model based on knowledge distillation according to claim 1, characterized in that, In S4, the second optimized Conformer-baseline model is pruned to compress the model.

9. The method for optimizing the deployment of a large visual model based on knowledge distillation according to claim 1, characterized in that, In S4, the second optimized Conformer-benchmark model is quantized to compress the model.

10. The method for optimizing the deployment of a large visual model based on knowledge distillation according to claim 1, characterized in that, TensorRT was used to accelerate inference, and the deployment optimization was performed on the second optimized Conformer-baseline model.

Images