Visual model compression method for intelligent equipment with body

Abstract

The invention discloses a visual model compression method for an intelligent device with a body, and the method comprises the steps: selecting a teacher model and a student model, carrying out the forward propagation of a data set through the teacher model and the student model, and obtaining the feature representation of an intermediate layer, and the original score of each class; comparing the probability distribution output by the teacher model with the probability distribution output by the student model, and calculating the divergence between the probability distribution output by the teacher model and the probability distribution output by the student model as response loss; comparing the middle layer feature representations of the teacher model and the student model as comparison loss; probability distribution output by the student model is compared with a real label, and cross entropy between the probability distribution and the real label is calculated as hard loss; adding the response loss, the comparison loss and the hard loss according to weights to form a total loss function, and training the student model; and adjusting parameters of the student model through stochastic gradient descent, and minimizing a total loss function to realize model compression. The method provides efficient visual perception capability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a visual model compression method based on knowledge distillation technology. Background Art

[0002] Embodied intelligent devices need to complete visual perception tasks in real time, but their resources (such as storage space and computing power) are limited, and traditional large-scale deep learning models are difficult to deploy directly. Therefore, developing lightweight model compression methods that can both meet the computing resource limitations of embodied intelligent devices and maintain high performance has become an important direction for current technological development.

[0003] The model can effectively learn a small student model from a large teacher model, and it has received increasing attention from the society. Knowledge distillation is a typical model compression technology that uses the knowledge of a larger model (teacher model) to train a smaller model (student model), while maintaining the model performance and reducing its size to adapt to edge environments. Knowledge distillation has been widely used in the field of computer vision, especially in model compression and acceleration. Its core idea is to improve the performance of the student model by transferring the knowledge from a large pre-trained model (teacher model) to a smaller model (student model).

[0004] Knowledge distillation, as a model compression technology, was first proposed in 2015. The initial research introduced a softened output probability distribution, which enabled the student model to better imitate the behavior of the teacher model, thus showing obvious effects and application prospects in model compression. This method marked the birth of the concept of knowledge distillation and opened up a new direction for subsequent related research.

[0005] In subsequent developments, research work has further expanded this concept, focusing on introducing intermediate layer hints in the student model. By incorporating the feature information of the teacher model into the intermediate layers of the student model, this type of method can guide the student model to learn more complex and sophisticated representations, thereby achieving significant performance improvements. This not only optimizes the distillation process, but also deepens people's understanding of the internal feature learning mechanism of deep neural networks.

[0006] As the technology continues to mature, researchers have systematically analyzed and improved knowledge distillation. From response distillation, feature distillation to relationship distillation, various methods have been proposed, clarifying the role and effect of different types of knowledge in the transfer process. At the same time, in specific areas such as visual tasks, there are also works that evaluate the effectiveness of different distillation strategies and explore the impact of teacher model output on student model learning, thus emphasizing the importance of choosing appropriate strategies for specific tasks.

[0007] In recent years, more new methods have emerged. For example, some studies focus on learning the relationship between samples, and improve the student model's understanding of the correlation information between samples through relational knowledge distillation; there are also methods that try to decouple different factors in the distillation process to enhance the flexibility and effectiveness of knowledge transfer. At the same time, there are also works that propose methods to dynamically adjust the learning temperature (i.e., course temperature), aiming to optimize the distillation effect according to the dynamics of the knowledge transfer process, thereby improving the performance of the student model in complex tasks.

[0008] In terms of specific application areas, knowledge distillation technology has also been deeply explored. For example, in the target detection task, by systematically analyzing the differences between the teacher model and the student model in the detection task, related research provides new ideas for practical applications. In addition, emerging methods such as attention transfer strategies and multi-objective knowledge distillation have also emerged, which further improve the effect of the student model in multi-task learning by focusing on specific category information and using self-reflection mechanisms.

[0009] Although knowledge distillation has shown significant potential in many fields, existing methods still face some challenges. First, many studies rely on specific teacher model architectures, which limits their application in heterogeneous environments. In addition, the effectiveness of knowledge selection and transfer mechanisms still needs to be further optimized to improve the adaptability and accuracy of student models to complex tasks. Especially when dealing with multi-task learning, the effect of knowledge distillation is often limited by the interference between tasks. How to effectively select and transfer knowledge remains an urgent problem to be solved.

[0010] In addition, knowledge distillation often faces challenges in computing resources and time efficiency when dealing with the diversity and complexity of practical applications. Although knowledge distillation can significantly improve the inference speed of the model and reduce the model size, in some high-load tasks, how to reduce computing and memory overhead while ensuring performance still requires in-depth research. Summary of the invention

[0011] To solve the above problems, the present invention discloses a visual model compression method for embodied intelligent devices, which is suitable for efficient visual task processing in embodied intelligent devices such as robots, drones and autonomous driving vehicles. It can comprehensively capture various types of knowledge of the teacher model, make full use of intermediate feature information, and is suitable for embodied intelligent resource-constrained scenarios.

[0012] To achieve the above object, the present invention proposes a visual model compression method for an embodied intelligent device, comprising the following steps: Step (1): Select a visual model with outstanding performance as the teacher model; Step (2): Design a lightweight student model; Step (3): Use the teacher model to forward propagate the data set to obtain the intermediate layer feature representation of the teacher model. The original scores of each category compared with the teacher model ; Step (4): Use the student model to forward propagate the data set to obtain the intermediate layer feature representation of the student model. The original scores of the student model for each category ; Step (5): The original scores of the teacher model for each category The raw scores of the student model for each category Passed separately Function activation is the probability distribution of the teacher model output , the probability distribution of the student model output ; Step (6): The probability distribution output by the teacher model and the probability distribution of the student model output Compare and calculate the difference between the two The divergence of , as the response loss; Step (7): Represent the intermediate layer features of the teacher model Intermediate layer feature representation with student model For comparison, calculate the difference in feature representation between the two as the contrast loss; Step (8): Compare the probability distribution output by the student model with the true label and calculate the cross entropy between the two as the hard loss; Step (9): Add the response loss, contrast loss and hard loss according to the weights to form the total loss function, and train the student model; Step (10): Adjust the parameters of the student model through stochastic gradient descent to minimize the total loss function and achieve model compression.

[0013] Furthermore, in step (5), by When the function calculates the probability distribution, the temperature parameter is used To smooth the probability distribution; the probability distribution output by the teacher model and the probability distribution of the student model output The specific calculation formulas are: ; ; in, and is the original score of the teacher model for each category, and is the original score of the student model for each category, is the temperature parameter, is the number of categories for the classification task.

[0014] Furthermore, in step (6), the specific calculation formula of the response loss is: ;in, is the number of categories in the classification task, is the probability distribution of the teacher model output, is the probability distribution of the student model output.

[0015] Furthermore, in step (7), the specific calculation formula of the contrast loss is: ;in, Indicates input samples, Representation and The input samples that form positive pairs, Represents the teacher model's response to the input The characteristic representation of Represents the student model for the input The characteristic representation of Represents the student model for the input The characteristic representation of is the total number of samples, is a balancing parameter used to adjust the weights.

[0016] Furthermore, in step (8), the specific calculation formula of hard loss is: ;in, is the number of categories in the classification task, A one-bit efficient encoding representation of the true label, is the probability distribution of the student model output.

[0017] Furthermore, in step (9), the total loss function is calculated as: ;in, is the temperature parameter, , 、 They are the weight coefficients of response loss, contrast loss, and hard loss respectively.

[0018] Furthermore, in step (10), the specific calculation formula for updating the stochastic gradient descent parameters is: ;in, are the model parameters after the t+1th iteration, are the model parameters after the tth iteration, The learning rate used to control the step size of parameter updates, is the loss function About parameters The gradient of is the size of the mini-batch samples.

[0019] Through the implementation of the above technical solution, the beneficial effects of the present invention are: (1) Significantly reduce the number of parameters and computational complexity of the student model, making it suitable for resource-constrained embodied intelligent devices such as robots, drones, and autonomous vehicles; (2) By introducing the contrastive loss of intermediate feature representations, we achieve a more comprehensive capture of the teacher model’s knowledge, enabling the student model to better learn complex feature representations, thereby improving the accuracy of its classification tasks; (3) It supports effective distillation when the teacher model and the student model have different architectures. By comprehensively utilizing response knowledge and feature knowledge, it significantly improves the adaptability and robustness of the student model in heterogeneous environments. (4) By combining the response loss and feature contrast loss, an adjustable total loss function is designed. By introducing temperature parameters and weight coefficients, it can flexibly balance performance and computational efficiency to meet the different task requirements in embodied intelligence scenarios. (5) It is not only applicable to classification tasks, but can also be extended to complex visual task scenarios such as target detection and semantic segmentation, which improves the practicality of embodied intelligent devices in multi-task learning; (6) Stochastic gradient descent or its variants are used to optimize the student model parameters, which is simple to implement and computationally efficient, facilitating rapid deployment in embodied intelligence scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a visual model compression method for embodied intelligent devices described in the present invention. DETAILED DESCRIPTION

[0021] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0022] like Figure 1As shown, the visual model compression method for an embodied intelligent device comprises the following steps: Step (1): Select resnet110 as the teacher model, which has a complex network structure, a large number of parameters, and performs well in visual tasks; Step (2): Select resnet32 as the student model, which has a simple network structure and a small number of parameters. Its performance on visual tasks is weaker than that of resnet110. Step (3): Use the teacher model to perform forward propagation on the CIFAR-100 dataset to obtain its intermediate layer feature representation Instead of the raw scores for each category ; Step (4): Use the student model to forward propagate the data set to obtain the intermediate layer feature representation of the student model. The original scores of the student model for each category ; Step (5): The original scores of the teacher model for each category The raw scores of the student model for each category Passed separately Function activation is the probability distribution of the teacher model output , the probability distribution of the student model output ; In step (5), by When the function calculates the probability distribution, the temperature parameter is used To smooth the probability distribution; the probability distribution output by the teacher model and the probability distribution of the student model output The specific calculation formulas are: ; ; in, and is the original score of the teacher model for each category, and is the original score of the student model for each category, is the temperature parameter, is the number of categories for the classification task.

[0023] Step (6): The probability distribution output by the teacher model and the probability distribution of the student model output Compare and calculate the difference between the two The divergence of is taken as the response loss; the specific calculation formula of the response loss is: ;in, is the number of categories in the classification task, is the probability distribution of the teacher model output, is the probability distribution of the student model output.

[0024] Step (7): Represent the intermediate layer features of the teacher model Intermediate layer feature representation with student model For comparison, the difference in feature representation between the two is calculated as the contrast loss. The specific calculation formula for the contrast loss is: ;in, Indicates input samples, Representation and The input samples that form positive pairs, Represents the teacher model's response to the input The characteristic representation of Represents the student model for the input The characteristic representation of Represents the student model for the input The characteristic representation of is the total number of samples, is a balancing parameter used to adjust the weights.

[0025] Step (8): Compare the probability distribution output by the student model with the true label and calculate the cross entropy between the two as the hard loss. The specific calculation formula of the hard loss is: ;in, is the number of categories in the classification task, A one-bit efficient encoding representation of the true label, is the probability distribution of the student model output.

[0026] Step (9): Add the response loss, contrast loss and hard loss according to the weights to form the total loss function, and train the student model; the total loss function calculation formula is: ;in, is the temperature parameter, , 、 They are the weight coefficients of response loss, contrast loss, and hard loss respectively.

[0027] Step (10): Adjust the parameters of the student model through stochastic gradient descent to minimize the total loss function and achieve model compression; the specific calculation formula for stochastic gradient descent parameter update is: ;in, are the model parameters after the t+1th iteration, are the model parameters after the tth iteration, The learning rate used to control the step size of parameter updates, is the loss function About parameters The gradient of is the size of the mini-batch samples.

[0028] Beneficial effects of this embodiment: (1) Significantly reduce the number of parameters and computational complexity of the student model, making it suitable for resource-constrained embodied intelligent devices such as robots, drones, and autonomous vehicles; (2) By introducing the contrastive loss of intermediate feature representations, we achieve a more comprehensive capture of the teacher model’s knowledge, enabling the student model to better learn complex feature representations, thereby improving the accuracy of its classification tasks; (3) It supports effective distillation when the teacher model and the student model have different architectures. By comprehensively utilizing response knowledge and feature knowledge, it significantly improves the adaptability and robustness of the student model in heterogeneous environments. (4) By combining the response loss and feature contrast loss, an adjustable total loss function is designed. By introducing temperature parameters and weight coefficients, it can flexibly balance performance and computational efficiency to meet the different task requirements in embodied intelligence scenarios. (5) It is not only suitable for classification tasks, but can also be extended to complex visual task scenarios such as target detection and semantic segmentation, which improves the practicality of embodied intelligent devices in multi-task learning. (6) It uses stochastic gradient descent or its variants to optimize the student model parameters, which is simple to implement and computationally efficient, making it easy to quickly deploy in embodied intelligent scenarios.

[0029]

[0030]

[0031] As can be seen from Tables 1 and 2, this method can not only significantly reduce the number of parameters and FLOPs of the model, but also has better performance than other distillation algorithms (such as KD, FitNet, and CRD). Specifically, this method achieves an accuracy of 73.75% on the resnet32 student model, which exceeds the KD, FitNet, and CRD methods. At the same time, the number of parameters dropped from 1736564 in the teacher model to 472756 in the student model, a decrease of about 72.73%; FLOPs dropped from 1029596416 in the teacher model to 281568512 in the student model, a decrease of about 72.87%. This shows that the student model has achieved significant compression in the number of parameters and computational complexity compared to the teacher model. This not only improves the operating efficiency of the model, but also greatly reduces the demand for computing resources, which is very suitable for application on resource-constrained embodied smart devices.

[0032] Unified annotations: KD: Knowledge Distillation, the most primitive knowledge distillation method, transfers knowledge from the teacher model to the student model; FitNet: A distillation method that guides the training of the student model by passing hints from the intermediate layers of the teacher network; CRD: Contrastive Representation Distillation, a distillation method that guides the student model to learn the intermediate layer representation of the teacher network through contrastive learning; FLOPs: Floating Point Operations Per Second, which indicates the number of floating-point operations required by the model for inference and is used to measure the computational complexity of the model.

[0033] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A visual model compression method for embodied intelligent devices, characterized in that: The following steps are involved: Step (1): Select a visual model with outstanding performance as the teacher model; Step (2): Design a lightweight student model; Step (3): Use the teacher model to forward propagate the data set to obtain the intermediate layer feature representation of the teacher model. The original scores of each category compared with the teacher model ; Step (4): Use the student model to forward propagate the data set to obtain the intermediate layer feature representation of the student model. The original scores of the student model for each category ; Step (5): The original scores of the teacher model for each category The raw scores of the student model for each category Passed separately Function activation is the probability distribution of the teacher model output , the probability distribution of the student model output ; Step (6): The probability distribution output by the teacher model and the probability distribution of the student model output Compare and calculate the difference between the two The divergence of , as the response loss; Step (7): Represent the intermediate layer features of the teacher model Intermediate layer feature representation with student model For comparison, calculate the difference in feature representation between the two as the contrast loss; Step (8): Compare the probability distribution output by the student model with the true label and calculate the cross entropy between the two as the hard loss; Step (9): Add the response loss, contrast loss and hard loss according to the weights to form the total loss function, and train the student model; Step (10): Adjust the parameters of the student model through stochastic gradient descent to minimize the total loss function and achieve model compression.

2. The visual model compression method for embodied intelligent devices according to claim 1, characterized in that: In step (5), by When the function calculates the probability distribution, the temperature parameter is used To smooth the probability distribution; the probability distribution output by the teacher model and the probability distribution of the student model output The specific calculation formulas are: ； ； in, and is the original score of the teacher model for each category, and is the original score of the student model for each category, is the temperature parameter, is the number of categories for the classification task.

3. The visual model compression method for embodied intelligent devices according to claim 1, characterized in that: In step (6), the specific calculation formula of response loss is: ;in, is the number of categories in the classification task, is the probability distribution of the teacher model output, is the probability distribution of the student model output.

4. The visual model compression method for embodied intelligent devices according to claim 1, characterized in that: In step (7), the specific calculation formula of contrast loss is: ;in, Indicates input samples, Representation and The input samples that form positive pairs, Represents the teacher model's response to the input The characteristic representation of Represents the student model for the input The characteristic representation of Represents the student model for the input The characteristic representation of is the total number of samples, is a balancing parameter used to adjust the weights.

5. The visual model compression method for embodied intelligent devices according to claim 1, characterized in that: In step (8), the specific calculation formula of hard loss is: ;in, is the number of categories in the classification task, A one-bit efficient encoding representation of the true label, is the probability distribution of the student model output.

6. The visual model compression method for embodied intelligent devices according to claim 1, characterized in that: In step (9), the total loss function is calculated as: ;in, is the temperature parameter, , 、 They are the weight coefficients of response loss, contrast loss, and hard loss respectively.

7. The visual model compression method for embodied intelligent devices according to claim 1, characterized in that: In step (10), the specific calculation formula for stochastic gradient descent parameter update is: ;in, are the model parameters after the t+1th iteration, are the model parameters after the tth iteration, The learning rate used to control the step size of parameter updates, is the loss function About parameters The gradient of is the size of the mini-batch samples.

Images