General quantitative perception training method and system based on knowledge feature distillation

Abstract

The invention discloses a generalized quantitative perception training method and system based on knowledge feature distillation, and belongs to the technical field of intelligent computing. The method comprises the following steps: constructing a full-precision floating-point teacher model and a mixed bit width quantification student model based on a neural network architecture and a target task scene; carrying out pre-training on the full-precision floating point teacher model; taking the pre-trained full-precision floating-point teacher model as a knowledge transmission subject and the mixed bit width quantification student model as a knowledge receiving subject to carry out quantification perception training; the quantitative perception training is integrated with an attention feature fusion module and a multi-level context knowledge module; integrating a knowledge feature distillation strategy in the quantitative perception training process to realize knowledge extraction, migration and fusion of multilayer features; and a knowledge distillation loss function is calculated, the knowledge distillation loss function is taken as an optimization target, training optimization of the mixed bit width quantification student model is completed, and universal quantification perception training is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent computing technology and relates to a generalized quantitative perception training method and system based on knowledge feature distillation. Background Technology

[0002] With the widespread deployment of artificial intelligence technology in terminal devices, model quantization, as a key technology for reducing computational overhead and storage costs, has received high attention from both industry and academia. Quantization-aware training, by introducing quantization error simulation during the training process, effectively mitigates the accuracy loss caused by post-quantization and has become the mainstream model quantization scheme. However, existing quantization-aware training techniques still have many shortcomings in practical applications that urgently need to be addressed, hindering their promotion and implementation in complex dynamic scenarios.

[0003] Current research largely focuses on simply applying knowledge distillation strategies to quantization-aware training. These methods only achieve basic hierarchical knowledge transfer between teacher and student models, failing to delve into the knowledge transfer patterns across different network layers and branch structures. Crucially, existing technologies generally overlook the significant natural differences in quantization parameters within network branch structures. Different branches, due to their different functions such as feature extraction and fusion, often exhibit significant differences in optimal quantization bit width and range. A simple, uniform quantization strategy inevitably leads to uneven quantization losses across branches, resulting in a decline in overall model accuracy. Furthermore, existing quantization-aware training techniques heavily rely on specific task scenarios and lack a universal training framework. For models in different task scenarios, quantization strategies need to be redesigned, training hyperparameters adjusted, and extensive debugging performed, drastically increasing the training cost of quantization-aware training. This limitation not only hinders the automation of new application deployment processes but also prevents it from adapting to more open and dynamic application environments. When facing constantly changing task scenarios, specialized quantization schemes make model deployment toolchains bloated and redundant, hindering rapid iterative development of new task models and severely reducing the toolchain's flexibility and adaptability.

[0004] In summary, existing quantitative perception training techniques suffer from problems such as a single knowledge transfer method, failure to consider differences in branch quantization parameters, and poor scenario adaptability, resulting in large quantization losses, high training costs, and insufficient toolchain flexibility. Therefore, there is an urgent need for a quantization perception training scheme that can optimize knowledge transfer patterns, adapt to branch structure characteristics, and improve scenario versatility to overcome the aforementioned technical bottlenecks. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problem of high training cost in existing quantitative perception training methods, and to provide a generalized quantitative perception training method and system based on knowledge feature distillation.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention discloses a generalized quantitative perception training method based on knowledge feature distillation, comprising: A full-precision floating-point teacher model and a hybrid bit-width quantized student model are constructed based on a neural network architecture and the target task scenario; the full-precision floating-point teacher model is pre-trained. The pre-trained full-precision floating-point teacher model is used as the knowledge transfer subject, and the hybrid bit-width quantized student model is used as the knowledge receiving subject to carry out quantization perception training; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. In the quantitative perception training process, a knowledge feature distillation strategy is integrated to achieve knowledge extraction, transfer and fusion of multi-layer features; Based on the attention feature fusion module and the multi-level context knowledge module, the knowledge distillation loss function is calculated. Using the knowledge distillation loss function as the optimization target, the training optimization of the hybrid bit-width quantization student model is completed, and generalized quantization perception training is realized.

[0007] Further improvements are made in the following aspects: The implementation process of the attention feature fusion module includes: Perform a convolution operation on the output feature map of the next layer after the current layer to convert its feature dimensions to the size of the current layer; I represents the I-th layer of the model; The output feature map of the current layer is connected to the transformed output feature map of the next layer in the channel dimension, and a 1x1 convolution operation is performed on it to generate a feature map with two channels. The feature maps of the two channels are multiplied by the two output feature maps before concatenation using a dot product operation. This operation is a weighted operation on the two output feature maps to distinguish the importance of the two paths. The two weighted output feature maps are then combined using a point-to-point addition operation to merge them into a single feature map.

[0008] The specific steps of performing a point-to-point addition operation on the two weighted output feature maps to fuse them into a single feature map are as follows:

[0009] The above formula represents the specific calculation operation of the attention feature fusion module. It extracts a comprehensive spatial attention feature map from the two output feature maps using a 1x1 convolution operation, and then performs weighted fusion. Here, i represents the i-th layer in the model. It is the output feature map of the current layer. It is the output feature map of the next layer after transformation and calculation. It is a 1x1 convolution operator. It's a dot product operation. It is the feature map output after fusion.

[0010] The implementation process of the multi-level context knowledge module includes: By using pooling layer operations of different scales, the feature maps output by the full-precision floating-point teacher model and the mixed bit-width quantized student model are pooled to different sizes. Then, the L2 loss is calculated for the two outputs of the same scale, and finally, the weighted sum is obtained to obtain the knowledge feature distillation loss function for the full-precision floating-point teacher model and the mixed bit-width quantized student model.

[0011] The knowledge feature distillation loss function for the full-precision floating-point teacher model and the mixed bit-width quantized student model is obtained in the following way:

[0012] Here, i represents the i-th layer of the model. It is the output feature map of the i-th layer of the full-precision floating-point teacher model. This is the output feature map of the i-th layer of the hybrid bit-width quantized student model. L2 refers to calculating the L2 loss of the two feature maps. Pool() refers to the pooling operator of the algorithm. size refers to different input sizes. cnt is a weighting factor that is gradually reduced as the size parameter decreases. When performing knowledge feature distillation, not only the local information of each pixel is considered, but also the global context information at the target level and the image level.

[0013] The integrated knowledge feature distillation strategy in the quantitative perception training process specifically includes: For the output layer of the mixed bit-width quantized student model, the loss between the output layer features of the full-precision floating-point teacher model and the mixed bit-width quantized student model is directly calculated to determine the similarity distance between them. For the remaining layers of the hybrid bit-width quantization student model, the attention feature fusion module is used to calculate the output feature map of the next layer with the output feature map of the current layer, that is, the attention mechanism of the feature maps is fused to form a new output feature map; then, the loss between it and the corresponding layer output feature map of the teacher model is calculated; and so on, except for the last layer, the multi-layer output feature maps of the hybrid bit-width quantization student model are fused with the output feature map information of the next layer and used to extract knowledge features with the full-precision floating-point teacher model.

[0014] The knowledge distillation loss function is:

[0015] Where Fs is the output feature map of the student model layer; Ft is the output feature map of the teacher model; L is the set of layers of the entire model; MCK is the multi-level context knowledge module; and finally, L2 distance is used to calculate the difference between the two models.

[0016] Secondly, this invention discloses a generalized quantitative perception training system based on knowledge feature distillation, comprising: The model building unit is used to build a full-precision floating-point teacher model and a hybrid bit-width quantized student model based on a neural network architecture and target task scenario; and to pre-train the full-precision floating-point teacher model. The quantization perception training unit is used to carry out quantization perception training with the pre-trained full-precision floating-point teacher model as the knowledge transfer subject and the hybrid bit-width quantized student model as the knowledge receiving subject; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. The knowledge feature distillation unit is used to integrate the knowledge feature distillation strategy during the quantitative perception training process to realize the knowledge extraction, transfer and fusion of multi-layer features; The student model training unit is used to calculate the knowledge distillation loss function based on the attention feature fusion module and the multi-level context knowledge module, and to optimize the training of the hybrid bit-width quantization student model by using the knowledge distillation loss function as the optimization target, thereby realizing generalized quantization perception training.

[0017] Thirdly, the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned generalized quantitative perception training method based on knowledge feature distillation.

[0018] Fourthly, the present invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described generalized quantitative perception training method based on knowledge feature distillation.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a generalized quantization perception training method based on knowledge feature distillation. Traditional quantization techniques are prone to accuracy degradation due to focused parameter compression, especially noticeable in low-bit scenarios. This invention uses a pre-trained full-precision teacher model as the knowledge source, extracts deep and shallow features through multi-level contextual knowledge modules, and achieves accurate alignment of teacher and student model features by combining an attention fusion module. This "layered extraction-focused fusion" knowledge transfer mode allows the hybrid bit-width student model to fully inherit the capabilities of the teacher model, significantly reducing the performance loss caused by quantization and ensuring core metrics. Fixed bit-width quantization is difficult to adapt to different device resources and task requirements, resulting in strong deployment limitations. The hybrid bit-width student model of this invention can dynamically adjust the quantization bit width according to the importance of features in each layer of the model by combining the optimization direction of the distillation loss function: high bit width for core layers to maintain accuracy, and low bit width for non-core layers to reduce resources. With a general training framework, the model can quickly adapt to various terminals such as mobile phones and IoT devices, breaking through the "one-size-fits-all" limitation and improving scenario versatility. Traditional deployments require designing separate quantization schemes for different hardware, resulting in low iteration efficiency and high costs. This invention constructs a standardized training process. The pre-trained teacher model serves as a general knowledge carrier, and only the student model configuration needs to be adjusted to quickly generate an adapted model for different needs, eliminating the need for repeated training of the teacher model. Simultaneously, the integrated module reduces manual intervention, improves training stability and convergence speed, shortens the development cycle, and reduces R&D costs. Traditional distillation easily leads to knowledge fragmentation, affecting the model's generalization ability. This invention constructs a full-link knowledge transfer channel through multi-level context modules, progressively refining knowledge from shallow basics to deep connections, forming a concise and abstract knowledge expression. This reduces parameter redundancy and improves inference efficiency. Combined with hybrid bit-width quantization to reduce computation, it significantly accelerates model inference speed, meeting the real-time application requirements of edge devices. This invention effectively solves the pain points of edge quantization—"large accuracy loss, difficult adaptation, and high cost"—and its standardized process and flexible solutions provide reliable support for edge applications such as smart homes and industrial monitoring, lowering the threshold for AI deployment. At the same time, it provides a new research direction for quantization technology, promoting its upgrade from "scenario-specific" to "general and efficient," and contributing to the large-scale development of the edge intelligent industry. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a generalized quantitative perception training method based on knowledge feature distillation in an embodiment of the present invention; Figure 2This is a schematic diagram of a generalized quantitative perception training method based on knowledge feature distillation in an embodiment of the present invention. Figure 3 This is a schematic diagram of the attention feature fusion module in a generalized quantitative perception training method based on knowledge feature distillation in an embodiment of the present invention; Figure 4 This is a multi-level contextual knowledge module diagram in a generalized quantitative perception training method based on knowledge feature distillation in an embodiment of the present invention; Figure 5 This is a block diagram of a generalized quantitative perception training system based on knowledge feature distillation in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 This invention discloses a generalized quantitative perception training method based on knowledge feature distillation, comprising: S1, Construct a full-precision floating-point teacher model and a hybrid bit-width quantized student model based on a neural network architecture and the target task scenario; pre-train the full-precision floating-point teacher model; S2, using the pre-trained full-precision floating-point teacher model as the knowledge transfer subject and the hybrid bit-width quantized student model as the knowledge receiving subject, carries out quantization perception training; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. S3, In the quantitative perception training process, a knowledge feature distillation strategy is integrated to realize the knowledge extraction, transfer and fusion of multi-layer features; S4. Based on the attention feature fusion module and the multi-level context knowledge module, calculate the knowledge distillation loss function. Using the knowledge distillation loss function as the optimization target, complete the training optimization of the hybrid bit-width quantization student model and realize generalized quantization perception training.

[0026] This invention discloses a generalized quantization perception training method based on knowledge feature distillation. Traditional quantization techniques are prone to accuracy degradation due to focused parameter compression, especially noticeable in low-bit scenarios. This invention uses a pre-trained full-precision teacher model as the knowledge source, extracts deep and shallow features through multi-level contextual knowledge modules, and achieves accurate alignment of teacher and student model features by combining an attention fusion module. This "layered extraction-focused fusion" knowledge transfer mode allows the hybrid bit-width student model to fully inherit the capabilities of the teacher model, significantly reducing the performance loss caused by quantization and ensuring core metrics. Fixed bit-width quantization is difficult to adapt to different device resources and task requirements, resulting in strong deployment limitations. The hybrid bit-width student model of this invention can dynamically adjust the quantization bit width according to the importance of features in each layer of the model by combining the optimization direction of the distillation loss function: high bit width for core layers to maintain accuracy, and low bit width for non-core layers to reduce resources. With a general training framework, the model can quickly adapt to various terminals such as mobile phones and IoT devices, breaking through the "one-size-fits-all" limitation and improving scenario versatility. Traditional deployments require designing separate quantization schemes for different hardware, resulting in low iteration efficiency and high costs. This invention constructs a standardized training process. The pre-trained teacher model serves as a general knowledge carrier, and only the student model configuration needs to be adjusted to quickly generate an adapted model for different needs, eliminating the need for repeated training of the teacher model. Simultaneously, the integrated module reduces manual intervention, improves training stability and convergence speed, shortens the development cycle, and reduces R&D costs. Traditional distillation easily leads to knowledge fragmentation, affecting the model's generalization ability. This invention constructs a full-link knowledge transfer channel through multi-level context modules, progressively refining knowledge from shallow basics to deep connections, forming a concise and abstract knowledge expression. This reduces parameter redundancy and improves inference efficiency. Combined with hybrid bit-width quantization to reduce computation, it significantly accelerates model inference speed, meeting the real-time application requirements of edge devices. This invention effectively solves the pain points of edge quantization—"large accuracy loss, difficult adaptation, and high cost"—and its standardized process and flexible solutions provide reliable support for edge applications such as smart homes and industrial monitoring, lowering the threshold for AI deployment. At the same time, it provides a new research direction for quantization technology, promoting its upgrade from "scenario-specific" to "general and efficient," and contributing to the large-scale development of the edge intelligent industry.

[0027] The present invention will be further described below with reference to specific embodiments: This invention employs a knowledge feature distillation training method in the quantization-aware training process of neural network models. More specifically, it integrates knowledge extraction techniques such as attention feature fusion modules and multi-level contextual knowledge modules into quantization-aware training, thus overcoming the task-specific constraints of traditional quantization. By utilizing knowledge distillation from a floating-point "teacher" model to a mixed-bit-width quantized "student" model, it promotes knowledge interaction, transfer, and fusion between different model layers and feature scales. This not only improves the expressiveness of the mixed-bit-width quantized model but also accelerates the training process. Since the training process does not need to reflect the training accuracy of the task and there is no actual task accuracy loss, this invention achieves knowledge distillation from a full-precision teacher model to a mixed-bit-width quantized student model solely based on knowledge distillation loss. By directly learning feature information from the full-precision teacher model, it is necessary to combine multi-layer features for knowledge extraction so that the mixed-bit-width quantized student model can better learn richer and more accurate output feature information. It also solves the problem of accuracy loss caused by significant differences in quantization parameters during the quantization process in the model branch structure. This is beneficial for reducing training costs and rapidly deploying and developing lightweight models.

[0028] The complete technical solution is as follows: (1) The multi-task generalized knowledge feature distillation method utilizes multi-layer feature information for feature fusion, such as Figure 2 As shown, this technique specifically includes two modules: an attention feature fusion module and a multi-level contextual knowledge module. The attention feature fusion module processes the output feature maps of each layer of the teacher model with the output of the next layer to achieve attention fusion of the feature maps. Then, the calculated result is matched and fused with the output feature maps of the student model in the multi-level contextual knowledge module. Finally, multi-scale pooling operations are used to calculate the multi-scale knowledge distillation loss of the two feature maps.

[0029] Specifically, for the output layer of the student model, the loss between the output layer features of the student model and the teacher model can be directly calculated to determine their similarity distance. For the remaining layers of the student model, an attention feature fusion module is needed to calculate the output feature map of the next layer with the output feature map of the current layer, i.e., the attention mechanism of the feature maps is used to fuse them to form a new output feature map. Then, the loss between this new output feature map and the corresponding layer output feature map of the teacher model is calculated. This process continues, except for the last layer. The multi-layer output feature maps of the student model are fused with the output feature map information of the next layer and used for knowledge feature extraction with the teacher model.

[0030] As shown in the formula, This is the knowledge feature distillation loss function used in the technical method of this invention, where Fs is the output feature map of the student model layer, Ft is the output feature map of the teacher model, L is the set of layers of the entire model, MCK is the multi-level context knowledge module, and finally L2 distance is used to calculate the difference between the two models.

[0031] (2) For example Figure 3 As shown, the specific calculation of the attention feature fusion module involves extracting a comprehensive spatial attention feature map from the two output feature maps (teacher model output and next-layer multi-level context output) through a 1x1 convolution operation, and then performing weighted fusion. The specific calculation process of this module is as follows: For the output feature map information of the next layer after the current layer, a convolution operation is first performed to convert its feature dimension to the dimension of the current layer. I represents the I-th layer of the model. Then, the output feature map of the current layer and the transformed output feature map of the next layer are concatenated along the channel dimension, and a 1x1 convolution operation is performed to generate two-channel feature maps. Next, these are multiplied by the two output feature maps before concatenation using a dot product operation. This operation actually weights the two output feature maps, distinguishing their importance. Finally, the weighted two output feature maps are added point-to-point to fuse them into a single feature map. As shown in the formula, The specific computational operations of the attention feature fusion module involve extracting a comprehensive spatial attention feature map from the two output feature maps using a 1x1 convolution operation, followed by weighted fusion; where i represents the i-th layer in the model. It is the output feature map of the current layer. It is the output feature map of the next layer after transformation and calculation. It is a 1x1 convolution operator. It's a dot product operation. It is the feature map output after fusion.

[0032] (3) such as Figure 4 As shown, the multi-level contextual knowledge module is specifically calculated by pooling the feature maps output by the teacher model and the student model to different sizes through pooling layer operations of different scales. Then, L2 loss is calculated for the two outputs of the same scale, and finally, the weighted sum is obtained to obtain the knowledge feature distillation loss function for the teacher model and the student model.

[0033] The specific calculation process of this module is as follows: Through pooling layer operations of different scales, the feature maps output by the teacher model and the student model are pooled to different sizes. Then, L2 loss is calculated for the two outputs at the same scale. Finally, a weighted sum is obtained to obtain the knowledge feature distillation loss function for the teacher model and the student model. As shown in the formula:

[0034] Here, i represents the i-th layer of the model. It is the output feature map of the i-th layer of the full-precision floating-point teacher model. This is the output feature map of the i-th layer of the hybrid bit-width quantization student model. L2 refers to calculating the L2 loss between the two feature maps. Pool() is the pooling operator of the algorithm. size refers to different input sizes (set to 1, 2, 4 in this experiment). cnt is a weighting factor, which is gradually reduced as the size parameter decreases. Thus, when performing knowledge feature distillation, not only the local information of each pixel is considered, but also the global context information at the target level and image level. This helps the model learn more accurate and robust semantic understanding, resulting in more refined and continuous model feature output results. The role played by this loss function reflects the importance of context in visual understanding. In summary, the loss function used in this module helps the model produce more accurate and continuous output feature map results through three levels of context information, demonstrating the important role of context in visual understanding.

[0035] Combining the attention feature fusion module and the multi-level contextual knowledge module allows the model to integrate pixel-level and multi-scale contextual information simultaneously. The attention feature fusion module focuses on low-level pixel relationships, generating smooth local context. The multi-level contextual knowledge module promotes semantic consistency at a higher level, producing more accurate global context. Compared to using either the attention feature fusion module or the multi-level contextual knowledge module alone, combining these two mechanisms allows the model to understand the semantic information of an image at multiple spatial scales. This also reflects a deeper understanding of the multi-task generalized knowledge feature distillation method in this invention, resulting in more refined and continuous knowledge distillation results.

[0036] Example 1 This invention applies the knowledge feature distillation method to a quantization-aware training framework and conducts multiple sets of comparative experiments. The results show that, compared to knowledge feature distillation methods targeting only the output layer, the 1-bit MobileNetV2 model improves classification performance by 71.1% on the MNIST dataset, and the 1-bit Yolov3-tiny model based on the VOC2007 dataset improves detection performance by 27.4%. Compared to fully floating-point models, the accuracy loss of 4-bit and 8-bit hybrid MobileNetV2 (image classification), Yolov3-tiny (object detection), and U-Net (semantic segmentation) is less than 1%. This demonstrates that this invention can significantly reduce accuracy loss during model quantization. By using the knowledge feature distillation method to decouple quantization-aware training from the task, it can significantly shorten the development cycle of embedded intelligent applications and reduce development and deployment costs. This has significant practical implications for deploying high-precision, low-bit-width models on edge devices.

[0037] See Figure 5 This invention discloses a generalized quantitative perception training system based on knowledge feature distillation, comprising: The model building unit is used to build a full-precision floating-point teacher model and a hybrid bit-width quantized student model based on a neural network architecture and target task scenario; and to pre-train the full-precision floating-point teacher model. The quantization perception training unit is used to carry out quantization perception training with the pre-trained full-precision floating-point teacher model as the knowledge transfer subject and the hybrid bit-width quantized student model as the knowledge receiving subject; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. The knowledge feature distillation unit is used to integrate the knowledge feature distillation strategy during the quantitative perception training process to realize the knowledge extraction, transfer and fusion of multi-layer features; The student model training unit is used to calculate the knowledge distillation loss function based on the attention feature fusion module and the multi-level context knowledge module, and to optimize the training of the hybrid bit-width quantization student model by using the knowledge distillation loss function as the optimization target, thereby realizing generalized quantization perception training.

[0038] A third objective of this invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the generalized quantitative perception training method based on knowledge feature distillation.

[0039] The generalized quantitative perception training method based on knowledge feature distillation includes the following steps: A full-precision floating-point teacher model and a hybrid bit-width quantized student model are constructed based on a neural network architecture and the target task scenario; the full-precision floating-point teacher model is pre-trained. The pre-trained full-precision floating-point teacher model is used as the knowledge transfer subject, and the hybrid bit-width quantized student model is used as the knowledge receiving subject to carry out quantization perception training; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. In the quantitative perception training process, a knowledge feature distillation strategy is integrated to achieve knowledge extraction, transfer and fusion of multi-layer features; Based on the attention feature fusion module and the multi-level context knowledge module, the knowledge distillation loss function is calculated. Using the knowledge distillation loss function as the optimization target, the training optimization of the hybrid bit-width quantization student model is completed, and generalized quantization perception training is realized.

[0040] A fourth objective of this invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the generalized quantitative perception training method based on knowledge feature distillation.

[0041] The generalized quantitative perception training method based on knowledge feature distillation includes the following steps: A full-precision floating-point teacher model and a hybrid bit-width quantized student model are constructed based on a neural network architecture and the target task scenario; the full-precision floating-point teacher model is pre-trained. The pre-trained full-precision floating-point teacher model is used as the knowledge transfer subject, and the hybrid bit-width quantized student model is used as the knowledge receiving subject to carry out quantization perception training; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. In the quantitative perception training process, a knowledge feature distillation strategy is integrated to achieve knowledge extraction, transfer and fusion of multi-layer features; Based on the attention feature fusion module and the multi-level context knowledge module, the knowledge distillation loss function is calculated. Using the knowledge distillation loss function as the optimization target, the training optimization of the hybrid bit-width quantization student model is completed, and generalized quantization perception training is realized.

[0042] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0044] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0046] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A generalized quantitative perception training method based on knowledge feature distillation, characterized in that, include: A full-precision floating-point teacher model and a hybrid bit-width quantized student model are constructed based on a neural network architecture and the target task scenario; the full-precision floating-point teacher model is pre-trained. The pre-trained full-precision floating-point teacher model is used as the knowledge transfer subject, and the hybrid bit-width quantized student model is used as the knowledge receiving subject to carry out quantization perception training; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. In the quantitative perception training process, a knowledge feature distillation strategy is integrated to achieve knowledge extraction, transfer and fusion of multi-layer features; Based on the attention feature fusion module and the multi-level context knowledge module, the knowledge distillation loss function is calculated. Using the knowledge distillation loss function as the optimization target, the training optimization of the hybrid bit-width quantization student model is completed, and generalized quantization perception training is realized.

2. The generalized quantitative perception training method based on knowledge feature distillation according to claim 1, characterized in that, The implementation process of the attention feature fusion module includes: Perform a convolution operation on the output feature map of the next layer after the current layer to convert its feature dimensions to the size of the current layer; I represents the I-th layer of the model; The output feature map of the current layer is connected to the transformed output feature map of the next layer in the channel dimension, and a 1x1 convolution operation is performed on it to generate a feature map with two channels. The feature maps of the two channels are multiplied by the two output feature maps before concatenation using a dot product operation. This operation is a weighted operation on the two output feature maps to distinguish the importance of the two paths. The two weighted output feature maps are then combined using a point-to-point addition operation to merge them into a single feature map.

3. The generalized quantitative perception training method based on knowledge feature distillation according to claim 2, characterized in that, The specific steps of performing a point-to-point addition operation on the two weighted output feature maps to fuse them into a single feature map are as follows: The above formula represents the specific calculation operation of the attention feature fusion module. It extracts a comprehensive spatial attention feature map from the two output feature maps using a 1x1 convolution operation, and then performs weighted fusion. Here, i represents the i-th layer in the model. It is the output feature map of the current layer. It is the output feature map of the next layer after transformation and calculation. It is a 1x1 convolution operator. It's a dot product operation. It is the feature map output after fusion.

4. The generalized quantitative perception training method based on knowledge feature distillation according to claim 1, characterized in that, The implementation process of the multi-level context knowledge module includes: By using pooling layer operations of different scales, the feature maps output by the full-precision floating-point teacher model and the mixed bit-width quantized student model are pooled to different sizes. Then, the L2 loss is calculated for the two outputs of the same scale, and finally, the weighted sum is obtained to obtain the knowledge feature distillation loss function for the full-precision floating-point teacher model and the mixed bit-width quantized student model.

5. The generalized quantitative perception training method based on knowledge feature distillation according to claim 1, characterized in that, The knowledge feature distillation loss function for the full-precision floating-point teacher model and the mixed bit-width quantized student model is obtained in the following way: Here, i represents the i-th layer of the model. It is the output feature map of the i-th layer of the full-precision floating-point teacher model. This is the output feature map of the i-th layer of the hybrid bit-width quantized student model. L2 refers to calculating the L2 loss of the two feature maps. Pool() refers to the pooling operator of the algorithm. size refers to different input sizes. cnt is a weighting factor that is gradually reduced as the size parameter decreases. When performing knowledge feature distillation, not only the local information of each pixel is considered, but also the global context information at the target level and the image level.

6. The generalized quantitative perception training method based on knowledge feature distillation according to claim 1, characterized in that, The integrated knowledge feature distillation strategy in the quantitative perception training process specifically includes: For the output layer of the mixed bit-width quantized student model, the loss between the output layer features of the full-precision floating-point teacher model and the mixed bit-width quantized student model is directly calculated to determine the similarity distance between them. For the remaining layers of the hybrid bit-width quantization student model, the attention feature fusion module is used to calculate the output feature map of the next layer with the output feature map of the current layer, that is, the attention mechanism of the feature maps is fused to form a new output feature map; then, the loss between it and the corresponding layer output feature map of the teacher model is calculated; and so on, except for the last layer, the multi-layer output feature maps of the hybrid bit-width quantization student model are fused with the output feature map information of the next layer and used to extract knowledge features with the full-precision floating-point teacher model.

7. The generalized quantitative perception training method based on knowledge feature distillation according to claim 6, characterized in that, The knowledge distillation loss function is: Where Fs is the output feature map of the student model layer; Ft is the output feature map of the teacher model; L is the set of layers of the entire model; MCK is the multi-level context knowledge module; and finally, L2 distance is used to calculate the difference between the two models.

8. A generalized quantitative perception training system based on knowledge feature distillation, characterized in that, include: The model building unit is used to build a full-precision floating-point teacher model and a hybrid bit-width quantized student model based on a neural network architecture and target task scenario; and to pre-train the full-precision floating-point teacher model. The quantization perception training unit is used to carry out quantization perception training with the pre-trained full-precision floating-point teacher model as the knowledge transfer subject and the hybrid bit-width quantized student model as the knowledge receiving subject; the quantization perception training integrates an attention feature fusion module and a multi-level contextual knowledge module. The knowledge feature distillation unit is used to integrate the knowledge feature distillation strategy during the quantitative perception training process to realize the knowledge extraction, transfer and fusion of multi-layer features; The student model training unit is used to calculate the knowledge distillation loss function based on the attention feature fusion module and the multi-level context knowledge module, and to optimize the training of the hybrid bit-width quantization student model by using the knowledge distillation loss function as the optimization target, thereby realizing generalized quantization perception training.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the generalized quantitative perception training method based on knowledge feature distillation as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the generalized quantitative perception training method based on knowledge feature distillation as described in any one of claims 1-7.

Images