A general performance enhancement distillation method for convolutional neural networks

Abstract

The application discloses a general performance enhancement distillation method for a convolutional neural network, and relates to the technical field of artificial intelligence.The method first acquires feature maps output by teacher-student models, generates an importance map through channel projection and activation, and calculates two types of complementary spatial gains, namely consistency and coverage; a learnable pixel-level gate consensus module is used to model the correlation between teacher and student features, and a mixed coefficient is adaptively output; a dynamic spatial consensus weight mask is generated through convex combination, a feature distillation loss term is constructed in combination with the mask, a gating regularization term is introduced to construct a total loss function, and a student model is iteratively optimized.The application solves the problems of rigid weighting rules, insufficient adaptation in the training stage, and inaccurate gradient distribution of existing methods, realizes adaptive matching of the distillation strategy and the training stage, significantly improves the precision and generalization ability of a lightweight student model in a dense target detection task without increasing inference overhead, and is suitable for resource-constrained platforms such as edge devices.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a general performance-enhancing distillation method for convolutional neural networks. Background Technology

[0002] In the field of deep learning technology, dense object detectors have been widely used in various visual perception tasks, mainly divided into two categories: single-stage detectors (such as RetinaNet and FCOS) and two-stage detectors (such as Faster R-CNN). While these models possess excellent detection performance, they typically have high computational complexity and a large number of parameters, making them difficult to deploy directly on platforms with limited computing resources, such as edge devices. To address this deployment challenge, knowledge distillation (KD) has become a mainstream method for model compression and performance enhancement. Its core idea is to leverage the rich training signals generated by a high-capacity, high-performance teacher model to guide a lightweight student model in learning, thereby reducing model complexity while preserving as much of the original model's detection accuracy as possible.

[0003] Unlike simple visual tasks such as image classification, dense object detection has significant unique characteristics: it requires predicting thousands of spatial locations on a single image and suffers from severe foreground-background class imbalance, resulting in highly uneven spatial distribution of responses generated by the teacher model. Against this backdrop, efficiently transferring knowledge from the teacher model to the student model becomes the core challenge of knowledge distillation in dense object detection. Existing techniques mostly employ feature-based knowledge distillation methods and attempt to optimize distillation performance by designing spatial weighting rules to distinguish the importance of different locations in the feature map. For example, typical methods such as Decoupled Feature Distillation (DeFeat), Gradient-guided Instance-aware Distillation (GID), Foreground-focused Distillation (FKD), and Foreground-focused Distillation (FGD) all generate spatial masks by using labeled regions (Ground Truth), teacher model confidence, IoU scores, or gradient information to reweight specific locations in the feature map, thereby achieving targeted knowledge transfer.

[0004] Although existing knowledge distillation methods provide a feasible path for model compression in dense object detection, they still reveal many significant shortcomings in practical applications, as follows:

[0005] First, the uniform distillation strategy leads to the dilution of effective information and noise interference: Traditional dense detection distillation methods apply equally weighted supervisory signals to all locations in the feature map. In scenarios with severe foreground-background imbalance and uneven distribution of teacher response space, key target cues are diluted by background features. At the same time, background prediction noise from the teacher model is introduced, which interferes with the normal convergence of the student model and reduces the efficiency of feature learning.

[0006] Second, spatial weighting rules lack flexibility: existing methods rely on manually designed spatial weighting rules, the mathematical form of which is fixed once set. This static weighting method is difficult to adapt to complex and ever-changing detection scenarios, cannot be dynamically adjusted according to actual task requirements, and lacks adaptability to different scenarios.

[0007] Third, it ignores the dynamic evolution of teacher-student relationships during training: Existing methods repeatedly use a single form of static weighted template throughout the training cycle, failing to consider the dynamic changes in the ability differences between teacher and student models as the training progresses. In the early stages of training, the student model has not yet converged, and the weights generated based on its high-noise response or related indicators lack reliability; in the later stages of training, static rules cannot adjust the focus in a timely manner, making it difficult to focus on high-order information regions that are only mastered by the teacher, resulting in the knowledge transfer efficiency not reaching its optimal level throughout the entire training process.

[0008] Fourth, it is difficult to balance the distillation needs of different regions: the weighting strategies of existing methods mostly focus on the foreground region or high confidence region, lacking a dynamic adjustment mechanism, and cannot simultaneously balance the distillation needs of three key regions, namely the advantageous region of the teacher model, the disadvantageous region of the student model, and the region of potentially useful information in the background, resulting in insufficient targeting and comprehensiveness of the distillation process.

[0009] Therefore, there is an urgent need for a knowledge distillation method that can adapt to the characteristics of dense target detection tasks, break through the limitations of static weighted rules, dynamically perceive the evolution of teacher-student model relationships during training, and balance the teacher's advantageous regions, the student's disadvantageous regions, and potentially useful background information. This method can solve problems such as effective information dilution, noise interference, and insufficient knowledge transfer efficiency in traditional strategies, and provide technical support for the efficient deployment of lightweight models on resource-constrained platforms. Summary of the Invention

[0010] Therefore, this invention aims to address the problems of rigid weighting rules, insufficient adaptability during the training phase, and lack of teacher-student interaction trust assessment in existing knowledge distillation methods for dense target detection. Specifically, static weighting rules cannot dynamically adjust the supervision intensity, and the lack of dynamic modeling capabilities based on teacher-student consensus leads to inaccurate gradient allocation in dense feature maps, dilution of effective information, and amplification of teacher prediction noise. Consequently, these methods fail to meet the high accuracy and robustness requirements of resource-constrained platforms. This invention provides a general performance-enhancing distillation method for convolutional neural networks.

[0011] The general performance enhancement distillation method for convolutional neural networks specifically includes the following steps:

[0012] S1: Obtain the image data to be processed, input the image data into the pre-trained teacher model and the student model to be trained respectively, and output the teacher feature map and the student feature map respectively through the teacher model and the student model;

[0013] S2: Project and activate the teacher feature map and the student feature map according to the channel dimension to generate the teacher importance map and the student importance map respectively;

[0014] S3: Based on the teacher importance map and the student importance map, calculate the same spatial location. The consistency gain and coverage gain are used to focus on consensus regions where both teacher and student models determine that the confidence level is greater than a threshold; the coverage gain is used to capture advantageous regions where the teacher model determines that the confidence level is greater than a threshold but the student model has not learned them sufficiently.

[0015] S4: For the same spatial location in the teacher importance map and the student importance map Calculation and spatial location The relevant mixing coefficient;

[0016] S5: Based on the mixing coefficient, the consistency gain, and the coverage gain, the spatial location is calculated using a convex combination. Spatial consensus weights are used to generate spatial weight masks that can balance the reinforcement needs of the consensus region of the teacher and student models with the supplementary needs of the teacher's advantageous region.

[0017] S6: Construct a feature distillation loss term using the spatial weight mask, and obtain an optimized total loss function by combining it with a gating regularization term. Iterate and train the student model using the optimized total loss function to obtain a performance-enhanced student model. In the inference stage, perform dense target detection on the input image data using the performance-enhanced student model.

[0018] In one embodiment of the present invention, in S4, for the same spatial location of the teacher importance map and the student importance map... Calculation and spatial location Related mixing coefficients The method is as follows:

[0019] For the same spatial location in the teacher importance map and the student importance map Construct feature descriptors in the form of quadruples Its expression is:

[0020] ;

[0021] in, The teacher importance map indicates the spatial location. The value at that location, The student importance graph represents the spatial location. The value at that location, Indicates spatial location Consistency gain at the location;

[0022] The feature descriptor The input is a learnable pixel-level gated consensus module, which is implemented through a lightweight convolutional mapping network with shared parameters. For the feature descriptor Processing and outputting spatial location Related mixing coefficients ,as follows:

[0023] .

[0024] In one embodiment of the present invention, the lightweight convolutional mapping network The output is configured to be constant during the initial training phase, and the mixing coefficient is dynamically adjusted during training through backpropagation gradients. The output value.

[0025] In one embodiment of the invention, the consistency gain The calculation method is as follows:

[0026] ;

[0027] in, This indicates an element-wise multiplication operation.

[0028] In one embodiment of the present invention, in step S5, the spatial position is calculated using a convex combination. Spatial consensus weight The method is as follows:

[0029] ;

[0030] in, Indication of spatial location The relevant mixing coefficient, Indicates spatial location Consistency gain at the location, Indicates spatial location Coverage gain at that location.

[0031] In one embodiment of the invention, the coverage gain The method is as follows:

[0032] ;

[0033] in, Indicates the corresponding spatial location The importance value of teachers in the department, Indicates the corresponding spatial location The importance value of students in the department.

[0034] In one embodiment of the present invention, the total loss function The expression is as follows:

[0035] ;

[0036] in, This represents the original detection loss term of the student model. For the first Each feature layer has a hierarchical loss term.

[0037] In one embodiment of the present invention, the first Hierarchical loss term for each feature layer The calculation method is as follows:

[0038] For the Each feature layer utilizes teacher feature maps. Student Feature Map and the spatial location of this layer Corresponding spatial consensus weight Through spatial location set Globally standardized summation is used to construct the characteristic distillation loss term. The formula is as follows:

[0039] ;

[0040] in, and They represent the first Spatial location of teacher feature maps and student feature maps The feature vector at that location, express Norms are used to quantify the degree of difference in teacher and student characteristics at a given spatial location.

[0041] Introducing lightweight gating regularization terms The calculation formula is as follows:

[0042] ;

[0043] Based on the characteristic distillation loss term and the gated regularization term , obtained the Hierarchical loss term for each feature layer The calculation formula is as follows:

[0044] ;

[0045] in, Representation of feature layer The total number of spatial locations; Indicates the first Teacher feature map in spatial location Eigenvalues ​​at; Indicates the first Student feature map in spatial location Eigenvalues ​​at; Indication of spatial location The relevant mixing coefficient; Indicates the target average threshold; , This is a balance coefficient used to adjust the contribution weights of characteristic distillation loss and regularization loss.

[0046] In one embodiment of the present invention, both the teacher model and the student model are convolutional neural networks used for dense object detection tasks, and the total number of network parameters, the number of feature extraction channels, and the network depth of the teacher model are all greater than those of the student model.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the general performance enhancement distillation method for convolutional neural networks.

[0048] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0049] The proposed stage-adaptive dense target detection distillation method based on learnable consensus masks breaks through the limitations of traditional static weighted rules. By modeling feature distillation as a spatial weighted regression problem, it introduces a learnable location-related spatial trust function and a lightweight pixel-level gating mechanism, achieving dynamic perception and weight redistribution of teacher-student response consistency, coverage, and training evolution status. The constructed dynamic teacher-student consensus mechanism can adaptively match the needs of different training stages. In the early stage of training, it can suppress the interference of high-noise responses from student models, and in the later stage of training, it can focus on key subtle features mastered by teachers, effectively solving the problems of rigid weighted rules, training stage mismatch, and weak noise resistance. At the same time, this scheme achieves accurate gradient allocation in dense feature maps with severe imbalance between foreground and background, avoiding the dilution of effective information and the amplification of teacher prediction noise. Moreover, the relevant mechanisms in the training stage do not increase inference time overhead. Finally, it significantly improves the accuracy and generalization ability of lightweight student models in complex detection tasks on resource-constrained platforms, combining adaptability, accuracy, and deployment efficiency. Attached Figure Description

[0050] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0051] Figure 1 This is a flowchart illustrating a general performance-enhancing distillation method for convolutional neural networks provided in an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of a specific process for a general performance enhancement distillation method for convolutional neural networks provided in an embodiment of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0054] Example 1:

[0055] Reference Figure 1 and Figure 2As shown, this invention provides a general performance enhancement distillation method for convolutional neural networks (hereinafter referred to as the "LCMD method"). The core objective is to solve the problem of dynamic adaptation of spatial weights in knowledge distillation for dense object detection tasks. By constructing a learnable consensus mechanism and a stage-adaptive optimization strategy, it improves the detection accuracy and generalization ability of lightweight student models without increasing inference overhead. This method is applicable to various dense object detectors, including single-stage detectors such as RetinaNet and FCOS, and two-stage detectors such as Faster R-CNN. It can be deployed on platforms with limited computing resources, such as edge devices and mobile robots.

[0056] The LCMD method specifically includes the following steps:

[0057] S1: Obtain the image data to be processed, input the image data into the pre-trained teacher model and the student model to be trained respectively, and output the teacher feature map and the student feature map respectively through the teacher model and the student model;

[0058] S2: Project and activate the teacher feature map and the student feature map according to the channel dimension to generate the teacher importance map and the student importance map respectively;

[0059] S3: Based on the teacher importance map and the student importance map, calculate the same spatial location. The consistency gain and coverage gain are used to focus on consensus regions where both teacher and student models determine that the confidence level is greater than a threshold; the coverage gain is used to capture advantageous regions where the teacher model determines that the confidence level is greater than a threshold but the student model has not learned them sufficiently.

[0060] S4: For the same spatial location in the teacher importance map and the student importance map Calculation and spatial location The relevant mixing coefficient;

[0061] S5: Based on the mixing coefficient, the consistency gain, and the coverage gain, the spatial location is calculated through convex combination. Spatial consensus weights are used to generate spatial weight masks that can balance the reinforcement needs of the consensus region of the teacher and student models with the supplementary needs of the teacher's advantageous region.

[0062] S6: Construct a feature distillation loss term using the spatial weight mask, and obtain an optimized total loss function by combining it with a gating regularization term. Iterate and train the student model using the optimized total loss function to obtain a performance-enhanced student model. In the inference stage, perform dense target detection on the input image data using the performance-enhanced student model.

[0063] As can be seen from the above technical solution, the core advantage of the LCMD method of the present invention lies in breaking through the limitations of traditional static weighting rules. By constructing a learnable consensus mechanism and a stage-adaptive optimization strategy, the teacher and student models first output feature maps and generate importance maps. Then, dynamic spatial weight masks are generated through dual spatial gain calculation, learnable mixing coefficient solution, and convex combination weighting. Finally, the total loss function is constructed by combining feature distillation loss and gating regularization term to optimize the student model. This method can accurately adapt to the scene of foreground and background imbalance in dense target detection, dynamically balance the reinforcement of teacher-student consensus region and the supplementation of teacher advantage region, and achieve adaptive matching between distillation strategy and training stage.

[0064] Further, in step S1, after acquiring the image data to be processed, conventional image preprocessing operations such as denoising, image scaling, normalization, and data augmentation are performed. The preprocessed image data is then input into the pre-trained teacher model and the student model to be trained, respectively. The model's feature pyramid (FPN) outputs a feature map covering multi-scale targets, i.e., the teacher feature map. Student Feature Map , For batch size, H represents the number of channels, and H and W represent the spatial dimensions of the feature map.

[0065] Both the teacher model and the student model are convolutional neural networks used for dense object detection tasks. The total number of network parameters, the number of feature extraction channels, and the network depth of the teacher model are all greater than those of the student model, ensuring that the teacher model has stronger feature representation and object detection capabilities.

[0066] Furthermore, this invention first constructs a dual-space trust model based on complementary perspectives, aiming to conduct a preliminary quantitative evaluation of the dense feature maps output by the teacher model and the student model from two dimensions: feature response intensity consistency and feature response coverage, providing basic decision signals for the subsequent construction of a dynamic weighting mechanism.

[0067] Specifically, in step S2, the teacher feature map output in step S1... Student Feature Map Lightweight projection processing along the channel dimension is performed using 1×1 convolution kernels to map high-dimensional multi-channel features into single-channel feature response maps. Then, the Sigmoid activation function is used to normalize the feature response values ​​to the [0,1] interval, thus obtaining the teacher importance map. Importance of students diagram Its mathematical expression is as follows:

[0068] , ;

[0069] in, This represents the set of spatial locations of the feature map. and Representing spatial location The importance values ​​of teachers and students at each location are used to quantitatively represent the criticality of feature information at the corresponding spatial locations.

[0070] Furthermore, in step S3, based on the teacher importance graph generated in step S2... Importance of students diagram For the same spatial location We construct two complementary spatial gain evaluation dimensions—consistency gain and coverage gain—to comprehensively analyze the response relationship of teacher-student models in dense feature spaces.

[0071] (1) Consistency Gain: Used to focus on consensus regions where both teacher and student models determine the confidence level to be greater than a threshold, thereby strengthening the distillation supervision intensity of such regions. It is calculated as the element-wise product of the teacher's importance value and the student's importance value, as shown in the following expression:

[0072] ;

[0073] in, This represents the element-wise multiplication operation, when and When all are in the high value range, The larger the value, the more confident the spatial location is as a key feature region with high confidence in the consensus between the teacher and student models;

[0074] (2) Coverage Gain: Used to capture areas where the teacher model determines the confidence level to be greater than a threshold, but the student model has not learned them sufficiently, thus supplementing distilled supervision in these areas. It is calculated as the arithmetic mean of the teacher importance value and the student importance value, as shown in the following expression:

[0075] ;

[0076] This formula ensures that any model will have a high response value at this spatial location. This means maintaining a significant level, thereby ensuring that the teacher model's strengths receive sufficient distillation attention to compensate for the student model's learning weaknesses.

[0077] Furthermore, in step S4, regarding the teacher importance graph... Importance of students diagram same spatial location The mixing coefficients associated with that location are calculated using a learnable pixel-level gated consensus module. The specific steps are as follows:

[0078] Construct feature descriptors in the form of quadruples This descriptor comprehensively captures the local relationships between teacher and student characteristics, including teacher importance, student importance, their overlap (i.e., consistency gain), and their differences, as expressed below:

[0079] ;

[0080] in, Indicates spatial location The absolute difference between the importance of teachers and students at a given location is used to accurately reflect the cognitive differences in the teacher-student model at that location.

[0081] The feature descriptor The input is a learnable pixel-level gated consensus module, which consists of a lightweight convolutional mapping network with shared parameters. Composition. Through convolutional mapping networks For the feature descriptor After feature extraction and nonlinear transformation, and using the Sigmoid activation function, the output is related to the spatial location. Related mixing coefficients The expression is as follows:

[0082] ;

[0083] in, This is used to adjust the contribution ratio of consistency gain and coverage gain in subsequent weight calculations.

[0084] The lightweight convolutional mapping network In the early stages of training, the network is configured to output a nearly constant value, making the initial distillation behavior close to uniform weighting. During training, the network dynamically adjusts the mixing coefficients along the training trajectory using the backpropagated gradient signal. The output value enables adaptive matching of local feature patterns, ensuring the rationality of weight allocation in different training stages.

[0085] Further, in step S5, based on the mixing coefficient obtained in step S4... The consistency gain obtained in step S3 and coverage gain Spatial position is calculated through convex combination operations. Spatial consensus weight The expression is:

[0086] ;

[0087] By integrating all spatial locations Spatial consensus weight A spatial weight mask M is generated, which can dynamically balance the reinforcement needs of the consensus region of the teacher and student models with the supplementary needs of the teacher model's advantageous region, providing accurate spatial weight guidance for subsequent feature distillation.

[0088] Furthermore, in step S6, by constructing a stage-adaptive loss function system under regularization constraints, the dynamic adaptation of the distillation strategy to the training stage and the stable convergence of the model are achieved. The specific implementation is as follows:

[0089] For the Each feature layer utilizes the teacher feature map of that layer. Student Feature Map and spatial location Corresponding spatial consensus weight Through this feature layer spatial location set Globally standardized summation is used to construct the characteristic distillation loss term. The difference in teacher and student characteristics at this feature layer is quantified by the following expression:

[0090] ;

[0091] in, Indicates the first The total number of spatial locations in the layer feature map. and They represent the first Teacher and student feature maps in spatial location eigenvalues ​​at that location express Norms are used to quantify the degree of difference in teacher and student characteristics at a given spatial location.

[0092] Due to spatial consensus weight Mixing coefficient For learnable parameters, their gradient is derived from the feature distillation loss term. Backpropagation automatically updates the gradient, which is expressed as:

[0093] ;

[0094] The gradient formula shows that, and The difference adjusts the loss for the mixing coefficient. Sensitivity: When there are significant differences in teacher and student characteristics and At locations with larger values, the gradient signal is stronger, prompting the gating network to adaptively optimize the mixing coefficients. This enables precise allocation of spatial weights and improves the efficiency of knowledge transfer.

[0095] To avoid the mixing coefficients of the gated network output If all values ​​approach 0 or 1, the model degenerates into a single coverage pattern or consistency pattern. This invention introduces a lightweight gating regularization term. Through constraints The mean distribution forces the model to reasonably integrate consistency gain and coverage gain at different spatial locations. The expression is:

[0096] ;

[0097] in, For the first Layer feature map spatial location The mixing coefficient at the location, In this embodiment, the target average threshold value is represented. The value is 0.5. This regularization term effectively avoids extreme weight distribution and ensures that the model reasonably integrates the two gain modes in different spatial locations.

[0098] Based on the characteristic distillation loss term and the gated regularization term The weighted sum is used to obtain the first... Hierarchical loss term for each feature layer This achieves synergistic optimization of feature distillation and weight regularization, and its calculation formula is as follows:

[0099] ;

[0100] in, and This is a balancing coefficient used to adjust the contribution weights of characteristic distillation loss and regularization loss, ensuring the balance of the dual-objective optimization.

[0101] The original detection loss term of the student model Hierarchical loss terms of all feature layers By performing a joint summation, the total loss function is obtained. The expression is:

[0102] ;

[0103] in, The standard loss function for dense object detection tasks is the weighted sum of classification loss and regression loss. This ensures that the model can transfer higher-order knowledge from teachers without deviating from the core objective of the detection task, thus achieving a dual guarantee of knowledge transfer and task performance.

[0104] The total loss function is optimized using a stochastic gradient descent (SGD) optimizer. Minimization is performed on the student model parameters, pixel-level gating consensus module parameters, and mixing coefficients. Iterative updates are performed; the teacher model parameters are kept constant during training to ensure the unidirectionality and effectiveness of knowledge transfer.

[0105] After training, the enhanced student model maintains the original network structure during the inference phase. Mechanisms such as pixel-level gating consensus module, spatial weight calculation, and hybrid coefficient gradient optimization only participate in the operation during the training phase, without introducing any additional computing modules or time overhead, ensuring the real-time deployment requirements of the model on edge devices.

[0106] To fully verify the effectiveness, versatility, and robustness of the Dense Target Detection Distillation Method (LCMD) based on Learnable Consensus Mask proposed in this invention, multiple sets of comparative and ablation experiments were designed in this embodiment. The specific experimental setup and result analysis are as follows.

[0107] The experiment uses the COCO2017 large-scale dataset, a mainstream dataset in the field of object detection, as the benchmark test set. This dataset covers 80 object categories and includes 83,000 training images, 40,000 validation images, and 40,000 test images. The dataset covers a variety of real-world application scenarios, such as dense crowds, large scale variations, severe object occlusion, and complex lighting conditions, which can fully examine the generalization performance of the algorithm in scenarios with extremely imbalanced foreground / background categories and multi-scale object detection.

[0108] We implement and reproduce various typical dense object detectors on mainstream open-source detection frameworks, including single-stage detectors such as RetinaNet and two-stage detectors such as Faster R-CNN. We construct multiple teacher-student network pairs to verify the adaptability of the method. The teacher model adopts a detection architecture based on a 101-layer residual network (Res101), while the student model adopts a detection architecture based on a 50-layer residual network (Res50), ensuring that the network capacity of the teacher model is significantly higher than that of the student model.

[0109] The model training process adopted the community-standard "2×" training scheme (a total of 24 training rounds), and the training and performance evaluation of the model were completed in a multi-GPU computing environment. In the optimization phase, while retaining the original objective function of the detector, the LCMD distillation loss function was introduced for joint optimization with the student network. To ensure the uniqueness of experimental variables, the data augmentation strategy, optimizer parameters (initial learning rate 0.02, momentum 0.9, weight decay 0.0001), and learning rate scheduling strategy (decaying to 1 / 10 of the original learning rate in rounds 16 and 22, respectively) were kept consistent in all experiments.

[0110] The evaluation index system adopted by COCO official standards specifically includes overall average precision (AP), intersection-over-union ratio (IoU), and average precision corresponding to a threshold of 0.5. ), the average precision corresponding to an IoU threshold of 0.75 ( ), and average accuracy for small-scale targets ( ), average accuracy of mesoscale targets ( ), average accuracy of large-scale targets ( This allows for a comprehensive evaluation of the model's detection performance under different target sizes and matching thresholds.

[0111] Representative feature distillation and dense detection distillation methods, such as Foreground Focusing Distillation (FGD), Feature Response Similarity Distillation (FRS), and Gradient Guided Instance Aware Distillation (GID), were selected as baselines for comparison. All comparison methods maintained identical teacher-student network structures, training hyperparameters, data augmentation strategies, and optimizer configurations, differing only in distillation loss and spatial weight calculation methods to ensure fairness in the experimental comparison. The LCMD method of this invention dynamically adjusts the distillation spatial weights by introducing a learnable spatial trust function and a pixel-level gating module, without altering the student model's forward inference structure, thus ensuring that the computational cost during the model's inference phase remains consistent with the original student model.

[0112] Table 1 Comparative experimental results of various distillation methods under the RetinaNet architecture

[0113]

[0114] As shown in Table 1, the experimental data demonstrates that the proposed LCMD method achieves an overall average precision (AP) of 0.3980, a significant improvement of 0.024 compared to the student network baseline model. This improvement is significantly better than mainstream comparative distillation methods such as FGD (+0.022), FRS (+0.019), and GID (+0.017). Furthermore, the LCMD method achieves the best performance with an AP75 of 0.4260 at an IoU threshold of 0.75, fully validating its superior target detection accuracy in high-matching-threshold scenarios and highlighting the technical advantages of the dynamic spatial weight allocation mechanism in complex feature distillation tasks.

[0115] Table 2 Comparative experimental results of various distillation methods under the Faster R-CNN architecture

[0116]

[0117] As shown in Table 2, the experimental results of the present invention demonstrate that the LCMD method achieves an AP value of 0.406 under the Faster R-CNN architecture, which is 0.022 higher than the benchmark and higher than the comparison methods such as FRS and GID. Furthermore, the AP75 index reaches 0.458, which shows the advantage of learning fine features of the target and verifies the universality and effectiveness of the present method under different detection architectures.

[0118] To verify the function of the core module of this invention, two sets of ablation experiments were designed to explore the effectiveness of the gating mechanism and the characteristic pyramid hierarchical distillation.

[0119] Table 3 Comparison of ablation experimental performance of gated mechanisms

[0120]

[0121] As can be seen from the experimental results in Table 3, the importance of teachers and students (…) , When the model's overall average precision (AP) is used as input, it reaches a minimum of 0.3910. Introducing either the "consistency intersection" feature or the "absolute difference" feature into the input improves the AP to 0.3930, showing a slight performance improvement. However, when using the gating mechanism with the complete four-dimensional feature descriptor (including teacher importance, student importance, consistency gain, and dissimilarity) proposed in this invention, the AP further increases to 0.3980, achieving optimal performance. These experimental results fully verify that dissimilarity and overlap features play complementary and irreplaceable roles in the distillation strategy decision-making of gating networks, demonstrating that the complete gating mechanism can achieve more accurate spatial weight allocation, thereby effectively improving knowledge transfer efficiency and model detection performance.

[0122] Table 4 Comparison of experimental performance of characteristic pyramid-shaped distillation ablation

[0123]

[0124] The experimental results in Table 4 show that the detection accuracy of the student model increases progressively with the increase in the number of feature pyramid levels involved in distillation. When distilling only the bottom-level features (P2), the AP value only increases by 0.003; while the distillation strategy covering all scale levels from P2 to P6 increases the AP value by 0.024 compared to the baseline, demonstrating that full-level distillation can fully utilize the multi-scale semantic information of the teacher model and effectively improve the model's ability to detect targets at different scales.

[0125] Comparative and ablation experiments show that the LCMD method of this invention achieves dynamic adaptive adjustment of distillation space weights through a learnable spatial trust function, pixel-level gating module, and full-level distillation strategy. It exhibits optimal performance under different detection architectures such as RetinaNet and Faster R-CNN, significantly improving the detection accuracy and generalization ability of lightweight student models without increasing computational overhead in the inference stage. It is suitable for dense target detection tasks on resource-constrained platforms.

[0126] Example 2:

[0127] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the general performance enhancement distillation method for convolutional neural networks described in Embodiment 1.

[0128] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A general performance-enhancing distillation method for convolutional neural networks, characterized in that, Includes the following steps: S1: Obtain the image data to be processed, input the image data into the pre-trained teacher model and the student model to be trained respectively, and output the teacher feature map and the student feature map respectively through the teacher model and the student model; S2: Project and activate the teacher feature map and the student feature map according to the channel dimension to generate the teacher importance map and the student importance map respectively; S3: Based on the teacher importance map and the student importance map, calculate the same spatial location. The consistency gain and coverage gain are used to focus on consensus regions where both teacher and student models determine that the confidence level is greater than a threshold; the coverage gain is used to capture advantageous regions where the teacher model determines that the confidence level is greater than a threshold but the student model has not learned them sufficiently. S4: For the same spatial location in the teacher importance map and the student importance map Calculation and spatial location The relevant mixing coefficient; S5: Based on the mixing coefficient, the consistency gain, and the coverage gain, the spatial location is calculated using a convex combination. Spatial consensus weights are used to generate spatial weight masks that can balance the reinforcement needs of the consensus region of the teacher and student models with the supplementary needs of the teacher's advantageous region. S6: Construct a feature distillation loss term using the spatial weight mask, and obtain an optimized total loss function by combining it with a gating regularization term. Iterate and train the student model using the optimized total loss function to obtain a performance-enhanced student model. In the inference stage, perform dense target detection on the input image data using the performance-enhanced student model. In step S4, for the same spatial location of the teacher importance map and the student importance map... Calculation and spatial location Related mixing coefficients The method is as follows: For the same spatial location in the teacher importance map and the student importance map Construct feature descriptors in the form of quadruples Its expression is: ； in, The teacher importance map indicates the spatial location. The value at that location, The student importance graph represents the spatial location. The value at that location, Indicates spatial location Consistency gain at the location; The feature descriptor The input is a learnable pixel-level gated consensus module, which is implemented through a lightweight convolutional mapping network with shared parameters. For the feature descriptor Processing and outputting spatial location Related mixing coefficients ,as follows: 。 2. The general performance enhancement distillation method for convolutional neural networks according to claim 1, characterized in that: The lightweight convolutional mapping network The output is configured to be constant during the initial training phase, and the mixing coefficient is dynamically adjusted during training through backpropagation gradients. The output value.

3. The general performance enhancement distillation method for convolutional neural networks according to claim 1, characterized in that: The consistency gain The calculation method is as follows: ； in, This indicates an element-wise multiplication operation.

4. The general performance enhancement distillation method for convolutional neural networks according to claim 1, characterized in that: In S5, the spatial position is calculated using a convex combination. Spatial consensus weight The method is as follows: ； in, Indication of spatial location The relevant mixing coefficient, Indicates spatial location Consistency gain at the location, Indicates spatial location Coverage gain at that location.

5. The general performance enhancement distillation method for convolutional neural networks according to claim 4, characterized in that: The coverage gain The method is as follows: ； in, Indicates the corresponding spatial location The importance value of teachers in the department, Indicates the corresponding spatial location The importance value of students in the department.

6. The general performance enhancement distillation method for convolutional neural networks according to claim 1, characterized in that: The total loss function The expression is as follows: ； in, This represents the original detection loss term of the student model. For the first Each feature layer has a hierarchical loss term.

7. The general performance enhancement distillation method for convolutional neural networks according to claim 6, characterized in that: The first Hierarchical loss term for each feature layer The calculation method is as follows: For the Each feature layer utilizes teacher feature maps. Student Feature Map and the spatial location of this layer Corresponding spatial consensus weight Through spatial location set Globally standardized summation is used to construct the characteristic distillation loss term. The formula is as follows: ； in, and They represent the first Spatial location of teacher feature maps and student feature maps The feature vector at that location, express Norms are used to quantify the degree of difference in teacher and student characteristics at a given spatial location. Introducing lightweight gating regularization terms The calculation formula is as follows: ； Based on the characteristic distillation loss term and the gated regularization term , obtained the Hierarchical loss term for each feature layer The calculation formula is as follows: ； in, Representation of feature layer The total number of spatial locations; Indicates the first Teacher feature map in spatial location Eigenvalues ​​at; Indicates the first Student feature map in spatial location Eigenvalues ​​at; Indication of spatial location The relevant mixing coefficient; Indicates the target average threshold; , This is a balance coefficient used to adjust the contribution weights of characteristic distillation loss and regularization loss.

8. The general performance-enhancing distillation method for convolutional neural networks according to claim 1, characterized in that, Both the teacher model and the student model are convolutional neural networks used for dense object detection tasks, and the total number of network parameters, the number of feature extraction channels, and the network depth of the teacher model are all greater than those of the student model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the general performance-enhancing distillation method for convolutional neural networks as described in any one of claims 1 to 8.

Images