Lightweight knowledge distillation method and system for feature redundancy suppression of transformer substation illegal behavior detection
By constructing a teacher-student network and screening high-value features, the problems of limited computing power and background interference in substation monitoring equipment were solved, and efficient, low-latency real-time detection of a lightweight knowledge distillation method was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- STATE GRID LIAONING ELECTRIC POWER CO LTD
- Filing Date
- 2025-12-08
- Publication Date
- 2026-04-24
AI Technical Summary
The edge computing power of substation monitoring equipment is limited, the video background is complex and highly interfered with, existing models are difficult to run in real time, and existing knowledge distillation methods cannot effectively filter redundant features, resulting in a decline in detection performance.
A teacher-student network is constructed, high-value features are selected through hierarchical mapping and channel contribution evaluation, redundant channels are shielded by gating mechanism, and a decoupled joint loss function is constructed for training to generate a lightweight student model.
Significantly reduces model complexity and training overhead, enabling high-precision, low-latency real-time detection on edge devices, adapting to the rapid iteration needs of substations.
Smart Images
Figure CN121920462A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and deep learning technology, specifically relating to a lightweight knowledge distillation method and system for feature redundancy suppression in substation violation detection. Background Technology
[0002] With the rapid application of artificial intelligence technology in the industrial field, deep neural networks have made significant progress in tasks such as object detection, behavior recognition, and video understanding. Mainstream high-performance models (such as Transformer, ResNet, YOLO, etc.) typically contain a large number of convolutional layers and self-attention structures, with parameter scales often reaching tens of millions to hundreds of millions. They have high computational demands, high GPU memory consumption, and obvious dependence on high-performance hardware such as GPUs.
[0003] In substation intelligent video monitoring scenarios, front-end data acquisition and real-time analysis primarily rely on resource-constrained devices such as camera-side AI modules, portable inspection terminals, drone gimbals, and edge computing boxes. These devices generally suffer from limited computing power, power sensitivity, storage limitations, and high real-time inference requirements, making it difficult to directly deploy large-scale deep learning models and easily leading to problems such as high detection latency, high energy consumption, and system instability. Therefore, model compression and acceleration methods have become an important direction for improving edge deployment efficiency.
[0004] Existing model compression techniques mainly include pruning, quantization, Neural Architecture Search (NAS), and Knowledge Distillation (KD). However, all of these techniques have significant limitations in substation monitoring scenarios, including the following drawbacks: Pruning: This achieves sparsity by removing unimportant weights, channels, or layers from the neural network. While it reduces the number of parameters, it makes it difficult to accurately assess parameter importance. Aggressive pruning can easily damage the model structure and usually requires multiple fine-tuning steps, increasing training costs. Quantization: Quantization methods reduce computational and storage overhead by lowering the precision of FP32 to lower precision forms such as INT8. However, in substation scenarios, there is strong reflection from metal equipment and complex background textures, leading to a greater accumulation of errors after quantization. The accuracy of numerically sensitive layers (such as attention modules or localization regression heads) decreases significantly. Furthermore, the quantization support capabilities of different edge device hardware vary greatly, making unified deployment difficult. Therefore, quantization methods are not easily able to achieve stable operation in complex industrial video scenarios. Neural Architecture Search (NAS): NAS improves efficiency by automatically searching for network structures, but it suffers from the following problems in engineering applications: the search process itself requires a large amount of GPU resources, resulting in high costs; search results are prone to overfitting to specific tasks and datasets, and have limited generalization performance in unstructured contexts such as substations; the overall development cycle is long, making it unsuitable for the rapid deployment and iteration requirements of substation monitoring systems. Therefore, NAS has insufficient engineering feasibility in real-world industrial scenarios. Knowledge Distillation (KD): Based on the teacher-student framework, it guides student model learning through a teacher model. Classical methods only utilize soft labels in the output layer to transfer knowledge, ignoring the rich hierarchical information contained in the intermediate layers of the teacher. Although subsequent research has introduced intermediate layer feature distillation, the following key problems still exist: it cannot filter redundant features, and noise is directly transferred; full feature distillation is computationally expensive and unsuitable for engineering needs; many distillation methods introduce new modules that cannot be separated during the inference stage.
[0005] In substation monitoring scenarios, video footage is characterized by features such as glare from metallic equipment, complex background textures, significant day-night lighting variations, and interference from wind-blown vegetation and construction backgrounds. This results in a large number of invalid or spurious features in the teacher network characteristics that are "not related to safety behaviors." When these redundant features are directly distilled into the student model, they interfere with the convergence of the lightweight model, increase the computational load of distillation, cause false alarms and missed detections, and lead to a performance degradation of the student model when deployed on edge devices.
[0006] Therefore, there is an urgent need for a distillation method that can "screen teacher characteristics and transfer only key knowledge" based on the actual characteristics of substation screens. Summary of the Invention
[0007] Therefore, the purpose of this invention is to provide a lightweight knowledge distillation method and system for feature redundancy suppression in substation violation detection, in order to solve problems such as limited edge computing power of substation monitoring equipment, complex and strong interference from video backgrounds, and difficulty in real-time model operation.
[0008] The technical solution provided by this invention is as follows: Firstly, this invention provides a lightweight knowledge distillation method for feature redundancy suppression in substation violation detection, comprising:
[0009] To address the task of detecting violations in substation monitoring videos, a teacher network and a student network are constructed. A hierarchical mapping relationship is established between the teacher network and the student network to align the key intermediate layer features of the teacher network with the corresponding layers of the student network.
[0010] The value of each channel in the output feature tensor of each layer of the teacher network is evaluated to obtain the comprehensive contribution score of each channel.
[0011] Based on the comprehensive contribution score of the channels, the top K% of channels are selected as the "high-value feature subset". A gating mechanism is used to dynamically shield redundant channels to obtain the selected teacher features.
[0012] A decoupled joint loss function is constructed to train the parameters of the student network end-to-end, resulting in a trained student model. The joint loss function includes task loss, feature distillation loss, and Logits distillation loss.
[0013] The feature redundancy detection module and gating module are removed from the trained student model to obtain a lightweight student model.
[0014] Lightweight student models are deployed to substation edge devices to enable real-time detection of violations.
[0015] Preferably, each channel in the output feature tensor of each layer of the teacher network includes: features highly correlated with the violation, and irrelevant or interfering redundant features;
[0016] The value of each channel in the output feature tensor of each layer of the teacher network is evaluated, including:
[0017] The feature tensor output by a certain intermediate layer of the teacher network is Where C is the number of channels, and H and W are the height and width;
[0018] For each channel c∈{1,...,C} of the feature tensor, two core contribution metrics are computed in parallel: information entropy. Gradient response intensity ;
[0019] The two indicators mentioned above are weighted and combined to obtain the overall contribution score for each channel. :
[0020]
[0021] in, This indicates a normalization operation. and This is a hyperparameter.
[0022] Preferably, the information entropy The entropy of all pixel values in the channel feature map is calculated using the following formula:
[0023]
[0024] in These are the normalized pixel values;
[0025] The gradient response intensity This means that in one iteration of training, the final task loss function is calculated for the channel features. The mean absolute value of the gradient can be calculated as follows:
[0026]
[0027] This indicates the detection loss.
[0028] Preferably, based on the comprehensive contribution score of the channels, the top K% of channels are selected as a "high-value feature subset," and a gating mechanism is used to dynamically shield redundant channels to obtain the selected teacher features, including:
[0029] Based on contribution score All feature channels are sorted in descending order, and the top K% of high-value feature channels are selected as effective feature channels, while the rest are considered redundant channels.
[0030] A binary mask M, consistent with the number of channels, is generated through a gating mechanism, marking high-value channels as 1 and redundant channels as 0;
[0031] During training, the mask M is multiplied channel-by-channel with the original teacher feature tensor F to mask redundant channels, thus obtaining the filtered teacher features. .
[0032] Preferably, the construction of the decoupled joint loss function :
[0033]
[0034] Losses incurred from detecting violations;
[0035] The selected teacher and student characteristics are used to construct the feature distillation loss.
[0036] The classic objective logic distillation loss is used to calculate the KL divergence of the final outputs of the teacher and student networks using a Softmax function with a temperature coefficient T, aiming to transfer the "soft-label knowledge" of the teacher network.
[0037] and This represents the hyperparameter used to balance the weights of various losses.
[0038] Preferably, the high-value characteristics of the teacher network after gating are selected. Features output by the corresponding layer of the student network The difference between them is used as the characteristic distillation loss. Among them, the features output by the corresponding layer of the student network After mask selection;
[0039] The calculation is performed using the L2 norm:
[0040]
[0041] in, This is the number of channels selected.
[0042] Secondly, this invention provides a lightweight knowledge distillation system for feature redundancy suppression in substation violation detection, comprising:
[0043] The teacher network module is used for monitoring video feature collection and behavior classification;
[0044] The student network module is deployed on edge devices to perform real-time detection;
[0045] The feature redundancy detection module is used to calculate entropy and gradient contribution.
[0046] The channel selection gating module is used to shield redundant features;
[0047] The distillation optimization module performs three joint loss optimizations.
[0048] The lightweight deployment module is used, and the auxiliary modules are removed after training to generate the final deployment model.
[0049] This invention provides a lightweight knowledge distillation method and system for feature redundancy suppression in substation violation detection. By constructing a "teacher-student" knowledge distillation framework and introducing a feature redundancy suppression mechanism, the method selects truly useful feature information for violation detection from the teacher network and selectively passes it to the student network (lightweight model). This significantly reduces model complexity and training overhead while ensuring detection accuracy, ultimately achieving efficient, low-latency, and zero-additional-overhead real-time deployment on edge devices.
[0050] This invention addresses the intelligent video monitoring scenario in substations by proposing a lightweight knowledge distillation method and system based on feature redundancy suppression. This method jointly evaluates the information entropy and gradient contribution of the teacher network feature channels and implements dynamic channel filtering during the training phase. This ensures that the student model acquires only high-value features related to violations, fundamentally reducing interference from background noise and irrelevant information.
[0051] This invention not only significantly reduces the computational burden of distillation training but also achieves complete decoupling between the training and inference phases, enabling lightweight student models to achieve high-precision, low-latency real-time violation detection on substation edge devices. This technology possesses extremely high engineering deployability, adaptability, and promotional value, and can be widely applied to various edge intelligence scenarios such as power line inspection, industrial safety monitoring, and video analytics. Attached Figure Description
[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a block diagram of the lightweight knowledge distillation system for feature redundancy suppression in substation violation detection provided by the present invention.
[0055] Figure 2 This is a schematic diagram of the feature redundancy detection module described in this invention;
[0056] Figure 3 This is a schematic diagram of the working principle of the channel gating module described in this invention;
[0057] Figure 4 This is a structural comparison diagram of the training phase and the inference phase described in this invention. Detailed Implementation
[0058] The present invention will be further explained below with reference to specific implementation schemes, but this explanation does not limit the scope of the invention.
[0059] In existing technologies, the following key technical problems are encountered in the actual deployment of intelligent video monitoring systems for substations:
[0060] 1) The computing power of edge devices in substations is limited, and existing detection models cannot run in real time. On edge computing devices such as smart cameras, portable inspection terminals, and drone inspection equipment at the substation site, the deep model cannot meet the real-time detection requirements due to its large size and high computational load.
[0061] 2) The substation footage contains a large number of interfering backgrounds, resulting in serious feature redundancy: The videos of violations may contain complex factors such as changes in lighting, background noise, and interference from the structure of power equipment. The large number of redundant features generated by the teacher model will interfere with the student model, making it difficult for the model to learn high-value information that is truly related to the violation.
[0062] 3) Distillation training involves large computational load and long iteration cycle of engineering projects: The existing distillation performs full feature alignment on all channels of the teacher, which is computationally intensive and difficult to adapt to the rapid iteration requirements of engineering systems.
[0063] To address common practical engineering problems in substation intelligent monitoring scenarios, such as insufficient computing power of edge devices, strong background interference in monitoring images, severe redundancy of network features, and difficulty in lightweight models learning key features, a lightweight knowledge distillation method and system for substation intelligent video monitoring is proposed. This method can be implemented by a program in a computer-readable storage medium and is applicable to resource-constrained devices such as intelligent cameras, drone inspection terminals, and edge computing modules, providing high-performance, low-latency deployment capabilities for substation violation detection.
[0064] First, this implementation plan proposes a lightweight knowledge distillation method based on feature redundancy suppression, including:
[0065] S1: Teacher-Student Network Construction and Hierarchical Mapping: Construct a “high-low combination” neural network, namely a high-precision but complex teacher network and a lightweight but unoptimized student network, and establish the hierarchical alignment relationship between them to lay the structural foundation for subsequent feature distillation.
[0066] Specifically, it includes:
[0067] Teacher Network: This uses currently popular high-performance deep learning models, such as the YOLO series (YOLOv5, YOLOv8), ResNet, and Swin Transformer. These models have powerful feature extraction and behavior recognition capabilities, and can learn high-quality "semantic features" and "behavioral patterns" with rich labeled data. However, they have a large number of parameters and high computational complexity, making them unsuitable for direct deployment on edge devices.
[0068] Student Network: Lightweight models are used, such as MobileNet-YOLO, ShuffleNet-YOLO, or other small CNNs or lightweight Transformer architectures. The goal is to deploy these in substation field devices with limited computing power, sensitive power consumption, and requiring real-time response, such as cameras, drones, and edge computing boxes; therefore, they must be small and fast.
[0069] Layer Alignment: Teacher and student networks typically have different network depths and channel structures, and not all layers in the teacher network are relevant to the student network. It's necessary to analyze the structures of both networks and connect the semantically rich key layers in the teacher network with corresponding layers in the student network in terms of size and semantic hierarchy. Therefore, it's crucial to establish a correspondence between intermediate layers (e.g., the output of the teacher's third residual block corresponds to the output of the student's second convolutional layer). This mapping ensures that subsequent knowledge distillation can be performed at the same semantic level; that is, the teacher and student transfer feature information between layers of similar depth, rather than arbitrarily aligning across layers.
[0070] The teacher network, as a "knowledge source," provides high-quality feature representations and behavioral judgment capabilities; the student network, as a "learner," aims to achieve lightweight and low latency while maintaining high detection performance; hierarchical mapping ensures that knowledge transfer occurs between "semantically equivalent" levels, avoiding information misalignment or misleading.
[0071] S2: Dynamic evaluation of feature channel contribution. During training, the value of each channel in the output feature tensor F of each layer of the teacher network in the substation monitoring scenario is evaluated to identify which channels carry information highly relevant to violation detection and which are redundant or interfering background information. The channels include:
[0072] Features that are highly correlated with violations (such as human body outlines, motion textures, helmet areas, hazard area markings, etc.);
[0073] Irrelevant or interfering redundant features (such as background texture of high-voltage equipment, reflected light, motion noise caused by wind blowing on animals, etc.).
[0074] Specifically, this includes: S21: The feature tensor output by a certain intermediate layer of the teacher network is... Where C is the number of channels, and H and W are the height and width, respectively. To distinguish between high-value and redundant channels, for each channel c∈{1,...,C} of this feature tensor, two core contribution metrics are calculated in parallel:
[0075] S22: Information Entropy Calculate the entropy of all pixel values in the feature map of this channel. A higher entropy value indicates that the channel contains richer and more uncertain information, potentially including key details of the target or a complex background; a lower entropy value indicates that the channel contains only simple information, possibly a smooth background region. Its calculation can be expressed as:
[0076]
[0077] in These are the normalized pixel values.
[0078] S23: Gradient Response Strength In one iteration of training, the final task loss function (such as detection loss) is calculated. ) characteristics of this channel The mean absolute value of the gradient. A larger gradient indicates a greater impact of small changes in that channel on the final task result, meaning a higher correlation between that channel and the violation detection task. Its calculation can be expressed as:
[0079] .
[0080] The two indicators mentioned above are weighted and combined to obtain the overall contribution score for each channel. :
[0081]
[0082] in, This indicates a normalization operation. and This is a hyperparameter used to balance the weight of information content and task relevance in the selection criteria. Through dynamic evaluation, the importance of each channel can be adaptively adjusted according to different input images.
[0083] By quantifying the "information value" and "task contribution" of each channel, a refined evaluation of the teacher's network characteristic channels can be achieved; the truly useful "high-value channels" and the interfering "redundant channels" can be distinguished, providing a basis for subsequent channel selection; and the problem of "grabbing everything indiscriminately" and transferring background noise in the traditional distillation process can be avoided.
[0084] Traditional distillation methods perform indiscriminate transfer of all intermediate layer channels in the teacher network, which contains a large number of redundant features such as background textures, lighting noise, and structural interference that are irrelevant to the target task. These redundant channels introduce noise interference to the student model and may even cause negative transfer, reducing training effectiveness. This invention evaluates the value of teacher feature channels using both information entropy and gradient contribution metrics, and automatically suppresses redundant channels during the training phase, ensuring that the student model learns only effective features relevant to violation detection, thereby significantly improving its training stability and final detection performance.
[0085] S3: High-value feature channel selection and gating: Based on the channel contribution scores obtained in step S2, the top K% of the most important channels are selected as the "high-value feature subset". A gating mechanism is designed to dynamically block redundant channels during training, allowing the student network to learn only the truly useful features.
[0086] S31: For all C channels of the teacher network feature map, score them according to their contribution. Sort in descending order.
[0087] The top K% (e.g., the top 30%~50%) of channels are selected as effective feature channels, and the rest are considered redundant channels. A learnable gating module is introduced, which generates a binary mask M (0 or 1) for each channel: 1 indicates that the channel is a high-value channel, allowing features to pass and participate in distillation; 0 indicates that the channel is a redundant channel, which is masked (i.e., the feature value is multiplied by 0 and no longer participates in subsequent calculations).
[0088] S32: During training, the original teacher feature tensor F is multiplied channel by channel with the mask M to obtain the filtered teacher features. In this way, only high-value features will be used to guide student networks in the future.
[0089] The gating module is only enabled during the training phase and is completely removed during the inference phase, without adding any computational or parameter burden.
[0090] Through the above gating mechanism, dynamic denoising and focusing of teacher features are achieved during the training process, and the student network only learns the truly important visual cues, avoiding interference from complex backgrounds, lighting changes, etc.
[0091] The gating module is highly flexible and can automatically learn which channels are more important, exhibiting a degree of adaptability; there is no additional overhead during inference, achieving true "zero-cost deployment".
[0092] Existing feature distillation methods perform feature alignment on all channels, requiring extensive tensor computation and gradient backpropagation, resulting in long training times and high memory consumption. This invention performs distillation loss calculation only on selected high-value feature channels, significantly reducing computational load and memory requirements. The overall training overhead is significantly reduced, making lightweight model training more efficient and suitable for rapid iterative development in engineering projects.
[0093] S4: Construct a decoupled joint loss function; design a joint optimization objective that comprehensively considers task loss, feature distillation loss, and Logits distillation loss, so that the student network can accurately complete the violation detection task and learn high-quality feature representations from the teacher network.
[0094] Specifically: High-value characteristics of teacher networks after being screened by the gating module. Features output by the corresponding layer of the student network The difference between (also masked) is used as the characteristic distillation loss. The L2 norm (mean squared error) is typically used for calculation.
[0095]
[0096] in, This is the number of channels selected.
[0097] Construct the total loss function for joint optimization It consists of three parts:
[0098]
[0099] Losses incurred from detecting violations.
[0100] The selected teacher and student features are used to construct a feature distillation loss.
[0101] The classic objective logits distillation loss uses a softmax function with a temperature coefficient T to calculate the KL divergence of the final outputs of the teacher and student networks, aiming to transfer the "soft-label knowledge" of the teacher network. and It is a hyperparameter used to balance the weights of various losses.
[0102] The joint loss function is used to train the parameters of the student network end-to-end. Through multi-task joint optimization, the student network not only learns "what" (task level), but also "how to see" (feature level) and "how to judge" (output level); significantly improving the generalization ability and detection accuracy of the lightweight model, while maintaining the consistency and stability of the training objectives.
[0103] S5: Training-Inference Decoupling and Lightweight Deployment: After the model training is completed, the auxiliary modules used in the training phase (such as feature redundancy detection module and gating module) are completely stripped away, and only the core lightweight student network structure is retained to ensure that there is no additional computational burden when inference on edge devices.
[0104] Specific operations:
[0105] During the training phase, such as Figure 4 As shown in the left half, all modules (feature redundancy evaluation, gating, joint loss) participate in the training, guiding the student's network parameter updates. After training, as shown... Figure 4 As shown in the right half, the gating module in S3 and the redundant evaluation module in S2 are removed from the student network; the final exported model is only a standard lightweight network (such as MobileNet-YOLO), whose structure is completely consistent with the ordinary inference model, without increasing any number of parameters, computation, or memory overhead; it can be directly deployed to various edge devices in substations, such as smart cameras, inspection drones, and edge AI boxes, to achieve real-time, low-latency detection of violations.
[0106] This invention truly achieves "complex training and simple inference." During training, it uses auxiliary modules to improve model quality, and during inference, it operates with minimal overhead, meeting industrial-grade real-time requirements. It is compatible with existing deployment frameworks, requiring no additional adaptation or optimization, and has extremely high engineering applicability.
[0107] Furthermore, this invention also provides a lightweight neural network knowledge distillation system based on feature redundancy suppression. The system includes: a teacher network module (for monitoring video feature acquisition and behavior classification); a student network module (deployed on edge devices to perform real-time detection); a feature redundancy detection module (for calculating entropy and gradient contribution); a channel selection gating module (for masking redundant features); a distillation optimization module (performing three-term joint loss optimization); and a lightweight deployment module (stripping the auxiliary module after training to generate the final deployment model).
[0108] Some improved distillation methods introduce additional modules during training, but these modules remain coupled to the main network during inference, leading to increased computation and hindering deployment on edge devices. The redundant detection and gating modules of this invention are used only during training and are completely decoupled after training. The resulting student model maintains the same lightweight structure as the original, without adding any extra parameters, memory, or computation. Therefore, real-time inference can be achieved on edge platforms such as substation smart cameras, portable inspection terminals, inspection robots, and drone gimbals.
[0109] The feature redundancy suppression mechanism of this invention is modularly designed and can be directly inserted into existing teacher-student frameworks without modifying the network structure. It is applicable to various tasks such as object detection, image classification, and behavior recognition, and is compatible with different architectures such as YOLO, CNN, and Transformer, thus having broad application potential.
[0110] Example 1:
[0111] A lightweight knowledge distillation method for substation safety helmet wearing detection:
[0112] S1: Building Teacher and Student Networks
[0113] In this embodiment, the teacher network uses a YOLOv5 model (pre-trained on the COCO dataset) with an input size of 640×640 to provide high-quality features. The student network uses MobileNetV2 as its backbone, coupled with a lightweight YOLO detection head. To achieve hierarchical knowledge alignment, the 20×20 feature layer of the teacher network backbone is aligned with the output of the 7th inverse residual module in the student network MobileNetV2; the 40×40 feature layer of the teacher network neck is aligned with the corresponding upsampled layer features of the student network.
[0114] S2: Evaluate the contribution of feature channels: for the feature tensors of the teacher network output Calculate for each channel:
[0115] (1) Information entropy
[0116]
[0117] in This represents the normalized pixel probability.
[0118] (2) Gradient response intensity
[0119]
[0120] (3) Overall contribution
[0121] After normalization, the fusion yields:
[0122]
[0123] S3: High-Value Channel Screening and Gating: Based on Sort the channels and select the top 40% as high-value channels. Generate a binary mask M, with M=1 for high-value channels and M=0 for redundant channels.
[0124] Filtered features:
[0125]
[0126] The gating module and contribution evaluation module are only enabled during the training phase; they are completely removed during the inference phase.
[0127] S4: Joint Loss Function Optimization:
[0128] Total loss function:
[0129]
[0130] in:
[0131] Student Network Object Detection Loss
[0132] MSE between teacher and student characteristics after screening
[0133] KL divergence distillation loss at temperature T=3
[0134] This embodiment adopts .
[0135] The Adam optimizer was used for training, with an initial learning rate of 1e−3, a batch size of 16, and a total of 100 epochs.
[0136] S5: Model Deployment: The exported model after training only contains the student network ontology and does not include any modules used in the training phase. The final student model is converted to TensorRT format and deployed to a Jetson development board at the substation edge to achieve real-time safety helmet detection.
[0137] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0138] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A lightweight knowledge distillation method for feature redundancy suppression in substation violation detection, characterized in that, include: To address the task of detecting violations in substation monitoring videos, a teacher network and a student network will be constructed. Establish a hierarchical mapping relationship between the teacher network and the student network, so that the key intermediate layer features of the teacher network are aligned with the corresponding layers of the student network; The value of each channel in the output feature tensor of each layer of the teacher network is evaluated to obtain the comprehensive contribution score of each channel. Based on the comprehensive contribution score of the channels, the top K% of channels are selected as the "high-value feature subset". A gating mechanism is used to dynamically shield redundant channels to obtain the selected teacher features. A decoupled joint loss function is constructed to train the parameters of the student network end-to-end, resulting in a trained student model. The joint loss function includes task loss, feature distillation loss, and Logits distillation loss. The feature redundancy detection module and gating module are removed from the trained student model to obtain a lightweight student model. Lightweight student models are deployed to substation edge devices to enable real-time detection of violations.
2. The lightweight knowledge distillation method for feature redundancy suppression in substation violation detection according to claim 1, characterized in that, Each channel in the output feature tensor of each layer of the teacher network contains: features highly correlated with violations, and irrelevant or interfering redundant features; The value of each channel in the output feature tensor of each layer of the teacher network is evaluated, including: The feature tensor output by a certain intermediate layer of the teacher network is Where C is the number of channels, and H and W are the height and width; For each channel c∈{1,...,C} of the feature tensor, two core contribution metrics are computed in parallel: information entropy. Gradient response intensity ; The two indicators mentioned above are weighted and combined to obtain the overall contribution score for each channel. : in, This indicates a normalization operation. and This is a hyperparameter.
3. The lightweight knowledge distillation method for feature redundancy suppression in substation violation detection according to claim 2, characterized in that, The information entropy The entropy of all pixel values in the channel feature map is calculated using the following formula: in These are the normalized pixel values; The gradient response intensity This means that in one iteration of training, the final task loss function is calculated for the channel features. The mean absolute value of the gradient can be calculated as follows: This indicates the detection loss.
4. The lightweight knowledge distillation method for feature redundancy suppression in substation violation detection according to claim 1, characterized in that, Based on the comprehensive contribution score of each channel, the top K% of channels are selected as a "high-value feature subset." A gating mechanism is then used to dynamically filter redundant channels, resulting in the selected teacher features, including: Based on contribution score All feature channels are sorted in descending order, and the top K% of high-value feature channels are selected as effective feature channels, while the rest are considered redundant channels. A binary mask M, consistent with the number of channels, is generated through a gating mechanism, marking high-value channels as 1 and redundant channels as 0; During training, the mask M is multiplied channel-by-channel with the original teacher feature tensor F to mask redundant channels, thus obtaining the filtered teacher features. .
5. The lightweight knowledge distillation method for feature redundancy suppression in substation violation detection according to claim 1, characterized in that, The construction of the decoupled joint loss function : Losses incurred from detecting violations; The selected teacher and student characteristics are used to construct the feature distillation loss. The classic objective logic distillation loss is used to calculate the KL divergence of the final outputs of the teacher and student networks using a Softmax function with a temperature coefficient T, aiming to transfer the "soft-label knowledge" of the teacher network. and This represents the hyperparameter used to balance the weights of various losses.
6. The lightweight knowledge distillation method for feature redundancy suppression in substation violation detection according to claim 5, characterized in that, High-value characteristics of the gating-screened teacher network Features output by the corresponding layer of the student network The difference between them is used as the characteristic distillation loss. Among them, the features output by the corresponding layer of the student network After mask selection; The calculation is performed using the L2 norm: in, This is the number of channels selected.
7. A lightweight knowledge distillation system for feature redundancy suppression in substation violation detection, characterized in that, include: The teacher network module is used for monitoring video feature collection and behavior classification; The student network module is deployed on edge devices to perform real-time detection; The feature redundancy detection module is used to calculate entropy and gradient contribution. The channel selection gating module is used to shield redundant features; The distillation optimization module performs three joint loss optimizations. The lightweight deployment module is used, and the auxiliary modules are removed after training to generate the final deployment model.