A knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channel

Abstract

This invention discloses a knowledge distillation method for real-time video and audio fusion perception in power transmission and distribution channels, comprising the following steps: S1, synchronously acquiring video and audio streams from the power transmission and distribution channel to obtain aligned multimodal sample pairs; S2, constructing and pre-training an image teacher model and an audio teacher model respectively; S3, constructing a lightweight student model with a cross-modal collaborative attention fusion mechanism; S4, designing a multi-level collaborative knowledge distillation loss function to transfer knowledge from the image and audio teacher models to the student model; S5, collaboratively training the student model by minimizing the multi-level collaborative knowledge distillation loss function; S6, deploying the trained student model on edge devices in the power transmission and distribution channel, acquiring video and audio streams in real time, inputting them into the student model, and outputting the fused perception event classification results. This invention solves three major problems in multimodal perception of power transmission and distribution channels: deep modal fusion, heterogeneous knowledge transfer, and lightweight model design.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for power transmission and distribution lines, and in particular to a knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels. Background Technology

[0002] With the deepening of smart grid construction, intelligent operation and maintenance of transmission and distribution lines has become a key link in ensuring the safe and stable operation of the power grid. Currently, intelligent inspection and monitoring of transmission and distribution lines mainly rely on video surveillance systems. By deploying high-definition cameras on poles, the environment of the transmission line is monitored in real time, and computer vision technology is used to analyze the video stream to achieve automatic identification and alarm for certain scenarios such as large-scale mechanical construction, smoke and fire, foreign objects, and conductor galloping within the transmission line.

[0003] However, after long-term practical application, the inherent limitations of single video sensing technology have gradually been exposed, which seriously restrict the sensing capabilities and reliability of intelligent monitoring systems.

[0004] First, there is the problem of limited information dimensions. Video information is essentially a visual signal, primarily reflecting the visible features of a scene. However, many critical events in the operation of power transmission and distribution lines often do not manifest visually in their early signs or core characteristics, or the visual features appear relatively late. For example, when partial discharge occurs in line insulators, a characteristic "buzzing" discharge sound is often produced first. This high-frequency sound signal can travel a long distance, but at this point, no abnormality may be visible on the video screen. Only when the discharge intensifies, producing corona, arcs, or even smoke, can the video surveillance capture visible visual features. Similarly, in cases of external damage, the impact sound of large machinery striking the tower foundation or tower itself is an earlier and more direct warning signal than the robotic arm entering the frame. Furthermore, when a wildfire occurs beneath a power line, the visual features of the flames may be obscured by thick smoke, but the audio features such as the crackling sound of burning and the sound of wind are still clearly discernible. Therefore, relying solely on video perception misses a large amount of valuable audio information, which is irreplaceable for early warning and accurate judgment.

[0005] Secondly, there is a problem with poor environmental adaptability. The quality of video surveillance is highly dependent on lighting conditions and weather conditions. At night, even with supplemental lighting, the clarity, color, and contrast of the image are still far inferior to those during the day, and many subtle visual features are difficult to discern. In severe weather conditions such as rain, snow, fog, and haze, visibility decreases significantly, images become blurry, and the accuracy of target detection and recognition is greatly reduced or even completely lost. In strong backlighting scenarios, the target area may be too dark or overexposed, also causing information loss. These environmental factors severely limit the all-weather working capability of video perception systems in complex natural environments.

[0006] Given the aforementioned limitations, introducing audio signals into the intelligent sensing system of power transmission and distribution channels, and constructing a sensing system that integrates video and audio, has become an important direction for overcoming existing technological bottlenecks. Audio signals have the characteristics of being unaffected by light, having strong penetration, and long propagation distance, which can provide all-weather, all-time information support for line status monitoring, and are especially suitable as a supplementary information source in harsh environments and an early warning signal for early events.

[0007] However, the fusion perception of video and audio faces a series of new technical challenges.

[0008] First, video and audio are completely heterogeneous modalities. Video data has a spatial structure, presented as image frames, while audio data has a temporal sequence structure, presented as waveforms or spectrograms. Designing an effective fusion mechanism that allows for deep interaction and mutual complementarity between the two modalities is a crucial scientific question. Simple feature concatenation or addition is insufficient to uncover deep correlations between modalities and cannot achieve true fusion perception.

[0009] Secondly, multimodal fusion models are typically complex in structure and have a huge number of parameters. Taking a typical two-stream network as an example, the visual branch and the audio branch each require a deep neural network for feature extraction. Adding the subsequent fusion and classification layers, the total number of parameters in the entire model can easily reach tens or even hundreds of megabytes. Such massive models are difficult to deploy directly on edge computing devices on power transmission line towers, which typically have limited computing power and storage space, and impose strict requirements on the model's real-time performance and resource consumption.

[0010] Knowledge distillation, as an effective model compression and acceleration technique, can transfer knowledge from large teacher models to small student models, significantly reducing model complexity while maintaining high accuracy. However, most existing knowledge distillation methods are designed for single-modal models. How to effectively co-distill knowledge from two heterogeneous modal teacher models (image and audio) to train a lightweight fusion perception model is a pressing technical problem. Specifically, there are three core challenges: First, the feature spaces of heterogeneous modal teacher models differ greatly; how to design distillation objectives that allow the student model to absorb knowledge from both modalities simultaneously? Second, there may be knowledge conflicts between the two teacher models, i.e., different understandings of the same event; how to reconcile these conflicts? Third, the knowledge of the teacher models is not only reflected in their individual outputs but also in their collaborative decision-making patterns; how to capture and inherit this collaborative knowledge?

[0011] In summary, there is a lack of a comprehensive solution in the existing technology that can simultaneously solve the problems of deep modal fusion, lightweight model, and collaborative transfer of heterogeneous knowledge.

[0012] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art. Summary of the Invention

[0013] The purpose of this invention is to provide a knowledge distillation method for real-time video and audio fusion sensing of power transmission and distribution channels. By deeply fusing audio and video information, a fusion sensing model capable of reliable monitoring around the clock is constructed. Furthermore, by utilizing multi-level collaborative knowledge distillation technology, knowledge from two high-precision single-modal teacher models is transferred to a lightweight student model. This ensures high precision while meeting the real-time requirements and resource constraints of edge devices, thus solving the problem of deploying multimodal models at the edge.

[0014] To achieve the above objectives, this invention provides a knowledge distillation method for real-time video and audio fusion sensing in power transmission and distribution channels, comprising the following steps: S1. Synchronously acquire video and audio streams from the power transmission and distribution channels to obtain aligned multimodal sample pairs; S2. Construct and pre-train the image teacher model and the audio teacher model respectively; S3. Construct a lightweight student model with a cross-modal collaborative attention fusion mechanism; S4. Design a multi-level collaborative knowledge distillation loss function to transfer the knowledge of the image teacher model and audio teacher model to the student model; S5. Train the student model collaboratively by minimizing the multi-level collaborative knowledge distillation loss function; S6. Deploy the trained student model on the edge device of the power transmission and distribution channel, collect video and audio streams in real time and input them into the student model, and output the event classification results of fusion perception.

[0015] Optionally, step S1 includes: The video stream is frame-by-frame extracted to obtain an image frame sequence. For each frame of the image to be analyzed... Extract the audio segments within each Δt time window before and after it on the time axis. ,image and audio clips Constituting aligned multimodal sample pairs ; audio clips Preprocessing is performed and the data is converted into a Mel spectrogram.

[0016] Optionally, the image teacher model is constructed based on a deep convolutional neural network, using images... As input, output the probability distribution of visual event categories. ; The audio teacher model is built on a convolutional neural network or Transformer architecture, using audio segments. The Mel spectrogram is used as input, and the output is the probability distribution of audio event categories. .

[0017] Optionally, the student model includes: The visual feature extraction module extracts features from the input image. Extracting multi-scale visual feature maps The corresponding global visual feature vector is obtained through global average pooling. ; The audio feature extraction module extracts features from audio segments. Extracting audio temporal feature sequences from Mel spectrograms The global audio feature vector is obtained through global average pooling. ; A cross-modal collaborative attention fusion module is used to achieve deep interaction and adaptive fusion of visual and audio features. This cross-modal collaborative attention fusion module includes: Audio-guided visual attention units are used to adjust the global audio feature vector. Visual feature maps The features of each spatial location are weighted for attention to obtain the weighted visual attention features. The calculation formula is as follows: In the formula, , , d is the learnable weight matrix; d is the dimension of the common space; This is an attention weight map, representing the correlation between each spatial location and audio features; The weighted visual attention features; softmax() represents the normalized exponential function; Visually guided audio attention units are used to adjust audio based on global visual feature vectors. Audio temporal feature sequences The features at each time step are weighted by attention to obtain the weighted audio attention features. The calculation formula is as follows: In the formula, , , The weight matrix is ​​a learnable matrix; This is the attention weight vector, representing the correlation between each time step and the visual features; The weighted audio attention features; The gated fusion unit is connected to both the audio-guided visual attention unit and the visual-guided audio attention unit, and is used to perform visual attention based on visual attention features input from the audio-guided visual attention unit. and audio attention features input from the visually guided audio attention unit Dynamically calculate fusion weights to obtain fusion features The calculation formula is as follows: In the formula, This is the gate value; The weight matrix is ​​a learnable matrix. The bias term is a learnable term; σ is the sigmoid activation function, which gates the value. Each element is mapped to the interval (0,1); ⊙ represents element-wise multiplication; The fused feature vector; [;] indicates vector concatenation operation; A multi-level residual connection unit, connected to the gated fusion unit, is used to process the fusion features calculated by the gated fusion unit. Residual connections are made with the linear transformation of the original features, and the final multimodal fusion features are obtained through layer normalization. The calculation formula is as follows: In the formula, Linear is a linear transformation layer that transforms the concatenated original features. Mapped to d-dimensional space; LayerNorm is a layer normalization operation.

[0018] Optionally, the multi-level collaborative knowledge distillation loss function includes three levels: intra-modal distillation loss, inter-modal relation distillation loss, and structured knowledge distillation loss. The calculation formula is: In the formula, The cross-entropy loss between the student model's predictions and the true labels is used as the underlying task loss. For visual feature distillation loss, For audio feature distillation loss, To visually predict distillation loss, To predict distillation loss for audio, To preserve cross-modal similarity loss, For mutual information distillation loss, To compare the learning distillation loss; α1, β1, β2, γ, δ, ε are the weighting coefficients of each loss term.

[0019] Optionally, the intramodal distillation loss includes: Visual feature distillation loss, used to approximate the intermediate layer features of the image teacher model with the visual features of the student model, is calculated as follows: Audio feature distillation loss, used to approximate the intermediate layer features of the audio teacher model with the audio features of the student model, is calculated as follows: In the formula, MSE is the mean squared error loss; and A linear projection layer to adapt to the dimensions; and These are the visual and audio features of the student model, respectively. and These are the intermediate layer features of the image teacher model and the audio teacher model, respectively; Visual prediction distillation loss, used to make the fused output of the student model approximate the output probability distribution of the image teacher model, is calculated as follows: The audio prediction distillation loss, used to make the fused output of the student model approximate the output probability distribution of the audio teacher model, is calculated as follows: In the formula, For KL divergence loss; The fusion output probability distribution for the student model; and These are the output probability distributions of the image teacher model and the audio teacher model, respectively.

[0020] Optionally, the intermodal relationship distillation loss includes a cross-modal similarity preservation loss, used to enable the student model to learn cross-modal association patterns between the image teacher model and the audio teacher model, and its calculation process is as follows: In the formula, This represents the similarity between the visual features of the i-th sample in the image teacher model and the audio features of the j-th sample in the audio teacher model. This represents the similarity between the visual features of the i-th sample and the audio features of the j-th sample in the student model. and These are the feature vectors extracted by the image teacher model and the audio teacher model for the i-th and j-th samples in the batch, respectively. and These are the visual features extracted by the student model for the i-th sample in the batch and the audio features extracted for the j-th sample in the batch, respectively, where cosine is the cosine similarity function.

[0021] Optionally, the structured knowledge distillation loss includes: The mutual information distillation loss, used to maximize the lower bound of the mutual information between the student fusion features and the features of the two teacher models, is calculated as follows: In the formula, I(·;·) represents mutual information estimation; sim represents the cosine similarity function; Features of fusion; and These are the intermediate layer features of the image teacher model and the audio teacher model, respectively; Negative sample features are the visual features of the teacher extracted by the image teacher model from other event samples in the current training batch, excluding the current event sample. Contrastive learning distillation loss, used to enhance the discriminative power of student model features, is calculated using the following formula: In the formula, These are positive sample features, i.e., the visual features of the teacher extracted from the image teacher model for the same current event sample.

[0022] Optionally, in calculating the KL divergence loss A temperature parameter T is introduced, initially set to T>1, and gradually decreased to 1 as the training progresses. This relates to the KL divergence loss. The calculation formula is: In the formula, This represents the probability distribution of the teacher model's output. This represents the probability distribution output by the student model, where i and j represent the indices of the event samples. This represents the probability value of the i-th sample output by the teacher model. This represents the probability value of the i-th sample output by the student model. The output of logits for the teacher model. This represents the logits output by the teacher model for the i-th sample. Let logits represent the output of the teacher model for the j-th sample.

[0023] Optionally, a progressive training strategy is adopted in the training process of step S5. In the early stage of training, larger α1 and β1 weight coefficients are set so that the model first learns the basic task and single-modal features. In the middle stage of training, β2 and γ weight coefficients are increased to introduce predictive distillation and relational distillation. In the later stage of training, δ and ε weight coefficients are introduced to start structured knowledge distillation.

[0024] Compared with the prior art, the present invention has the following beneficial effects: First, the perception capability is significantly enhanced. This invention deeply integrates audio and video information and applies it to event monitoring in power transmission and distribution channels. The introduction of audio compensates for the shortcomings of visual perception in harsh environments and early warning stages. By synchronously analyzing audio characteristics such as discharge sounds and impact sounds, the system can issue an early warning at the initial stage of partial discharge on the line, several seconds to tens of seconds earlier than relying solely on visual detection of arcs or smoke, thus gaining valuable response time for maintenance personnel. In scenarios where video quality degrades, such as rainy, foggy weather or at night, the audio information can still operate stably, ensuring the reliability of the monitoring system around the clock.

[0025] Second, the innovative cross-modal collaborative attention fusion mechanism enables deep modal interaction. The audio-guided visual attention mechanism (using sonication to locate images) and the visual-guided audio attention mechanism (using images to locate time) designed in this invention achieve precise alignment in the spatiotemporal dimensions, enabling the model to understand "when a sound is heard corresponds to which image." This fine-grained interaction far surpasses traditional feature splicing or addition, truly uncovering complementary information between modalities. The gated fusion unit allows the model to adaptively adjust modal weights according to environmental conditions, emphasizing visual when the video is clear and audio when audio features are significant, achieving dynamic and intelligent modal fusion. Multi-level residual connections preserve original feature information, avoiding information loss that may occur with attention mechanisms.

[0026] Third, the innovative multi-level collaborative knowledge distillation framework enables comprehensive transfer of heterogeneous knowledge. This invention designs a distillation framework comprising three levels and six loss components, systematically solving the challenge of transferring knowledge from heterogeneous teacher models to lightweight student models. Intramodal distillation (feature level + prediction level) ensures that the student model inherits the single-modal professional capabilities of the two teacher models; intermodal relation distillation inherits the collaborative decision-making patterns between teacher models, enabling the student model to learn "how to combine seeing and hearing in thinking"; structured knowledge distillation (mutual information + contrastive learning) further strengthens the feature representation of the student model from the perspectives of information theory and discriminativity. This multi-level framework overcomes the shortcomings of traditional distillation methods, such as incomplete knowledge transfer, difficulty in handling heterogeneous modalities, and inability to reconcile knowledge conflicts.

[0027] Fourth, a balance between lightweight model and high accuracy. Through the innovative fusion mechanism and distillation framework described above, the resulting lightweight student model (approximately 5-10M parameters) achieves accuracy comparable to or even better than complex teacher models (over 100M parameters) in power transmission and distribution event detection tasks, while requiring only 1 / 10 to 1 / 20 of the computational cost. This allows the model to run smoothly on edge devices with limited computing power (e.g., achieving real-time inference speeds of over 25 frames per second) while maintaining high detection accuracy, truly achieving a balance between accuracy and efficiency.

[0028] Fifth, the system exhibits strong robustness. Because the model relies on both visual and auditory features and achieves adaptive decision-making through attention mechanisms and gating fusion, even when the quality of information from one modality deteriorates (e.g., video is occluded or audio is subject to strong noise interference), the model can still rely on information from another modality to make a judgment. This multimodal redundancy design gives the model extremely strong environmental adaptability and anti-interference capabilities, making it particularly suitable for complex and variable outdoor environments such as power transmission and distribution channels.

[0029] In summary, this invention systematically solves three key technical problems in multimodal sensing of power transmission and distribution channels—deep modal fusion, heterogeneous knowledge transfer, and lightweight model—through two core innovations: cross-modal collaborative attention fusion and multi-level collaborative knowledge distillation. It has significant technological advancements and broad industrial application prospects. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This is a structural block diagram of the cross-modal collaborative attention fusion module in this invention; Figure 3 This is a structural block diagram of the multi-level collaborative knowledge distillation loss function in this invention; Figure 4 This is a schematic diagram of the progressive training strategy in this invention. Detailed Implementation

[0031] The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, only for the purpose of conveniently and clearly illustrating the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0032] This invention provides a knowledge distillation method for real-time video and audio fusion sensing of power transmission and distribution channels, such as... Figure 1 As shown, it includes the following steps: S1. Data Acquisition and Preprocessing: Simultaneously acquire video and audio streams from the power transmission and distribution channels, and extract audio segments from the time windows before and after each frame of image to form multimodal sample pairs.

[0033] Specifically, real-time video and audio streams from the power transmission and distribution channel are acquired synchronously. The video stream is acquired by video acquisition equipment (such as high-definition cameras) deployed on the towers, while the audio stream is acquired by audio acquisition equipment (such as high-sensitivity microphones) deployed at the same location, ensuring the alignment of the two signals in time and space. Then, the video stream is processed by frame extraction to obtain an image frame sequence. The frame rate can be set according to the actual application requirements, typically 1-25 frames per second.

[0034] For each frame of the image to be analyzed Extract the audio segments within each Δt time window before and after it on the time axis. The value of Δt is typically 1-5 seconds to ensure that the audio clip contains sufficient event context information. (Image) and audio clips Constituting aligned multimodal sample pairs .

[0035] image As input features for visual modalities; for audio segments Preprocessing is performed, including pre-emphasis, framing, and windowing, and the data is converted into Mel spectrograms as input features for the audio modalities.

[0036] For the multimodal sample pairs constructed above Event categories are labeled, including but not limited to normal, crane, excavator, fireworks, conductor galloping, insulator discharge, and external impact. The labeled multimodal samples are then mapped to their corresponding ground truth labels. Together they form the training dataset, which is used for the pre-training of the teacher model and the distillation training of the student model.

[0037] S2. Constructing Teacher Models: Two independent, structurally complex, and highly accurate single-modal teacher models are constructed and pre-trained using a large amount of labeled data. The two teacher models are the image teacher model Teacher-V and the audio teacher model Teacher-A.

[0038] The Teacher-V image teacher model is built upon a deep convolutional neural network, preferably using large-scale network architectures such as ResNet-152 or Vision Transformer (ViT). The input is an image. Feature extraction is performed through multiple convolutional or self-attention layers, and finally, the visual event category probability distribution is output through a fully connected layer and a softmax function. The visual event categories may include cranes, excavators, fireworks, conductor galloping, insulator discharge (visible light), and normal events.

[0039] The audio teacher model, Teacher-A, is built on a convolutional neural network or Transformer architecture, preferably a hybrid network combining convolution and self-attention, such as the Audio Spectrogram Transformer or ResNet-50 for Audio. The input is an audio segment. The resulting Mel spectrogram is processed by a feature extraction network, and finally outputs the audio event category probability distribution through a fully connected layer and a softmax function. The audio event categories may include discharge sounds, mechanical impact sounds, digging sounds, explosion sounds, wind sounds, rain sounds, and normal sounds.

[0040] Both teacher models were pre-trained on large-scale labeled datasets containing tens of thousands to hundreds of thousands of samples and fine-tuned on power transmission and distribution professional datasets to ensure that their recognition accuracy in their respective modalities reached more than 95%.

[0041] S3. Constructing a student model: Construct a student model with a cross-modal collaborative attention fusion mechanism. Achieve precise alignment of spatiotemporal dimensions through audio-guided visual attention and visual-guided audio attention, and achieve adaptive adjustment of modal weights through a gating fusion unit.

[0042] Specifically, a lightweight multimodal fusion perception student model is constructed for deployment on edge devices. This student model consists of three main modules: a visual feature extraction module, an audio feature extraction module, and a cross-modal collaborative attention fusion module.

[0043] The visual feature extraction module employs a lightweight convolutional neural network, preferably MobileNetV3 or EfficientNet-Lite. Input image This network extracts multi-scale visual feature maps. H, W, These represent the height, width, and number of channels of the feature map, respectively. Simultaneously, a global visual feature vector is obtained through global average pooling. , as a global representation of the visual modality.

[0044] The audio feature extraction module employs a lightweight 1D convolutional neural network combined with temporal pooling. The input audio Mel-spectrum is processed through several layers of 1D convolution to extract local temporal features, and then a Transformer coding layer is used to capture long-range dependencies, resulting in an audio temporal feature sequence. Where T is the time step, The feature dimension is [not specified]. Simultaneously, a global audio feature vector is obtained through temporal global average pooling. , as a global representation of audio modalities.

[0045] The features extracted by the visual feature extraction module and the audio feature extraction module are both output to the cross-modal collaborative attention fusion module. The cross-modal collaborative attention fusion module is used to achieve deep interaction and adaptive fusion of visual and audio features, such as... Figure 2 As shown, this module consists of four sub-units: Audio-guided visual attention units: utilizing global audio feature vectors As a query, visual feature map The features of each spatial location are weighted for attention to obtain the weighted visual attention features. The calculation method is shown in formula (1): In the formula, , , d is a learnable weight matrix used to map features from different modalities to the same common space; d is the dimension of the common space. This is an attention weight map, representing the correlation between each spatial location and audio features; The weighted visual attention features are represented by ; softmax() represents the normalized exponential function.

[0046] This audio-guided visual attention unit focuses visual features on spatial regions related to the current audio event, enabling the "sound map" function. For example, when the model detects a discharge sound, the attention map highlights the insulator string region in the image.

[0047] Visually Guided Audio Attention Unit: Utilizing Global Visual Features As a query, the audio time-series feature sequence The features at each time step are weighted by attention to obtain the weighted audio attention features. The calculation method is shown in formula (2): In the formula, , , The weight matrix is ​​a learnable matrix; This is the attention weight vector, representing the correlation between each time step and the visual features; The weighted audio attention features.

[0048] This visually guided audio attention unit focuses audio features on time segments related to the current visual event, achieving the function of "finding time from images". For example, when the model detects an excavator entering the frame, the attention weight will reach its peak near the corresponding time point.

[0049] Gated fusion unit: Adaptively fuses visual attention features and audio attention features through a learnable gating mechanism. Specifically, this gated fusion unit is connected to both the audio-guided visual attention unit and the visual-guided audio attention unit, and fuses visual attention features input from the audio-guided visual attention unit. and audio attention features input from the visually guided audio attention unit Dynamically calculate fusion weights to obtain fusion features The calculation method is shown in formula (3): In the formula, This is the gate value; The weight matrix is ​​a learnable matrix. The bias term is a learnable term; σ is the sigmoid activation function, which gates the value. Each element is mapped to the interval (0,1); ⊙ represents element-wise multiplication; The fused feature vector; [;] indicates vector concatenation operation.

[0050] The gating value is dynamically calculated based on the attention features of the two modalities, enabling the student model to dynamically determine the degree of modality dependence based on the confidence level of the current scene. In well-lit daytime conditions, the gating value may be more biased towards the visual modality; while at night or in foggy weather, the gating value will automatically increase the weight of the audio modality, achieving adaptive fusion.

[0051] Multi-level residual connection unit: Connected to the gated fusion unit, in order to retain rich information of the original features, this multi-level residual connection unit converts the fused features calculated by the gated fusion unit into a multi-level residual connection unit. Compared with the original visual features (global visual feature vector) ) and raw audio features (global audio feature vector) The linear transformation of the model is used to perform residual connections, and the final multimodal fusion features are obtained through layer normalization. Then, the multimodal features are fused. The input is fed into the classification layer, and the output is a multimodal fusion event category probability distribution. .

[0052] Among them, the multimodal fusion features The calculation method is shown in formula (4): In the formula, Linear is a linear transformation layer that transforms the concatenated original features. Mapped to d-dimensional space; LayerNorm is a layer normalization operation.

[0053] Final multimodal fusion features The input is fed into a classification layer, then through a fully connected layer and a softmax function, and outputs a multimodal fusion event category probability distribution. .

[0054] S4. Design a multi-level collaborative knowledge distillation loss function, which comprehensively transfers the knowledge of the pre-trained image teacher model and audio teacher model to the student model from three levels: intra-modal feature distillation, inter-modal relation distillation, and structured knowledge distillation.

[0055] Specifically, to achieve comprehensive knowledge transfer from two heterogeneous teacher models to a lightweight student model, a multi-level collaborative knowledge distillation loss function with three levels and six loss components is designed. , Figure 3 The organizational structure and its relationship with the student and teacher models are illustrated below. Each level will be described in detail below.

[0056] The first level is Intra-modal Distillation, which aims to ensure that the student model inherits the single-modal recognition capabilities of the two teacher models respectively, including feature-level distillation and prediction-level distillation.

[0057] The feature-level distillation process enables the visual and audio features of the student model to approximate the intermediate-layer features of the corresponding teacher model, thereby inheriting the teacher model's powerful feature representation capabilities. Specifically, this includes: Visual feature distillation loss is used to make the visual features of the student model approximate the intermediate layer features of the image teacher model. Its calculation method is shown in formula (5): The audio feature distillation loss is used to make the audio features of the student model approximate the intermediate layer features of the audio teacher model. Its calculation method is shown in formula (6): In the formula, MSE is the mean squared error loss; and A linear projection layer for adapting dimensions, used to adapt student feature dimensions to teacher feature dimensions; and These are the visual and audio features of the student model, respectively. and These are the intermediate layer features of the image teacher model and the audio teacher model, respectively (in this embodiment, they are the feature vectors of the penultimate layer of the image teacher model and the audio teacher model, respectively).

[0058] The predictive-level distillation enables the fused output of the student model to approximate the output probability distributions of the two teacher models, thereby inheriting the decision boundaries of the teacher models. Specifically, this includes: The visual prediction distillation loss is used to make the fused output of the student model approximate the output probability distribution of the image teacher model. Its calculation method is shown in Equation (7): The audio prediction distillation loss is used to make the fused output of the student model approximate the output probability distribution of the audio teacher model. Its calculation method is shown in Equation (8): In the formula, For KL divergence loss; The fusion output probability distribution of the student model is obtained from the output of the classification layer of the student model; and These are the output probability distributions of the image teacher model and the audio teacher model, respectively.

[0059] The second level is the intermodal relationship distillation loss, which includes cross-modal similarity preservation loss. This forces the student model to maintain a cross-modal association pattern consistent with that of the teacher model, thereby learning the "collaborative decision-making wisdom" between the two teacher models.

[0060] The cross-modal similarity preservation loss is used to enable the student model to learn the cross-modal association patterns between the image teacher model and the audio teacher model, and its calculation process is as follows: First, calculate the similarity matrix between the visual features and audio features of the same batch of samples in the teacher model, as shown in formula (9): In the formula, The image teacher model extracts the visual features of the i-th sample in the batch. denoted as , where is the audio feature extracted by the audio teacher model for the j-th sample in the batch; cosine is the cosine similarity function. This represents the similarity between the visual features of the i-th sample in the image teacher model and the audio features of the j-th sample in the audio teacher model.

[0061] Similarly, the similarity matrix between visual features and audio features in the student model is calculated as shown in formula (10): In the formula, and These are the visual features extracted by the student model for the i-th sample in the batch and the audio features extracted for the j-th sample in the batch, respectively. This represents the similarity between the visual features of the i-th sample and the audio features of the j-th sample in the student model.

[0062] The cross-modal similarity preservation loss is achieved by minimizing the difference between the two similarity matrices, as shown in Equation (11): This loss ensures that the student model learns cross-modal correspondences consistent with the teacher model. For example, if the teacher model considers "arc image" to be highly correlated with "discharge sound audio" but not highly correlated with "wind sound audio", the student model must also learn this cross-modal association pattern.

[0063] The third level: Structured knowledge distillation loss. This level extracts deeper structured knowledge from the teacher model from the perspectives of information theory and contrastive learning, further enhancing the feature representation capabilities of the student model. This level includes: The mutual information distillation loss is used to maximize the lower bound of the mutual information between the student fusion features and the features of the two teacher models. From the perspective of information theory, it ensures that the student model can retain the effective information in the teacher model to the greatest extent. Its calculation formula is shown in formulas (12) and (13): In the formula, I(·;·) represents mutual information estimation, achieved through InfoNCE loss; sim is the cosine similarity function; These are negative sample features, typically taken from the visual features of teachers in other samples within the same batch.

[0064] The contrastive learning distillation loss enhances the discriminative power of the student model features by constructing positive and negative sample pairs. Its calculation formula is shown in formula (14): In the formula, The anchor feature is specifically the student fusion feature extracted from the student model for the current event sample. These are positive sample features, specifically the visual features of the teacher extracted from the image teacher model for the same current event sample; The negative sample features are specifically the visual features of the teacher extracted by the image teacher model from other event samples in the current training batch, excluding the current event sample.

[0065] This loss forces the fused c-features of the student model to be closer to the visual features of the corresponding teacher model in the feature space, while being farther away from the teacher features of other samples, thereby enhancing the discriminative power of the features.

[0066] Finally, the multi-level collaborative knowledge distillation loss function is a weighted combination of the loss components of the above three levels, and a temperature parameter T is introduced to soften the probability distribution of the teacher model, so that the student model can learn "tacit knowledge". The calculation method of the multi-level collaborative knowledge distillation loss function is shown in formula (15): In the formula, Predicting for student models The cross-entropy loss between the student model's fusion output probability distribution (obtained from the classification layer output of the student model) and the true label y serves as the basic task loss; α1, β1, β2, γ, δ, and ε are the weight coefficients of each loss term, determined through experimental tuning, with typical values ​​of α1=1.0, β1=0.1, β2=0.5, γ=0.2, δ=0.01, and ε=0.01.

[0067] Specifically, in the calculation of the predictive distillation stage, the KL divergence loss is calculated. A temperature parameter T is introduced to soften the probability distribution of the teacher model, enabling the student model to learn the "tacit knowledge" in the teacher model. The calculation method is shown in formula (16): In the formula, This represents the probability distribution of the teacher model's output. This represents the probability distribution output by the student model, where i and j represent the indices of the event samples. This represents the probability value of the i-th sample output by the teacher model. This represents the probability value of the i-th sample output by the student model. The output of logits for the teacher model. This represents the logits output by the teacher model for the i-th sample. Let T represent the logits output by the teacher model for the j-th sample, and T be the temperature parameter. Initially, T is set to > 1 (e.g., T = 4) to allow the student model to learn the similarity structure between categories. As the training progresses, T is gradually reduced to 1, allowing the student model to gradually focus on fine-grained category boundaries, thus achieving the transition from broad learning to fine-grained learning.

[0068] S5. Collaborative Training of Student Models: Freeze the parameters of the image teacher model and the audio teacher model, use the multimodal sample pairs constructed and labeled in step S1 and their corresponding ground truth labels, and minimize the multi-level collaborative knowledge distillation loss function described in step S4. The student model is trained end-to-end.

[0069] The training process specifically includes the following steps: S51. Parameter initialization: The visual feature extraction module and audio feature extraction module of the student model are initialized with weights pre-trained on ImageNet and AudioSet, while the remaining modules are initialized randomly.

[0070] S52, Teacher Model Freeze: Load the pre-trained image teacher model and audio teacher model from step S2, and freeze all their parameters so that they remain unchanged during training.

[0071] S53. Progressive Training: The training process employs a progressive training strategy, moving from easy to difficult, allowing student models to gradually absorb knowledge at different levels. This is divided into three stages. Figure 4 The diagram illustrates the changing trends of the weights of each loss term during different training stages, as well as the annealing process using the temperature parameter. In the early stages of training (the first 20% of iterations): set larger task losses and feature distillation weights (i.e., α1 and β1 weight coefficients, for example, set the weight coefficients of each loss term to α1=1.0, β1=0.5, β2=0, γ=0, δ=0, ε=0) so that the student model first learns basic tasks and single-modal features. Mid-training phase (the middle 50% of iterations): Increase the weights for predictive distillation and relational distillation (i.e., the weight coefficients of β2 and γ, for example, β2=0.5 and γ=0.2) to enable the student model to learn the unimodal decision boundary and intermodal correlation of the teacher model. In the later stages of training (the last 30% of iterations): structured knowledge distillation is initiated by introducing mutual information distillation and contrastive learning distillation (i.e., introducing δ and ε weight coefficients) to further refine the feature representation.

[0072] S54. Temperature Annealing: In conjunction with a progressive training strategy, the temperature parameter T starts from an initial value of 4 and decreases linearly to 1 as the training progresses, allowing the student model to gradually transition from learning general inter-class relationships to learning fine-grained class boundaries.

[0073] S55. Optimizer Settings: The AdamW optimizer is used, with an initial learning rate of 0.001. Cosine annealing is used to dynamically adjust the learning rate. The batch size is set to 64, and the total number of training epochs is 200.

[0074] S56. Data Augmentation: Apply data augmentation strategies such as random cropping, horizontal flipping, and color dithering to the visual modality; apply augmentation strategies such as temporal masking, frequency masking, and adding environmental noise to the audio modality to improve the generalization ability of the student model.

[0075] S6. Model Deployment and Application: Deploy the trained lightweight student model on the edge devices of the power transmission and distribution channel, collect video and audio in real time and input them into the student model, and output the event classification results of fusion perception.

[0076] Specifically, the trained lightweight student model is exported to a format suitable for edge devices (such as TensorFlow Lite or ONNX) and quantized and compressed (such as INT8 quantization compression) to further reduce the size and computational cost of the student model. The optimized student model is then deployed to edge devices on power transmission and distribution towers, including edge computing devices (such as NVIDIA Jetson series, HiSilicon Hi35xx series, etc.).

[0077] In actual operation, the edge device acquires video streams from the camera and audio streams from the microphone in real time, extracts frames from the video stream, and extracts audio segments from the preceding and following time windows for each frame. These are then input into the student model for forward inference. The student model outputs fused perception event classification results, including event category and confidence score. When a preset alarm event (such as discharge, impact, smoke, etc.) is detected and the confidence score exceeds a threshold, alarm information is automatically generated and uploaded to the monitoring center via a 4G / 5G network, enabling high-precision and robust real-time event perception.

[0078] The following section uses the detection of external force damage and discharge events in power transmission and distribution channels as a specific application scenario to explain in detail how to implement the knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels proposed in this invention, and to verify its effectiveness. Those skilled in the art should understand that this specific embodiment is only for explaining the invention and should not be considered as a limitation thereof.

[0079] S1. Data Acquisition and Preprocessing Industrial-grade high-definition network cameras (such as DS-2CD7A47EWD-XZ, 1920×1080 resolution, 25fps frame rate) and high-sensitivity microphones (48kHz sampling rate, 16-bit quantization) are deployed on 220kV transmission line towers. The camera and microphone are synchronously triggered to ensure audio and video timestamp alignment.

[0080] Frame extraction is performed on the video stream: the frame rate is 5fps, meaning one frame is extracted every 200 milliseconds; for each frame to be analyzed... Extract audio signal segments within a 2-second time window before and after it. (4 seconds of audio). The audio segment is preprocessed as follows: First, it is pre-emphasized (emphasis coefficient 0.97), then it is framed (frame length 25ms, frame shift 10ms), a Hamming window is added, and finally it is converted into a Mel spectrogram through short-time Fourier transform. The number of Mel filter banks is 64, resulting in a spectrogram with a dimension of T×64, where T is the number of time frames (4 seconds of audio corresponds to approximately 400 frames).

[0081] S2, Teacher Model Construction and Pre-training The Teacher-V image teacher model adopts the Vision Transformer (ViT-B / 16) architecture. The input image size is 224×224, which passes through a 12-layer Transformer encoder, and the output feature dimension is 768. Finally, it is mapped to the number of event categories through a fully connected layer (in this embodiment, there are 10 event categories: normal, crane, excavator, fireworks, conductor galloping, insulator discharge, external impact, wildfire, bird nest, and foreign object). The model is pre-trained on ImageNet-21K and then fine-tuned for 50 epochs on a self-built power transmission and distribution image dataset (containing 50,000 labeled images). The optimizer is AdamW, the initial learning rate is 1e-4, and the batch size is 64.

[0082] The Teacher-A audio teacher model employs an Audio Spectrogram Transformer (AST) architecture. The input Mel spectrogram is 400×64 pixels, and the model passes through a 12-layer Transformer encoder, outputting a feature dimension of 768, which is also mapped to 10 event classes. The model is pre-trained on AudioSet-2M and then fine-tuned for 50 epochs on a self-built power transmission and distribution audio dataset (containing 30,000 labeled audio segments). The optimizer is AdamW, with an initial learning rate of 1e-4 and a batch size of 32.

[0083] The two teacher models achieved 96.5% accuracy (images) and 94.2% accuracy (audio) on their respective test sets.

[0084] S3, Student Model Construction The visual feature extraction module of the Student-Fusion model uses MobileNetV3-Large, with an input image of 224×224 and an output feature map. The size is 7×7×160, and the visual global features are obtained after global average pooling. Dimension 160.

[0085] The audio feature extraction module employs a lightweight 1D convolutional network: The input is a 400×64 Mel spectrogram, which is processed through four 1D convolutional layers (kernel sizes of 3, 3, 5, and 5, stride of 2, and output channels of 32, 64, 128, and 256 respectively). Each layer is followed by batch normalization and ReLU activation. Then, a Transformer encoding layer (4 heads, feedforward dimension 512) is used to capture long-range dependencies, yielding the audio temporal feature sequence. The size is 50×256 (time steps compressed to 50), and the audio global features are obtained after temporal global average pooling. Dimension 256.

[0086] The parameter settings for the cross-modal collaborative attention fusion module are as follows: Final fusion features The event probability distribution is obtained through a fully connected classification layer (128 dimensions → 10 classes). .

[0087] S4. Multi-level collaborative knowledge distillation loss settings The loss function is shown in Equation (15), and the weight coefficients are determined by grid search: α1=1.0, β1=0.1, β2=0.5, γ=0.2, δ=0.01, ε=0.01. The temperature parameter T is initially set to 4, and decreases linearly to 1 as the training progresses.

[0088] During feature distillation, the feature locations extracted from the teacher model are: for the image teacher model, the output of the last layer [CLS] token (768-dimensional) of the ViT layer is selected; for the audio teacher model, the output of the last layer [CLS] token (768-dimensional) of the AST layer is selected. Visual features of the student model are also considered. (160-dimensional) via linear projection Mapped to 768 dimensions, audio features (256-dimensional) via linear projection Map to 768 dimensions, then calculate the MSE loss.

[0089] When predicting distillation, the KL divergence is calculated by dividing the logits output of the teacher model by the temperature T.

[0090] In relational distillation, the batch size is set to 64. The cosine similarity matrix between all sample pairs is calculated to obtain a 64×64 matrix. Then, the MSE of the student and teacher matrices is calculated.

[0091] Mutual information distillation uses InfoNCE loss, with a temperature set to 0.1 and 63 negative samples (other samples within the same batch). Contrastive learning distillation pairs positive samples are student fusion features and teacher visual features from the same event sample, while negative samples are teacher visual features from other samples.

[0092] S5, Collaborative Training Student Model A progressive training strategy was adopted, with a total of 200 training epochs and a batch size of 64. The optimizer was AdamW, with an initial learning rate of 0.001, and a cosine annealing strategy was used with weight decay of 0.01.

[0093] Image and audio data enhancement: Images are randomly cropped (random ratio 0.8~1.0), randomly horizontally flipped, and color jittered (brightness 0.2, contrast 0.2, saturation 0.2); audio is enhanced with time masking (maximum mask length 20 frames, 2 mask blocks), frequency masking (maximum mask length 8 frequency bands, 2 mask blocks), and Gaussian white noise is added (signal-to-noise ratio 15~25dB random).

[0094] Then, three more phases of training will be conducted: Phase 1 (rounds 0-40): α1=1.0, β1=0.5, β2=0, γ=0, δ=0, ε=0, T=4, student model learns basic classification tasks and unimodal features.

[0095] Second stage (rounds 41-140): Introduce predictive distillation and relational distillation, α1=1.0, β1=0.1, β2=0.5, γ=0.2, δ=0, ε=0, T linearly decreases from 4 to 2.

[0096] Phase 3 (rounds 141-200): α1=1.0, β1=0.1, β2=0.5, γ=0.2, δ=0.01, ε=0.01, T decreases linearly from 2 to 1, initiating structured knowledge distillation.

[0097] Training was performed on an NVIDIA Tesla V100 GPU and took approximately 3 days. Student model weights were saved after training was complete.

[0098] S6, Model Deployment and Testing The trained student model was exported to ONNX format and quantized using INT8 (using TensorRT). The quantized model size was approximately 5.2MB, and the inference speed on the NVIDIA Jetson Xavier NX reached 32ms / frame (31fps), meeting the real-time requirements.

[0099] Deployed on edge computing devices on power transmission and distribution towers, the system collects video and audio streams in real time for event detection. In real-world testing, the student model achieved a comprehensive recognition accuracy of 93.8% and a recall rate of 92.1% for 10 types of events, with a false alarm rate of 0.5 times per hour. Compared to standalone video perception models (87.3% accuracy) and audio perception models (83.6% accuracy), the fusion model significantly improves accuracy and maintains over 90% accuracy even in harsh environments such as nighttime and rain / fog, demonstrating the effectiveness of this invention.

[0100] Furthermore, to verify the effectiveness of each innovative module, ablation experiments were conducted on the same dataset. The results are shown in the table below: As can be seen from the table above, the multi-level collaborative knowledge distillation framework proposed in this invention improves the performance of the student model layer by layer, and finally achieves an accuracy comparable to that of the teacher model. At the same time, the number of parameters is only 1 / 35 of that of the teacher model, thus achieving a balance between accuracy and efficiency.

[0101] Furthermore, in the design of the fusion module, different attention mechanisms were compared, and the results are shown in the table below: The results show that the cross-modal collaborative attention fusion module proposed in this invention is significantly superior to other fusion methods.

[0102] The student model of this invention was deployed on a 220kV transmission line tower in Jiangsu Province and operated continuously for three months. During this period, it successfully provided early warnings for 15 external damage events (including crane contact with the line and excavator construction), 8 early discharge warnings, and 3 wildfire warnings. The average warning time was approximately 8 seconds earlier than traditional video monitoring, effectively preventing line tripping accidents. The student model also operated stably under severe weather conditions such as nighttime, heavy fog, and torrential rain, verifying the engineering practicality of this invention.

[0103] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] In the description of this invention, it should be understood that the terms "center," "height," "thickness," "upper," "lower," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0105] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0106] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0107] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A power transmission channel real-time video and audio fusion perception knowledge distillation method, characterized in that, Includes the following steps: S1. Synchronously acquire video and audio streams from the power transmission and distribution channels to obtain aligned multimodal sample pairs; S2. Construct and pre-train the image teacher model and the audio teacher model respectively; S3. Construct a lightweight student model with a cross-modal collaborative attention fusion mechanism; S4. Design a multi-level collaborative knowledge distillation loss function to transfer the knowledge of the image teacher model and audio teacher model to the student model; S5. Train the student model collaboratively by minimizing the multi-level collaborative knowledge distillation loss function; S6. Deploy the trained student model on the edge device of the power transmission and distribution channel, collect video and audio streams in real time and input them into the student model, and output the event classification results of fusion perception.

2. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 1, characterized in that, Step S1 includes: The video stream is frame-by-frame extracted to obtain an image frame sequence. For each frame of the image to be analyzed... Extract the audio segments within each Δt time window before and after it on the time axis. ,image and audio clips Constituting aligned multimodal sample pairs ; audio clips Preprocessing is performed and the data is converted into a Mel spectrogram.

3. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 2, characterized in that, The image teacher model is constructed based on a deep convolutional neural network, taking images as input and outputting a visual event class probability distribution ; The audio teacher model is built on a convolutional neural network or Transformer architecture, using audio segments. The Mel spectrogram is used as input, and the output is the probability distribution of audio event categories. .

4. The power transmission corridor real-time video and audio fusion perception knowledge distillation method of claim 2, wherein, The student model includes: a visual feature extraction module extracts a multi-scale visual feature map from an input image and obtains a corresponding global visual feature vector through global average pooling and obtains a corresponding global visual feature vector through global average pooling ; The audio feature extraction module extracts features from audio segments. Extracting audio temporal feature sequences from Mel spectrograms The global audio feature vector is obtained through global average pooling. ; A cross-modal collaborative attention fusion module is used to achieve deep interaction and adaptive fusion of visual and audio features. This cross-modal collaborative attention fusion module includes: Audio-guided visual attention units are used to adjust the global audio feature vector. Visual feature maps The features of each spatial location are weighted for attention to obtain the weighted visual attention features. The calculation formula is as follows: In the formula, , , d is the learnable weight matrix; d is the dimension of the common space; This is an attention weight map, representing the correlation between each spatial location and audio features; The weighted visual attention features; softmax() represents the normalized exponential function; Visually guided audio attention units are used to adjust audio based on global visual feature vectors. Audio temporal feature sequences The features at each time step are weighted by attention to obtain the weighted audio attention features. The calculation formula is as follows: In the formula, , , The weight matrix is ​​a learnable matrix; This is the attention weight vector, representing the correlation between each time step and the visual features; The weighted audio attention features; The gated fusion unit is connected to both the audio-guided visual attention unit and the visual-guided audio attention unit, and is used to perform visual attention based on visual attention features input from the audio-guided visual attention unit. and audio attention features input from the visually guided audio attention unit Dynamically calculate fusion weights to obtain fusion features The calculation formula is as follows: In the formula, This is the gate value; The weight matrix is ​​a learnable matrix. The bias term is a learnable term; σ is the sigmoid activation function, which gates the value. Each element is mapped to the interval (0,1); ⊙ represents element-wise multiplication; The fused feature vector; [;] indicates vector concatenation operation; A multi-level residual connection unit, connected to the gated fusion unit, is used to process the fusion features calculated by the gated fusion unit. Residual connections are made with the linear transformation of the original features, and the final multimodal fusion features are obtained through layer normalization. The calculation formula is as follows: In the formula, Linear is a linear transformation layer that transforms the concatenated original features. Mapped to d-dimensional space; LayerNorm is a layer normalization operation.

5. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 1, characterized in that, The multi-level collaborative knowledge distillation loss function includes three levels: intra-modal distillation loss, inter-modal relation distillation loss, and structured knowledge distillation loss. The calculation formula is: In the formula, The cross-entropy loss between the student model's predictions and the true labels is used as the underlying task loss. For visual feature distillation loss, For audio feature distillation loss, To visually predict distillation loss, To predict distillation loss for audio, To preserve cross-modal similarity loss, For mutual information distillation loss, To compare the learning distillation loss; α1, β1, β2, γ, δ, ε are the weighting coefficients of each loss term.

6. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 5, characterized in that, The intramodal distillation losses include: Visual feature distillation loss, used to approximate the intermediate layer features of the image teacher model with the visual features of the student model, is calculated as follows: Audio feature distillation loss, used to approximate the intermediate layer features of the audio teacher model with the audio features of the student model, is calculated as follows: In the formula, MSE is the mean squared error loss; and A linear projection layer to adapt to the dimensions; and These are the visual and audio features of the student model, respectively. and These are the intermediate layer features of the image teacher model and the audio teacher model, respectively; Visual prediction distillation loss, used to make the fused output of the student model approximate the output probability distribution of the image teacher model, is calculated as follows: The audio prediction distillation loss, used to make the fused output of the student model approximate the output probability distribution of the audio teacher model, is calculated as follows: In the formula, For KL divergence loss; The fusion output probability distribution for the student model; and These are the output probability distributions of the image teacher model and the audio teacher model, respectively.

7. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 5, characterized in that, The intermodal relationship distillation loss includes a cross-modal similarity preservation loss, which enables the student model to learn cross-modal association patterns between the image teacher model and the audio teacher model. Its calculation process is as follows: In the formula, This represents the similarity between the visual features of the i-th sample in the image teacher model and the audio features of the j-th sample in the audio teacher model. This represents the similarity between the visual features of the i-th sample and the audio features of the j-th sample in the student model. and These are the feature vectors extracted by the image teacher model and the audio teacher model for the i-th and j-th samples in the batch, respectively. and These are the visual features extracted by the student model for the i-th sample in the batch and the audio features extracted for the j-th sample in the batch, respectively, where cosine is the cosine similarity function.

8. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 5, characterized in that, The structured knowledge distillation loss includes: The mutual information distillation loss, used to maximize the lower bound of the mutual information between the student fusion features and the features of the two teacher models, is calculated as follows: In the formula, I(·;·) represents mutual information estimation; sim represents the cosine similarity function; Features of fusion; and These are the intermediate layer features of the image teacher model and the audio teacher model, respectively; Negative sample features are the visual features of the teacher extracted by the image teacher model from other event samples in the current training batch, excluding the current event sample. Contrastive learning distillation loss, used to enhance the discriminative power of student model features, is calculated using the following formula: In the formula, These are positive sample features, i.e., the visual features of the teacher extracted from the image teacher model for the same current event sample.

9. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 6, characterized in that, In calculating the KL divergence loss A temperature parameter T is introduced, initially set to T>1, and gradually decreased to 1 as the training progresses. This relates to the KL divergence loss. The calculation formula is: In the formula, This represents the probability distribution of the teacher model's output. This represents the probability distribution output by the student model, where i and j represent the indices of the event samples. This represents the probability value of the i-th sample output by the teacher model. This represents the probability value of the i-th sample output by the student model. This is the logits output of the teacher model. This represents the logits output by the teacher model for the i-th sample. Let logits represent the output of the teacher model for the j-th sample.

10. The knowledge distillation method for real-time video and audio fusion perception of power transmission and distribution channels as described in claim 5, characterized in that, In the training process of step S5, a progressive training strategy is adopted. In the early stage of training, larger α1 and β1 weight coefficients are set so that the model first learns the basic task and single-modal features. In the middle stage of training, β2 and γ weight coefficients are increased to introduce predictive distillation and relational distillation. In the later stage of training, δ and ε weight coefficients are introduced to start structured knowledge distillation.

Images