Power transmission line bird damage prevention and control system based on space-time semantic information model

Abstract

The invention provides a power transmission line bird damage prevention and control system based on a space-time semantic information model. The system comprises a bird damage evolution mechanism based on a knowledge graph, a bird nest refined detection model, a bird nest form trend prediction model fused with space-time semantic information and a bird damage grade evaluation model combined with the space-time semantic information. According to the bird damage evolution mechanism based on the knowledge graph, bird data is used as input, a power transmission line bird damage knowledge graph is constructed, and the bird damage evolution mechanism is obtained; the bird nest fine detection model is based on an NFFB-YOLOv9 algorithm and a bird nest deep semantic information extraction module, and image feature and semantic information extraction is carried out on the input picture to obtain a bird nest fine detection result; the spatio-temporal semantic information fused bird nest form trend prediction model predicts the change trend of the bird nest form along with time based on the historical data of the bird nest image; and finally, performing bird damage grade evaluation through the bird damage grade evaluation model. The bird damage grade and harm can be comprehensively judged through influence factors such as time and environment.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, image processing, and UAV power transmission line inspection; specifically, it relates to a bird damage prevention system for power transmission lines based on a spatio-temporal semantic information model. Background Art

[0002] As the main way of power transmission and an important part of the power system, maintaining the stable and high-quality power supply of power transmission lines plays a key role in the safe and stable operation of the entire power system. With the rapid development of the power industry and the continuous improvement of the natural ecological environment, the overlapping degree between the coverage area of overhead power transmission lines and the bird activity area has been continuously expanding. Coupled with the moderate height and wide field of vision of power transmission line towers, the nesting rate of birds on power transmission line towers has been continuously rising. Overhead power transmission lines pass through various complex natural environments such as forests, rivers, and farmlands. The daily activities of birds around the lines, such as perching, nesting, and defecating on the towers or high-voltage lines, have brought quite serious interference to the safe and stable operation of the power grid. After birds successfully build nests, it is easy to cause power transmission line failures. For example, it indirectly causes bird droppings flashover, power outages when birds fly in and out of the nests and touch live conductors, and tripping caused by the shedding of nest materials. According to data statistics, line failures caused by bird activities are second only to lightning strikes and external damage, ranking the 3rd in the total number of line failures. How to conduct refined detection, real-time evaluation, and feedback on bird damage to power transmission lines has become an urgent problem to be solved in the operation and maintenance of power transmission lines.

[0003] With the development of technology and the continuous innovation of power grid operation and maintenance methods, the UAV power transmission line inspection method has gradually replaced manual inspection. The inspection mode combining UAVs and image technology transmits the massive power transmission line image data collected by UAVs to the backend server in a wireless or manual copy manner, and the backend server runs a deep learning model to detect the image data, improving the work efficiency of power transmission line inspection and reducing the work intensity of line inspection personnel. Existing deep learning-based bird's nest detection algorithms are mainly divided into two categories: one is the two-stage algorithm represented by R-CNN (Regions with Convolutional Neural Network); the other is the one-stage algorithm represented by YOLO (You Only Look Once). The current bird's nest detection algorithms have a certain accuracy and precision, but the detected information is only whether there is a formed or unformed bird's nest.

[0004] Currently, the types of bird damage faults mainly include bird body short - circuit, bird nesting, bird predation, bird droppings flashover, bird natural enemy body short - circuit, bird pecking composite insulator faults, etc. Among them, bird droppings flashover is the most typical form of bird damage faults, with the highest proportion, and it occurs on transmission lines of various voltage levels below 500 kV. The assessment of bird damage level mainly involves reasonably allocating weights to various evaluation indicators such as geographical features and tower structure types, then determining the weight coefficients of each sub - indicator, and finally constructing a bird damage risk assessment model.

[0005] Traditional bird's nest detection is carried out through manual inspections. This not only has a high labor intensity and low efficiency, but also relies solely on manual visual judgment. For some bird's nests located at high positions, far away, or hidden in complex structures, misjudgment or missed judgment may occur due to factors such as observation angle and eyesight. Moreover, restricted by geographical environment, weather conditions, etc., it is difficult to conduct comprehensive and timely inspections on all lines, and it is easy to have inspection blind spots, resulting in some bird's nests not being discovered in time. Although drone - based inspections can obtain a large number of line images, manually screening and analyzing these images to identify bird's nests is still a time - consuming and laborious task, and there are also accuracy problems, and it is easy to miss some less obvious bird's nests. And it requires professional drone operators and supporting equipment maintenance personnel. At the same time, there are high requirements for the performance, battery life, and shooting equipment of the drone, increasing the detection cost.

[0006] Existing detection algorithms based on deep learning for drone image recognition require a large amount of labeled data for training to learn various features and patterns of bird's nests. The existing technical means for bird's nest detection are relatively single and have limited functions. They can only detect and judge the basic information of whether a bird's nest exists, but cannot further deeply explore the specific structural features of the bird's nest, such as the shape, size, internal structure, and materials used in the bird's nest. It is also difficult to know detailed information such as these, nor can it conduct a comprehensive analysis of the environmental conditions where the bird's nest is located. For example, valuable information such as the specific location of the bird's nest on the transmission line or tower and the degree of impact on the transmission line cannot be obtained.

[0007] The existing bird damage level assessment system has obvious defects, and one of the particularly prominent points is the lack of input of spatio-temporal information. In the time dimension, due to the lack of consideration of the seasonal variation of bird activity cycles, for example, some birds may carry out more frequent activities such as nesting and foraging during the breeding season, thus posing a greater threat to transmission lines, but the system is unable to incorporate these dynamic changes over time into the assessment system. From a spatial perspective, the bird damage level assessment system does not fully integrate specific information. For nests located in different positions, made of different materials, and in different geographical environments on the transmission tower, the bird damage risks are very different. This makes it difficult for the existing bird damage level assessment system to accurately determine the bird damage level in the face of complex and changing actual situations, greatly reducing the accuracy and reliability of its assessment, and thus it is difficult to play an efficient and powerful guiding role in the prevention and control of bird damage on transmission lines.

[0008] In view of this, the present invention proposes a bird damage prevention and control system for transmission lines based on a spatio-temporal semantic information model. Summary of the Invention

[0009] The object of the present invention is to propose a bird damage prevention and control system for transmission lines based on a spatio-temporal semantic information model. During the UAV inspection process, through a target detection algorithm based on deep learning, the nests on the inspection line are identified and analyzed to obtain more information about the nests, such as the material of the nest, the shape and state of the nest, the position of the nest on the transmission line or tower, and the degree of influence on the transmission line, etc. And through influencing factors such as time and environment, the bird damage level and harm are comprehensively judged. Reduce the failures of transmission lines caused by bird damage, reduce the consumption of human resources, and promote the technological progress and transformation and upgrading in the field of power transmission line maintenance.

[0010] To achieve the above object, the technical solution of the present invention is: a bird damage prevention and control system for transmission lines based on a spatio-temporal semantic information model, the system includes a bird damage evolution mechanism based on a knowledge graph, a refined nest detection model, a nest shape trend prediction model integrating spatio-temporal semantic information, and a bird damage level assessment model combining spatio-temporal semantic information;

[0011] The bird damage evolution mechanism based on a knowledge graph takes bird data as input, constructs a knowledge graph of bird damage on transmission lines, and obtains the bird damage evolution mechanism based on the knowledge graph of bird damage on transmission lines;

[0012] The refined nest detection model is based on the NFFB-YOLOv9 algorithm and a deep semantic information extraction module for nests, respectively extracts image features and semantic information from the input pictures, and obtains refined nest detection results;

[0013] The nest shape trend prediction model integrating spatio-temporal semantic information predicts the change trend of the nest shape over time based on the historical data of nest images;

[0014] The bird damage level evaluation model combining spatio-temporal semantic information evaluates the bird damage level based on the output results of other modules.

[0015] Preferably, constructing a transmission line bird damage knowledge graph with bird data as input specifically includes the following steps:

[0016] S1.1. Collect data related to bird damage, and perform preprocessing operations on the collected data, including cleaning, denoising, and format unification;

[0017] S1.2. Identify key entities related to bird damage from the preprocessed data, including bird names, crop types, geographical regions, prevention and control measures, and time information, and construct a bird damage knowledge data set with the help of the common sense knowledge base ConceptNet;

[0018] S1.3. Based on the bird damage knowledge data set, use the restart random walk algorithm RWR to model the semantic consistency matrix to quantify the knowledge graph, and form a transmission line bird damage knowledge graph.

[0019] Preferably, using the restart random walk algorithm RWR to model the semantic consistency matrix based on the bird damage knowledge data set is specifically as follows:

[0020] Starting from a certain entity as the starting node, move to the neighbor node of this node with probability 1 - c, or jump back to the starting point with probability c. Define c as the restart probability, and the mathematical expression is as follows:

[0021]

[0022] In the formula is an n×1 score vector, where n is the number of nodes in the knowledge graph; c is the restart probability; W = [W i,j is a graph with directed weights, and its element W ij represents the directed weight from node i to node j, reflecting the strength and direction information of the connection between nodes; is the matrix after normalizing W; is an n×1 initial vector.

[0023] Preferably, obtaining the evolution mechanism of bird damage based on the transmission line bird damage knowledge graph is specifically as follows: Use the cognitive graph network CGN to extract features and embed the transmission line bird damage knowledge graph to generate a high-dimensional vector representation with semantic associations, so as to combine the historical data of various bird damages and their relationships with environmental factors, use clustering algorithms in machine learning and time series analysis models in deep learning to mine the evolution patterns of bird damage, and predict the possible evolution paths of bird damage under different environmental and seasonal conditions.

[0024] Preferably, NFFB-YOLOv9 includes a backbone network, a neck network, and a head network;

[0025] The backbone network embeds the Nest Feature Focusing Module (NFFB), and the Nest Feature Focusing Module (NFFB) integrates the PANet module of YOLOv5, the RepVGGBlock module of YOLOv6, and the E-ELAN module of YOLOv7;

[0026] The neck network uses a bidirectional feature pyramid structure for feature enhancement;

[0027] The head network completes multi-scale prediction and post-processing, and outputs bounding box coordinates, class probabilities, and confidences.

[0028] Preferably, the deep nest semantic information extraction module includes shape semantic information extraction, structural semantic information extraction, and sag state semantic information extraction, and uses deep learning methods, geometric analysis methods, and dynamically corrected extraction strategies to extract key semantic information including nest structure symmetry, shape normality, and sag state;

[0029] Among them, the extraction of nest shape normality uses a binary classification multi-layer perceptron (MLP), the extraction of nest structure symmetry uses a geometric analysis method based on centroid calculation and mirror flipping, and then considering the results of both, a semantic information correction model is constructed to dynamically correct the semantic information and obtain the nest sag state information.

[0030] Preferably, the backbone network adopts a three-level feature processing architecture:

[0031] First, the input image is initially feature-extracted through a multi-modal feature fusion module, and the multi-modal feature fusion module includes an initial convolution group of 1×1 compression convolution, 3×3 stride convolution, and BN+SiLU activation;

[0032] Subsequently, the initial features are subjected to hybrid feature extraction and feature fusion through parallel 3×3, 5×5, and 7×7 convolution branches, and the feature maps are output to the Nest Feature Focusing Module (NFFB);

[0033] The Nest Feature Focusing Module (NFFB) processes the received feature maps in parallel through three branches; the first branch uses the bidirectional feature pyramid structure of PANet to achieve efficient fusion of features at different levels; the second branch integrates the multi-branch convolution of RepVGGBlock and performs re-parameterization, and the multi-branch convolution includes 3×3 and 1×1 convolution branches; the third branch introduces the feature shunting mechanism of E-ELAN, and generates enhanced features through 3 layers of 3×3 convolution and cross-layer connection;

[0034] After the three branch output features are concatenated in the channel dimension, they are reduced in dimension by a 1×1 convolution and weighted by the Adaptive Feature Gating Module (AFM). Finally, a feature map containing multi-scale context information is generated. Subsequently, the backbone network generates multi-scale features through a Progressive Feature Pyramid.

[0035] Preferably, the fine-grained bird's nest detection model is made lightweight through the Adaptive Feature Compression Network (AFCN). AFCN achieves efficient optimization of model size and computational complexity through a three-level dynamic compression strategy:

[0036] First, a multi-scale feature pyramid is constructed through the Adaptive Feature Selection Module. Parallel convolutional branches with 3×3, 5×5, and 7×7 kernels are used to extract bird's nest features at different scales. Channel attention and spatial attention mechanisms are combined to weight and fuse the features. Then, the Feature Scoring Network dynamically calculates the importance scores of each channel. Finally, the key feature channels with high contribution to detection are selected.

[0037] Subsequently, it enters the Dynamic Channel Pruning Module. The Dynamic Channel Pruning Module uses the learnable soft pruning mask technique to generate a continuous-value pruning mask through the sigmoid activation function and automatically adjusts the pruning rate according to the validation set loss during training. To compensate for the feature loss caused by pruning, the Dynamic Channel Pruning Module introduces cross-layer compensation connections and adds a 1×1 convolutional layer after the pruning layer to reconstruct the features. During training, a progressive pruning strategy is adopted, and the pruning intensity is dynamically optimized every 5 epochs according to the validation accuracy. Finally, the pruning mask is converted into a binary pruning decision through a preset threshold.

[0038] Finally, model compression is achieved through the Hierarchical Quantization Distillation Module. The Hierarchical Quantization Distillation Module adopts a three-level progressive quantization strategy and introduces residual connections during quantization to compensate for quantization errors. In the knowledge distillation stage, multi-scale feature matching losses are used, including the L1 loss of shallow features, the cosine similarity loss of deep features, and the KL divergence loss of the output layer. By dynamically adjusting the distillation weights and combining cross-modal knowledge transfer techniques, the RGB image features are transferred to the infrared modality to improve the model's generalization ability.

[0039] Preferably, the bird's nest morphological trend prediction model that fuses spatio-temporal semantic information adopts the spatio-temporal information algorithm STIFA to predict the change trend of the bird's nest morphology over time based on the historical data of bird's nest images. The STIFA algorithm processes and extracts the historical data of the bird's nest morphology in time and space sequences, weights and re-encodes the time and space features, fuses the time and space features, and then uses a multi-layer perceptron for prediction. Finally, the predicted bird's nest morphological features are obtained at the output layer.

[0040] Preferably, the bird damage level evaluation model adopts the Adaptive Voting Ensemble Algorithm (AVEA): The criticality of bird damage is divided into three levels: critical, severe, and general. The evolution mechanism of bird damage, the refined detection results of bird nests, and the changing trend of bird nest morphology are used as sample labels, and are labeled through the bird damage criticality classification index to construct a bird damage level evaluation dataset; The bird damage level evaluation model based on the Adaptive Voting Ensemble Algorithm (AVEA) achieves accurate evaluation through dynamic weight allocation and multi-modal feature fusion strategies.

[0041] The AVEA algorithm adopts a hierarchical ensemble architecture, including three core modules: the base model generation layer, the dynamic weight adjustment layer, and the multi-modal decision fusion layer.

[0042] In the model training stage, first, a multi-modal feature space is constructed. The original image features and environmental parameter features are standardized, and key features are selected through feature importance analysis. The base model generation layer uses three heterogeneous models: Gradient Boosting Decision Tree (XGBoost), Support Vector Machine (SVM), and Deep Neural Network (DNN), and independently trains on the training set to generate initial prediction results.

[0043] The dynamic weight adjustment layer introduces an adaptive weight update mechanism. By calculating the Mean Squared Error (MSE) and Chi-Square statistic of each base model on the validation set, the weight coefficient of each base model in the ensemble voting is dynamically adjusted, so that the base models with excellent performance can obtain higher weights in the ensemble decision-making.

[0044] The multi-modal decision fusion layer adopts a three-level voting mechanism: First, perform single-modal internal voting, conduct morphological evaluation based on image features, and conduct risk assessment based on environmental parameters; Then, generate a comprehensive feature vector through cross-modal feature fusion; Finally, conduct global weighted voting, and map the voting results to three bird damage levels.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention constructs a knowledge graph of transmission line bird damage, and uses machine learning and deep learning algorithms to realize the deep mining of the category and spatial relationships between transmission line bird damages, and predicts the possible evolution paths of bird damages under different environmental and seasonal conditions. By introducing time period and solar term factors, it can more effectively and accurately master the behavior and nesting rules of birds, reveal the evolution mechanism of bird damage, and thus better construct a bird damage prevention and control system.

[0047] The present invention adopts a lightweight deep neural network NFFB-YOLOv9, innovatively integrates multi-dimensional information such as the nest morphology, material, and spatial position of different birds, and introduces model lightweight technologies such as an adaptive feature compression network, and proposes a refined bird nest detection model that takes both high accuracy and low computational resource consumption into account, making the data at the output end more refined and rich.

[0048] Compared with traditional bird damage assessment methods, spatio-temporal semantic information is innovatively introduced. Combining factors such as bird species, spatial location, nesting patterns, and solar terms, a bird damage level assessment model for transmission lines considering spatio-temporal semantic information is proposed to achieve accurate prediction of the trend of bird damage hazards, thereby better preventing and eliminating bird damage.

[0049] Integrate the refined detection of bird nests and the assessment of bird damage levels, and develop an intelligent bird damage prevention and control software module. Through the intelligent analysis of inspection images, historical data, solar terms and other elements, provide accurate and dynamic bird damage prevention and control decision-making suggestions for power grid maintenance personnel. Description of the Drawings

[0050] Figure 1 It is the overall design flow chart of the transmission line bird damage prevention and control system based on the spatio-temporal semantic information model of the present invention;

[0051] Figure 2 It is the network structure diagram of the NFFB-YOLOv9 of the present invention. Detailed Embodiments

[0052] The following combines the attached Figure 1-2 , and specifically describes the technical solutions of the present invention.

[0053] What this application proposal realizes is a transmission line bird damage prevention and control system based on a spatio-temporal semantic information model, which mainly includes the bird damage evolution mechanism based on a knowledge graph, a refined bird nest detection model of the YOLOv9 algorithm based on the Nest Feature Focus Block (NFFB) module, and a bird damage level assessment model combined with spatio-temporal semantic information.

[0054] The bird damage evolution mechanism based on the knowledge graph uses knowledge graph reasoning and machine learning methods, combines the time period, grasps the bird behavior characteristics and nesting patterns during different seasons, analyzes and mines the high-incidence parts of bird damage, and analyzes and classifies the bird damage of different bird species, material types, and spatial positions, and evaluates its harm level to the transmission line, providing theoretical support for preventing bird damage.

[0055] The refined bird nest detection model based on the NFFB-YOLOv9 algorithm introduces deep neural networks, image segmentation and multimodal fusion technologies, designs a refined bird nest detection model using the NFFB-YOLOv9 algorithm, and deeply analyzes the spatial position information and sag state of the bird nest. Introduce bird species classification, and conduct refined identification of the nest characteristics of different birds, such as volume, shape, materials, etc. Adopt model lightweight technology to reduce the volume and computational resource consumption of the model, while ensuring the detection accuracy to achieve efficient detection.

[0056] The bird damage level assessment model combining spatio-temporal semantic information analyzes the morphological change trends of different bird nests based on historical bird nest images. By combining the spatial location of the bird nests and the bird species information, it predicts their future change trends and the possible degree of harm. Spatio-temporal semantic information is introduced, and seasonal factors are incorporated into the model to study the impact of bird behaviors in different seasons on the changes of bird nests, and dynamically evaluate the bird damage levels of different bird species in different time periods.

[0057] The following specifically describes the transmission line bird damage prevention and control system based on the spatio-temporal semantic information model.

[0058] 1. Construction of bird damage knowledge graph

[0059] Collect data related to bird damage from various channels, including ornithological research reports, bird damage statistics of departments, meteorological data, geographical information data, crop planting information, and news, forum posts, blog articles, etc. about bird damage on the Internet. For example, obtain the crop disaster situation and bird damage loss data in different regions and seasons through departments; collect information such as the species, habits, and distribution ranges of birds from ornithological research institutions.

[0060] Then perform preprocessing operations on the collected data, such as cleaning, denoising, and unifying the format; remove duplicate, incorrect, or incomplete data, and convert the data into a format suitable for subsequent processing, such as performing word segmentation and part-of-speech tagging on text data.

[0061] Identify key entities related to bird damage from the preprocessed data, such as various bird names (sparrow, magpie, woodpecker, etc.), crop species (wheat, rice, corn, etc.), geographical regions (the environment where the transmission line is located, such as woods, fields, rivers, etc.), prevention and control measures (bird repellers, bird nets, scarecrows, etc.), and time information (seasons, months, etc.). With the help of the common sense knowledge base ConceptNet, construct a bird damage knowledge data set, and use the restart random walk algorithm to model the semantic consistency matrix to quantify the knowledge graph, forming a transmission line bird damage knowledge graph.

[0062] By using the restart random walk algorithm (Random Walk with Restart, RWR) to iteratively explore the overall structure of the knowledge graph and estimate the semantic consistency between two entities. If there is a direct connection or more paths between two entities, the two entities are closer. The RWR algorithm is an improvement of the random walk algorithm. Starting from a certain node in the graph, it moves to the neighbor node of this node with a probability of 1 - c, or jumps back to the starting point with a probability of c. Define c as the restart probability, and the mathematical expression is shown in Equation (1):

[0063]

[0064] In the formula is an n×1 score vector, r i,j is the correlation score between node i and node j; c is the restart probability; W = [W i,j is a graph with directed weights, is the matrix after W is normalized; is an n×1 initial vector, and n is the number of nodes in the graph.

[0065] The correlation score between two nodes is obtained by restarting the random walk, and various aspects of information between the two nodes are captured. Compared with those algorithms that only use two nodes to measure, the RWR algorithm can capture the global structural information of the image to construct a knowledge graph of bird damage to transmission lines.

[0066] 2. Bird damage evolution mechanism

[0067] The above research has successfully constructed a knowledge graph of bird damage to transmission lines and represented the types and spatial correlations of bird damage to transmission lines in the form of a semantic consistency matrix. The Cognitive Graph Network (CGN) is used for feature embedding to mine the complex relationships and evolution mechanisms between bird damages. The Cognitive Graph Network (CGN) is a network model specifically used for graph-structured data. Different from the traditional Convolutional Neural Network (CNN), CGN is applicable to non-Euclidean data structures, such as irregular graphs like knowledge graphs and molecular structures. CGN can effectively capture the complex relationships between graph nodes and node attribute information, realizing efficient graph data processing and feature extraction.

[0068] The core idea of CGN is based on the topological structure of the graph. By aggregating the information of neighbor nodes, the feature representation of the nodes is updated. It uses the edge and node information in the graph to learn the high-dimensional features of each node, regarding the graph operation as an information propagation and aggregation process between the node and its neighborhood nodes. Each layer of CGN gradually updates the features of the current node by aggregating the features of neighbor nodes, thereby capturing the global information in the graph structure.

[0069] The Cognitive Graph Network (CGN) is used to extract and embed features from the knowledge graph of bird damage to transmission lines to deeply capture the features in the semantic consistency matrix. CGN fuses and learns information such as bird damage categories, bird behaviors, environmental factors, and spatial distributions by aggregating the features of node neighborhoods, generating a high-dimensional vector representation with semantic associations, thus laying a foundation for further analysis of the bird damage evolution mechanism.

[0070] After completing the knowledge graph embedding, pattern recognition and analysis of the evolution trend of bird damage are carried out through the high-dimensional vector representation of the cognitive graph network. Combining the historical data of various types of bird damage and their relationships with environmental factors, clustering algorithms in machine learning and time series analysis models in deep learning are used to mine the evolution patterns of bird damage, and the possible evolution paths of bird damage are predicted under different environmental and seasonal conditions.

[0071] 3. Fine-grained detection of bird nests

[0072] Construct the NFFB-YOLOv9 algorithm for detecting bird nests. The algorithm model structure is as Figure 2 shown;

[0073] NFFB-YOLOv9 is mainly composed of a backbone network, a neck network, and a head network, and is used for precise bird nest detection tasks. Among them, the core module NestEnhanceNet of the backbone network is embedded with a Nest Feature Focus Block (NFFB) on the basis of deeply integrating the PathAggregation Network (PANet) module of YOLOv5, the RepVGGBlock module of YOLOv6, and the E-ELAN module of YOLOv7.

[0074] The NFFB-YOLOv9 deep detection architecture consists of a backbone network, a neck network, and a head network to form a complete detection pipeline. The backbone network NestEnhanceNet integrates multiple modules and innovatively embeds the NFFB module. Through three-level feature processing and multi-branch parallel operations, multi-scale features are generated; the neck network is based on a bidirectional feature pyramid structure, and the features are refined through sampling branch processing and feature enhancement operations; the head network outputs detection results based on specific channel feature maps and completes multi-scale prediction and post-processing. In addition, the algorithm uses a variety of strategies to extract deep semantic information such as the shape, structure, and sagging state of bird nests, and constructs model dynamic correction to achieve comprehensive and precise detection of bird nests.

[0075] After the input image enters the backbone network, the NestEnhanceNet module extracts deep features from the input image, generating a multi-dimensional preliminary feature map containing information about the shape, texture, and location of the bird's nest. In the backbone network, PANet constructs bottom-up and top-down feature transfer paths. The bottom-up path transfers information from shallow feature maps to deep feature maps. Shallow feature maps usually contain more detailed information, such as the texture and edges of the bird's nest; deep feature maps contain more abstract and high-level semantic information, such as the overall shape and category information of the bird's nest. The top-down path does the opposite, transferring deep semantic information to the shallow layer to help the shallow feature maps fuse more semantic information. And the preliminary feature map is transmitted to the neck network; The RepVGGBlock module uses 3x3 and 1x1 convolutional structures to improve the feature expression ability; The E-ELAN (Extended Efficient Layer Aggregation Networks) module extracts features from the input feature map through convolutional layers with multiple branches. Each branch can use different convolutional kernel sizes and numbers of convolutional layers to capture features of different scales and complexities, forming a more powerful feature representation.

[0076] The backbone network NestEnhanceNet adopts a three-level feature processing architecture: First, the input image is initially feature-extracted through a multi-modal feature fusion module. In this embodiment, an input image with a size of 3×640×640 RGB is used as an example for illustration; The multi-modal feature fusion module includes an initial convolution group with 1×1 compression convolution, 3×3 stride convolution, and BN+SiLU activation. Subsequently, mixed feature extraction and feature fusion are performed through parallel 3×3 / 5×5 / 7×7 convolution branches, and a 64-channel 320×320 feature map is output.

[0077] After receiving the above feature map, the core innovation module NFFB (Feature Focusing Module for Birds' nests) processes it through three-branch parallel processing: The first branch adopts the bidirectional feature pyramid structure of PANet to achieve efficient fusion of features at different levels; The second branch integrates the multi-branch convolution (3×3 + 1×1) of RepVGGBlock and performs re-parameterization to improve the feature extraction ability while maintaining the inference efficiency; The third branch introduces the feature splitting mechanism of E-ELAN to generate enhanced features through 3 layers of 3×3 convolution and cross-layer connections. After the outputs of the three branches are concatenated in the channel dimension, they are reduced in dimension through 1×1 convolution and weighted through an adaptive feature gating module (AFM). Finally, a feature map containing multi-scale context information is generated. Subsequently, the backbone network generates multi-scale features through a progressive feature pyramid: First, a 256-channel 160×160 feature map is generated through 3×3 stride convolution. Subsequently, the PANet path aggregation structure is used to generate 512-channel 80×80 and 1024-channel 40×40 features, and finally a feature pyramid with 256 / 512 / 1024 channels is formed.

[0078] The neck network feature refiner adopts a bidirectional feature pyramid structure, processes the upsampling branch (transpose convolution × 2 + feature concatenation + 3×3 convolution) and the downsampling branch (3×3 stride 2 + feature concatenation + 3×3 convolution), and generates refined features with 512 channels at 80×80 and 160×160. Subsequently, feature enhancement is achieved through a dynamic convolution group (deformable 3×3 convolution + SE attention) and cross-scale feature fusion (downsampling to 256 channels by 1×1).

[0079] The head network multi-scale detector is constructed based on a 256-channel feature map, and outputs 6-channel detection results (xywh + conf + cls) through 3×3 → 512 channels → 1×1 convolution, and performs multi-scale prediction on 80×80 / 40×40 / 20×20 grids. The bounding box decoding uses 9 kinds of K-means anchor box matching to calculate coordinates, and finally outputs the bounding box coordinates (x1, y1, x2, y2), class probability and confidence after NMS post-processing.

[0080] The extraction of deep semantic information of the bird's nest is mainly divided into the extraction of shape semantic information, the extraction of structural semantic information and the extraction of sag state semantic information, which respectively adopt deep learning method, geometric analysis method and dynamically corrected extraction strategy, and effectively extract key semantic information such as the structural symmetry, shape normality and sag state of the bird's nest. The extraction of the shape normality of the bird's nest uses a binary multi-layer perceptron MLP, and the extraction of the structural symmetry of the bird's nest uses a geometric analysis method based on centroid calculation and mirror flipping. Then, considering the results of both, a semantic information correction model is constructed to dynamically correct the semantic information and obtain the sag state information of the bird's nest.

[0081] 4. Model lightweight technology

[0082] The refined detection model of the bird's nest based on multi-modal fusion can effectively detect the bird damage of the transmission line bird's nest, and output refined recognition results such as the bounding box coordinates, shape, structure and sag trend of the bird's nest. On the basis of object detection, it further analyzes the characteristics of the structure, shape and material of the bird damage of the bird's nest. The model already has high resolution and the ability to recognize fine features, but the model parameters are relatively complex, which is too computationally burdensome for software modules. Therefore, a model lightweight technology called Adaptive Feature Compression Network (AFCN for short) is proposed to reduce the volume and computational resource consumption of the model, while ensuring the detection accuracy and achieving efficient detection.

[0083] Adjust the paragraph structure to ensure the natural transition of the description of each module, and present the key technical indicators and implementation effects as independent paragraphs.

[0084] The model lightweight technology based on Adaptive Feature Compression Network (AFCN) realizes efficient optimization of model volume and computational complexity through a three-level dynamic compression strategy. This technology first constructs a multi-scale feature pyramid through an adaptive feature selection module, extracts nest features of different scales using 3×3, 5×5, and 7×7 parallel convolution branches, combines the channel attention (SE Block) and spatial attention (CBAM) mechanisms to perform weighted fusion on the features, then dynamically calculates the importance scores of each channel through a feature scoring network, and finally selects the key feature channels with high contribution to detection.

[0085] Subsequently, it enters the dynamic channel pruning module. This module innovatively adopts a learnable soft pruning mask technology, generates a continuous value pruning mask (between 0 and 1) through the sigmoid activation function, and automatically adjusts the pruning rate (dynamic range of 0.3 - 0.7) according to the validation set loss during the training process. To compensate for the feature loss caused by pruning, the module introduces cross-layer compensation connections and adds a 1×1 convolution layer after the pruning layer to reconstruct the features. During the training process, a progressive pruning strategy is adopted, and the pruning intensity is dynamically optimized every 5 epochs according to the validation accuracy. Finally, the soft mask is converted into a binary pruning decision through a 0.5 threshold.

[0086] Finally, the model compression is achieved through a hierarchical quantization distillation module. This module adopts a three-level progressive quantization strategy (32bit → 16bit → 8bit), and introduces residual connections during the quantization process to compensate for quantization errors. In the knowledge distillation stage, a multi-scale feature matching loss (MSFM Loss) is adopted, including the L1 loss of shallow features, the cosine similarity loss of deep features, and the KL divergence loss of the output layer. By dynamically adjusting the distillation weight (teacher weight is 0.8 in the first 10 epochs and drops to 0.5 in the next 10 epochs), combined with cross-modal knowledge transfer technology, the RGB image features are transferred to the infrared modality, effectively improving the model generalization ability.

[0087] 5. Prediction of Nest Shape Trend

[0088] The refined nest detection model based on the NFFB-YOLOv9 algorithm can well detect information such as the structure, shape, and sagging state of nest bird damage. After completing the detection and positioning of nest targets, the model further analyzes the material, structure, and shape characteristics of the nests, providing a more abundant basis for nest bird damage assessment. Considering that the nesting behavior of birds usually requires a relatively long period, it is necessary to refer to historical images of the same nest at the same location, analyze the morphological change trend of the nest, and predict its future change trend according to the location of the nest.

[0089] The bird's nest morphology trend prediction model that fuses spatio-temporal semantic information is implemented using the Spatio-Temporal Information Fusion Algorithm (STIFA). This algorithm predicts the changing trend of the bird's nest morphology over time based on the historical data of bird's nest images. Further, based on the historical data of bird's nest images, the changing trend of the bird's nest morphology over time is predicted.

[0090] The STIFA algorithm processes and extracts the historical data of the bird's nest morphology in time and space series. It weights and re-encodes the time and space features, then fuses the time and space features, and finally uses a multi-layer perceptron for prediction. Ultimately, the predicted bird's nest morphology features are obtained at the output layer. In this way, STIFA can extract and learn features from the historical bird's nest image data within a unified framework, achieving an accurate analysis of the changing trend of the bird's nest.

[0091] 6. Bird damage level assessment

[0092] The research on mining the evolution mechanism of bird damage by integrating knowledge graphs realizes the effective association between bird damage and bird species by constructing a knowledge graph of transmission line bird damage and mining the deep types and spatial connections between bird damages. The refined bird's nest detection model based on the NFFB module provides refined detection information such as the material, morphology, and sag state of the bird's nest. The bird's nest morphology trend prediction model that fuses spatio-temporal semantic information predicts the changing trend of the bird's nest morphology based on the historical data of bird's nest graphics. Based on the above three bird damage detection and prediction results, a bird damage level assessment model that combines spatio-temporal semantic information is established, and the influence of bird behavior with seasonal factors is incorporated into the model. The impact of bird behavior in different seasons on the changes in bird's nests is studied, and the bird damage levels of different bird species within different time periods are dynamically evaluated.

[0093] The bird damage level assessment model that combines spatio-temporal semantic information uses the Adaptive Ensemble Voting Algorithm (AVEA). AEVA is an ensemble learning algorithm that fuses the prediction results of multiple base learners and adaptively assigns weights according to the spatio-temporal semantic features of the data to achieve an accurate assessment of the bird damage level.

[0094] This model classifies the criticality of bird damage into three levels: critical, severe, and general. By evaluating the criticality of bird damage, the potential impact of bird damage on the safety of transmission lines can be analyzed, so as to take preventive maintenance measures early, thereby reducing the risk of bird damage faults in transmission lines, which is of great significance for ensuring the safe operation of transmission lines, improving operation efficiency, and protecting the environment. The judgment of the criticality of bird damage is comprehensively evaluated based on the above-mentioned bird species causing bird damage, the material of the bird's nest, the spatial location of bird damage, the sag state of the bird's nest, the changing trend of the bird's nest morphology, and seasonal factors.

[0095] Take the bird damage evolution mechanism, bird species, bird nest material, bird nest sagging state, and bird nest morphological change trend obtained from the above model inference as sample labels, annotate them through the bird damage criticality division index, construct a bird damage level evaluation dataset, and the bird damage level evaluation model based on the Adaptive Ensemble Voting Algorithm (AVEA) achieves accurate evaluation through dynamic weight allocation and multi-modal feature fusion strategies. This algorithm adopts a hierarchical integration architecture, including three core modules: the base model generation layer, the dynamic weight adjustment layer, and the multi-modal decision fusion layer.

[0096] In the model training stage, first construct a multi-modal feature space. Standardize features such as original image features (RGB / infrared) and environmental parameters (temperature / humidity / wind speed), and select 8 key features through feature importance analysis. The base model generation layer uses three heterogeneous models: Gradient Boosting Tree (XGBoost), Support Vector Machine (SVM), and Deep Neural Network (DNN), and independently trains them on the training set to generate initial prediction results.

[0097] The dynamic weight adjustment layer introduces an adaptive weight update mechanism. By calculating the mean square error (MSE) and chi-square statistic of each base model on the validation set, dynamically adjust its weight coefficient in the ensemble voting. This mechanism enables base models with excellent performance to obtain higher weights in the ensemble decision-making.

[0098] The multi-modal decision fusion layer adopts a three-level voting mechanism: first, conduct single-modal internal voting (image features → morphology evaluation, environmental parameters → risk evaluation), then generate a comprehensive feature vector through cross-modal feature fusion (Concatenate operation), and finally conduct global weighted voting. The voting results are mapped to three bird damage levels: general (blue), severe (orange), and critical (red).

[0099] This application proposal makes full use of deep learning algorithms for bird nest recognition and refined analysis, and combines spatio-temporal information to predict the evolution of bird nests, so as to better evaluate the bird damage level. A system integrating bird nest refined detection and bird damage level evaluation is proposed.

[0100] In the overall system architecture design: This system innovatively integrates multiple functional modules based on spatio-temporal semantic information to build a complete bird damage prevention system for transmission lines. The system covers the entire process functions from bird nest recognition and detection to bird damage level assessment and prediction of the development trend of bird damage. The modules do not exist in isolation, but work together through carefully designed interfaces and data interaction protocols. Through the construction of a bird damage knowledge graph and the input of more information about bird nests and spatio-temporal information obtained from the refined detection of bird nests, the resulting bird damage prevention system is more accurate and effective.

[0101] (2) NFFB-YOLOv9 algorithm: By deeply integrating the PathAggregation Network (PANet) module of YOLOv5, the RepVGGBlock module of YOLOv6, and the E-ELAN module of YOLOv7, and innovatively embedding the Bird Nest Feature Focusing Module (NFFB), the algorithm has achieved a significant improvement in feature extraction ability. The NFFB module interacts with other modules to achieve the focusing and enhanced extraction of bird nest features. In terms of output results, the NFFB-YOLOv9 algorithm can output a large amount of basic attribute data about bird nests. In addition to the traditional bounding box coordinates (the upper left and lower right coordinates or the center point coordinates and width and height information), it can also provide the approximate position of the bird nest in three-dimensional space (such as the specific part of the pole tower where the bird nest is located), as well as the specific structure and composition of the bird nest, providing more valuable information for bird damage level assessment and bird damage prevention.

[0102] The above are the preferred embodiments of the present invention. All changes made according to the technical solutions of the present invention that do not exceed the scope of the technical solutions of the present invention in terms of the functions and effects produced belong to the protection scope of the present invention.

Claims

1. A power transmission line bird damage prevention system based on spatiotemporal semantic information model, characterized in that: The system includes a bird damage evolution mechanism based on knowledge graph, a bird nest refinement detection model, a bird nest morphology trend prediction model integrating spatiotemporal semantic information, and a bird damage level assessment model combining spatiotemporal semantic information; The bird damage evolution mechanism based on the knowledge graph is to use bird data as input, construct a knowledge graph of bird damage to power transmission lines, and obtain the bird damage evolution mechanism based on the knowledge graph of bird damage to power transmission lines; The bird's nest refined detection model is based on the NFFB-YOLOv9 algorithm and the bird's nest deep semantic information extraction module, which respectively extracts image features and semantic information from the input image and obtains the bird's nest refined detection result; The bird's nest morphology trend prediction model integrating spatiotemporal semantic information predicts the changing trend of the bird's nest morphology over time based on the historical data of the bird's nest images; The bird damage level assessment model combining spatiotemporal semantic information performs bird damage level assessment based on output results of other modules.

2. A power transmission line bird damage control system based on a spatiotemporal semantic information model according to claim 1, characterized in that: The construction of the knowledge graph of bird damage to power transmission lines using bird data as input specifically includes the following steps: S1.

1. Collect data related to bird damage and perform pre-processing operations on the collected data, including cleaning, denoising, and formatting; S1.

2. Identify key entities related to bird damage from the preprocessed data, including bird names, crop species, geographical areas, control measures, and time information, and construct a bird damage knowledge data set with the help of the common sense knowledge base ConceptNet; S1.

3. Based on the bird damage knowledge data set, the restarted random walk algorithm RWR is used to model the semantic consistency matrix to quantify the knowledge graph and form a knowledge graph of bird damage to power transmission lines.

3. A power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 2, characterized in that: The semantic consistency matrix modeled using the restarted random walk algorithm RWR based on the bird damage knowledge data set is specifically: Starting from a certain entity as the starting node, move to the neighboring node of this node with probability 1-c, or jump back to the starting point with probability c. Define c as the restart probability. The mathematical expression is as follows: In the formula is an n×1 score vector, n is the number of knowledge graph nodes; c is the restart probability; W=[W i,j ] is a graph with directed weights, whose elements W ij Represents the directed weight from node i to node j, reflecting the strength and direction of the connection between nodes; is the matrix after W is standardized; is the n×1 initial vector.

4. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 1, characterized in that: The method for obtaining the evolution mechanism of bird damage based on the knowledge graph of bird damage to power transmission lines is specifically as follows: the cognitive graph network CGN is used to extract and embed features of the knowledge graph of bird damage to power transmission lines, and a high-dimensional vector representation with semantic association is generated to combine the historical data of various types of bird damage and their relationship with environmental factors, and the clustering algorithm in machine learning and the time series analysis model in deep learning are used to mine the evolution pattern of bird damage, and the possible evolution path of bird damage is predicted under different environmental and seasonal conditions.

5. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 1, characterized in that: NFFB-YOLOv9 includes backbone network, neck network and head network; The backbone network is embedded with a bird's nest feature focusing module NFFB, and the bird's nest feature focusing module NFFB integrates the PANet module of YOLOv5, the RepVGGBlock module of YOLOv6, and the E-ELAN module of YOLOv7; The neck network adopts a bidirectional feature pyramid structure for feature enhancement; The head network performs multi-scale prediction and post-processing, and outputs bounding box coordinates, category probabilities, and confidence levels.

6. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 1, characterized in that: The deep-level semantic information extraction module of the bird's nest includes shape semantic information extraction, structural semantic information extraction and drooping state semantic information extraction. It uses deep learning method, geometric analysis method and dynamic correction extraction strategy to extract key semantic information including the structural symmetry, shape normality and drooping state of the bird's nest. Among them, the normality of the bird's nest shape is extracted using a binary classification multi-layer perceptron MLP, and the symmetry of the bird's nest structure is extracted using a geometric analysis method based on center of mass calculation and mirror flipping. Then, considering the results of both, a semantic information correction model is constructed to dynamically correct the semantic information and obtain the drooping state information of the bird's nest.

7. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 5, characterized in that: The backbone network adopts a three-level feature processing architecture: First, an initial feature extraction is performed on the input image through a multimodal feature fusion module, wherein the multimodal feature fusion module includes a 1×1 compressed convolution, a 3×3 stride convolution, and an initial convolution group activated by BN+SiLU; Then the initial features are mixed and fused through parallel 3×3, 5×5, and 7×7 convolution branches, and the feature map is output to the Bird's Nest feature focusing module NFFB; The Bird's Nest feature focusing module NFFB processes the received feature maps in parallel through three branches; the first branch uses the bidirectional feature pyramid structure of PANet to achieve efficient fusion of features at different levels; the second branch integrates the multi-branch convolution of RepVGGBlock and re-parameterizes it, and the multi-branch convolution includes 3×3 and 1×1 convolution branches; the third branch introduces the feature diversion mechanism of E-ELAN, and generates enhanced features through 3 layers of 3×3 convolution and cross-layer connection; After the output features of the three branches are concatenated in the channel dimension, they are reduced in dimension by 1×1 convolution and weighted by the adaptive feature gating module AFM, finally generating a feature map containing multi-scale context information. The backbone network subsequently generates multi-scale features through a progressive feature pyramid.

8. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 1, characterized in that: The bird's nest refined detection model achieves model lightweighting through the adaptive feature compression network AFCN; AFCN achieves efficient model volume and computational optimization through a three-level dynamic compression strategy: First, a multi-scale feature pyramid is constructed through an adaptive feature selection module. The bird's nest features of different scales are extracted using 3×3, 5×5, and 7×7 parallel convolution branches. The features are weighted and fused by combining channel attention and spatial attention mechanisms. The importance score of each channel is dynamically calculated through a feature scoring network, and finally the key feature channels with high contribution to detection are screened out. Then enter the dynamic channel pruning module, which adopts the learnable soft pruning mask technology, generates continuous value pruning masks through the sigmoid activation function, and automatically adjusts the pruning rate according to the validation set loss during the training process; to compensate for the feature loss caused by pruning, the dynamic channel pruning module introduces cross-layer compensation connection, and adds a 1×1 convolution layer after the pruning layer to reconstruct the features; A progressive pruning strategy is adopted during the training process. The pruning intensity is dynamically optimized every 5 epochs according to the verification accuracy, and the pruning mask is finally converted into a binary pruning decision through a preset threshold. Finally, model compression is achieved through a hierarchical quantization distillation module; the hierarchical quantization distillation module adopts a three-level progressive quantization strategy and introduces residual connections to compensate for quantization errors during the quantization process; multi-scale feature matching losses are used in the knowledge distillation stage, including shallow feature L1 loss, deep feature cosine similarity loss and output layer KL divergence loss; by dynamically adjusting the distillation weights and combining cross-modal knowledge transfer technology, the RGB image features are migrated to the infrared modality to improve the generalization ability of the model.

9. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 1, characterized in that: The bird's nest morphology trend prediction model integrating spatiotemporal semantic information adopts the spatiotemporal information algorithm STIFA to predict the changing trend of the bird's nest morphology over time based on the historical data of the bird's nest images; the STIFA algorithm processes and extracts the historical data of the bird's nest morphology in time and space sequences, weights and re-encodes the time and space features, integrates the time and space features, and then uses the multi-layer perceptron to predict, and finally obtains the predicted bird's nest morphology features in the output layer.

10. The power transmission line bird damage control system based on spatiotemporal semantic information model according to claim 1, characterized in that: The bird damage level assessment model adopts the adaptive integrated voting algorithm AVEA: the severity of bird damage is divided into three levels: critical, severe, and general. The bird damage evolution mechanism, the refined detection results of bird nests, and the trend of bird nest morphology changes are used as sample labels. The bird damage level assessment data set is constructed by annotating the bird damage severity classification indicators. The bird damage level assessment model based on the adaptive integrated voting algorithm AVEA achieves accurate assessment through dynamic weight allocation and multimodal feature fusion strategy. The AVEA algorithm adopts a layered integrated architecture, which includes three core modules: base model generation layer, dynamic weight adjustment layer and multimodal decision fusion layer; In the model training phase, we first construct a multimodal feature space, standardize the original image features and environmental parameter features, and select key features through feature importance analysis. The base model generation layer uses three heterogeneous models: gradient boosting tree XGBoost, support vector machine SVM, and deep neural network DNN. These three models are trained independently on the training set to generate initial prediction results. The dynamic weight adjustment layer introduces an adaptive weight update mechanism. By calculating the mean square error (MSE) and chi-square statistic of each base model on the validation set, it dynamically adjusts its weight coefficient in the ensemble voting, so that the base model with excellent performance can obtain a higher weight in the ensemble decision. The multimodal decision fusion layer adopts a three-level voting mechanism: first, single-modal internal voting is performed to conduct morphological assessment based on image features and risk assessment based on environmental parameters; then, a comprehensive feature vector is generated through cross-modal feature fusion; finally, a global weighted vote is performed, and the voting results are mapped into three bird damage levels.

Images