A grounding-dino combined with a dynamic sparse attention mechanism model lightening method in the power field

Abstract

The present application relates to the technical field of power equipment image data processing, in particular to a model lightening method of Grounding-DINO combined with a dynamic sparse attention mechanism in the power field.The present application introduces a dynamic sparse attention mechanism through a power equipment special data set, prunes and screens key tokens in SwinTransformer, reduces the computational complexity to a low order through axial decomposition, designs LoRA and weight sharing, reduces the parameters in BERT, and fine-tunes and adapts power terms, fine-tunes part of the parameters to reduce the computing power, multi-scale training consolidates the accuracy, dynamically adjusts the parameters to adapt to the hardware during deployment, and avoids accuracy loss.It can be seen that the present application realizes lightening in terms of complexity, parameter quantity and the like through cooperation of each step, guarantees accuracy from multiple aspects, and finally realizes double improvement of lightening and accuracy.

Description

Technical Field

[0001] This invention relates to the field of power equipment image data processing technology, specifically a lightweight modeling method for the power industry that combines a dynamic sparse attention mechanism with Grounding-DINO. Background Technology

[0002] Power equipment inspection is a core component of ensuring the safe and stable operation of power systems. With the promotion of technologies such as drone inspection and edge real-time monitoring, the industry has put forward a dual demand for inspection models: "lightweight deployment" and "high-precision detection." The models need to be adapted to the limited computing power and storage resources of edge devices such as Jetson Xavier NX carried by drones, and they also need to accurately identify equipment defects (such as broken insulators and broken conductor strands) in complex power scenarios such as obstruction, severe weather (rain, snow, fog and haze), and changes in lighting, so as to avoid safety accidents caused by detection errors. Therefore, how to balance the lightweight nature of the model and the detection accuracy has become the core research direction of current power inspection technology.

[0003] Currently, there are two main types of technical solutions in the field of power equipment inspection: one is based on traditional convolutional neural network detection models, such as the YOLO series and Faster R-CNN. These models achieve target detection through multi-scale feature extraction and have certain practicality in general scenarios. However, they lack the ability to identify fine-grained defects in power equipment (such as missing pins) and are easily affected by background interference in complex inspection scenarios, making it difficult to meet the required detection accuracy. The other type is based on Transformer architecture detection models, such as DINO and Grounding-DINO. These models improve feature association capabilities through a global self-attention mechanism and have significantly better accuracy than traditional models in cross-modal detection of power equipment (text command and image matching). However, they rely on a large number of parameters and complex computational logic, and are currently the mainstream high-precision detection solution.

[0004] However, all the existing methods mentioned above share the common problem of "difficulty in balancing lightweight design and accuracy": Traditional convolutional models require additional feature extraction layers and expanded receptive fields to improve detection accuracy, leading to a simultaneous increase in both parameter count and computational load, failing to meet the lightweight requirements of edge devices. While Transformer-based models offer higher accuracy, the computational complexity of their global self-attention mechanism is O(n²). When dealing with high-resolution images of power lines and substations in power line inspections, the number of tokens surges, further amplifying the computational burden. Achieving lightweight design through simple feature cropping or reducing the number of attention heads results in the loss of key equipment features (such as insulator arrangement structures and minor conductor strand breaks), leading to a significant decrease in detection accuracy. Therefore, existing methods are either too "unlightweight" to be deployed on edge devices or sacrifice detection accuracy due to "crude lightweighting methods," severely hindering the practical application of power equipment inspection technology. This issue urgently needs to be addressed. Summary of the Invention

[0005] To avoid and overcome the technical problems existing in the prior art, this invention provides a model lightweighting method for the power field that combines a dynamic sparse attention mechanism with Grounding-DINO. This invention can improve detection accuracy while achieving model lightweighting.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A lightweight model simplification method for Grounding-DINO in the power field, incorporating a dynamic sparse attention mechanism, includes the following lightweighting steps:

[0008] S1. Construct a power dataset adapted to the power industry. The power dataset includes images of high-voltage lines and substation equipment, and after annotation and data augmentation, it is divided into training set, validation set and test set according to a set ratio.

[0009] S2. Design a dynamic sparse attention mechanism, embed a pruning module in the image encoder SwinTransformer of the Grounding-DINO model to dynamically filter the visual tokens of the feature map obtained during processing, and reduce the computational complexity of the model by axial attention decomposition.

[0010] S3. Design a LoRA adapter and weight sharing scheme. Insert a low-rank adapter into the text encoder BERT of the Grounding-DINO model and optimize the text encoding process by combining the inter-layer weight sharing strategy.

[0011] S4. Train and fine-tune the Grounding-DINO model. Initialize the Grounding-DINO model based on the pre-trained weights, adopt a partial parameter fine-tuning strategy, and complete the model training by combining a specific loss function and training parameters.

[0012] S5. Optimize the training model for edge device deployment, including model format conversion, hardware adaptation and dynamic parameter adjustment, and verify the detection performance after deployment.

[0013] As a further aspect of the present invention, the dynamic filtering specifically includes: embedding a pruning module in each downsampling layer of the SwinTransformer, and setting a set differential ratio for the retention of visual tokens in each downsampling layer; the pruning module filters key visual tokens by calculating the importance score of the visual tokens, and the filtering criteria are to retain visual tokens with an importance score higher than a set threshold and discard visual tokens with an importance score lower than the threshold.

[0014] As a further aspect of the present invention, the sub-steps for designing the LoRA adapter and weight sharing scheme are as follows:

[0015] S3A1. Insert a low-rank adapter LoRA into the self-attention layer of BERT. LoRA consists of matrix A and matrix B. The dimension of matrix A is d×r and the dimension of matrix B is r×d. d is the feature dimension of BERT and r is the low-rank dimension.

[0016] S3A2. Initialize LoRA. Matrix A is initialized using a random Gaussian distribution with a set mean and a set variance, and matrix B is initialized as a zero matrix.

[0017] S3A3: Freeze the pre-trained weights of the BERT backbone network, retaining only the trainable parameters of the LoRA adapter and higher-order semantic coding layers.

[0018] As a further aspect of the present invention, the specific steps for optimizing the edge device deployment of the trained model are as follows:

[0019] S5A1, Model Format Conversion: Convert the trained model to ONNX format, enable operator fusion during conversion, and then perform quantitative optimization on the ONNX model with set precision.

[0020] S5A2, Hardware Adaptation: For target edge devices, TensorRT is used to accelerate inference, the inference accuracy is set to the set accuracy, the workspace size is set to the set space value, and a memory paging loading mechanism is used to load the model weights into the device memory layer by layer in batches.

[0021] S5A3, Dynamic Pruning Adjustment: Adjust the pruning threshold α according to the real-time computing load of the edge device. That is, when the CPU utilization is lower than the set low threshold, α takes the set high precision retention value; when the CPU utilization is higher than the set high threshold, α takes the set high speed retention value.

[0022] As a further aspect of the present invention, the formula for calculating the importance score of the visual token is as follows:

[0023]

[0024] In the formula, t i Score(t) represents the i-th visual token. i ) represents t i Importance score; H represents the number of attention heads in the SwinTransformer; N represents the length of the token sequence in the current downsampling layer; Attn h (I,j) represents the attention weight between the i-th visual token and the j-th visual token in the h-th attention head;

[0025] The set threshold satisfies: Threshold = α•max(Scores), where Scores represents the set of importance scores for all visual tokens in the current layer; α represents the pruning threshold; and max(•) represents the operation of taking the maximum value.

[0026] As a further aspect of the present invention, the content of reducing computational complexity through axial attention decomposition specifically includes: decomposing the two-dimensional feature map output by the Swin Transformer into two one-dimensional sequences along the height and width axes, calculating the horizontal axis attention along the row and the vertical axis attention along the column respectively, and then adding the attention outputs in the two directions element by element to obtain the final feature map; and in the process of axial attention calculation, introducing a learnable relative position encoding consistent with the token feature dimension, which is initialized by a set method and updated with model training.

[0027] As a further aspect of this invention, the inter-layer weight sharing strategy specifically involves: within the defined higher-order semantic coding layers of BERT, inter-layer weight reuse is employed, with each subsequent layer reusing the basic or optimized weights of the previous layer; and during the model inference phase, the weight increment ΔW of LoRA is calculated using ΔW=A×B, and then... nem =W pretrained +△W will increment the weights by △W and then add the pre-trained weights W of BERT. pretrained Merge to obtain the new weight W for inference. nem .

[0028] As a further aspect of the present invention, the sub-steps of step S4 are as follows:

[0029] S41. Initialize the model backbone network using the original pre-trained weights of Grounding-DINO;

[0030] S42. The initial learning rate is set to a set value, and the total number of training rounds is set to a set number of rounds. In the first set number of rounds, a linear warming strategy is used to raise the learning rate from the set initial low learning rate to the set target learning rate. In the last set number of rounds, a cosine annealing strategy is used to lower the learning rate from the set target learning rate to the set final low learning rate.

[0031] S43. Construct a combined loss function, which includes FocalLoss for class imbalance optimization, GIoULoss for bounding box accuracy improvement, and InfoNCE cross-modal contrast loss for text-image alignment.

[0032] S44. Perform partial parameter fine-tuning: Freeze the settings of the Swin Transformer's pre-layer and the BERT backbone network, and only fine-tune the parameters of the dynamic sparse attention module, LoRA adapter, cross-modal fusion module and detection head.

[0033] S45. Multi-scale training is adopted, and the input image size is dynamically adjusted during the training process. The size range is the set image size interval.

[0034] As a further aspect of the present invention, the sub-steps of step S1 are as follows:

[0035] S11. Collect power data sets through appropriate technical means. The power data sets include a set number of high-voltage lines, substation equipment, and images of obstructions, rain, snow, and nighttime conditions.

[0036] S12. Use the LabelImg tool to manually annotate the image, including the device type, target bounding box location, and abnormal status.

[0037] S13. Expand the labeled image through multi-dimensional data enhancement. Multi-dimensional data enhancement methods include: translation amplitude within a set range, cropping within a set ratio range, random horizontal / vertical flipping with a set probability, random rotation within a set angle range, contrast adjustment within a set multiple range, brightness adjustment within a set multiple range, saturation adjustment within a set multiple range, Gaussian noise addition within a set variance range, fog effect within a set concentration range, and rain line simulation within a set density range.

[0038] S14. Divide the augmented images into training set, validation set and test set according to a set ratio.

[0039] As a further aspect of this invention, the verification of the detection performance after deployment is as follows: Two types of tests are performed on the target edge device:

[0040] The first type of test: single image test: select a set number of images in the test set for iterative inference, the average inference speed is not lower than the set image inference speed threshold, and the average accuracy is not lower than the set accuracy threshold.

[0041] The second type of test: video detection test: real-time detection of power inspection videos at a set resolution, with the inference speed not lower than the set video inference speed threshold and the defect identification delay not exceeding the set delay threshold.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This invention constructs a full-link optimization system encompassing "scenario adaptation, mechanism innovation, efficient training, and deployment adaptation," achieving a synergistic improvement in both lightweight design and detection accuracy from multiple dimensions: First, by constructing a dedicated dataset for power equipment, specifically covering high-voltage lines, substation equipment, and complex inspection scenarios (such as obstruction and severe weather), it provides a foundation for accurate model training, avoiding detection bias caused by general datasets and ensuring accuracy from the data source; Second, it innovatively introduces a dynamic sparse attention mechanism, embedding a pruning module in SwinTransformer to dynamically filter key visual tokens, while simultaneously reducing computational complexity from O(n²) to a lower order through axial attention decomposition, significantly reducing redundant computation (achieving lightweight design). Furthermore, this method retains the fine-grained features required for device detection (ensuring accuracy); secondly, it designs a LoRA adapter and weight sharing scheme, reducing the number of parameters in the BERT text encoder through low-rank adaptation and inter-layer weight reuse (enhancing lightweighting), while freezing the backbone network to retain general encoding capabilities and fine-tuning to adapt to power domain terminology, ensuring text-image cross-modal alignment accuracy; in addition, it focuses on optimizing key modules through partial parameter fine-tuning strategies to avoid the computational consumption of full training (lightweight extension), and combines multi-scale training to adapt to different inspection scenarios, further consolidating detection accuracy; finally, dynamic parameter adjustment in edge device deployment optimization can adapt the model's running state in real time according to hardware computing power, avoiding accuracy loss while achieving lightweight deployment. In summary, this method, through the collaborative design of each step, achieves model lightweighting from the perspectives of computational complexity, number of parameters, and deployment adaptation, while ensuring and improving detection accuracy from the perspectives of data adaptation, feature preservation, cross-modal alignment, and scene adaptation, ultimately achieving the dual goals of "improved lightweighting and enhanced detection accuracy," perfectly adapting to the real-time detection needs of power inspection edge devices. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Please see Figure 1 This implementation method, based on the high-efficiency detection requirements of power equipment inspection scenarios, takes "dataset adaptation - mechanism innovation - model training - deployment" as the core link to fully realize the technical solution described in the claims, ensuring a balance between detection accuracy, computational efficiency, and edge deployment adaptability of the model. The specific steps are as follows:

[0047] I. Constructing a power equipment dataset

[0048] This step aims to build a dedicated dataset that covers the entire power inspection scenario and is adapted to the model training requirements, solving the problem of insufficient adaptation of traditional general datasets in terms of power equipment types, defect states, and environmental diversity. The specific content is as follows:

[0049] 1.1 Raw Image Data Collection

[0050] 1.1.1 Data Sources and Scenarios Covered

[0051] Focusing on core scenarios of power line inspection, a set number of raw image data (e.g., 4290 raw images) will be collected through the following channels:

[0052] On-site drone data collection: For high-voltage lines (including conductors, insulators, fittings, pins) and substation equipment (transformers, circuit breakers, disconnect switches, instrument transformers), drones of appropriate brands are used to take pictures at different altitudes (0-1500m) and terrains (plains, mountains, hills). The flight altitude is controlled between 5-30m to ensure that the image resolution meets the inspection requirements (2048×1536 pixels).

[0053] Substation monitoring video capture: Extracting static frames from high-definition monitoring of 110kV-500kV substations, covering normal equipment operation, component aging, minor defects (such as stains on the surface of insulators), etc.

[0054] Historical inspection archive data: Screen the inspection images of the power company in recent years, and focus on supplementing data of extreme scenarios, including obstructed scenarios (tree branches blocking the power lines, overlapping equipment parts), severe weather (rain, snow, fog, haze, sandstorm), and changes in lighting (direct sunlight, low light at night), to ensure that the scene coverage is ≥90%.

[0055] 1.1.2 Data Filtering Criteria

[0056] Sharpness filtering: Remove blurry images (peak signal-to-noise ratio PSNR < 30dB) and motion-blurred images (edge ​​blur caused by drone shaking).

[0057] Equipment integrity screening: retain images with the main body of the equipment accounting for ≥15% to avoid difficulties in annotation due to the small size of the equipment.

[0058] Defect diversity screening: Ensure that the ratio of normal equipment to defective equipment images is 3:1, and cover high-frequency defects in power inspection such as broken strands, icing, broken insulators, missing pins, and bushing ruptures.

[0059] 1.2 Manual annotation

[0060] 1.2.1 Annotation Tools and Personnel Allocation

[0061] The LabelImg annotation tool was used, and several professional annotation personnel with many years of experience in power equipment inspection were deployed to ensure annotation accuracy.

[0062] Pre-labeling training: Conduct one week of training on labeling standards for power equipment types (5 types of core equipment) and defect status (8 types of typical defects), and standardize labeling terminology (e.g., "damaged insulator" requires specifying the boundary of the damaged area, "broken strand" requires specifying the specific location of the broken strand of the conductor).

[0063] The annotation process is standardized as follows: a three-level process of "initial annotation - re-annotation - cross-verification" is adopted. After the initial annotation is completed by the personnel, the re-annotation personnel check the integrity of the bounding box and the accuracy of the label. Finally, the three annotation personnel cross-verify to ensure that the annotation accuracy rate is ≥98%.

[0064] 1.2.2 Definition of Annotation Content

[0065] Category labels: Labeled by the secondary category of "Equipment type - Defect status", such as "Insulator - Normal" and "Conductor - Broken strand".

[0066] Bounding box annotation: Use a rectangular bounding box to ensure that the bounding box completely encloses the target area (the distance between the target edge and the bounding box is ≤5 pixels).

[0067] Defect description: For images with defects, add the degree of defect in the annotation notes (such as "insulator damage - minor" or "icing - thickness 5-10mm") to provide a basis for fine-grained identification of the model.

[0068] 1.3 Multi-dimensional data enhancement

[0069] To address the issues of insufficient data volume and incomplete scene coverage for power equipment annotations, a commonly used multi-dimensional data augmentation method is employed to expand the original data to a set scale (e.g., 12,870 images after expansion).

[0070] 1.4 Dataset Partitioning and Validation

[0071] The augmented dataset is divided into training, validation, and test sets according to a set ratio (e.g., 7:2:1), following these principles:

[0072] Class balance principle: Ensure that the proportion of equipment type and defect status in each subset is consistent with the total dataset (e.g., the proportion of "insulator-damaged" images in the training set is the same as that in the total dataset), and avoid class imbalance that leads to model bias.

[0073] The principle of non-overlapping scenarios: Images of the same inspection route and the same equipment are not distributed across subsets (e.g., monitoring images of a substation are only assigned to the training set or the test set), to ensure that the test set can truly reflect the model's generalization ability.

[0074] Post-split validation: Statistically analyze the number of images, category distribution, and scene coverage of each subset, and generate a dataset split report to ensure that the functions of the training set for model parameter learning, the validation set for hyperparameter tuning, and the test set for final performance evaluation are clearly defined.

[0075] II. Designing a Dynamic Sparse Attention Mechanism

[0076] To address the high computational complexity of the SwinTransformer image encoder in the Grounding-DINO model, a lightweight approach is adopted using a dual strategy of "dynamic token pruning + axial attention decomposition" while retaining the fine-grained features required for power equipment detection. The specific details are as follows:

[0077] 2.1 Infrastructure Adaptation and Pruning Module Embedding

[0078] 2.1.1 Selecting the Model Infrastructure

[0079] Based on Grounding-DINOv1.5, and using the core architecture of "image encoder (SwinTransformer-Tiny) + text encoder (BERT-Base) + cross-modal fusion module (Cross-Attention)," the focus is on optimizing the image encoder:

[0080] SwinTransformer configuration: It adopts 4 downsampling stages (Stage1-Stage4), corresponding to 4x, 8x, 16x and 32x feature map scaling. Each stage contains 2 SwinTransformerBlocks, with 12 attention heads H and 768 feature dimensions d.

[0081] The pruning module is embedded in the following location: after the output of the SwinTransformerBlock in each downsampling stage and before the feature map enters the next stage, the "token importance score - threshold filtering" pruning module is inserted to ensure that the pruning operation gradually removes redundant information during the feature abstraction process.

[0082] 2.1.2 Setting differentiated pruning ratios

[0083] Based on the importance of features in each downsampling stage (Stage 1 features are highly fine-grained and contain local device details; Stage 4 features are highly abstract and contain global device information), set differentiated token retention ratios (e.g., 85% for Stage 1, 75% for Stage 2, 65% for Stage 3, and 55% for Stage 4):

[0084] Basis for determining the ratio: Through preliminary experiments, the model accuracy (mAP) and computational cost (FLOPs) under different retention ratios were tested. When the retention ratio was lower than the above value, the mAP decreased by more than 2%; when it was higher than the above value, the FLOPs decreased by less than 5%. Therefore, this ratio range was determined as the balance point between accuracy and efficiency.

[0085] Ratio adjustment mechanism: A ratio adjustment interface is reserved, which can dynamically modify the retention ratio of each stage according to the type of power equipment (such as small target equipment "pins" which require a higher retention ratio).

[0086] 2.2 Token Importance Scoring and Screening

[0087] 2.2.1 Calculate the importance score

[0088] The importance score for each token is calculated using the following formula:

[0089] (1)

[0090] In the formula, t i Score(t) represents the i-th visual token. i ) represents t i Importance score; H represents the number of attention heads in the SwinTransformer; N represents the length of the token sequence in the current downsampling layer; Attn h(I,j) represents the attention weight between the i-th visual token and the j-th visual token in the h-th attention head, reflecting t i The higher the weight of the correlation with global features, the better t i The greater the contribution to device identification.

[0091] Calculation process: After each SwinTransformerBlock completes the attention calculation, the weight matrix of all attention heads is extracted, and the average is calculated according to formula (1) to obtain the importance score of each token, ensuring that the importance score can comprehensively reflect the global importance of the token.

[0092] 2.2.2 Determination and Implementation of Pruning Thresholds

[0093] Threshold calculation: Set the pruning threshold α according to the formula Threshold=α•max(Scores), where α is set in the range of 0.3~0.6 (e.g., initial α=0.5).

[0094] When α=0.6, the threshold is relatively high, the number of tokens retained is large, the accuracy is high but the computational load is large, which is suitable for the training stage or edge devices with sufficient computing power.

[0095] When α=0.3, the threshold is low, the number of tokens retained is small, the amount of computation is small but the accuracy is slightly reduced, which is suitable for edge devices with limited computing power (such as low-power chips carried by drones).

[0096] Filtering execution: Compare the importance score of each token with a threshold, retain tokens with an importance score ≥ the threshold, and discard tokens with an importance score < the threshold. For the retained tokens, record their location information using an index to ensure accurate association with spatial features during subsequent axial attention calculations.

[0097] 2.3 Axial attention decomposition and relative position coding

[0098] 2.3.1 Implementation of Axial Attention Decomposition

[0099] To address the computational complexity issue of O(n²) for global self-attention in traditional Transformers, an axial attention decomposition strategy is adopted:

[0100] Feature map decomposition: The current two-dimensional feature map x∈R^{h×w×d} (where h is the feature map height, w is the width, and d is the feature dimension) is decomposed into two one-dimensional sequences along the height axis (y-axis) and the width axis (x-axis):

[0101] Horizontal axis sequence: The feature map is split by rows to obtain h one-dimensional sequences of length w (each sequence corresponds to a row of pixels); Vertical axis sequence: The feature map is split by columns to obtain w one-dimensional sequences of length h (each sequence corresponds to a column of pixels).

[0102] Attention Calculation: Horizontal Axis Attention: Self-attention is calculated separately for each horizontal axis sequence to obtain the horizontal feature response (capturing the horizontal structure of the device, such as the linear distribution of wires). Vertical Axis Attention: Self-attention is calculated separately for each vertical axis sequence to obtain the vertical feature response (capturing the vertical structure of the device, such as the columnar distribution of power towers). Feature Fusion: The horizontal and vertical axis attention outputs are element-wise added to obtain the final axial attention feature map, reducing the computational complexity to O(2n) (n = h × w, where n is the total number of tokens).

[0103] 2.3.2 Adding relative position encoding

[0104] To avoid spatial information loss due to axial attention decomposition, the spatial structure of power equipment is crucial for detection, such as the arrangement of insulators and the direction of conductors. Therefore, learnable relative position coding is introduced.

[0105] Encoding initialization: The relative position encoding matrix is ​​initialized using a sine function, and the encoding dimension is consistent with the token feature dimension d (768).

[0106] Encoding fusion: During the axial attention calculation process, the relative position encoding is added to the token features and then input into the attention layer to ensure that the model can learn the spatial relationship between power equipment.

[0107] Training Update: The relative position encoding is a learnable parameter that is updated along with the model training. The encoding value is optimized through backpropagation to adapt it to the spatial characteristics of the power equipment (e.g., if the insulator spacing is fixed, the encoding value will be gradually optimized to highlight the spacing feature).

[0108] III. LoRA Adapter and Weight Sharing Design

[0109] To address the issues of large parameter count and high fine-tuning cost in the Grounding-DINO text encoder BERT, a dual optimization strategy of "Low-Rank Adapter (LoRA) + Inter-layer Weight Sharing" is employed. This approach reduces both the number of parameters and computational cost while preserving the model's general text understanding capabilities. The specific details are as follows:

[0110] 3.1 Basic Configuration of BERT Text Encoder

[0111] The BERT-Base model was chosen as the text encoder. Its basic configuration is as follows: 12 TransformerBlock layers, hidden layer dimension d=768, number of attention heads H=12, and total number of parameters approximately 110M. The focus is on the self-attention layer of BERT (each TransformerBlock contains 1 self-attention layer), which is the core of text feature encoding and also the module with the highest parameter count (approximately 60% of the total parameters).

[0112] 3.2 LoRA Adapter Design and Insertion

[0113] 3.2.1 LoRA Matrix Dimensions and Initialization

[0114] Matrix dimension setting: The LoRA adapter consists of a low-rank matrix A (dimension d×r) and a matrix B (dimension r×d), where d=768 (BERT feature dimension), and r is the low-rank dimension, set in the range of 4~16 (e.g., r=8): The selection of r is based on the experimental test of model performance when r=4, 8, 12, and 16. When r=8, the number of parameters is reduced by 35% (from 110M to 71.5M), and the text-image alignment accuracy (cross-modal retrieval accuracy) only decreases by 0.8%, which is the balance point between "number of parameters and accuracy".

[0115] Matrix initialization: Matrix A: Initialized using a random Gaussian distribution with a mean of 0 and a variance of 0.01 to ensure small initial weights and avoid drastic interference with BERT pre-trained features. Matrix B: Initialized as an all-zero matrix, so that the LoRA adapter's initial output is 0, and the model's initial performance is equivalent to the original BERT, facilitating subsequent fine-tuning and stable convergence.

[0116] 3.2.2 LoRA Insertion Location and Mechanism of Action

[0117] Insertion location: Insert the LoRA adapter after the QKV linear transformation and before the attention calculation in the BERT self-attention layer. The specific process is as follows: BERT pre-trained weights generate QKV features → LoRA adapter generates incremental features → fused features.

[0118] Mechanism of action: By using the low-rank incremental features of the LoRA adapter, text features are fine-tuned to adapt to the terminology of the power equipment field (such as professional terms like "insulator" and "broken strand") without modifying the pre-trained weights, while significantly reducing the number of trainable parameters (only the LoRA matrix parameters are trainable, accounting for about 5% of the total number of parameters in BERT).

[0119] 3.3 Inter-layer weight sharing strategy

[0120] 3.3.1 Selection and Implementation of Shared Layers: Layers 6-12 of BERT (higher-order semantic coding layers) are selected for inter-layer weight sharing because: Layers 1-5 (lower-order semantic layers) are responsible for basic vocabulary encoding (such as "equipment" and "line"), which need to retain their universality and are not suitable for sharing; Layers 6-12 (higher-order semantic layers) are responsible for semantic combination and domain adaptation (such as "electric insulator" and "broken strand of high-voltage conductor"), and the semantic logic between layers is similar, making weight reuse suitable.

[0121] Sharing method: "Progressive weight reuse" is adopted, that is, the weights of the 7th layer are reused from the 6th layer (initially the pre-trained weights of the 6th layer, and the 7th layer is updated synchronously after the weights of the 6th layer are updated during training), the 8th layer reuses the optimized weights of the 7th layer, and so on up to the 12th layer, to ensure the consistency of semantic encoding between layers and the efficiency of parameter reuse.

[0122] 3.3.2 Gradient Update Mechanism

[0123] To avoid gradient conflicts caused by weight sharing between layers (where multiple backpropagation gradients update the same weight simultaneously), a "gradient accumulation" mechanism is adopted: During training, the loss of layers 6-12 is calculated sequentially, and the gradient of each layer's loss is calculated separately and temporarily stored; after all shared layers have completed forward calculation and loss calculation, the temporarily stored gradients of each layer are averaged, and then backpropagation is used to update the shared weights; Advantages: Balances the gradient contributions of each shared layer, avoids excessive gradients in one layer causing the weights to be biased towards the semantics of that layer, and ensures the overall adaptability of high-order semantic encoding.

[0124] 3.4 Weighting during the inference phase

[0125] To improve the inference efficiency of edge devices (avoid loading pre-trained weights and LoRA matrices simultaneously during inference), the LoRA adapter and BERT pre-trained weights are merged after model training is completed. The specific process is as follows: Calculate the LoRA weight increment: Calculate the LoRA weight increment of the Q, K, and V linear layers respectively according to ΔW=A×B.

[0126] Merge weights: Add the increment to the pre-trained weights to obtain the new weight W. nem =W pretrained +△W.

[0127] Replace and save: Use the merged W nem Replace the original pre-trained weights to generate an "integrated" BERT model, and then save it in ONNX format. During inference, only the merged model needs to be loaded, reducing memory usage and the number of operator calls (inference speed is improved by about 15%).

[0128] IV. Model Training and Fine-tuning

[0129] Based on the constructed dataset, the designed dynamic sparse attention mechanism, and the LoRA adapter, the model achieves high-precision convergence in the power equipment detection task through "partial parameter fine-tuning + multi-strategy training," while controlling training cost and time. Details are as follows:

[0130] 4.1 Training Environment and Initial Configuration

[0131] 4.1.1 Hardware and Software Environment

[0132] Hardware environment: NVIDIA TITAN XP GPU (single or multi-card parallel, distributed training when using multiple cards), 32GB DDR4 memory, 512GB SSD storage (for storing datasets, model weights and training logs).

[0133] Software environment: Ubuntu 16.04 operating system, PyTorch 1.12 framework, CUDA 11.6, CuDNN 8.4. Dependencies include: Computer vision libraries: torchvision 0.13.1 (data loading and augmentation), mmdet 2.28.0 (detection model tool); Natural language processing libraries: transformers 4.24.0 (BERT model loading and fine-tuning); Log and evaluation libraries: tensorboard 2.11.0 (training log visualization), pycocotools 2.0.6 (detection performance evaluation).

[0134] 4.1.2 Model Weight Initialization

[0135] Basic weights: The model backbone network (SwinTransformer and BERT) is initialized using the official pre-trained weights from Grounding-DINO (trained on the COCO dataset, mAP=49.5%) to ensure that the model has general object detection and text understanding capabilities.

[0136] Custom module initialization: Dynamic sparse attention module: The scoring calculation layer of the pruning module is initialized using Xavier to ensure stable scoring calculation. LoRA adapter: Initialize matrices A and B as described in Section 3.2.1. Detection head (BoxHead): Initialize using a normal distribution (mean 0, variance 0.01) to ensure that the detection head parameters are adapted to the target size of the power equipment (e.g., insulator diameter approximately 10-20cm, conductor diameter approximately 2-5cm).

[0137] 4.2 Training Parameter Settings

[0138] 4.2.1 Learning Rate and Training Rounds

[0139] Learning rate strategy: A combination of linear warming and cosine annealing is adopted, with the following specific settings:

[0140] Initial low learning rate: 1e-6 (for the first 10 training epochs, to avoid initial weight oscillations); Linear warm-up: For the first 50 epochs, linearly increase from 1e-6 to the set target learning rate (exemplarily 2e-5), because the model is in the parameter adaptation period during the first 50 epochs, and gradually increasing the learning rate can accelerate convergence; Cosine annealing: From epochs 51 to 300 (total training epochs are the set number of epochs), reduce from 2e-5 to 1e-6, because the model is close to convergence in the later stages, and reducing the learning rate can avoid overfitting. At the same time, cosine annealing is more likely to find the optimal solution than a fixed learning rate; Learning rate adaptation: Set differentiated learning rates for different modules: Dynamic sparse attention module, LoRA adapter, detector head: learning rate = 2e-5 (key fine-tuning modules, requiring a higher learning rate); BERT shared layer weights: learning rate = 1e-5 (only fine-tuning adaptation, requiring a lower learning rate to avoid destroying generality).

[0141] 4.2.2 Batch Size and Optimizer

[0142] Batch size: Set the batch size to 8 for a single card (based on GPU memory limitations, 32GB of memory can support training 8 2048×1536 images simultaneously). When multiple cards are used in parallel, the batch size is increased by the number of cards (e.g., batch size for 2 cards = 16).

[0143] Optimizer selection: The AdamW optimizer is used with the following parameters: weight decay = 0.01 (to suppress overfitting; weight decay is not applied to the LoRA matrix and shared layer weights to avoid excessive parameter shrinkage); β1 = 0.9, β2 = 0.999 (standard AdamW momentum parameters to ensure stable gradient updates).

[0144] 4.3 Design of Combined Loss Function

[0145] To balance the accuracy of "category recognition", "boundary box localization", and "text-image alignment" in power equipment detection, a combined loss function is adopted, with a total loss L. total =L cls +L reg +L cross The loss design for each part is as follows:

[0146] 4.3.1 Classification Loss

[0147] Applicable scenarios: Solving the problem of imbalanced categories in power equipment datasets (e.g., 60% of images are of normal equipment, and 40% are of defective equipment); Formula: L cls =-α t (1-p t )γlogp t α tThis represents the category weight; normal equipment has a weight of 0.4, and defective equipment has a weight of 0.6. Increasing the weight of the defective category... t L represents the model's predicted class probability; γ represents the focusing parameter; γ=2 reduces the loss contribution of easily classified samples and focuses on difficult-to-classify samples such as "slight icing". Compared to traditional cross-entropy loss, L... cls This significantly improves the model's accuracy in identifying defective equipment.

[0148] 4.3.2 Regression Loss

[0149] Applicable scenarios: Improving the positioning accuracy of power equipment bounding boxes (such as small target equipment like insulators and pins, where bounding box offset can lead to detection failure); Formula: L reg =1-GIoU, GIoU=IoU-|C-(A∪B)| / |C|, where A is the predicted bounding box, B is the ground truth bounding box, and C is the minimum bounding box between A and B. Compared to traditional IoULoss, L reg It can handle situations where the predicted bounding box and the ground truth bounding box do not overlap (such as box offset caused by drone shooting angle deviation), and the bounding box positioning error is significantly reduced.

[0150] 4.3.3 Cross-modal contrast loss

[0151] Applicable Scenarios: Enhancing the alignment accuracy between text features and image features (e.g., the model needs to align the text "damaged insulator" with the damaged insulator target in the image); Implementation: Constructing positive and negative sample pairs: using "power equipment text description - image target" as the unit, text and image of the same target are positive sample pairs, and text and image of different targets are negative sample pairs (each positive sample is paired with 10 negative samples); Calculating contrastive loss: L cross =-log(exp(s pos / τ) / (exp(s pos / τ)+∑exp(s neg,i / τ)), where s pos For positive sample pairs, s neg,i The similarity is set to negative sample pairs; τ=0.1, a temperature parameter controlling the similarity distribution. This contrastive loss function significantly improves the accuracy of text-image alignment in the power equipment domain, addressing the problem of insufficient alignment of domain terms in traditional models.

[0152] 4.4 Fine-tuning of some parameters and multi-scale training

[0153] 4.4.1 Strategies for Fine-tuning Some Parameters

[0154] To avoid degradation of the model's general capabilities (such as a decrease in general object detection performance due to modifications to low-order feature layers of the SwinTransformer), a "freeze-fine-tuning" layered strategy is adopted:

[0155] Freeze parameters: Layers 1-5 of SwinTransformer (low-order feature layers): retain its general image feature extraction capability; Layers 1-5 of BERT (low-order semantic layers): retain its general text encoding capability.

[0156] Fine-tuning parameters: Dynamic sparse attention module (pruning module, axial attention layer); LoRA adapter and BERT 6-12 layers (high-order semantic layer); cross-modal fusion module and detection head (BoxHead).

[0157] Advantages: The number of trainable parameters accounts for only 15% of the total number of parameters in the model, reducing training time by about 40%, while ensuring a balance between the model's adaptability and versatility in power equipment detection tasks.

[0158] 4.4.2 Implement multi-scale training

[0159] To adapt to the resolution differences in equipment images in power inspection scenarios (e.g., high-resolution images of insulators taken by drones at close range, and low-resolution images of power towers taken at a distance), multi-scale training is adopted:

[0160] Size range: The input image size is dynamically adjusted during training, and the size range is set to 640×640~960×960 (for example, it includes five sizes: 640×640, 720×720, 800×800, 880×880, and 960×960).

[0161] Adjustment method: Randomly select a size every 10 rounds as the input size for the current round, and use "adaptive padding" (fill according to the image aspect ratio to avoid stretching and deformation); Advantages: The detection accuracy fluctuation of the model for power equipment images of different resolutions is reduced from ±8% to ±3%, significantly improving the generalization ability.

[0162] 4.5 Training Process Monitoring and Model Selection

[0163] 4.5.1 Monitoring Indicators and Log Recording

[0164] Key monitoring metrics: Training metrics: total loss, classification loss, regression loss, cross-modal loss for each training round, and mAP (mean precision) and recall for the training set; Validation metrics: every 50 rounds, mAP (calculated by device type, such as insulator mAP, conductor mAP) and FPS (frames per second, a preliminary assessment of inference speed) are evaluated on the validation set.

[0165] Log recording: Tensorboard is used to visualize training curves in real time, and the model weights (.pth format) and evaluation reports for each round are saved. The recorded content includes: Weight file: named in the format "model_epochXXX_mAPXXX.pth" (XXX is the round number and mAP value); Evaluation report: includes mAP, Recall, FPS for each device type, and loss curve trend analysis.

[0166] 4.5.2 Selecting the Optimal Model

[0167] Selection criteria: The validation set mAP is used as the core indicator, while FPS is also taken into account (it must meet the real-time requirements of edge devices). The model with the highest mAP and FPS ≥ the set threshold is selected as the optimal model. Model validation: The optimal model is finally evaluated on the test set to ensure that the difference between the test set mAP and the validation set mAP is <2% (to avoid overfitting). At the same time, the model's open set detection capability for unseen power equipment types (such as new smart sensors) is tested (zero-sample mAP decrease <10%).

[0168] V. Edge Device Deployment Optimization

[0169] To address the computing power and memory limitations of edge devices used in power line inspection (such as Jetson Xavier NX mounted on drones), a method involving "model format conversion, hardware adaptation, dynamic parameter adjustment, and performance verification" is employed to achieve efficient model deployment and real-time detection. The specific details are as follows:

[0170] 5.1 Model Format Conversion and Quantization Optimization

[0171] 5.1.1 ONNX Format Conversion

[0172] To improve model compatibility on edge devices (most edge inference frameworks support ONNX format), the trained PyTorch model (.pth format) is converted to ONNX format. The specific process is as follows:

[0173] Conversion tool: The `torch.onnx.export` function, which comes with PyTorch, is used. The parameters are set as follows: `input_names`: Input node names ("image" for image input, "text" for text input); `output_names`: Output node names ("boxes" for bounding box output, "scores" for confidence score output, "labels" for category output); `dynamic_axes`: Dynamic dimension setting (image input dimension is set to dynamic, supporting resolutions from 640×640 to 960×960); `opset_version`: Operator version is set to 12 (compatible with the TensorRT version of Jetson Xavier NX).

[0174] Operator fusion: During the transformation process, “Conv-BN-ReLU” operator fusion is enabled (implemented by do_constant_folding=True via torch.onnx.export), which combines convolution, batch normalization, and activation functions into one operator, reducing the number of operator calls during inference (improving inference speed by about 10%).

[0175] 5.1.2 INT8 Quantization Optimization

[0176] To further reduce model storage size and computational load (edge ​​devices have limited memory, such as Jetson XavierNX with 8GB of memory), INT8 quantization is performed on the ONNX model. Specific implementation details are as follows:

[0177] Quantization tool: TensorRT's INT8 quantization tool is used, which supports "calibration quantization" (requires a small amount of calibration data); Calibration data selection: 100 images are randomly selected from the test set as calibration data, covering different device types and scenarios (to ensure that the quantized model is compatible with all scenarios).

[0178] Quantization process: Building the calibrator: Implementing the IInt8Calibrator interface, loading calibration data and propagating forward to collect the activation value distribution of each layer of the model; Calculating quantization parameters: Calculating the quantization scale and zero point of each layer based on the activation value distribution (ensuring minimal quantization error); Generating the quantized model: Converting the ONNX model to an INT8 quantized model, reducing the storage volume from approximately 500MB to 166MB (a reduction of approximately 67%), while ensuring that the mAP decreases by ≤0.5% after quantization (meeting accuracy requirements).

[0179] 5.2 Edge device hardware adaptation

[0180] Hardware adaptation and optimization were performed using Jetson XavierNX (a commonly used edge device for power line inspection, with an 8-core ARMv8.2 CPU, a 128-core Volta architecture GPU, and 8GB of memory) as the target device:

[0181] 5.2.1 TensorRT Inference Acceleration

[0182] Inference Engine Construction: The inference engine is built using TensorRT 8.4, with the following parameters set: Inference Accuracy: FP16 accuracy is preferred (balancing accuracy and speed; compared to FP32, inference speed is improved by about 2 times, and mAP decreases by ≤0.3%); Workspace Size: Set to 8GB (maximizing the use of Jetson XavierNX's memory to avoid insufficient memory during inference); Maximum Batch Size: Set to 1 (power line inspection is a real-time single-frame detection, no batch processing is required).

[0183] Engine optimization: Enable TensorRT's "Layer Fusion" and "Auto Kernel Tuning" to optimize the GPU kernel for Jetson Xavier NX's TensorCore architecture, improving computing power utilization (GPU utilization increased from 60% to 85%).

[0184] 5.2.2 Memory Optimization Strategy: To avoid memory overflow during model loading and inference (Jetson Xavier NX has limited memory), a strategy of "memory paging loading" and "feature map cache reuse" is adopted:

[0185] Memory paging loading: The model weights are split into layers (such as storing each layer's weights separately in SwingTransformer). During inference, they are loaded into memory layer by layer. After the inference of the previous layer is completed, the memory of the weights of that layer is released, and then the weights of the next layer are loaded. The peak memory usage is reduced from 8GB to 4.5GB.

[0186] Feature map cache reuse: In axial attention calculation, intermediate feature maps of horizontal and vertical axis attention are cached and reused (e.g., the feature map of horizontal axis attention can be used as the input reference of vertical axis attention), reducing the amount of feature map storage (memory usage is reduced by about 20%).

[0187] 5.3 Dynamic Pruning Adjustment and Real-time Adaptation

[0188] 5.3.1 Dynamic Pruning Triggering Conditions

[0189] The pruning threshold α is dynamically adjusted based on the real-time computing load of Jetson Xavier NX (CPU utilization is monitored via NVIDIA Jetson Utilities tools). Specific triggering conditions are as follows:

[0190] High-precision mode: When CPU utilization is <70% (sufficient computing power), α is set to the high-precision retention value (for example, α=0.6), retaining more tokens (e.g., 85% for Stage 1) to ensure detection accuracy (mAP≥89%). High-speed mode: When CPU utilization is >90% (computing power is tight, such as when a drone is simultaneously transmitting images and detecting), α is set to the high-speed retention value (for example, α=0.3), discarding more redundant tokens (e.g., 55% for Stage 1) to prioritize real-time performance (FPS≥17 frames / s). Balanced mode: When CPU utilization is between 70% and 90%, α is set to the middle value (for example, α=0.45) to balance accuracy and speed.

[0191] 5.3.2 Dynamic Adjustment Implementation

[0192] Monitoring frequency: CPU utilization is checked every 10 frames, and the α value is adjusted according to the utilization change; Smooth transition: To avoid fluctuations in detection results caused by abrupt changes in α (such as sudden offset of bounding boxes), the "exponential moving average" is used to smooth the α value, that is, new α value = 0.7 × old α value + 0.3 × target α value, to ensure that the α adjustment process is smooth; Effect verification: After dynamic adjustment, the FPS fluctuation of the model in the computing power fluctuation scenario is reduced from ±3 frames / s to ±1 frame / s, and the stability of the detection results is significantly improved.

[0193] 5.4 Deployment Performance Verification

[0194] 5.4.1 Single Image Detection Test

[0195] Test data: A set number of images (exemplary, 1000 images) are randomly selected from the test set, covering 5 types of core power equipment, 8 types of typical defects, and 6 types of complex scenarios.

[0196] Test metrics: Inference speed: Calculate the average inference time for 1000 images, converted to FPS, and it must be ≥ the set image inference speed threshold (for example, 17.6 frames / s); Detection accuracy: Calculate the mAP of various devices and the overall mAP, and the overall mAP must be ≥ the set accuracy threshold (for example, 89.1%); Defect recognition rate: For key defects such as broken strands and insulator damage, the recall rate must be ≥90%.

[0197] Test results: For example, the average inference speed per image is 18.2 frames / s, the overall mAP is 89.5%, and the critical defect recall rate is 92%, which meets the accuracy and speed requirements of power line inspection.

[0198] 5.4.2 Real-time Video Detection Test

[0199] Test data: Power line inspection video with a set resolution (exemplary, 1080P resolution, frame rate 30 frames / s) was taken by a drone. The video is 30 minutes long and includes two scenes: high-voltage line inspection and substation equipment inspection.

[0200] Test metrics: Video inference speed: ≥ set video inference speed threshold (e.g., 14.1 frames / s), ensuring no stuttering (stuttering rate < 1%); Defect recognition latency: latency from image frame acquisition to defect result output, ≤ set latency threshold (e.g., 100ms), ensuring real-time feedback; Continuous operation stability: continuous operation for 30 minutes, monitoring memory leaks and computing power usage, requiring memory fluctuation < 5% and no program crashes.

[0201] Test results: For example, the video inference speed is 14.8 frames / s, the defect recognition latency is 85ms, and the memory is stable after 30 minutes of continuous operation (fluctuation of 3%), which meets the real-time and stability requirements of power inspection.

[0202] 5.4.3 Open set detection capability test

[0203] To verify the model's adaptability to unseen types of power equipment (such as novel smart sensors), a "zero-shot test" was employed:

[0204] Test data: Collect 100 images of new power equipment (not used in training) and label the equipment type and status.

[0205] Test metric: mAP in the zero-sample case, requiring a decrease of ≤ a set decrease threshold (e.g., 10%).

[0206] Test results: For example, the zero-sample mAP is 81.2%, which is 8.3% lower than the known equipment mAP (89.5%), meeting the open set detection requirements and reducing the labeling and fine-tuning costs of adding new equipment.

[0207] VI. Summary of Implementation Results

[0208] Through the above specific embodiments, the present invention achieves the following technical effects and solves the core technical problems in the field of power equipment inspection:

[0209] Significantly reduced computational complexity: By leveraging axial attention decomposition technology, the computational complexity of the traditional Transformer is reduced from O(n²) to O(2n). At the same time, combined with dynamic token pruning strategy, redundant computation is further reduced, resulting in a significant reduction in model FLOPs and making it fully adaptable to the limited computing resources of edge devices.

[0210] Parameter and storage volume optimization: Through LoRA adapter and inter-layer weight sharing strategy, the number of model parameters is significantly reduced; after INT8 quantization optimization, the model storage volume is controlled within a small range, without occupying too much edge device storage resources, which facilitates rapid deployment.

[0211] Balancing detection accuracy and real-time performance: On edge devices such as Jetson Xavier NX, the model can achieve efficient single-image inference and real-time video inference while maintaining high overall detection accuracy. This meets the requirements of power line inspection for accurate defect identification and ensures real-time feedback.

[0212] Enhanced open set detection capability: For newly added power equipment types, the model's accuracy decline in zero-sample detection scenarios is kept to a low level. It can quickly adapt to new equipment without a large amount of labeled data and repeated fine-tuning, significantly reducing data labeling costs and model iteration cycles.

[0213] Strong adaptability to complex scenarios: Through multi-scale training and data augmentation strategies for complex scenarios, the model can keep the fluctuation of detection accuracy within a small range in typical complex scenarios of power inspection, such as occlusion, severe weather, and changes in lighting. It can stably cope with diverse inspection environments and ensure reliable detection results.

[0214] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A lightweight modeling method for the power industry using Grounding-DINO combined with a dynamic sparse attention mechanism, characterized in that, This includes the following lightweighting steps: S1. Construct a power dataset adapted to the power industry. The power dataset includes images of high-voltage lines and substation equipment, and after annotation and data augmentation, it is divided into training set, validation set and test set according to a set ratio. S2. Design a dynamic sparse attention mechanism, embed a pruning module in the image encoder SwinTransformer of the Grounding-DINO model to dynamically filter the visual tokens of the feature map obtained during processing, and reduce the computational complexity of the model by axial attention decomposition. The dynamic filtering specifically includes: embedding pruning modules in each downsampling layer of SwinTransformer, and the retention ratio of visual tokens in each downsampling layer is a set differentiation ratio; the pruning module filters key visual tokens by calculating the importance score of the visual tokens, and the filtering criteria are to retain visual tokens with an importance score higher than a set threshold and discard visual tokens with an importance score lower than the threshold. The formula for calculating the importance score of visual tokens is as follows: ； In the formula, t i Score(t) represents the i-th visual token. i ) represents t i Importance score; H represents the number of attention heads in the SwinTransformer; N represents the length of the token sequence in the current downsampling layer; Attn h (I,j) represents the attention weight between the i-th visual token and the j-th visual token in the h-th attention head; The set threshold satisfies: Threshold = α max(Scores), where Scores represents the set of importance scores for all visual tokens in the current layer; α represents the pruning threshold; max( () indicates the operation of retrieving the maximum value; The reduction in computational complexity through axial attention decomposition specifically includes: decomposing the two-dimensional feature map output by the Swin Transformer into two one-dimensional sequences along the height and width axes, calculating the horizontal axis attention along the row and the vertical axis attention along the column respectively, and then adding the attention outputs in the two directions element by element to obtain the final feature map; and in the process of axial attention calculation, introducing a learnable relative position encoding consistent with the token feature dimension, which is initialized by a set method and updated as the model is trained. S3. Design a LoRA adapter and weight sharing scheme. Insert a low-rank adapter into the text encoder BERT of the Grounding-DINO model and optimize the text encoding process by combining the inter-layer weight sharing strategy. The inter-layer weight sharing strategy is as follows: within the scope of the higher-order semantic coding layers defined in BERT, inter-layer weight reuse is adopted, with each subsequent layer reusing the basic or optimized weights of the previous layer; and during the model inference phase, the weight increment ΔW of LoRA is calculated using ΔW=A×B, and then W is used to... nem =W pretrained +△W will increment the weights by △W and then add the pre-trained weights W of BERT. pretrained Merge to obtain the new weight W for inference. nem ; S4. Train and fine-tune the Grounding-DINO model. Initialize the Grounding-DINO model based on the pre-trained weights, adopt a partial parameter fine-tuning strategy, and complete the model training by combining a specific loss function and training parameters. S5. Optimize the training model for edge device deployment, including model format conversion, hardware adaptation and dynamic parameter adjustment, and verify the detection performance after deployment.

2. The lightweight modeling method for Grounding-DINO in the power field, combining dynamic sparse attention mechanism as described in claim 1, is characterized in that... The sub-steps for designing the LoRA adapter and weight sharing scheme are as follows: S3A1. Insert a low-rank adapter LoRA into the self-attention layer of BERT. LoRA consists of matrix A and matrix B. The dimension of matrix A is d×r and the dimension of matrix B is r×d. d is the feature dimension of BERT and r is the low-rank dimension. S3A2. Initialize LoRA. Matrix A is initialized using a random Gaussian distribution with a set mean and a set variance, and matrix B is initialized as a zero matrix. S3A3: Freeze the pre-trained weights of the BERT backbone network, retaining only the trainable parameters of the LoRA adapter and higher-order semantic coding layers.

3. The lightweight modeling method for Grounding-DINO in the power field, combining dynamic sparse attention mechanism as described in claim 2, is characterized in that... The specific steps for optimizing the edge device deployment of the trained model are as follows: S5A1, Model Format Conversion: Convert the trained model to ONNX format, enable operator fusion during conversion, and then perform quantitative optimization on the ONNX model with set precision. S5A2, Hardware Adaptation: For target edge devices, TensorRT is used to accelerate inference, the inference accuracy is set to the set accuracy, the workspace size is set to the set space value, and a memory paging loading mechanism is used to load the model weights into the device memory layer by layer in batches. S5A3, Dynamic Pruning Adjustment: Adjust the pruning threshold α according to the real-time computing load of the edge device. That is, when the CPU utilization is lower than the set low threshold, α takes the set high precision retention value; when the CPU utilization is higher than the set high threshold, α takes the set high speed retention value.

4. The lightweight modeling method for Grounding-DINO in the power field, combining dynamic sparse attention mechanism as described in claim 1, is characterized in that... The sub-steps of step S4 are as follows: S41. Initialize the model backbone network using the original pre-trained weights of Grounding-DINO; S42. The initial learning rate is set to a set value, and the total number of training rounds is set to a set number of rounds. In the first set number of rounds, a linear warming strategy is used to raise the learning rate from the set initial low learning rate to the set target learning rate. In the last set number of rounds, a cosine annealing strategy is used to lower the learning rate from the set target learning rate to the set final low learning rate. S43. Construct a combined loss function, which includes FocalLoss for class imbalance optimization, GIoULoss for bounding box accuracy improvement, and InfoNCE cross-modal contrast loss for text-image alignment. S44. Perform partial parameter fine-tuning: Freeze the settings of the Swin Transformer's pre-layer and the BERT backbone network, and only fine-tune the parameters of the dynamic sparse attention module, LoRA adapter, cross-modal fusion module and detection head. S45. Multi-scale training is adopted, and the input image size is dynamically adjusted during the training process. The size range is the set image size interval.

5. A lightweight model simplification method for Grounding-DINO in the power field, combining dynamic sparse attention mechanism as described in claim 4, characterized in that... The sub-steps of step S1 are as follows: S11. Collect power data sets through appropriate technical means. The power data sets include a set number of high-voltage lines, substation equipment, and images of obstructions, rain, snow, and nighttime conditions. S12. Use the LabelImg tool to manually annotate the image, including the device type, target bounding box location, and abnormal status. S13. Expand the labeled image through multi-dimensional data enhancement. Multi-dimensional data enhancement methods include: translation amplitude within a set range, cropping within a set ratio range, random horizontal / vertical flipping with a set probability, random rotation within a set angle range, contrast adjustment within a set multiple range, brightness adjustment within a set multiple range, saturation adjustment within a set multiple range, Gaussian noise addition within a set variance range, fog effect within a set concentration range, and rain line simulation within a set density range. S14. Divide the augmented images into training set, validation set and test set according to a set ratio.

6. A lightweight model simplification method for Grounding-DINO in the power field, combining dynamic sparse attention mechanism as described in claim 5, is characterized in that... The following steps are performed to verify the detection performance after deployment: Two types of tests are executed on the target edge device: The first type of test: single image test: select a set number of images in the test set for iterative inference, the average inference speed is not lower than the set image inference speed threshold, and the average accuracy is not lower than the set accuracy threshold. The second type of test: video detection test: real-time detection of power inspection videos at a set resolution, with the inference speed not lower than the set video inference speed threshold and the defect identification delay not exceeding the set delay threshold.

Images