Transformer model lightening method and power transmission line image analysis method

Abstract

The application discloses a kind of Transformer model light weight method and power transmission line image analysis method, including obtaining target Transformer model;Sensitivity analysis is carried out to target Transformer model;The generation of pruning strategy is carried out to target Transformer model;Modeling is carried out to hardware environment and pruning strategy is optimized according to modeling result;Importance evaluation is carried out and dynamic bit allocation and joint optimization are carried out according to evaluation result, to complete the light weight of target Transformer model.The application is analyzed, modeled and optimized to target Transformer model, not only completes the light weight of target Transformer model, but also higher reliability, better accuracy, and better effect.

Description

Technical Field

[0001] This invention belongs to the field of digital signal processing, specifically relating to a lightweight method for Transformer models and a method for analyzing images of power transmission lines. Background Technology

[0002] With the rapid development of deep learning technology, the Transformer model has achieved remarkable results in fields such as natural language processing and computer vision. However, the high computational complexity and large size of the Transformer model, along with its ever-increasing demands for storage space, RAM, and computing resources, have become major bottlenecks limiting its deployment on edge devices.

[0003] Currently, methods for lightweighting Transformer models mainly include model pruning, knowledge distillation, and quantization. Among these, pruning techniques, by removing redundant weights or neurons from the network to reduce model size and computational requirements, have become an effective model compression method. Existing pruning techniques are mainly divided into two categories: structured pruning and unstructured pruning. Unstructured pruning operates directly on individual weights, which can achieve high accuracy, but it produces irregular sparse matrices, leading to irregular memory access and computation patterns, and increasing storage access latency. While structured pruning is more suitable for hardware acceleration, because each pruning involves a large number of objects, the model accuracy drops rapidly as the pruning rate increases.

[0004] As a result, existing Transformer models and their corresponding lightweight solutions suffer from poor reliability and accuracy when performing image analysis of power system transmission lines, severely impacting the normal and stable operation of the power system. This deficiency is particularly pronounced when these models are deployed on drones and monitoring equipment. Summary of the Invention

[0005] The purpose of this invention is to provide a lightweight method for Transformer models that is highly reliable, accurate, and effective.

[0006] The second objective of this invention is to provide a transmission line image analysis method that includes the aforementioned Transformer model lightweighting method.

[0007] The lightweight Transformer model method provided by this invention includes the following steps:

[0008] S1. Obtain the target Transformer model;

[0009] S2. Based on model hierarchy sensitivity analysis, attention head importance analysis, and channel importance analysis, perform sensitivity analysis on the target Transformer model;

[0010] Specifically, based on a random noise scheme, sensitivity analysis is performed on each layer of the target Transformer model to assess the degree of influence of the corresponding layer on the output of the target Transformer model; based on mutual information and attention graph entropy schemes, the importance of each attention head of the target Transformer model is evaluated to assess the contribution of the corresponding attention head to the model output; based on the L1 norm, the importance of the channels of each linear layer of the target Transformer model is evaluated.

[0011] S3. Based on the set pruning scheme and sensitivity analysis scheme, generate the pruning strategy for the target Transformer model;

[0012] S4. For the hardware environment in which the target Transformer model runs, model the hardware environment and optimize the pruning strategy based on the modeling results;

[0013] S5. Evaluate the importance of the parameters of the target Transformer model, and perform dynamic bit allocation and joint optimization based on the evaluation results to achieve the lightweighting of the target Transformer model.

[0014] Step S2 specifically includes the following steps:

[0015] Hierarchical sensitivity analysis:

[0016] Random noise is added to each layer of the target Transformer model, and the corresponding performance changes are obtained. The sensitivity value of the corresponding layer is calculated using the following formula:

[0017]

[0018] In the formula The sensitivity value of the layer; This represents the model's original accuracy. The accuracy of the model after adding noise; Let L2 be the noise norm; The L2 norm of the layer's output;

[0019] Attention-based head importance assessment:

[0020] For each attention head in the model, the corresponding mutual information is calculated using the following formula:

[0021]

[0022] In the formula X represents the mutual information of the attention heads; X is the input of the attention head; Y is the output of the attention head. Let X be the joint probability distribution of the input X and the output Y. The marginal probability distribution of input X; The marginal probability distribution of output Y;

[0023] For each attention head in the model, the corresponding attention map entropy is calculated using the following formula:

[0024]

[0025] In the formula, H is the attention map entropy of the attention head; This is a single attention weight value in the attention map, which represents the degree of attention one element pays to another element in the model;

[0026] For each attention head in the model, the corresponding attention head influence score is calculated using the following formula:

[0027]

[0028] In the formula The attention of the head affects the score; The performance degradation of the target Transformer model after resetting the weights of the attention head to zero;

[0029] For each attention head in the model, the importance evaluation result of the corresponding attention head is calculated using the following formula:

[0030]

[0031] In the formula The results of the attention head importance assessment; This is the first weight value set. This is the set second weight value; This is the set third weight value;

[0032] Channel importance assessment:

[0033] For each channel in the model, the corresponding channel importance value is calculated using the following formula:

[0034]

[0035] In the formula This represents the channel importance value. The weight value corresponding to the channel; It is an L1 norm;

[0036] The correlation between any two channels can be calculated using the following formula:

[0037]

[0038] In the formula The correlation between the i2 channel and the j2 channel; This is the function for calculating cosine similarity. This represents the weight value of the i2 channel; This represents the weight value of the j2 channel.

[0039] Step S3 includes the following steps:

[0040] For the attention head, channels, blocks, and elements of the Transformer model, several candidate pruning granularity schemes are generated;

[0041] For each attention head, channel, block, and element in the target Transformer model, calculate the corresponding sensitivity-efficiency index based on sensitivity and efficiency, and select the candidate pruning granularity scheme with the highest sensitivity-efficiency index as the corresponding pruning granularity scheme.

[0042] For each attention head, channel, block, and element in the target Transformer model, the corresponding pruning ratio is calculated based on the corresponding sensitivity value.

[0043] Step S3 specifically includes the following steps:

[0044] Candidate pruning granularity scheme generation:

[0045] The generated candidate pruning granularity schemes include attention head-level pruning, channel-level pruning, block-level pruning, and element-level pruning. Among them, attention head-level pruning is used to directly remove the entire attention head; channel-level pruning is used to remove the entire channel; block-level pruning is used to remove blocks that include several channels; and element-level pruning is used to retain the K most important elements in each sub-row and remove all remaining elements according to the Top-K strategy.

[0046] Selection of pruning particle size scheme:

[0047] For each attention head, channel, block, and element in the target Transformer model, the sensitivity-efficiency index of the corresponding candidate pruning granularity scheme is calculated using the following formula:

[0048]

[0049] In the formula The sensitivity-efficiency index when using candidate pruning granularity scheme g; The sensitivity enhancement value when using candidate pruning granularity scheme g; The computational efficiency improvement when using candidate pruning granularity scheme g; This represents the maximum sensitivity improvement value among all candidate pruning granularity schemes. This represents the maximum computational efficiency improvement among all candidate pruning granularity schemes. This is the fourth weight value that is set; This is the fifth weight value that is set;

[0050] Determining the pruning ratio:

[0051] The pruning ratio is calculated using the following formula:

[0052]

[0053] In the formula The pruning ratio for the i3th part; The set basic pruning ratio; The sensitivity of the i3th part after normalization;

[0054] Considering accuracy constraints, a binary search scheme is used to obtain the maximum pruning ratio that meets the set accuracy requirements:

[0055] An optimization method based on binary search is employed to determine the optimal basic pruning ratio under specific constraints. ;

[0056] Precision constraints are expressed as:

[0057]

[0058] In the formula The accuracy on the defined validation set; This represents the accuracy of the original model. The set threshold for the decrease;

[0059] The maximum pruning ratio is obtained by following these steps:

[0060] (1) Initialize the search interval: set Search scope: lower bound The upper bound of the search is 0. =1;

[0061] (2) Iterative search: Within a preset number of iterations, or when the difference between the upper and lower bounds is less than a set minimum value, repeat the following operation:

[0062] a. Calculate the pruning ratio based on the midpoint of the interval as a candidate base. for ;

[0063] b. As Substitute the branches into the pruning ratio formula to calculate the pruning ratio of each part of the model. ;

[0064] c. Based on the obtained The model is pruned, and the pruned model is adjusted to restore its performance.

[0065] d. Calculate the accuracy of the adjusted model on the defined validation set. ;

[0066] e. If If the set precision constraints are met, then update the search lower bound. for ;like If the set precision constraints are not met, update the upper bound of the search. for ;

[0067] (3) Determine the final ratio: After the iterative search is completed, the lower bound of the search is obtained. As the final .

[0068] Step S4 includes the following steps:

[0069] Hardware performance modeling is performed on the target Transformer model's running hardware environment, based on computational cost, memory access cost, and data locality.

[0070] Based on the established hardware performance model, the pruning granularity scheme is optimized with the goal of maximizing hardware performance indicators and with accuracy as the constraint.

[0071] Choose the optimal sparse representation format based on the hardware environment in which the target Transformer model runs.

[0072] Step S4 specifically includes the following steps:

[0073] Hardware performance modeling:

[0074] The following formula is used as a hardware performance model to predict the execution time of the pruned Transformer model on the target hardware. :

[0075]

[0076] In the formula, l is the index of the model layer; L is the total number of model layers; g represents a pruning granularity scheme; This represents the number of floating-point operations in the l-th layer under the pruning granularity scheme g. The peak floating-point performance of the target hardware; To determine the amount of memory data required to execute the l-th layer under the pruning granularity scheme g; The memory bandwidth of the target hardware;

[0077] Optimization of pruning granularity scheme:

[0078] The following formula is used as the objective function for optimizing the pruning granularity scheme:

[0079]

[0080] The following formula is used as a constraint condition for optimizing the pruning granularity scheme:

[0081]

[0082] In the formula The accuracy achieved on the set validation set after the model adopts and adjusts the pruning granularity scheme g. This is the minimum acceptable accuracy rate set.

[0083] Selection of sparse representation format:

[0084] If the hardware environment in which the target Transformer model runs supports structured sparse computation, then structured pruning is preferred.

[0085] If the hardware environment in which the target Transformer model is running does not support structured sparse computation, then unstructured pruning is preferred.

[0086] Based on the characteristics of the target Transformer model's running hardware environment, adjust the size of the target Transformer model's Block so that the adjusted Block size matches the size of the computing unit of the target Transformer model's running hardware.

[0087] Step S5 includes the following steps:

[0088] The importance of the parameters of the target Transformer model is evaluated based on the importance of pruning and the importance of quantification.

[0089] Based on the importance assessment results, different numbers of quantization bits are assigned to different parameters;

[0090] With minimizing the model size as the optimization objective and accuracy as the constraint, a joint optimization model for optimizing pruning and quantization parameters is constructed and solved to achieve lightweighting of the target Transformer model.

[0091] Step S5 specifically includes the following steps:

[0092] Parameter importance assessment:

[0093] The parameter importance of each parameter in the target Transformer model is calculated using the following formula:

[0094]

[0095] In the formula The result of the parameter importance assessment; The importance of pruning results, and , For a single weight parameter in the model; To quantify the importance of the results, and , For the loss function L with respect to the parameters The second-order partial derivative; This is the set sixth weight value; This is the set seventh weight value;

[0096] Quantization bit allocation:

[0097] The number of quantization bits allocated is calculated using the following formula:

[0098]

[0099] In the formula The number of quantization bits allocated; The minimum number of quantization bits to be allocated; The number of quantization bits allocated to the set baseline; The importance of the parameters after normalization;

[0100] Joint optimization:

[0101] The following formula is used as the objective function of the joint optimization model:

[0102]

[0103] In the formula, S is the set of pruning decision variables; B is the set of bit allocation variables; and N is the total number of model parameters. It is a binary variable, and when the i-th parameter is reserved... When the i-th parameter is not retained ; The number of quantization bits allocated to the i-th parameter;

[0104] The following formula is used as the constraint condition for the joint optimization model:

[0105]

[0106] In the formula The accuracy achieved by the model on a given validation set under pruning decisions S and bit allocation B;

[0107] The joint optimization model is solved using the alternating direction multiplier method.

[0108] The present invention also provides a transmission line image analysis method including the aforementioned lightweight Transformer model method, comprising the following steps:

[0109] A. Obtain the trained Transformer model for transmission line image analysis; this model can analyze images captured by drones or monitoring towers to identify defects or abnormal conditions in key components such as insulators, vibration dampers, and conductors.

[0110] B. Using the Transformer model obtained in step A as the target Transformer model, perform lightweighting processing using the aforementioned Transformer model lightweighting method;

[0111] C. Deploy the lightweight Transformer model obtained in step B to achieve image analysis of transmission lines.

[0112] The lightweight Transformer model and transmission line image analysis method provided by this invention not only achieves lightweighting of the target Transformer model by performing multi-dimensional analysis, modeling and optimization, but also has higher reliability, better accuracy and better results. Attached Figure Description

[0113] Figure 1 This is a schematic diagram of the lightweight method of the present invention.

[0114] Figure 2 This is a schematic diagram comparing the accuracy and compression ratio of the lightweighting method of the present invention with existing solutions.

[0115] Figure 3 This is a schematic diagram comparing the FLOPs and compression ratio of the lightweighting method of the present invention with existing solutions.

[0116] Figure 4This is a schematic diagram of the process flow of the transmission line image analysis method of the present invention. Detailed Implementation

[0117] like Figure 1 The diagram shows a flowchart of the lightweighting method of the present invention: The lightweighting method for Transformer models disclosed in this invention includes the following steps:

[0118] S1. Obtain the target Transformer model;

[0119] S2. Based on model hierarchy sensitivity analysis, attention head importance analysis, and channel importance analysis, perform sensitivity analysis on the target Transformer model; including the following steps:

[0120] Based on a random noise scheme, sensitivity analysis is performed on each layer of the target Transformer model to assess the degree of influence of the corresponding layer on the output of the target Transformer model.

[0121] Based on the mutual information and attention graph entropy scheme, the importance of each attention head of the target Transformer model is evaluated to assess the contribution of the corresponding attention head to the model output.

[0122] Based on the L1 norm, the importance of channels in each linear layer of the target Transformer model is evaluated.

[0123] In practice, the following steps can be taken:

[0124] Hierarchical sensitivity analysis:

[0125] Random noise is added to each layer of the target Transformer model, and the corresponding performance changes are recorded. Larger changes indicate a more sensitive layer. The sensitivity value for each layer is calculated using the following formula:

[0126]

[0127] In the formula The sensitivity value of the layer; This represents the model's original accuracy. The accuracy of the model after adding noise; Let L2 be the noise norm; The L2 norm of the layer's output;

[0128] Attention-based head importance assessment:

[0129] For each attention head in the model, the corresponding mutual information is calculated using the following formula:

[0130]

[0131] In the formula X represents the mutual information of the attention heads; X is the input of the attention head; Y is the output of the attention head. Let X be the joint probability distribution of the input X and the output Y. The marginal probability distribution of input X; This represents the marginal probability distribution of the output Y.

[0132] For each attention head in the model, the corresponding attention map entropy is calculated using the following formula:

[0133]

[0134] In the formula, H is the attention map entropy of the attention head; This is a single attention weight value in the attention map, which represents the degree of attention one element pays to another element in the model;

[0135] For each attention head in the model, the corresponding attention head influence score is calculated using the following formula:

[0136]

[0137] In the formula The attention of the head affects the score; The performance degradation of the target Transformer model after resetting the weights of the attention head to zero;

[0138] For each attention head in the model, the importance evaluation result of the corresponding attention head is calculated using the following formula:

[0139]

[0140] In the formula The results of the attention head importance assessment; The first weight value is set (preferably 0.3); The second weight value is set (preferably 0.3); The third weight value is set (preferably 0.4).

[0141] Channel importance assessment:

[0142] For each channel in the model, the corresponding channel importance value is calculated using the following formula:

[0143]

[0144] In the formula This represents the channel importance value. The weight value corresponding to the channel; It is an L1 norm;

[0145] Simultaneously considering the correlation between channels, complementary channels with low similarity are avoided from being pruned; the correlation between any two channels is calculated using the following formula:

[0146]

[0147] In the formula The correlation between the i2 channel and the j2 channel; This is the function for calculating cosine similarity. This represents the weight value of the i2 channel; This represents the weight value of the j2 channel.

[0148] S3. Based on the defined pruning scheme and sensitivity analysis scheme, generate the pruning strategy for the target Transformer model; including the following steps:

[0149] For the attention head, channels, blocks, and elements of the Transformer model, several candidate pruning granularity schemes are generated;

[0150] For each attention head, channel, block, and element in the target Transformer model, calculate the corresponding sensitivity-efficiency index based on sensitivity and efficiency, and select the candidate pruning granularity scheme with the highest sensitivity-efficiency index as the corresponding pruning granularity scheme.

[0151] For each attention head, channel, block, and element in the target Transformer model, the corresponding pruning ratio is calculated based on the corresponding sensitivity value.

[0152] In practice, the following steps can be taken:

[0153] Candidate pruning granularity scheme generation:

[0154] The generated candidate pruning granularity schemes include attention head-level pruning, channel-level pruning, block-level pruning, and element-level pruning. Among them, attention head-level pruning is used to directly remove the entire attention head; channel-level pruning is used to remove the entire channel; block-level pruning is used to remove blocks that include several channels; and element-level pruning is used to retain the K most important elements in each sub-row and remove all remaining elements according to the Top-K strategy.

[0155] Selection of pruning particle size scheme:

[0156] For each attention head, channel, block, and element in the target Transformer model, the sensitivity-efficiency index of the corresponding candidate pruning granularity scheme is calculated using the following formula:

[0157]

[0158] In the formula The sensitivity-efficiency index when using candidate pruning granularity scheme g; The sensitivity enhancement value when using candidate pruning granularity scheme g; The computational efficiency improvement when using candidate pruning granularity scheme g; This represents the maximum sensitivity improvement value among all candidate pruning granularity schemes. This represents the maximum computational efficiency improvement among all candidate pruning granularity schemes. This is the fourth weight value that is set; This is the fifth weight value that is set;

[0159] Ultimately, the choice The pruning granularity scheme with the highest value is selected as the corresponding scheme.

[0160] Determining the pruning ratio:

[0161] Areas with high sensitivity are allocated a lower pruning ratio, while areas with low sensitivity are allocated a higher pruning ratio; the pruning ratio is calculated using the following formula:

[0162]

[0163] In the formula The pruning ratio for the i3th part; The set basic pruning ratio; The sensitivity of the i3th part after normalization;

[0164] Considering accuracy constraints, a binary search scheme is used to obtain the maximum pruning ratio that meets the set accuracy requirements:

[0165] An optimization method based on binary search is employed to determine the optimal basic pruning ratio under specific constraints. ;

[0166] Precision constraints are expressed as:

[0167]

[0168] In the formula The accuracy on the defined validation set; This represents the accuracy of the original model. The set threshold for the decrease;

[0169] The maximum pruning ratio is obtained by following these steps:

[0170] (1) Initialize the search interval: set Search scope: lower bound The upper bound of the search is 0. =1;

[0171] (2) Iterative search: Within a preset number of iterations, or when the difference between the upper and lower bounds is less than a set minimum value, repeat the following operation:

[0172] a. Calculate the pruning ratio based on the midpoint of the interval as a candidate base. for ;

[0173] b. As Substitute the branches into the pruning ratio formula to calculate the pruning ratio of each part of the model. ;

[0174] c. Based on the obtained The model is pruned, and the pruned model is adjusted to restore its performance.

[0175] d. Calculate the accuracy of the adjusted model on the defined validation set. ;

[0176] e. If If the set precision constraints are met, then update the search lower bound. for ;like If the set precision constraints are not met, update the upper bound of the search. for ;

[0177] (3) Determine the final ratio: After the iterative search is completed, the lower bound of the search is obtained. As the final ;

[0178] This step is related to the previous calculation of the pruning ratio. The relationship is one of global optimization and local application: the aforementioned formula defines the rules for how the pruning amount is distributed among different parts of the model based on sensitivity, but the basic proportions within it... It is an undetermined global variable; the purpose of this binary search step is to find this global variable. Finding an optimal solution determines the total amount of pruning for the entire model; combining the two transforms the relative pruning strategy into an absolute, specific implementation plan that satisfies performance constraints.

[0179] S4. For the hardware environment in which the target Transformer model runs, model the hardware environment and optimize the pruning strategy based on the modeling results; including the following steps:

[0180] Hardware performance modeling is performed on the target Transformer model's running hardware environment, based on computational cost, memory access cost, and data locality.

[0181] Based on the established hardware performance model, the pruning granularity scheme is optimized with the goal of maximizing hardware performance indicators and with accuracy as the constraint.

[0182] Choose the optimal sparse representation format based on the hardware environment in which the target Transformer model runs.

[0183] In practice, the following steps can be taken:

[0184] Hardware performance modeling:

[0185] The following formula is used as a hardware performance model to predict the execution time of the pruned Transformer model on the target hardware. :

[0186]

[0187] In the formula, l is the index of the model layer; L is the total number of model layers; g represents a pruning granularity scheme; This represents the number of floating-point operations in the l-th layer under the pruning granularity scheme g. The peak floating-point performance (operations per second) of the target hardware. To determine the amount of memory data required to execute the l-th layer under the pruning granularity scheme g; The memory bandwidth (bytes per second) of the target hardware;

[0188] Optimization of pruning granularity scheme:

[0189] The following formula is used as the objective function for optimizing the pruning granularity scheme:

[0190]

[0191] The following formula is used as a constraint condition for optimizing the pruning granularity scheme:

[0192]

[0193] In the formula The accuracy achieved on the set validation set after the model adopts and adjusts the pruning granularity scheme g. This is the minimum acceptable accuracy rate set.

[0194] Selection of sparse representation format:

[0195] If the hardware environment in which the target Transformer model runs supports structured sparse computation, then structured pruning is preferred.

[0196] If the hardware environment in which the target Transformer model is running does not support structured sparse computation, then unstructured pruning is preferred.

[0197] Based on the characteristics of the target Transformer model's running hardware environment, adjust the size of the target Transformer model's Block so that the adjusted Block size matches the size of the computing unit of the target Transformer model's running hardware.

[0198] S5. Evaluate the importance of the parameters of the target Transformer model, and perform dynamic bit allocation and joint optimization based on the evaluation results to achieve lightweighting of the target Transformer model; including the following steps:

[0199] The importance of the parameters of the target Transformer model is evaluated based on the importance of pruning and the importance of quantification.

[0200] Based on the importance assessment results, different numbers of quantization bits are assigned to different parameters;

[0201] With minimizing the model size as the optimization objective and accuracy as the constraint, a joint optimization model for optimizing pruning and quantization parameters is constructed and solved to achieve lightweighting of the target Transformer model.

[0202] In practice, the following steps can be taken:

[0203] Parameter importance assessment:

[0204] The parameter importance of each parameter in the target Transformer model is calculated using the following formula:

[0205]

[0206] In the formula The result of the parameter importance assessment; The pruning importance result is based on the absolute value of the weight parameters (L1 norm), assuming that parameters with larger absolute values ​​contribute more to the model results. , For a single weight parameter in the model; To quantify the importance of the results, the sensitivity of the loss function to parameter changes is measured. This sensitivity is approximated by the second derivative of the loss function with respect to the parameter (i.e., the diagonal elements of the Hessian matrix). A larger second derivative indicates a greater impact of small parameter variations (such as quantization errors) on the loss function. , For the loss function L with respect to the parameters The second-order partial derivative; This is the set sixth weight value; This is the set seventh weight value;

[0207] Quantization bit allocation:

[0208] More bits are allocated to important parameters, and fewer bits are allocated to unimportant parameters; the number of quantization bits allocated is calculated using the following formula:

[0209]

[0210] In the formula The number of quantization bits allocated; The minimum number of quantization bits to be allocated; The number of quantization bits allocated to the set baseline; The importance of the parameters after normalization;

[0211] Joint optimization:

[0212] The following formula is used as the objective function of the joint optimization model:

[0213]

[0214] In the formula, S is the set of pruning decision variables; B is the set of bit allocation variables; and N is the total number of model parameters. It is a binary variable, and when the i-th parameter is reserved... When the i-th parameter is not retained ; The number of quantization bits allocated to the i-th parameter;

[0215] The following formula is used as the constraint condition for the joint optimization model:

[0216]

[0217] In the formula The accuracy achieved by the model on a given validation set under pruning decisions S and bit allocation B;

[0218] The alternating direction multiplier method is used to solve the constructed joint optimization model. The simplified process of the alternating direction multiplier method is as follows: pruning (applying sparsity constraints) and quantization (applying discretization constraints) are regarded as two independent constraints. The original problem is decomposed by introducing auxiliary variables. During the iteration process, the model weights, pruning-related auxiliary variables and quantization-related auxiliary variables are optimized alternately, and penalty terms are gradually applied. Finally, the model weights simultaneously satisfy the sparsity and discretization constraints, thereby obtaining an efficient joint optimization solution that meets the target accuracy.

[0219] The lightweight method of this invention selects the optimal pruning granularity and pruning ratio for different parts of the model through multi-granularity sensitivity analysis and adaptive pruning strategy generation, achieving high compression ratio while maintaining model accuracy. Hardware-aware adaptive mapping makes the pruned model structure more suitable for the target hardware characteristics, resulting in higher practical acceleration. Co-optimization of pruning and quantization significantly reduces model size and computational complexity, lowering memory usage and energy consumption. Attention mechanism sensitivity analysis accurately assesses the importance of attention heads and channels, better preserving key information during pruning. Under the same pruning rate, this method better preserves the model's expressive power and performs better on tasks heavily reliant on attention mechanisms. Furthermore, this lightweight method is not only applicable to the ViT model but also to other Transformer models such as Cait, demonstrating good applicability.

[0220] To further illustrate the beneficial effects of the present invention, the present invention will be described in detail in conjunction with the following embodiments.

[0221] This embodiment establishes a unified testing environment. The experiment selects the VisionTransformer base model (ViT-Base), widely used in computer vision, as the target Transformer model to be lightweighted, and evaluates its performance on the validation set of the large-scale public image classification dataset ImageNet. To simulate the deployment scenario of the technical solution in real-world applications, the experiment is conducted on a typical edge computing device (NVIDIA Jetson AGX Xavier) to verify the model's real-world performance under resource-constrained environments.

[0222] Two existing technical solutions were proposed. The first solution (hereinafter referred to as "Comparison Method 1") is an unstructured pruning method based on the L1 norm. This method measures the importance of weights based on their absolute values ​​and removes weights with smaller values. The second solution (hereinafter referred to as "Comparison Method 2") is a conventional structured pruning method. This method prunes the model at a fixed granularity, such as attention heads or feedforward network channels, but its process does not incorporate multi-dimensional sensitivity analysis and hardware environment adaptation optimization.

[0223] Three key metrics were used to evaluate the lightweighting effect: Top-1 accuracy, which measures the accuracy of the model's classification prediction; Pruning ratio, which quantifies the degree of model size compression; and Floating-Point Operations (FLOPs), which evaluates the model's computational complexity and inference efficiency.

[0224] The specific performance data comparison results of the experiment are as follows: Figure 2 As shown; according to Figure 2 Experimental data shows that, under the same compression ratio, the method of this invention has a significant performance advantage. For example, at a compression ratio of 40%, the model processed by the method of this invention achieves a Top-1 accuracy of 81.5%, significantly higher than 81.1% for comparison method 1 and 80.5% for comparison method 2. This result indicates that, through multi-dimensional sensitivity analysis of model hierarchy, attention head, and channels, this invention can more effectively identify and retain key structures that play a decisive role in model performance, thereby achieving better accuracy preservation under the same compression intensity. When the compression ratio is further increased to a high compression ratio of 60%, the advantages of the method of this invention become even more prominent, with its accuracy (80.3%) far exceeding that of the two comparison methods, demonstrating the excellent robustness of the technical solution of this invention under high-intensity compression.

[0225] In terms of computational efficiency, the method of this invention also demonstrates high computational efficiency. Figure 3 This is a diagram illustrating the comparison of computational efficiency; by Figure 3 It is evident that the method of this invention achieves higher accuracy while also having the lowest FLOPs among its peers. This is attributed to the fact that the pruning strategy generation process of this invention incorporates performance modeling and adaptive optimization of the target hardware environment, enabling the final generated sparse model structure to better adapt to the computational characteristics of the hardware, thereby achieving greater practical computational acceleration.

[0226] like Figure 4 The diagram shown is a flowchart of the transmission line image analysis method of the present invention: The transmission line image analysis method disclosed in this invention, which includes the aforementioned lightweight Transformer model method, includes the following steps:

[0227] A. Obtain the trained Transformer model for transmission line image analysis; this model can analyze images captured by drones or monitoring towers to identify defects or abnormal conditions in key components such as insulators, vibration dampers, and conductors.

[0228] B. Using the Transformer model obtained in step A as the target Transformer model, perform lightweighting processing using the aforementioned Transformer model lightweighting method;

[0229] C. Deploy the lightweight Transformer model obtained in step B to achieve image analysis of transmission lines.

[0230] The transmission line image analysis method provided by this invention adopts a lightweight Transformer model, which enables faster and more efficient image analysis, and is particularly suitable for applications of edge devices such as drones and monitoring equipment.

[0231] Furthermore, the lightweight Transformer model method provided by this invention can also be applied to scenarios such as power grid load forecasting, intelligent monitoring of substation equipment status, intelligent identification and analysis of power dispatching instructions, forecasting of new energy power generation, analysis of power marketing customer behavior, power grid operation risk assessment, and fault location and diagnosis of distribution networks.

Claims

1. A transmission line image analysis method based on a lightweight Transformer model, characterized in that... Includes the following steps: (i) Obtain the trained Transformer model for transmission line image analysis; (ii). The Transformer model obtained in step (i) is used as the target Transformer model, and the Transformer model is lightweighted using a lightweighting method. (iii) Deploy the lightweight Transformer model obtained in step (ii) to achieve image analysis of transmission lines; The lightweight Transformer model method specifically includes the following steps: S1. Obtain the target Transformer model; S2. Based on model hierarchy sensitivity analysis, attention head importance analysis, and channel importance analysis, perform sensitivity analysis on the target Transformer model; Specifically, based on a random noise scheme, sensitivity analysis is performed on each layer of the target Transformer model to assess the degree of influence of the corresponding layer on the output of the target Transformer model; based on mutual information and attention graph entropy schemes, the importance of each attention head of the target Transformer model is evaluated to assess the contribution of the corresponding attention head to the model output; based on the L1 norm, the importance of the channels of each linear layer of the target Transformer model is evaluated. S3. Based on the defined pruning scheme and sensitivity analysis scheme, generate the pruning strategy for the target Transformer model; including the following steps: For the attention head, channels, blocks, and elements of the Transformer model, several candidate pruning granularity schemes are generated; For each attention head, channel, block, and element in the target Transformer model, calculate the corresponding sensitivity-efficiency index based on sensitivity and efficiency, and select the candidate pruning granularity scheme with the highest sensitivity-efficiency index as the corresponding pruning granularity scheme. For each attention head, channel, block, and element in the target Transformer model, the corresponding pruning ratio is calculated based on the corresponding sensitivity value. The specific implementation includes the following steps: Candidate pruning granularity scheme generation: The generated candidate pruning granularity schemes include attention head-level pruning, channel-level pruning, block-level pruning, and element-level pruning. Among them, attention head-level pruning is used to directly remove the entire attention head; channel-level pruning is used to remove the entire channel; block-level pruning is used to remove blocks that include several channels; and element-level pruning is used to retain the K most important elements in each sub-row and remove all remaining elements according to the Top-K strategy. Selection of pruning particle size scheme: For each attention head, channel, block, and element in the target Transformer model, the sensitivity-efficiency index of the corresponding candidate pruning granularity scheme is calculated using the following formula: In the formula The sensitivity-efficiency index when using candidate pruning granularity scheme g; The sensitivity enhancement value when using candidate pruning granularity scheme g; The computational efficiency improvement when using candidate pruning granularity scheme g; This represents the maximum sensitivity improvement value among all candidate pruning granularity schemes. This represents the maximum computational efficiency improvement among all candidate pruning granularity schemes. This is the fourth weight value that is set. This is the fifth weight value that is set. Determining the pruning ratio: The pruning ratio is calculated using the following formula: In the formula The pruning ratio for the i3th part; The set basic pruning ratio; The sensitivity of the i3th part after normalization; Considering accuracy constraints, a binary search scheme is used to obtain the maximum pruning ratio that meets the set accuracy requirements: An optimization method based on binary search is employed to determine the optimal basic pruning ratio under specific constraints. ; Precision constraints are expressed as: In the formula The accuracy on the defined validation set; This represents the accuracy of the original model. The set threshold for the decrease; The maximum pruning ratio is obtained by following these steps: (1) Initialize the search interval: set Search scope: lower bound The upper bound is 0. =1; (2) Iterative search: Within a preset number of iterations, or when the difference between the upper and lower bounds is less than a set minimum value, repeat the following operation: a. Calculate the pruning ratio based on the midpoint of the interval as a candidate base. for ; b. As Substitute the branches into the pruning ratio formula to calculate the pruning ratio of each part of the model. ; c. Based on the obtained The model is pruned, and the pruned model is adjusted to restore its performance. d. Calculate the accuracy of the adjusted model on the defined validation set. ; e. If If the set precision constraints are met, then update the search lower bound. for ;like If the set precision constraints are not met, update the upper bound of the search. for ; (3) Determine the final ratio: After the iterative search is completed, the lower bound of the search is obtained. As the final ; S4. For the hardware environment in which the target Transformer model runs, model the hardware environment and optimize the pruning strategy based on the modeling results; S5. Evaluate the importance of the parameters of the target Transformer model, and perform dynamic bit allocation and joint optimization based on the evaluation results to achieve the lightweighting of the target Transformer model.

2. The transmission line image analysis method based on the lightweight method of the Transformer model according to claim 1, characterized in that... Step S2 specifically includes the following steps: Hierarchical sensitivity analysis: Random noise is added to each layer of the target Transformer model, and the corresponding performance changes are obtained. The sensitivity value of the corresponding layer is calculated using the following formula: In the formula The sensitivity value of the layer; This represents the model's original accuracy. The accuracy of the model after adding noise; The L2 norm of the noise; The L2 norm of the layer's output; Attention-based head importance assessment: For each attention head in the model, the corresponding mutual information is calculated using the following formula: In the formula X represents the mutual information of the attention heads; X is the input of the attention head; Y is the output of the attention head. Let X be the joint probability distribution of the input X and the output Y; The marginal probability distribution of input X; The marginal probability distribution of output Y; For each attention head in the model, the corresponding attention map entropy is calculated using the following formula: In the formula, H is the attention map entropy of the attention head; For each individual attention weight in the attention graph; For each attention head in the model, the corresponding attention head influence score is calculated using the following formula: In the formula The attention of the head affects the score; The performance degradation of the target Transformer model after resetting the weights of the attention head to zero; For each attention head in the model, the importance evaluation result of the corresponding attention head is calculated using the following formula: In the formula The results of the attention head importance assessment; This is the first weight value set. This is the set second weight value; This is the set third weight value; Channel importance assessment: For each channel in the model, the corresponding channel importance value is calculated using the following formula: In the formula This represents the channel importance value. The weight value corresponding to the channel; It is an L1 norm; The correlation between any two channels can be calculated using the following formula: In the formula The correlation between the i2 channel and the j2 channel; This is the function for calculating cosine similarity. The weight value for the i2 channel; This represents the weight value of the j2 channel.

3. The transmission line image analysis method based on the lightweight method of the Transformer model according to claim 2, characterized in that... Step S4 includes the following steps: Hardware performance modeling is performed on the target Transformer model's running hardware environment, based on computational cost, memory access cost, and data locality. Based on the established hardware performance model, the pruning granularity scheme is optimized with the goal of maximizing hardware performance indicators and with accuracy as the constraint. Choose the optimal sparse representation format based on the hardware environment in which the target Transformer model runs.

4. The transmission line image analysis method based on the lightweight method of the Transformer model according to claim 3, characterized in that... Step S4 specifically includes the following steps: Hardware performance modeling: The following formula is used as a hardware performance model to predict the execution time of the pruned Transformer model on the target hardware. : In the formula, l is the index of the model layer; L is the total number of model layers; g represents a pruning granularity scheme; This represents the number of floating-point operations in the l-th layer under the pruning granularity scheme g. The peak floating-point performance of the target hardware; To determine the amount of memory data required to execute the l-th layer under the pruning granularity scheme g; The memory bandwidth of the target hardware; Optimization of pruning granularity scheme: The following formula is used as the objective function for optimizing the pruning granularity scheme: The following formula is used as a constraint condition for optimizing the pruning granularity scheme: In the formula The accuracy achieved on the set validation set after the model adopts and adjusts the pruning granularity scheme g. This is the minimum acceptable accuracy rate set. Selection of sparse representation format: If the hardware environment in which the target Transformer model is running supports structured sparse computation, then structured pruning is selected. If the hardware environment in which the target Transformer model is running does not support structured sparse computation, then unstructured pruning is selected. Based on the characteristics of the target Transformer model's running hardware environment, adjust the size of the target Transformer model's Block so that the adjusted Block size matches the size of the computing unit of the target Transformer model's running hardware.

5. The transmission line image analysis method based on the lightweight method of the Transformer model according to claim 4, characterized in that... Step S5 includes the following steps: The importance of the parameters of the target Transformer model is evaluated based on the importance of pruning and the importance of quantification. Based on the importance assessment results, different numbers of quantization bits are assigned to different parameters; With minimizing the model size as the optimization objective and accuracy as the constraint, a joint optimization model for optimizing pruning and quantization parameters is constructed and solved to achieve lightweighting of the target Transformer model.

6. The transmission line image analysis method based on the lightweight method of the Transformer model according to claim 5, characterized in that... Step S5 specifically includes the following steps: Parameter importance assessment: The parameter importance of each parameter in the target Transformer model is calculated using the following formula: In the formula The result of the parameter importance assessment; The importance of pruning results, and , For a single weight parameter in the model; To quantify the importance of the results, and , For the loss function L with respect to the parameters The second-order partial derivative; This is the set sixth weight value; This is the set seventh weight value; Quantization bit allocation: The number of quantization bits allocated is calculated using the following formula: In the formula The number of quantization bits allocated; The minimum number of quantization bits to be allocated; The number of quantization bits allocated to the set baseline; The importance of the parameters after normalization; Joint optimization: The following formula is used as the objective function of the joint optimization model: In the formula, S is the set of pruning decision variables; B is the set of bit allocation variables; N represents the total number of parameters in the model; It is a binary variable, and when the i-th parameter is reserved... When the i-th parameter is not retained ; The number of quantization bits allocated to the i-th parameter; The following formula is used as the constraint condition for the joint optimization model: In the formula The accuracy achieved by the model on a given validation set under pruning decisions S and bit allocation B; The joint optimization model is solved using the alternating direction multiplier method.

Images