Transformer fault diagnosis method based on low-rank fine-tuning of adaptive attention guidance

Abstract

The present application relates to a transformer fault diagnosis method based on low-rank fine-tuning of adaptive attention guidance, belonging to the technical field of artificial intelligence, the present application introduces attention score mechanism, adaptive attention score mechanism, dynamic rank allocation strategy and hierarchical learning rate adjustment mechanism, and further fuses the dynamic updating strategy of context perception, dynamically intelligently triggers the update of key parameters according to the real-time performance, loss and gradient in the model training process, and constructs a large model light adaptation framework suitable for transformer fault diagnosis task, which significantly reduces the model fine-tuning and deployment cost under the premise of ensuring the diagnosis accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a transformer fault diagnosis method based on low-rank fine-tuning of adaptive attention guidance. BACKGROUND

[0002] With the acceleration of the digitalization and intelligentization process of the power system, the importance of the operation state monitoring and fault diagnosis of the transformer, as a core equipment of the power grid operation, is increasingly prominent. At present, power enterprises have accumulated a large amount of text data about the operation of the transformer, such as inspection logs, fault reports, maintenance records and alarm records, etc. These data contain rich implicit knowledge and expert experience. However, how to use these unstructured information for intelligent analysis is still a technical bottleneck that needs to be broken through in the field of power operation and maintenance.

[0003] In recent years, large language models (LLMs) have made significant progress in natural language processing tasks and are gradually applied to fault question answering, operation and maintenance knowledge extraction and other scenarios. Although general pre-training models have strong semantic understanding ability, their direct application to the transformer fault diagnosis task still has significant challenges: on the one hand, the special terminology and complex context of the power industry make it difficult for general models to adapt to the context; on the other hand, complete fine-tuning of large models requires a large amount of computing resources, which is not conducive to deployment in actual industrial edge devices.

[0004] Therefore, how to reduce the fine-tuning cost while maintaining the performance of the model and quickly adapt to the data characteristics of the transformer field has become a hot and difficult problem in current research. To solve the above problems, the industry has gradually explored lightweight fine-tuning strategies, such as parameter efficient fine-tuning (PEFT), low-rank adaptation (LoRA) and other methods. However, these methods often fail to fully utilize the importance differences of input data at different levels or lack flexibility in parameter allocation, resulting in unstable fine-tuning results or performance bottlenecks. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application proposes a low-rank fine-tuning method based on adaptive attention guidance, which is particularly suitable for the field of transformer fault diagnosis, especially in the application of power systems, such as fault detection and early warning of transformers. The method of the present application can effectively fine-tune the pre-trained model to improve the performance of the model in the transformer fault diagnosis task, and is particularly suitable for industrial applications with large amounts of data and limited computing resources.

[0006] The application introduces an attention score mechanism, an adaptive attention score mechanism, a dynamic rank allocation strategy and a hierarchical learning rate adjustment mechanism, and further integrates a context-aware dynamic updating strategy, dynamically intelligently triggers the update of key parameters according to real-time performance, loss and gradient in the model training process, and constructs a large model lightweight adaptation framework suitable for transformer fault diagnosis tasks, which significantly reduces the model fine-tuning and deployment cost on the premise of ensuring the diagnosis accuracy.

[0007] The technical scheme of the application is:

[0008] The transformer fault diagnosis method based on low-rank fine-tuning of adaptive attention guidance comprises the following steps:

[0009] S1, data acquisition and preprocessing stage;

[0010] S101, acquiring relevant operation and maintenance data of the transformer in the target power system;

[0011] S102, standardizing, cleaning and preprocessing the collected data, and constructing a localized transformer fault diagnosis data set;

[0012] S2, initialization stage;

[0013] S201, sampling data from the constructed local transformer fault diagnosis data set, performing forward propagation and back propagation on the original pre-trained language model, and collecting the gradient, attention activation graph and hidden state information of each layer;

[0014] S202, based on the collected information, the gradient importance score, the activation importance score and the value change importance score of each layer are calculated respectively, and a unified attention score index is constructed according to the learnable weight coefficient combination;

[0015] S203, according to the obtained attention score, the rank value and learning rate required by each layer are distributed, and the LoRA parameter matrices A and B of the corresponding layer are initialized to realize the initial configuration of the low-rank structure;

[0016] S3, fine-tuning stage;

[0017] S301, performing a standard training batch process on the fault diagnosis data, i.e., the relevant operation and maintenance data of the transformer, including forward propagation, loss calculation and back propagation, and completing local update of parameters;

[0018] S302, real-time monitoring of the neural network model training dynamics based on attention mechanism and low-rank representation, including performance indicators, training loss and gradient dynamics;

[0019] S303, according to the monitoring of the performance of the neural network model based on attention mechanism and low rank representation and the training dynamics, when the preset context perception trigger condition is met, the attention score and the rank allocation strategy are dynamically updated, and the self-adaptive adjustment of the low rank structure is realized;

[0020] S304, at the end of each training, the overall effect is evaluated based on the performance evaluation of the validation set, and whether to adjust the rank allocation strategy is determined according to the feedback of the neural network model based on attention mechanism and low rank representation;

[0021] S4, adaptive adjustment stage;

[0022] S401, monitor the change trend of the verification index, if the performance improvement is small after continuous multiple evaluations, automatically adjust the key hyperparameters or implement the learning rate restart strategy;

[0023] S402, if the performance still has no significant improvement after multiple adjustments, trigger the early stop mechanism, and select the model with the best verification performance as the final model;

[0024] S5, deployment and real-time diagnosis stage;

[0025] The fine-tuned neural network model based on attention mechanism and low rank representation is deployed to the edge server to realize online analysis and diagnosis of transformer operation and maintenance logs or real-time monitoring data.

[0026] Preferably, in S101, the related operation and maintenance data of the transformer includes inspection records, maintenance logs, alarm information, running parameters, environmental parameters, unstructured or semi-structured text data.

[0027] Preferably, in S102, the collected data is standardized, cleaned and preprocessed; that is, the collected data is sequentially denoised, normalized, missing value filled, classified and labeled.

[0028] Preferably, in S201, the data is sampled from the constructed local transformer fault diagnosis data set, and the original pre-training language model is forward propagated and backward propagated, and the gradients, attention activation maps and hidden state information of each layer are collected; including:

[0029] Collecting the gradients of each layer: in the backward propagation process, the original pre-training language model calculates the error according to the loss function, and reversely transmits from the output layer to the input layer, so as to calculate the gradient of each layer weight matrix and collect the gradient information , n represents the nth layer, and the gradient information represents the sensitivity of the nth layer parameter to the loss function under the current task;

[0030] Collecting attention activation maps of each layer: the original pre-trained language model synchronously collects key signals, i.e. extracts attention activation information, during the forward propagation process; the attention activation map, i.e. the attention activation information, refers to the attention weight matrix generated by each attention head in the original pre-trained language model when processing the input sequence;

[0031] Collecting hidden state information of each layer; the hidden state information refers to the intermediate representation result output by each encoder or decoder layer of the Transformer model during the forward propagation process, including the context semantic features of the input data after nonlinear transformation in the current layer.

[0032] Preferably, the original pre-trained language model is BERT.

[0033] Preferably, BERT includes 12 encoder layers, and the input text is first processed by an embedding layer and then sequentially passes through each encoder layer; the output of each encoder layer, i.e. the hidden state of the encoder layer, is a three-dimensional tensor with a shape of [batch_size, seq_len, hidden_size], where batch_size represents the sample batch size; seq_len represents the length of the input sequence; and hidden_size represents the feature dimension of the current layer.

[0034] Preferably, in S202, the gradient importance score G(n) of each layer is calculated by the following formula:

[0035] ;

[0036] wherein, represents the gradient of the weight matrix of the nth layer, represents the Frobenius norm, i.e. the square root of the sum of squares of all elements in the corresponding matrix.

[0037] Preferably, in S202, the activation importance score is calculated using the activation map of the self-attention mechanism, and the calculation steps are as follows:

[0038] Normalizing by the number of heads, dividing the total difference by the number of attention heads in the layer to calculate the average single-head difference; the total difference refers to the sum of L1 differences between the attention matrices of each attention head in the same layer;

[0039] Normalizing by the sequence length, dividing the average single-head difference by the square of the input sequence length;

[0040] Processing the sequence length normalized result by a dynamic normalization function to obtain the activation importance score.

[0041] Preferably, the results after sequence length normalization are processed by a dynamic normalization function, including:

[0042] In the initial training stage, i.e. t < 0.4T, t is the current training step number, and T is the total training step number, global maximum normalization is used for processing;

[0043] In the middle training stage, i.e. 0.4T≤t < 0.6T, a mixed function is used for processing;

[0044] In the late training stage, i.e. t≥0.6T, quantile normalization is used for processing.

[0045] Preferably, in S202, the value change importance score V(n) is calculated by the following formula:

[0046] ;

[0047] Where H(n) and H0(n) are the hidden state outputs of the nth layer in the fine-tuning and the original model respectively; H(i) and H0(i) are the hidden state outputs of the ith layer in the fine-tuning and the original model respectively;

[0048] Alternatively, the value change importance score V(n) is calculated by the following formula:

[0049] ;

[0050] Where α1, α2, α3 are weight coefficients, and V(n) represents the value change importance score of the nth layer;

[0051] Preferably, in S202, the attention score S(n) is calculated by the following formula:

[0052] Where G(n) represents the gradient importance score of the nth layer, A(n) represents the activation importance score of the nth layer, and V(n) represents the value change importance score of the nth layer, are learnable weight coefficients.

[0053] Preferably, in S203, resource allocation, based on the attention score S(n), rank values r(n) and learning rates lr(n) of each layer are allocated according to a dynamic rank allocation strategy and a hierarchical adaptive learning rate mechanism, including the following steps:

[0054] Basic parameter setting and rank value and learning rate allocation:

[0055] Set the basic parameters, including the basic rank value , the maximum rank value , the scaling factor , the basic learning rate and a learning rate adjustment factor , based on the calculated attention score S(n), rank value r(n) and learning rate lr(n) are assigned to each layer;

[0056] Rank value assignment, rank value r(n) is assigned by the following formula:

[0057] ;

[0058] Learning rate assignment, hierarchical adaptive learning rate lr(n) is assigned by the following formula:

[0059] ;

[0060] For each layer, the corresponding LoRA parameter matrix A and B are initialized according to the assigned rank value, and the operation is as follows,

[0061] LoRA parameter matrix initialization, the corresponding LoRA parameter matrix A and B are initialized according to the assigned rank value, where the A matrix is initialized with normal distribution and the B matrix is initialized with all zeros.

[0062] Preferably, in S203, the attention score and rank assignment strategy are periodically updated according to the performance of the neural network model based on attention mechanism and low-rank representation after every N training batches, realizing dynamic adaptation of low-rank structure; including:

[0063] The attention score of each layer is updated by sliding window average:

[0064] ;

[0065] S(l) old indicates the attention score at the last update of the l-th layer, S(l) current indicates the attention just calculated in the current training cycle of the l-th layer;

[0066] Periodically update the rank assignment strategy, including:

[0067] The total rank budget constraint is: ;

[0068] Rank value calculation logic: ;

[0069] Adaptive adjustment of LoRA matrix dimension is performed, and the specific strategy is as follows:

[0070] Rank expansion: when r increases to r+Δr, the new parameter matrix block is initialized with orthogonalization, and the principal component of the original matrix is inherited through SVD decomposition;

[0071] Rank shrinkage: when r increases to r-Dr, the feature components corresponding to the top Dr singular values of the singular value decomposition are retained.

[0072] Preferably, in S4, the adaptive adjustment stage; comprising:

[0073] When the model performance is slightly improved or stagnates during fine-tuning, that is, the evaluation index is improved less than the preset threshold for N consecutive times, it is determined that the performance is slightly improved, and it is determined that the model may be trapped in a local optimum; automatically adjust the key hyperparameters or implement the learning rate restart strategy.

[0074] If the performance is still slightly improved after multiple adjustments, trigger the early stopping mechanism, and select the model with the best validation performance as the final model.

[0075] Preferably, in S5, the deployment and real-time diagnosis stage; comprising:

[0076] ①Edge computing device deployment: the low-rank adaptive model completed by fine-tuning is deployed on the NVIDIA Jetson XavierAGX edge computing device;

[0077] ②Data input and real-time monitoring: real-time monitoring data of the transformer is collected through sensors, and the sensors transmit real-time data to the edge server through the Modbus TCP / IP protocol; the real-time monitoring data of the transformer includes: input voltage, output current, oil temperature, load current, alarm state;

[0078] ③Real-time fault diagnosis and early warning: the neural network model based on attention mechanism and low-rank representation completed by fine-tuning processes the real-time monitoring data of the transformer; specifically including: receiving multi-source data input from the transformer online monitoring system, converting the multi-source data input into a standard format acceptable by the neural network model based on attention mechanism and low-rank representation according to the preset data preprocessing process, and constructing a time series input feature tensor;

[0079] The neural network model based on attention mechanism and low-rank representation fuses historical data distribution and current input features, captures the coupling relationship and dynamic change trend between key variables using the attention mechanism, and discriminates the current state based on the enhanced feature expression ability of the low-rank structure; at the output end, the neural network model based on attention mechanism and low-rank representation classifies or grades the running state of the transformer, and can output specific fault types and confidence scores in combination with abnormal trends;

[0080] At the same time, the neural network model based on attention mechanism and low-rank representation supports a dynamic scoring mechanism based on a sliding time window, continuously judges the health state trend at multiple times, and realizes early identification and early warning of potential faults; if the model continuously determines that the state is abnormal for multiple times, trigger the early warning mechanism;

[0081] The diagnostic result includes:

[0082] Fault prediction: if the current or oil temperature of the transformer exceeds the set threshold value, a pre-warning will be automatically issued, prompting that the transformer has a fault risk;

[0083] Fault type identification: the fine-tuned neural network model based on attention mechanism and low-rank representation identifies the fault type, and the fault type includes overload, short circuit and temperature anomaly; the specific determination basis is:

[0084] If the real-time current exceeds the maximum allowable current, it is determined as an overload fault;

[0085] If the oil temperature exceeds the set threshold value, it is determined as a temperature anomaly;

[0086] 4. Real-time reasoning and decision support: when the transformer has a fault or a potential fault, the edge server generates a fault diagnosis report, and the report includes the fault type, fault time and fault location, and fault severity.

[0087] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above-mentioned transformer fault diagnosis method based on adaptive attention guidance and low-rank fine-tuning when executing the computer program.

[0088] A computer readable storage medium has a computer program stored thereon, and the computer program implements the steps of the above-mentioned transformer fault diagnosis method based on adaptive attention guidance and low-rank fine-tuning when executed by a processor.

[0089] The beneficial effects of the present application are:

[0090] By dynamically allocating the rank value, the calculation resource allocation is optimized, better performance is obtained under the same parameter amount, higher parameter efficiency is realized; based on the hierarchical learning rate mechanism, the optimization of important parameters can be accelerated, the training time is shortened, and the convergence speed is improved; through the attention mechanism, the task key parameters are identified, and the adaptability of the model to specific tasks is improved; in addition, the optimized rank allocation reduces the overall parameter amount, reduces the storage demand, and is more suitable for resource-limited scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0091] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below.

[0092] Figure 1 A flowchart of a transformer fault diagnosis method based on adaptive attention guidance and low-rank fine-tuning is provided.

[0093] Figure 2A flowchart of the data acquisition and preprocessing stage in the transformer fault diagnosis method of adaptive attention-guided low-rank fine-tuning proposed in the present application is shown in the figure.

[0094] Figure 3 A flowchart of the initialization stage in the transformer fault diagnosis method of adaptive attention-guided low-rank fine-tuning proposed in the present application is shown in the figure.

[0095] Figure 4 A flowchart of the fine-tuning training stage in the transformer fault diagnosis method of adaptive attention-guided low-rank fine-tuning proposed in the present application is shown in the figure.

[0096] Figure 5 A flowchart of the adaptive adjustment stage in the transformer fault diagnosis method of adaptive attention-guided low-rank fine-tuning proposed in the present application is shown in the figure.

[0097] Figure 6 A flowchart of the deployment and real-time diagnosis stage in the transformer fault diagnosis method of adaptive attention-guided low-rank fine-tuning proposed in the present application is shown in the figure.

[0098] Figure 7 The score function S(n) in the present application acts on the encoder-decoder architecture diagram.

[0099] Figure 8 The dynamic rank r(n) in the present application acts on the encoder-decoder architecture diagram.

[0100] Figure 9 The hierarchical adaptive learning rate mechanism in the present application acts on the encoder-decoder architecture diagram. DETAILED DESCRIPTION

[0101] The present application is further limited by the following description and drawings, but is not limited thereto.

[0102] Terminology explanation:

[0103] 1. Power system: refers to an organic whole for power generation, power transmission, power transformation, power distribution and power consumption, including power plants, substations, transmission lines, distribution networks and terminal power consumption equipment, etc. Its main function is to realize the production, transmission and distribution of electric energy, and to guarantee the stable and reliable power consumption of users in various application scenarios.

[0104] 2. Global maximum normalization: a common data preprocessing method, the basic idea of which is to divide the value of a certain feature in all samples by the maximum value of the feature in the entire data set, so as to compress the value range of the feature to the interval [0, 1]. This method is suitable for scenarios where the feature numerical range is large but uniform scale processing is desired.

[0105] 3. Quantile normalization: is a method of standardizing data by considering the position of the characteristic value in the distribution of the data set (such as median, quartile, etc.). It maps the characteristic value of each sample according to its position in the whole sample, effectively reducing the influence of outliers on data distribution, and is suitable for processing data with skew distribution or noise interference.

[0106] 4. Hybrid function: refers to a mathematical structure that combines multiple different types of functions together to enhance the expression ability of the model. In algorithm or model design, hybrid functions are often used to fuse multiple features or process data with different distributions to more comprehensively extract information and improve overall prediction or classification performance.

[0107] 5. Pytorch, the python version of torch, is a neural network framework open sourced by Facebook, which is specially designed for GPU accelerated deep neural network (DNN) programming. Torch is a classic tensor library for operating on multi-dimensional matrix data, which has been widely used in machine learning and other math-intensive applications. The computation graph of pytorch is dynamic and can be changed in real time according to the needs of calculation.

[0108] Example 1

[0109] The transformer fault diagnosis method based on adaptive attention guide low-rank fine-tuning, as shown in Figure 1 , includes the following steps:

[0110] S1, data acquisition and preprocessing stage; as Figure 2 shown:

[0111] S101, collect the relevant operation and maintenance data of the transformer in the target power system;

[0112] S102, standardize and clean the collected data and pre-process to build a localized transformer fault diagnosis data set;

[0113] S2, initialization stage; as Figure 3 shown:

[0114] S201, sample data from the constructed local transformer fault diagnosis data set, perform forward propagation and back propagation on the original pre-trained language model, and collect the gradient, attention activation map and hidden state information of each layer;

[0115] S202, based on the collected information, calculate the gradient importance score, activation importance score and value change importance score of each layer respectively, and combine to build a unified attention score index according to the learnable weight coefficient group;

[0116] S203, assign the rank value and learning rate required by each layer according to the obtained attention score, and initialize the LoRA parameter matrices A and B of the corresponding layer to realize the initial configuration of the low-rank structure;

[0117] S3, fine-tuning stage; as Figure 4 shown:

[0118] S301, perform a standard training batch process on the fault diagnosis data, i.e., the relevant operation and maintenance data of the transformer, including forward propagation, loss calculation and back propagation, to complete local update of parameters;

[0119] S302, real-time monitoring of the neural network model training dynamics based on attention mechanism and low-rank representation, including performance indicators, training loss and gradient dynamics;

[0120] S303, according to the performance and training dynamics of the neural network model based on attention mechanism and low-rank representation, when the preset context-aware trigger condition is met, dynamically update the attention score and rank allocation strategy to realize adaptive adjustment of the low-rank structure;

[0121] S304, at the end of each training round, evaluate the overall effect based on the validation set diagnosis performance, and decide whether to adjust the rank allocation strategy according to the feedback of the neural network model based on attention mechanism and low-rank representation;

[0122] S4, adaptive adjustment stage; as Figure 5 shown:

[0123] S401, monitor the change trend of the validation indicators, if the performance improvement is small after continuous multiple evaluations, automatically adjust the key hyperparameters or implement the learning rate restart strategy;

[0124] S402, if the performance still has no significant improvement after multiple adjustments, trigger the early stopping mechanism, and select the model with the best validation performance as the final model;

[0125] S5, deployment and real-time diagnosis stage; as Figure 6 shown:

[0126] S501, deploy the fine-tuned neural network model based on attention mechanism and low-rank representation to the edge server to realize online analysis and diagnosis of transformer operation and maintenance logs or real-time monitoring data.

[0127] Embodiment 2

[0128] The transformer fault diagnosis method based on adaptive attention guidance and low-rank fine-tuning according to embodiment 1 is different in that:

[0129] In S101, the related operation and maintenance data of the transformer includes inspection records, maintenance logs, alarm information, operation parameters, environment parameters, unstructured or semi-structured text data. Real-time data is obtained from related sensors and monitoring devices in the power system to ensure the timeliness and accuracy of the data.

[0130] In S102, the collected data is standardized, cleaned and preprocessed; that is, the collected data is sequentially denoised, normalized, missing value filled, classified and labeled. A localized transformer fault diagnosis data set that meets the task requirements is formed.

[0131] In S201, sample data from the constructed local transformer fault diagnosis data set, forward propagate and back propagate the original pre-trained language model, collect the gradient, attention activation map and hidden state information of each layer; including:

[0132] Collecting the gradient of each layer: in the back propagation process, the original pre-trained language model calculates the error from the loss function, and reversely transmits from the output layer to the input layer, so as to calculate the gradient of each layer weight matrix, and collect the gradient information , n represents the nth layer, and the gradient information of the nth layer parameter indicates the sensitivity of the parameter to the loss function under the current task; the specific implementation includes:

[0133] First, in the forward propagation process, the original pre-trained language model generates a prediction result according to the input sample, and calculates the loss function value (such as cross-entropy loss) to measure the error between the prediction and the true label;

[0134] Then, in the back propagation stage, using the chain rule, starting from the partial derivative of the loss function to the output layer, the error signal is sequentially transmitted forward, and the partial derivative of each layer parameter including the weight matrix and the bias term is calculated, which is in the form of:

[0135] ;

[0136] Wherein, represents the loss function, represents the error signal of the layer, The output of the previous layer (or the input of the current layer). In implementation, PyTorch automatically constructs a computational graph during model definition and backpropagation, and automatically completes the calculation of partial derivatives and the accumulation of gradients. By calling the.backward() method, the system will automatically calculate the gradient of each layer parameter along the computational graph and store it in the.grad attribute of each parameter tensor. By collecting these gradient information, the sensitivity of each layer parameter to the current task loss can be quantified, which can be further used to analyze the model structure adaptability or perform low-rank decomposition, rank allocation strategy and other subsequent steps.

[0137] For example, for a language model based on the Transformer architecture, when processing the text translation task, the gradient of some layers is large, indicating that the adjustment of the parameters of these layers has a greater impact on the accuracy of the translation result, which needs to be focused on and optimized in the subsequent process.

[0138] Collecting attention activation maps of each layer: the original pre-trained language model synchronously collects key signals during forward propagation, that is, the extraction of attention activation information; the attention activation map, that is, the attention activation information, is the attention weight matrix generated by each attention head in the original pre-trained language model when processing the input sequence; it reflects the attention degree of the model to different position inputs in each layer.

[0139] Taking BERT as an example, in each Transformer encoder, the input sequence is first mapped to three matrices of query (Q), key (K) and value (V). The formula for calculating the attention weight is:

[0140] ;

[0141] Where, The generated matrix after normalization is the attention activation map, and each item of the matrix represents the attention score of the current word to other position words.

[0142] ​In a specific implementation, the activation map of each layer and each attention head can be collected by calling the forward function of the model and registering the attention_weights output of the intermediate layer. For example, in PyTorch, the attention map of the corresponding layer can be extracted by model.encoder.layer[i].attention.self.get_attention_map(). The dimensions of these attention weight tensors are usually [batch_size, num_heads, seq_len, seq_len], representing the attention relationship between each position in the input sequence for each head. The collection of this attention activation map is of great significance for subsequent analysis of the model structure, implementation of attention pruning and rank adjustment, and other low-rank structure optimization work.

[0143] The attention activation information records the degree of association between different positions and features when the model processes data. In natural language processing tasks, by analyzing the attention activation map, we can understand the focus of the model on different words in the input text when generating text, thereby providing a basis for evaluating the importance of each layer in information processing.

[0144] Collecting the hidden state information of each layer; the hidden state information refers to the intermediate representation result output by the encoder or decoder of each layer during the forward propagation process of the Transformer model, including the context semantic features of the input data after nonlinear transformation in the current layer. The hidden state of each layer preserves the feature expression of the input sequence in the current representation space and is the basis for the model to build semantic representation and make final predictions.

[0145] As the last link of forward calculation information collection, the feature representation of each layer of the model needs to be systematically captured: hidden state information collection, the hidden state is the output of each layer after feature extraction and transformation of the input information during the model processing data process. Collecting hidden state information H(n) can understand the representation of data by the model at different layers. During fine-tuning, by comparing the changes in hidden state before and after fine-tuning, the role and contribution of each layer in adapting to new tasks can be evaluated.

[0146] The original pre-trained language model is BERT.

[0147] For example, for a Transformer architecture-based language model BERT, when processing English to Chinese text translation tasks, the gradient values of some intermediate layers (e.g., the 6th and 9th layers) are large, indicating that the parameter adjustment of these layers has a significant impact on the accuracy of the translation results, and subsequent attention and optimization are needed.

[0148] The original pre-trained language model refers to a basic model that has been trained on a large-scale general corpus (such as English Wikipedia, BooksCorpus). The language model used in the present application is BERT, which has a network structure composed of 12 layers of Transformer encoder stacks. Each layer contains substructures such as multi-head self-attention mechanism, feedforward fully connected network, residual connection, and layer normalization, which can effectively model the context association between words in the input sequence.

[0149] After the model is pre-trained, it is fine-tuned for a specific task (in this case, English to Chinese translation) to optimize the parameters. During the fine-tuning process, the gradient of each layer parameter is calculated by the standard backpropagation algorithm, and then the sensitivity of different layers to the task loss is judged to support the subsequent rank adjustment strategy and low-rank modeling steps.

[0150] BERT includes 12 encoder layers. The input text is first processed by the embedding layer, and then passes through each encoder layer in turn. The output of each encoder layer is the hidden state of that encoder layer, which is a three-dimensional tensor with a shape of [batch_size, seq_len, hidden_size], where batch_size represents the sample batch size; seq_len represents the length of the input sequence; hidden_size represents the feature dimension of the current layer (for example, 768 in BERT-base).

[0151] In the implementation process, the forward inference function of the model can be called, and the option of outputting the intermediate layer results can be set to collect the hidden states of all layers. For example, when using the HuggingFace Transformers framework in PyTorch, by setting output_hidden_states=True, a list hidden_states containing all layer hidden states can be obtained in the model return result, and the structure is as follows:

[0152] ;

[0153] Where is the embedding layer output, to is the hidden state of each encoder layer. These hidden state information can be used for subsequent analysis of the expression ability and task relevance of each layer, and plays an important supporting role in structure optimization operations such as layer importance evaluation, low-rank dimension reduction, and pruning strategy formulation.

[0154] In S202, the gradient importance score G(n) of each layer is calculated by the following formula:

[0155] ;

[0156] wherein, denotes the gradient of the weight matrix of the nth layer, denotes the Frobenius norm, i.e. the square root of the sum of the squares of all elements in the corresponding matrix. The importance score reflects the sensitivity of the nth layer parameter to the loss function under the current task, and is specifically obtained by normalizing the overall amplitude of the gradient (gradient norm). The introduction of the Frobenius norm makes the evaluation index have dimensional consistency and comparability, so that it can be used for importance ranking between different layers and low-rank rank allocation strategy making.

[0157] In S202, the activation importance score is calculated using the activation map calculation of the self-attention mechanism, and the calculation steps are as follows:

[0158] In order to measure the difference in attention distribution of different layers and eliminate the influence of the number of attention heads in different layers, the total difference is divided by the number of attention heads in the layer to calculate the average single-head difference. The total difference is the sum of the L1 differences between each pair of attention matrices in the same layer;

[0159] The average single-head difference is divided by the square of the input sequence length (i.e. the total number of elements in each attention matrix) to perform sequence length normalization.

[0160] The result after sequence length normalization is processed by a dynamic normalization function to obtain the activation importance score.

[0161] The result after sequence length normalization is processed by a dynamic normalization function, including:

[0162] In the early stage of training, i.e. t < 0.4T, t is the current training step, and T is the total training step, the global maximum value normalization is used for processing; this method can preserve extreme signals and help quickly locate key layers;

[0163] In the middle stage of training, i.e. 0.4T≤t<0.6T, the mixed function is used for processing; a smooth transition is realized.

[0164] In the later stage of training, i.e. t≥0.6T, the quantile normalization is used for processing. The 25% quantile and 75% quantile of the data are used to normalize the inter-quartile range to suppress noise interference and prevent overfitting;

[0165] In S202, the value change importance score V(n) is calculated by the following formula:

[0166] ;

[0167] where H(n) and H0(n) are the hidden state outputs of the nth layer in the fine-tuned model and the original model, respectively; H(i) and H0(i) are the hidden state outputs of the ith layer in the fine-tuned model and the original model, respectively;

[0168] Alternatively, the value change importance score V(n) is calculated by the following formula:

[0169]

[0170] where a1, a2, a3 are weight coefficients, and V(n) represents the value change importance score of the nth layer;

[0171] In S202, the attention score S(n) is calculated by the following formula:

[0172] where G(n) represents the gradient importance score of the nth layer, A(n) represents the activation importance score of the nth layer, V(n) represents the value change importance score of the nth layer, are learnable weight coefficients.

[0173] where the weight coefficients are learnable parameters trained and updated by the optimizer in the fine-tuning stage (i.e., step S301). This mechanism enables the system to adaptively learn how to balance different types of importance signals based on the feedback of model performance and data characteristics during training, thus more accurately assessing the true importance of each layer. To ensure the reasonableness and stability of the learnable weight coefficients, non-negativity constraints or L1 / L2 regularization can be imposed on them.

[0174] Sigmoid refers to a commonly used activation function, whose input is an arbitrary real number, and the output value is between 0 and 1, with the characteristics of smoothness, continuity and monotonicity. It can compress the output of the model into a probability space, commonly used in the output layer of binary classification tasks, or used in mapping attention scores, gating mechanisms, etc., so that the output has a probabilistic meaning.

[0175] In S203, resource allocation, based on the attention score S(n), the rank value r(n) and the learning rate lr(n) of each layer are allocated according to the dynamic rank allocation strategy and the hierarchical adaptive learning rate mechanism, including the following steps:

[0176] Basic parameter setting and rank value, learning rate allocation:

[0177] Set the basic parameters, including the basic rank value (the minimum rank of all layers), the maximum rank value , the scaling factor , the basic learning rate​ and a learning rate adjustment factor , based on the calculated attention score S(n), rank value r(n) and learning rate lr(n) are assigned to each layer;

[0178] After completing the basic parameter setting, the core resource quantization allocation link is entered. The first step is: rank value allocation. The rank value r(n) is allocated through the following formula:

[0179] ;

[0180] This formula is based on the total rank constraint (R is the pre-set total rank budget) and each layer constraint. By constructing the Lagrange function and using the Lagrange multiplier method, it is obtained that the layers with high importance will be allocated higher rank values, thereby obtaining larger parameter space for adjustment; the layers with low importance will be allocated lower rank values, realizing the optimized utilization of computing resources.

[0181] After the rank value allocation is completed, the adaptive allocation operation of the learning rate parameter is carried out immediately: learning rate allocation. The layer adaptive learning rate lr(n) is allocated through the following formula:

[0182] ;

[0183] This formula introduces an importance weighting mechanism, designs the learning rate as a weak inverse proportional function of the gradient amplitude, and balances the sensitivity and stability through a linear term, so that the learning rate is positively correlated with the importance score of each layer, which can accelerate the optimization of important parameters.

[0184] After completing the differentiated allocation of rank values and learning rates, the specific configuration stage of the model parameters is entered. For each layer, the corresponding LoRA parameter matrices A and B are initialized according to the allocated rank value, and the operation is as follows,

[0185] The core operation in the parameter initialization stage is the customized configuration of the LoRA parameter matrix, which is as follows: LoRA parameter matrix initialization. The corresponding LoRA parameter matrices A and B are initialized according to the allocated rank value. The A matrix is initialized with a normal distribution, which can make the matrix elements randomly distributed around zero mean, which is helpful for the model to explore different parameter spaces in the early training stage; the B matrix is initialized with all zeros. In the training process, the B matrix will be updated gradually according to the data and the learning of the model, so that the model can adapt to new tasks by adding a low-rank matrix without changing most of the parameters of the original pre-trained model.

[0186] After completing the initialization and configuration of the model parameters, the iteration training stage based on dynamic resource allocation is entered.

[0187] In S203, the attention score and rank allocation strategy are periodically updated according to the performance of the neural network model based on attention mechanism and low-rank representation after every N training batches, realizing dynamic adaptation of the low-rank structure; including:

[0188] The attention score of each layer is updated by sliding window averaging:

[0189]

[0190] S (l) old S (l) current S (l)

[0191] The rank allocation strategy is periodically updated, including:

[0192] The total rank budget constraint is: (unchanged);

[0193] Rank value calculation logic:

[0194] After determining the new rank value of each layer, adaptive adjustment of the LoRA matrix dimension is performed, and the specific strategy is as follows:

[0195] Rank expansion: when r increases to r+Δr, the new parameter matrix block is initialized orthogonally, and the principal component of the original matrix is inherited through SVD decomposition; when the rank increases from r to r+Δr (where r represents the rank value of the current layer), new parameter matrix blocks need to be introduced for the newly added Δr rank components. The new parameter block is initialized orthogonally to maintain numerical stability, and the principal component (i.e., the feature direction corresponding to the largest singular value) of the original matrix is used as the basis of the new matrix to inherit existing knowledge through SVD (singular value decomposition).

[0196] Rank contraction: when r increases to r-Δr, the feature components corresponding to the first Δr largest singular values before SVD decomposition are retained.

[0197] By performing SVD decomposition on the original weight matrix, only the feature components corresponding to the first r−Δr largest singular values are retained, i.e., the most representative subspace information is retained, thereby minimizing the accuracy loss.

[0198] In S301, after obtaining the hierarchical attention score S(n), the LoRA module and part or all of the parameters in the Transformer model are fine-tuned. In this stage, the model iteratively updates the LoRA parameters, other trainable parameters of the Transformer model, and the learnable weight coefficients ​​The iterative process aims to minimize the loss of the diagnostic task and improve the performance of the model on transformer fault diagnosis.

[0199] Specifically, the update includes the following parameters:

[0200] ① LoRA Parameters: This mainly refers to the low-rank decomposition matrices A and B added to each attention layer and feedforward network layer of the Transformer model. These matrices are the core of the invention to achieve efficient fine-tuning of parameters. For example, for the linear layer in Transformer , which is frozen during LoRA fine-tuning, by introducing two low-rank matrices, its update is modeled as During the fine-tuning process, only the parameters of matrices A and B need to be trained, and their dimensions and initial allocation are controlled by the dynamic rank allocation strategy in step S203. The original pre-trained Transformer weights (such as the self-attention weights of BERT, the feedforward network weights, etc.) are usually kept frozen during the fine-tuning process, thereby greatly reducing the amount of parameters that need to be updated and the consumption of computing resources.

[0201] ② Additional Trainable Parameters within the Transformer Model Itself: Although the core of LoRA lies in freezing the original pre-trained weights, in order to further optimize performance or adapt to specific needs, sometimes a small number of parameters other than the main weight matrices in the Transformer model are also fine-tuned. This may include but is not limited to the bias terms of each linear layer in the Transformer (such as the Q, K, V projection layers, the output projection layer, and the linear layers in the feedforward network); the layer normalization parameters of the layer normalization operation in the Transformer model, i.e. its scaling factor (gamma) and offset (beta); and the task-specific output layers (Task-Specific Output Layers / Heads) added to map the feature representation of the Transformer model to specific transformer fault diagnosis results. The weights and bias terms of these task-specific output layers (for example, a linear layer followed by a Softmax activation function to output the probability of different fault types) are fully trainable parameters that convert the semantic features output by the Transformer into the final fault type prediction or diagnosis score.

[0202] ③ Learnable Weight Coefficients: These are the coefficients used in step S202 to combine the gradient importance, activation importance, and value change importance scores. These coefficients are also optimized as parameters of the model during fine-tuning, allowing the model to adaptively learn how to trade off different types of importance signals to better assess the true importance of each layer.

[0203] Specifically, the iterative process is as follows:

[0204] ① Batch Loading: Load a batch of data containing multiple training samples from the constructed localized transformer fault diagnosis dataset. Each sample includes the operational data of the transformer (such as text logs, operating parameters, etc.) and its corresponding true fault label.

[0205] ② Forward Pass: The loaded batch data is used as input, first passing through the embedding layer and position encoding layer of the Transformer model. Then, the data is sequentially processed through each encoder layer (or decoder layer, depending on the specific architecture) of the Transformer model. In each layer, the frozen parameters of the original pre-trained model are used to perform core calculations (such as self-attention calculations, feedforward network transformations), while the LoRA module fine-tunes the feature representation by adding the product of the input and low-rank matrices A and B to the output of the original weight matrix. For example, when an input tensor X passes through a LoRA-enhanced linear layer, its output is After processing all Transformer layers, the semantic features of the output are passed to the task-specific output layer (such as the classification head). The task-specific output layer converts these features into a prediction result for the transformer fault type or a diagnostic score (e.g., a probability distribution of each fault type).

[0206] ③ Loss Calculation: Compare the model's forward propagation prediction results (e.g., predicted fault type probabilities) with the true fault labels in the batch data. According to the pre-defined loss function (such as cross-entropy loss for multi-classification tasks, which quantifies the difference between the predicted probability distribution and the true label), the current prediction error is calculated. The smaller the loss value, the closer the model's prediction is to the true situation.

[0207]

[0208]

[0209]

[0210]

[0211] ​​​​​In S303, instead of the original fixed periodic update mechanism, the application dynamically updates the attention score and rank allocation strategy according to the model performance, training loss and gradient dynamics monitored in real time in S302 when the preset context-aware trigger condition is met, to realize the adaptive adjustment of the low-rank structure. Once any trigger condition is met, the system will automatically perform the following operations: recalculate the attention score (corresponding to S202 step), and reallocate the rank value and learning rate of each layer (corresponding to S203 step), thereby realizing the dynamic adaptation of the low-rank structure. The trigger conditions are as follows:

[0212] If the diagnostic accuracy rate improvement amplitude of continuous 5 times evaluation on the validation set is less than 0.01%.

[0213] If the standard deviation of the training loss in the last 100 batches is less than 0.001.

[0214] If the validation loss starts to rise continuously for 3 times, or the average gradient norm of any key layer LoRA module drops by more than 90% of its historical average within 50 batches.

[0215] The S303 specifically further includes the following update operations:

[0216] Attention score update: the attention score S(n) of each layer is updated by exponential weighted moving average (EWMA) method, which integrates historical data and the score calculated by the latest batch. The specific formula is:

[0217]

[0218] Where, is set to 0.1.

[0219] Rank allocation strategy update:

[0220] Total rank budget constraint: the total effective rank budget of the model is set to 256.

[0221] Rank value calculation logic: according to the updated attention score and total rank budget, the rank value of each layer is reallocated. First, normalize the attention score of all layers:

[0222]

[0223] Then, the rank value n allocated to each layer is calculated as:

[0224]

[0225] Where, is the total rank budget, which refers to the upper limit of the total rank that can be allocated to all layers modified by LoRA technology during the model fine-tuning process.

[0226] Ensure that the rank of each layer is at least 4. Finally, for all layers... Make adjustments to ensure No more than .

[0227] Perform adaptive adjustment of LoRA matrix dimensions: Dynamically adjust the dimensions of the corresponding layer LoRA parameter matrices A and B based on the reallocated r_n.

[0228] Layer-by-layer learning rate adjustment: Learning rate of each LoRA module layer According to its assigned rank Adjustments will be made.

[0229] S304. Evaluation and Decision: At the end of each epoch, the overall performance is evaluated based on the diagnostic performance of an independent validation set. Based on the model's diagnostic performance feedback, determine whether any of the following conditions are met:

[0230] If the accuracy of the validation set does not improve for three consecutive epochs.

[0231] If the validation set accuracy reaches 98%.

[0232] The system makes a decision when any of the above conditions are met. If the validation set accuracy does not improve for three consecutive epochs, the current learning rate is adjusted to 0.5 times the original learning rate.

[0233] In S4, the adaptive adjustment phase includes:

[0234] The system monitors and verifies the trend of the performance indicators. If, after the context-aware dynamic update in S303, the model performance improvement is still small or stagnant (i.e., the performance improvement in N consecutive evaluations is less than a preset threshold, such as the improvement in the indicator in 5 consecutive evaluations being less than the preset threshold of 0.5%), it is determined that the performance improvement is small and the model may be trapped in a local optimum. At this time, the system will automatically adjust the hyperparameters used to optimize the learnable weight coefficients (e.g., prioritize adjusting their specific learning rate or reinitialization strategy), or adjust other key hyperparameters (e.g., LoRA scaling factor α and learning rate adjustment factor β). The adjustment range can be set according to experience, or a learning rate restart strategy can be implemented to try to break out of the current predicament.

[0235] If performance improvement remains minimal after multiple adjustments, an early stopping mechanism is triggered, and the model with the best validation performance is selected as the final model; priority is given to adjusting the learnable weight coefficients used to optimize them. hyperparameters (such as its specific learning rate or reinitialization strategy), or other key hyperparameters (such as the LoRA scaling factor a and learning rate adjustment factor b), the adjustment range can be set according to experience. The purpose of this stage is to ensure that the model can continue to learn and optimize its performance on the transformer fault diagnosis task through intelligent hyperparameter adjustment;

[0236] In S5, the real-time diagnosis stage is deployed; including:

[0237] ①Edge computing device deployment: the fine-tuned low-rank adaptive model is deployed on the NVIDIA Jetson Xavier AGX edge computing device; this device has high-performance computing capabilities, uses Volta GPU architecture and 32GB of memory, and can meet the computing needs of the transformer fault diagnosis model in real-time data processing. The edge server is connected to the power system where the transformer is located through Ethernet / IP protocol and receives real-time monitoring data in real time.

[0238] ②Data input and real-time monitoring: real-time monitoring data of the transformer is collected through sensors, and the sensors transmit real-time data to the edge server through Modbus TCP / IP protocol; real-time monitoring data of the transformer includes: input voltage (unit: volt, V), output current (unit: ampere, A), oil temperature (unit: Celsius, ℃), load current (unit: ampere, A), alarm state (such as overload, short circuit, temperature anomaly, etc.);

[0239] ③Real-time fault diagnosis and early warning: the fine-tuned neural network model based on attention mechanism and low-rank representation processes the real-time monitoring data of the transformer after receiving it; specifically including: receiving multi-source data input from the transformer online monitoring system, such as winding temperature, oil temperature, load current, voltage, frequency, partial discharge signal, vibration signal, and gas content (DGA) etc.; converting the multi-source data input into a standard format acceptable by the neural network model based on attention mechanism and low-rank representation according to the pre-set data preprocessing process (including normalization, denoising, outlier rejection, etc.), and constructing a time series input feature tensor;

[0240] The neural network model based on attention mechanism and low-rank representation captures the coupling relationship and dynamic trend between key variables by fusing historical data distribution and current input features, and discriminates the current state based on the enhanced feature expression ability of low-rank structure; at the output end, the neural network model based on attention mechanism and low-rank representation classifies or grades the running state of the transformer, such as "normal", "mild abnormality", "early warning", "serious fault", etc., and can output specific fault types (such as winding overheating, iron core grounding, electrical partial discharge, mechanical looseness, etc.) and confidence scores in combination with abnormal trends;

[0241] At the same time, the neural network model based on attention mechanism and low-rank representation supports a dynamic scoring mechanism based on a sliding time window, continuously judges the health status trend at multiple times, and realizes the early identification and warning of potential faults; if the model continuously determines that the state is abnormal for multiple times, the warning mechanism is triggered; output the alarm signal and link to the upper operation and maintenance system, prompt the maintenance personnel to intervene or adjust the operation strategy, so as to realize the intelligent fault management ability of "running while diagnosing".

[0242] The model performs real-time fault diagnosis based on collected voltage, current, oil temperature and other data, and the diagnosis results include:

[0243] Fault prediction: if the current or oil temperature of the transformer exceeds the set threshold value, a warning will be automatically issued, indicating that the transformer has a risk of failure; Specifically, the current data of the transformer usually comes from the load current sensor in the real-time monitoring system, which reflects the working load and electrical performance of the transformer; The oil temperature is obtained through the oil temperature sensor, which reflects the thermal state inside the transformer oil tank, which is closely related to the heat dissipation capacity of the transformer, the load condition and the efficiency of the cooling system.

[0244] Current threshold:

[0245] The threshold of current data (for example: load current) is usually closely related to the rated current of the transformer. The standard threshold range is usually:

[0246] Normal operation: the current should be kept between 80%-100% of the rated current.

[0247] Warning threshold: if the current continuously exceeds 110%-120% of the rated current, there may be an overload risk.

[0248] Fault risk threshold: if the current exceeds 120% of the rated current (or the upper limit set according to the specific model), it may cause the transformer to overload, overheat or electrical failure, and immediate warning measures need to be taken.

[0249] Oil temperature threshold:

[0250] Oil temperature data (for example: transformer oil temperature) mainly reflects the cooling effect and heat dissipation capacity of the transformer. Its set threshold value is generally based on the rated oil temperature of the transformer and the design standard of the cooling system.

[0251] Normal oil temperature: usually between 40°C to 80°C (the specific value is related to the design specifications, operating environment and type of the transformer).

[0252] Warning oil temperature: when it exceeds 85°C, it may cause the internal temperature of the transformer to be too high, affecting the insulation performance, and entering the warning stage.

[0253] Risk of failure: If the oil temperature consistently exceeds 95°C or 100°C, a high-temperature failure warning should be triggered, indicating a risk of insulation oil degradation, localized overheating, or cooling system failure.

[0254] Fault type identification: The fine-tuned attention mechanism and low-rank representation-based neural network model identifies the fault type, which includes overload, short circuit, and temperature anomaly. The specific determination basis is as follows:

[0255] If the real-time current exceeds the maximum allowable current (e.g., 300A), it is determined as an overload fault;

[0256] If the oil temperature exceeds the set threshold (e.g., 85°C), it is determined as a temperature anomaly;

[0257] Fault type identification: The fine-tuned attention mechanism and low-rank representation-based neural network model is used to identify the fault type of the transformer. The specific process is as follows: Based on the multi-dimensional data (such as current, oil temperature, partial discharge, etc.) from the real-time monitoring system, combined with historical data and features learned during training, the model uses deep learning and attention mechanism to comprehensively analyze the current operating state. First, the model extracts and reduces the dimension of the multi-dimensional input data, and uses the low-rank adaptation mechanism to optimize the representation ability of each layer of features; then, the processed data is determined by the classifier module.

[0258] Fault types include but are not limited to:

[0259] Overload fault: When the transformer load current exceeds the set threshold of the rated current, it indicates that the transformer may have an overload risk;

[0260] Short circuit fault: When the transformer current abnormally rises and continuously exceeds the rated value, accompanied by other data characteristics such as partial discharge, vibration anomaly, etc., a short circuit fault may occur;

[0261] Temperature anomaly: When the oil temperature of the transformer exceeds the set safety threshold (e.g., 85°C or 95°C), it indicates that there may be insufficient heat dissipation or cooling system failure, resulting in temperature anomaly;

[0262] Partial discharge fault: By monitoring the partial discharge signal, when the amplitude of the signal exceeds the normal range, it indicates that the internal insulation material of the transformer may be partially broken or damaged;

[0263] Mechanical failure: When the vibration signal or noise signal is abnormal, it indicates that the mechanical part (such as fan, motor bearing, etc.) of the transformer may have a failure;

[0264] Insulation fault: By monitoring gas content (such as hydrogen, ethylene gas concentration, etc.), a high concentration of gas may indicate that the transformer's insulation material has deteriorated and there is a risk of breakdown.

[0265] The specific criteria for judgment are as follows:

[0266] Current threshold: Based on load current data, if the current continuously exceeds 110%-120% of the rated value, it is judged as an overload fault; if the current fluctuates drastically and matches the characteristics of a short circuit, it is judged as a short circuit fault.

[0267] Oil temperature threshold: If the oil temperature exceeds 85°C, it enters the warning stage; if it exceeds 95°C, it is considered an abnormal temperature, which may lead to a decrease in the insulation performance of the transformer.

[0268] Partial discharge: If the amplitude and frequency of the partial discharge signal exceed the set safety threshold, it is determined to be a partial discharge fault.

[0269] Vibration and noise signals: When the amplitude and frequency of the vibration signal change significantly and exceed the normal range, it is determined to be a mechanical fault.

[0270] ④ Real-time reasoning and decision support: When a transformer fails or has a potential failure, the edge server generates a fault diagnosis report, which includes the fault type, fault time, fault location, and fault severity.

[0271] Figure 7 This invention illustrates the scoring function S(n) applied to the encoder-decoder architecture, which is based on a Transformer model with an encoder-decoder structure. This model consists of N stacked encoder blocks and N stacked decoder blocks. The encoder is responsible for converting the input transformer operation and maintenance data (after embedding and position encoding) into a contextual representation, while the decoder uses the contextual information output by these encoders and its own output embedding (also position-encoded) to progressively generate the target fault diagnosis result. Sub-layers within each encoder and decoder block (such as attention layers and feedforward networks) are connected through residual connections and layer normalization (Add & Normalize) operations to ensure stable training and information flow of the deep network.

[0272] The central to each encoder and decoder block is a multi-head self-attention (MHSA) layer and a feed-forward network (FFN) sublayer. In the decoder, an additional encoder-decoder attention layer is also included to bridge the information between the encoder and the decoder. The structural innovation is that the LoRA module is ingeniously integrated into the linear transformation matrix of these core layers, including the Q, K, V projection and output linear layer of MHSA, the weight matrix of FFN, and the encoder-decoder attention layer. LoRA achieves efficient fine-tuning while freezing most of the original parameters by adding a pair of small low-rank matrices (such as Figure 7 The arrows marked "low-rank adaptation" in the figure show the process of LoRA.

[0273] In addition, another structural innovation of the model is its adaptive attention guidance mechanism. This mechanism dynamically guides the parameter configuration (such as the rank value of the low-rank matrix) of the LoRA module integrated in each layer through the dynamically calculated attention score S(n). This means that the attention score S(n) is not an independent layer, but rather a high-level control signal that dynamically adjusts the "size" and "strength" of multiple LoRA modules within the model, thereby optimizing the allocation of model resources at the structural level and enabling the model to prioritize the allocation of computational resources and parameter updates to more critical feature paths for transformer fault diagnosis. Finally, the output of the decoder passes through a linear layer with LoRA and a Softmax function to produce the final fault diagnosis probability.

[0274] Figure 8 The dynamic rank r(n) in the encoder-decoder architecture diagram of the present application; Figure 8 The figure highlights the extensive integration of the LoRA module in the model and how its core parameter (rank r) is adaptively allocated according to the dynamic strategy. The model as a whole follows the encoder-decoder structure of the Transformer, processing input and output embeddings through positional encoding to capture the sequential information of the sequence.

[0275] The present application integrates the LoRA module extensively into the core linear transformation layer. As shown in the figure, Figure 8As shown, this includes the query (Q), key (K), value (V) projection layers inside the Multi-Head Attention (MHSA), and the linear layer after the MHSA output. In addition, the key linear layers inside the Feed Forward Network (FFN) in the encoder and decoder, and the masked multi-head attention and encoder-decoder attention in the decoder, all integrate LoRA modules. LoRA greatly reduces the number of parameters that need to be updated during training by adding a pair of trainable low-rank matrices (BA) to the original weight matrix (which remains frozen during the fine-tuning stage), thereby achieving efficient fine-tuning of parameters.

[0276] The most core innovation of the present application is reflected in the dynamic rank r(n) mechanism of LoRA. As shown in the annotations beside each module in the figure, Figure 8 As shown in the annotations beside each module in the figure,

[0277] Figure 9 The hierarchical adaptive learning rate mechanism in the present application acts on the encoder-decoder architecture diagram. Figure 9 The hierarchical adaptive learning rate mechanism in the present application acts on the encoder-decoder architecture diagram. Figure 9 The hierarchical adaptive learning rate mechanism in the present application acts on the encoder-decoder architecture diagram. Figure 7 The hierarchical adaptive learning rate mechanism in the present application acts on the encoder-decoder architecture diagram. Figure 8 The hierarchical adaptive learning rate mechanism in the present application acts on the encoder-decoder architecture diagram.

[0278] As shown in the figure, the present application widely integrates Low Rank Adaptation (LoRA) modules into the core components of the Transformer architecture: including Q, K, V projection layers inside Multi-Head Self-Attention (MHSA) and the final output linear layer, linear layers in Feed-Forward Networks (FFN), and the masked multi-head attention and encoder-decoder attention in the decoder part. These integrations achieve parameter-efficient fine-tuning by adding a pair of trainable low-rank matrices (BA) to the original weight matrices.

[0279] The most core structural innovation point of the present application is embodied in the mechanisms of "adaptive low rank adaptation" and "adaptive fine-tuning adaptation". Figure 9 The annotations and connections in the figure explicitly indicate that the low rank adaptation parameters and their learning rates of these key modules (such as multi-head attention, feed-forward network) are not fixed, but are dynamically regulated by a hierarchical adaptive learning rate mechanism. This means that the model can intelligently adjust the rank values of the low rank matrices and their learning rates of different layers or different parameters according to the feedback during the training process or pre-set strategies, so as to realize more fine and efficient parameter updates. This adaptive strategy ensures that the model maintains strong expressive ability while minimizing the fine-tuning cost and optimizing the convergence speed and final performance of the model on the transformer fault diagnosis task.

[0280] The effect data of the transformer fault diagnosis method based on adaptive attention-guided low rank fine-tuning of the present embodiment is shown in Table 1.

[0281] Table 1 Effect data table

[0282]

[0283] Embodiment 3

[0284] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the transformer fault diagnosis method based on adaptive attention-guided low rank fine-tuning of embodiment 1 or 2 when executing the computer program.

[0285] Embodiment 4

[0286] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the transformer fault diagnosis method based on adaptive attention-guided low rank fine-tuning of embodiment 1 or 2.

Claims

1. A transformer fault diagnosis method based on low-rank fine-tuning of adaptive attention guidance, characterized in that, Comprise the following steps: S1, data acquisition and pretreatment stage; S101, collect the relevant operation and maintenance data of the transformer in the target power system; S102, standardize and pretreat the collected data, and construct a localized transformer fault diagnosis data set; S2, initialization stage; S201, sample data from the constructed local transformer fault diagnosis data set, and perform forward propagation and back propagation on the original pre-trained language model, collect the gradient, attention activation map and hidden state information of each layer; S202, based on the collected information, calculate the gradient importance score, activation importance score and value change importance score of each layer respectively, and construct a unified attention score index according to the learnable weight coefficients λ1, λ2, λ3, λ4, λ5, λ6, λ7, λ8, λ9 combination; S203, according to the obtained attention score, allocate the rank value and learning rate required by each layer, and initialize the LoRA parameter matrix A and B of the corresponding layer to realize the initial configuration of the low-rank structure; S3, fine-tuning stage; S301, execute the standard training batch process on the fault diagnosis data, i.e., the relevant operation and maintenance data of the transformer, including forward propagation, loss calculation and back propagation, and complete the local update of the parameters; S302, real-time monitoring of the neural network model training dynamics based on attention mechanism and low-rank representation, including performance indicators, training loss and gradient dynamics; S303, according to the performance and training dynamics of the neural network model based on attention mechanism and low-rank representation monitored, when the preset context-aware trigger condition is met, dynamically update the attention score and rank allocation strategy to realize the adaptive adjustment of the low-rank structure; S304, at the end of each training, evaluate the overall effect based on the validation set diagnosis performance, and decide whether to adjust the rank allocation strategy according to the feedback of the neural network model based on attention mechanism and low-rank representation; S4, adaptive adjustment stage; S401, monitor the change trend of the validation indicators, if the performance improvement is small after continuous evaluation, automatically adjust the key hyperparameters or implement the learning rate restart strategy; S402, if the performance still has no significant improvement after multiple adjustments, trigger the early stopping mechanism, and select the model with the best validation performance as the final model; S5, deployment and real-time diagnosis stage; Deploy the fine-tuned neural network model based on attention mechanism and low-rank representation to the edge server to realize online analysis and diagnosis of transformer operation logs or real-time monitoring data; In S203, resource allocation, based on the attention score S(n), according to the dynamic rank allocation strategy and hierarchical adaptive learning rate mechanism, allocate the rank value r(n) and learning rate lr(n) of each layer, including the following steps: Basic parameter setting and rank value, learning rate allocation: Setting base parameters including base rank value r base , maximum rank value r max , scaling factor a, base learning rate lr base , and learning rate adjustment factor b, assigning rank value r(n) and learning rate lr(n) to each layer based on the calculated attention score S(n); Rank value allocation, the rank value r(n) is allocated by the following formula: r(n) = r base + a · (S(n) · (r max - r base )); Learning rate allocation, the hierarchical adaptive learning rate lr(n) is allocated by the following formula: lr(n) = lr base x (1 + β - s(n)); For each layer, initialize the corresponding LoRA parameter matrix A and B according to the allocated rank value, as follows: LoRA parameter matrix initialization, the corresponding LoRA parameter matrix A and B are initialized according to the assigned rank value, wherein the A matrix is initialized by normal distribution, and the B matrix is initialized by all zero; In S203, the attention score and the rank allocation strategy are periodically updated according to the performance of the neural network model based on attention mechanism and low rank representation after every N training batches, so as to realize dynamic adaptation of the low rank structure; including: The attention scores of each layer are updated by sliding window averaging: S(l) new = 0.7 · S(l) old + 0.3 · S(l) current ; S(l) old S(l) denotes the attention score at the last update of the l-th layer current S(l) denotes the attention score at the last update of the l-th layer The rank allocation strategy is periodically updated, including: The total rank budget constraint is: Rank value computation logic:

2. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 1, characterized in that, In S101, the related operation and maintenance data of the transformer include inspection records, maintenance logs, alarm information, running parameters, environmental parameters, unstructured or semi-structured text data; In S102, the collected data is standardized, cleaned and preprocessed; that is, the collected data is sequentially denoised, normalized, missing value filled, classified and labeled.

3. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 1, characterized in that, In S201, the gradient, attention activation map and hidden state information of each layer are collected by forward propagation and backward propagation of the original pre-trained language model on the sampled data in the constructed local transformer fault diagnosis data set; including: Collect the gradient of each layer: during back propagation, the original pre-trained language model calculates the error according to the loss function, and the error is transmitted from the output layer to the input layer in the reverse direction, so as to calculate the gradient of each layer weight matrix and collect the gradient information n represents the nth layer, and the gradient information The sensitivity of the nth layer parameter to the loss function under the current task The attention activation map of each layer is collected: the original pre-trained language model synchronously collects key signals in the forward propagation process, that is, the extraction of attention activation information; the attention activation map, that is, the attention activation information, is the attention weight matrix generated by each attention head in the original pre-trained language model when processing the input sequence; The hidden state information of each layer is collected; the hidden state information refers to the intermediate representation result output by each encoder or decoder in the forward propagation process of the Transformer model, including the context semantic features of the input data after nonlinear transformation in the current layer.

4. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 1, characterized in that, The original pre-trained language model is BERT; BERT includes 12 encoder layers, and the input text is first processed by the embedding layer and then sequentially passes through each encoder layer; the output of each encoder layer is the hidden state of the encoder layer, which is a three-dimensional tensor with a shape of [batch_size, seq_len, hidden_size], wherein batch_size represents the sample batch size; seq_len represents the length of the input sequence; hidden_size represents the feature dimension of the current layer.

5. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 1, characterized in that, In S202, the gradient importance score G(n) of each layer is calculated by the following formula: wherein, denotes the gradient of the nth layer weight matrix, || · ||Fdenotes the Frobenius norm, i.e. the square root of the sum of the squares of all elements in the corresponding matrix; In S202, the activation importance score is calculated using the activation map of the self-attention mechanism, and the calculation steps are as follows: Standardize by head number, divide the total difference by the number of attention heads in the layer to calculate the average single-head difference; the total difference is the sum of the L1 difference between each attention matrix in the same layer; Standardize by sequence length, divide the average single-head difference by the square of the input sequence length; The result after sequence length standardization is processed by a dynamic normalization function to obtain the activation importance score.

6. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 5, characterized in that, The result after sequence length standardization is processed by a dynamic normalization function, including: In the initial training stage, i.e. t < 0.4T, t is the current training step, and T is the total training step, global maximum normalization is used for processing; In the middle training stage, i.e. 0.4T ≤ t < 0.6T, a hybrid function is used for processing; In the late training stage, i.e. t ≥ 0.6T, quantile normalization is used for processing.

7. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 1, characterized in that, In S202, the value change importance score V(n) is calculated by the following formula: Wherein, H(n) and H0(n) are the hidden state outputs of the nth layer in the fine-tuning and the original model respectively; H(i) and H0(i) are the hidden state outputs of the ith layer in the fine-tuning and the original model respectively; Or, the value change importance score V(n) is calculated by the following formula: Wherein, α1, α2, α3 are weight coefficients, and α1+α2+α3=1, V(n) represents the value change importance score of the nth layer; In S202, the attention score S(n) is calculated by the following formula: S(n) = Sigmoid(λ1 · G(n) + λ2 A(n) + λ3 · V(n) + λ4 G(n) · A(n) + λ5 · A(n) · V(n) + λ6 · G(n) · V(n) + λ7 · G(n) 2 + λ8 A(n) 2 + λ9 · V(n) 2 ) Wherein, G(n) represents the gradient importance score of the nth layer, A(n) represents the activation importance score of the nth layer, V(n) represents the value change importance score of the nth layer, λ1, λ2, λ3, λ4, λ5, λ6, λ7, λ8, λ9 are learnable weight coefficients.

8. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 5, characterized in that, Adaptive adjustment of LoRA matrix dimension is performed, and the specific strategy is as follows: Rank expansion: when increasing from r to r+Δr, the newly added parameter matrix block is initialized orthogonally, and the principal components of the original matrix are inherited through SVD decomposition; Rank contraction: when increasing from r to r-Δr, the feature components corresponding to the first Δr largest singular values before singular value decomposition are retained.

9. The transformer fault diagnosis method based on adaptive attention-guided low-rank fine-tuning according to claim 1, characterized in that, In S4, the adaptive adjustment stage; Including: When detecting that the model performance improves slightly or stagnates during the fine-tuning process, i.e. the evaluation index improves less than the preset threshold for consecutive N times, it is determined that the performance improves slightly, and it is determined that the model may fall into local optimum; automatically adjust the key hyperparameters or implement the learning rate restart strategy; If the performance still improves slightly after multiple adjustments, trigger the early stopping mechanism, and select the model with the best validation performance as the final model.

10. The transformer fault diagnosis method based on low-rank fine-tuning of adaptive attention guidance according to any one of claims 1-9, characterized in that, In S5, the deployment and real-time diagnosis stage; including: ①Edge computing device deployment: the low-rank adaptation model after fine-tuning is deployed on the NVIDIA Jetson Xavier AGX edge computing device; ②Data input and real-time monitoring: real-time monitoring data of the transformer is collected through sensors, and the sensors transmit real-time data to the edge server through the ModbusTCP / IP protocol; the real-time monitoring data of the transformer includes: input voltage, output current, oil temperature, load current, alarm state; ③Real-time fault diagnosis and early warning: the neural network model based on attention mechanism and low-rank representation after fine-tuning processes the real-time monitoring data of the transformer; specifically including: receiving multi-source data input from the transformer online monitoring system, converting the multi-source data input into a standard format acceptable by the neural network model based on attention mechanism and low-rank representation according to the preset data preprocessing process, and constructing a time series input feature tensor; The neural network model based on attention mechanism and low-rank representation fuses the historical data distribution and the current input features, uses the attention mechanism to capture the coupling relationship and dynamic change trend between key variables, and discriminates the current state based on the enhanced feature expression ability of the low-rank structure; at the output end, the neural network model based on attention mechanism and low-rank representation classifies or grades the running state of the transformer, and can output the specific fault type and confidence score in combination with the abnormal trend; At the same time, the neural network model based on attention mechanism and low-rank representation supports a dynamic scoring mechanism based on a sliding time window, continuously judges the health state trend at multiple times, and realizes the early identification and warning of potential faults; if the model continuously determines that the state is abnormal for multiple times, the warning mechanism is triggered; The diagnosis results include: Fault prediction: if the current or oil temperature of the transformer exceeds the set threshold, a warning will be automatically issued, prompting that the transformer has a fault risk; Fault type identification: the fine-tuned neural network model based on attention mechanism and low-rank representation identifies the fault type, and the fault type includes overload, short circuit and temperature anomaly; the specific determination basis is: If the real-time current exceeds the maximum allowable current, it is determined as an overload fault; If the oil temperature exceeds the set threshold, it is determined as a temperature anomaly; Real-time reasoning and decision support: when the transformer has a fault or a potential fault, the edge server generates a fault diagnosis report, which includes the fault type, fault time and fault location, and fault severity.

Images