A multi-site transformer fault diagnosis model collaborative training method and system

Abstract

This invention provides a collaborative training method and system for a multi-site transformer fault diagnosis model, comprising: collecting runtime sequence data of multi-site transformers; generating initial site summary vectors by performing intra-site self-attention during forward propagation; generating global context-aware site vectors through cross-site cross-attention; and feeding back to modulate the next round of intra-site attention; partitioning the model weight tensor during training; calculating activity scores based on the average L1 norm normalization result of the gradient of the previous training batch and the weight density; updating partition parameters by skipping partitions with inverse probability according to the score; scaling the learning rate of non-skipped partitions according to the score; and unloading and reloading low-level partitions between GPU and CPU; after backpropagation, decomposing the gradients of each site into consensus components and novel components; weighting the consensus components according to the fault rarity; dynamically adjusting the weights of the novel components according to the total activity score and the site contribution; and updating the global model after fusion to balance diagnostic accuracy, cross-site generalization ability, and training resource utilization efficiency.

Description

Technical Field

[0001] This application belongs to the field of training, and in particular relates to a collaborative training method and system for a multi-site transformer fault diagnosis model. Background Technology

[0002] As a core hub in the power system, the operating status of power transformers is crucial to the safety and stability of the entire power grid. Deep learning-based fault diagnosis technology has been widely applied due to its powerful data fitting and feature extraction capabilities. In actual power grid operation, transformers are widely distributed across multiple sites in different regions. Each site is affected by numerous factors such as operating environment, load characteristics, equipment model, and service life, resulting in highly diverse and complex operational data across time and space. Traditional fault diagnosis models are trained independently based on local data from a single site, making it difficult to utilize the global data features of a wide-area power grid. This leads to insufficient generalization ability when facing unknown operating conditions or across sites. To break down data silos, multi-site collaborative training is gradually becoming a development trend. How to uncover the deep correlations between multi-site sequence data during collaborative training, enabling the model to overcome the limitations of a local perspective and possess global contextual awareness, thus achieving higher-precision fault diagnosis, is a crucial problem that urgently needs to be solved.

[0003] In the process of leveraging complex model architectures to improve fault diagnosis performance, the scale of model parameters expands dramatically, posing severe challenges to hardware resources and training efficiency for multi-site collaborative training. Large-scale deep learning models not only require massive computational power during training but also consume enormous amounts of GPU memory during backpropagation and gradient calculation, making it difficult for many substations with limited hardware to reliably support the entire training process. Traditional distributed collaborative training strategies employ a simplistic gradient averaging aggregation method when dealing with complex and non-independent transformer operating data from multiple sites. This approach fails to account for the imbalance in fault data distribution across different sites, easily leading to the submersion of globally rare fault features by local common faults and hindering reliable incentivization of the model to explore novel fault features. Furthermore, all model parameters participate in equally intensive computation and updates in each iteration, resulting in wasted computational and memory resources. Therefore, there is an urgent need for a comprehensive method that can intelligently schedule underlying hardware resources, manage memory and computational load, and balance global consensus and novel feature exploration during collaborative updates, in order to break through computational bottlenecks and improve model training efficiency and diagnostic performance. Summary of the Invention

[0004] To improve the global recognition accuracy and hardware adaptation efficiency of power transformer fault diagnosis models under multi-site collaborative training, this invention provides a method and system for collaborative training of multi-site transformer fault diagnosis models.

[0005] The first aspect of this invention provides a collaborative training method for a multi-site transformer fault diagnosis model, comprising the following steps:

[0006] The system acquires transformer operation data sequences from multiple sites and performs iterative attention processing during forward propagation: intra-site self-attention is performed on each site sequence to generate an initial site summary vector; cross-site cross-attention is performed based on each initial site summary vector to obtain a global context-aware site vector; this vector is then fed back to the next round of intra-site self-attention input features for feature modulation.

[0007] During training, the model weight tensor is partitioned, and the tensor activity score is determined based on the normalized result of the average L1 norm of the gradient of each partition in the previous training batch and the weight density. After backpropagation, the parameter update of the corresponding partition is skipped with a probability inversely proportional to the tensor activity score. For partitions that are not skipped, the learning rate is adjusted by a dynamic factor proportional to the tensor activity score. When the tensor activity score is lower than the unloading threshold, the corresponding partition is unloaded from the graphics processor to the central processing unit and reloaded when needed in subsequent calculations.

[0008] After backpropagation, the gradient vector of each site relative to the global average gradient is decomposed into consensus components and novel components. The consensus components are weighted according to the global rarity of fault type, and the novel components are dynamically weighted according to the global sum of activity scores of each region and the contribution of the novel components of each site. When the sum of the norms of the novel components is lower than the preset norm threshold, differentiated site exploration weights are introduced, and the weight amplitude is limited by smoothing terms and exploration coefficient ranges. When the global sum of activity scores is low, the weight of the novel components is enhanced, and the weighted consensus components and novel components are aggregated to update the global model.

[0009] A second aspect of the present invention provides a collaborative training system for a multi-site transformer fault diagnosis model, comprising the following modules:

[0010] The generation module is used to acquire multi-site transformer operation data sequences and perform iterative attention processing during forward propagation: perform intra-site self-attention on each site sequence to generate initial site summary vectors; perform cross-site cross-attention based on each initial site summary vector to obtain global context-aware site vectors; and feed this vector back to the input features of the next round of intra-site self-attention for feature modulation.

[0011] The computation module is used to partition the model weight tensor during training. It determines the tensor activity score based on the normalized result of the average L1 norm of the gradient of each partition in the previous training batch and the weight density. After backpropagation, the corresponding partition parameter update is skipped with a probability inversely proportional to the tensor activity score. For partitions that are not skipped, the learning rate is adjusted by a dynamic factor proportional to the tensor activity score. When the tensor activity score is lower than the unloading threshold, the corresponding partition is unloaded from the graphics processor to the central processing unit and reloaded when needed in subsequent calculations.

[0012] The aggregation module, after backpropagation, decomposes the gradient vector of each site relative to the global average gradient into consensus components and novel components. It weights the consensus components according to the global rarity of fault types and dynamically weights the novel components according to the global sum of activity scores in each region and the contribution of each site's novel components. When the sum of the norms of the novel components is lower than a preset norm threshold, it introduces differentiated site exploration weights and uses smoothing terms and exploration coefficient ranges to limit the weight amplitude. When the global sum of activity scores is low, it enhances the weight of the novel components and aggregates the weighted consensus components and novel components to update the global model.

[0013] This invention improves fault diagnosis accuracy by performing iterative attention calculations during data forward propagation, combining site-specific self-attention with global cross-attention, and utilizing global context-aware vectors for feedback modulation to extract global and local correlation features across multiple sites. A partitioned memory management mechanism based on tensor activity scores is employed. Activity scores are calculated based on the normalized parameter gradient activity and weight density. Parameter updates for some partitions are skipped based on probability, and the learning rate is scaled proportionally using this score. Low-score tensors are offloaded to the central processing unit, reducing GPU memory usage and computational overhead, and improving collaborative training efficiency. During backpropagation, gradients are decomposed into consensus and novelty components, which are weighted differently based on the global rarity of faults, the sum of activity scores, and the contribution of site novelty components. This strategy balances the model's learning of various fault features, enhancing its generalization ability in complex multi-site scenarios. Attached Figure Description

[0014] Figure 1 A flowchart of a collaborative training method for a multi-site transformer fault diagnosis model;

[0015] Figure 2 This is a schematic diagram of global average pooling;

[0016] Figure 3 This is a diagram illustrating attention weights. Detailed Implementation

[0017] The terms "first," "second," and "third," etc., used in this application specification, claims, and the aforementioned drawings are used to distinguish different objects, not to limit a specific order.

[0018] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0019] Example 1

[0020] In Embodiment 1 of the present invention, as Figure 1 As shown, a collaborative training method for a multi-site transformer fault diagnosis model includes the following steps:

[0021] S1. Obtain the multi-site transformer operation data sequence and perform iterative attention processing during forward propagation: perform intra-site self-attention on each site sequence to generate an initial site summary vector; perform cross-site cross-attention based on each initial site summary vector to obtain a global context-aware site vector; feed this vector back to the next round of intra-site self-attention input features for feature modulation.

[0022] The pandas data analysis library was used to read time-series data from oil temperature, current, and voltage sensors at each station. The StandardScaler algorithm was applied for zero-mean standardization, converting the data into a floating-point tensor format supported by the PyTorch framework. This tensor data was then input into the embedding layer to generate a high-dimensional feature matrix. During the model's forward propagation phase, a linear mapping layer was used to generate a query matrix, key matrix, and value matrix for each station. The scaled dot product of the query matrix and the transpose of the key matrix was calculated, and probability normalization was performed along a specified dimension to obtain the local self-attention weight matrix. This weight matrix was then multiplied by the value matrix to obtain the hidden state representation of the current Transformer layer.

[0023] The function performs average pooling on the hidden state sequence dimension to compress the sequence length and extract the initial site summary vector for that site. The master node collects the initial site summary vectors of all sites and concatenates them along either the site dimension or the site index dimension to form the global key matrix and global value matrix. The initial site summary vector of the current site is used as the global query matrix, and the multi-head attention mechanism algorithm is called again to perform cross-attention calculation, generating a site vector that incorporates the global state context. Using the Feature Linear Modulation (FiLM) algorithm, a two-layer multilayer perceptron maps this context-aware site vector into scaling coefficient vectors and offset coefficient vectors. These two coefficient vectors are broadcast and applied element-wise to the input feature tensor of the current site before it enters the next round of internal self-attention calculation. This modulated input feature tensor is then used to generate the query matrix, key matrix, and value matrix for the next round, achieving signal feedback injection and modulation before residual connection.

[0024] In one embodiment, the step of performing intra-site self-attention on each site sequence to generate an initial site summary vector includes:

[0025] The output feature matrix of the self-attention within each site is input into a feedforward neural network with GeLU activation function to perform nonlinear mapping of the multidimensional vector space and obtain the mapped feature tensor.

[0026] A global average pooling operation is performed on the mapped feature tensor along the time step dimension to compress the time step dimension, retain the feature dimension, and extract the one-dimensional site summary vector corresponding to each batch of samples as the initial site summary vector.

[0027] In the forward propagation stage of the Transformer model architecture for multi-site power transformer fault diagnosis, when the output feature matrix of the self-attention within each site is obtained, the tensor dimension of the feature matrix is ​​typically set to batch size B × time step length T × feature dimension. For example, the parameters can be configured as B=32, T=128, The feedforward neural network structure with GeLU activation function consists of a first layer of fully connected linear network, a GeLU activation function layer, and a second layer of fully connected linear network connected in sequence. The input of this feedforward neural network is the output feature matrix of the self-attention within each station, and the output is a mapped feature tensor. This output feature matrix is ​​input into the feedforward neural network. The first layer of linear network expands the feature dimension from 512 to a hidden layer dimension of 2048, preferably four times the original dimension. The GeLU function is used to achieve a non-linear smooth activation mapping that preserves small negative values. Then, the second layer of linear network reduces the dimension of the activated hidden layer feature tensor and maps it back to the original feature dimension of 512, resulting in a mapped feature tensor that preserves deep local spatiotemporal interaction information.

[0028] For the 32×128×512 mapped feature tensor, a global average pooling operation is performed along the 128-time-step dimension, i.e., dimension T. Specifically, the calculation logic involves accumulating the 512-dimensional feature vectors at each time step using the tensor reduction operator and then uniformly dividing by the total number of time steps, T=128. Through this operation, the average temporal feature attributes of each observation time step within the corresponding time window are extracted, reducing the dimensionality dependency of the sequence due to the time step length and compressing the three-dimensional feature tensor into a shape of... This refers to a 32×512 two-dimensional tensor. The 512-dimensional one-dimensional sequence vectors corresponding to each batch of slices from this two-dimensional tensor are extracted and output as the initial site summary vector for the current iteration round. This vector is stored in a memory register for direct use in subsequent global cross-site information exchange phases. Figure 2 As shown.

[0029] In one embodiment, the step of performing cross-site cross-attention based on each initial site summary vector to obtain a global context-aware site vector includes:

[0030] The initial site summary vector of the target site is converted into a query vector matrix through a first linear mapping network;

[0031] The initial site summary vectors of all sites are concatenated and then converted into key vector matrices and value vector matrices through the second and third linear mapping networks, respectively.

[0032] Calculate the dot product of the query vector matrix and the key vector matrix, divide by the scaling factor, and then process using the Softmax function to obtain the attention weight matrix;

[0033] Multiplying the attention weight matrix with the value vector matrix yields the global context-aware site vector.

[0034] When implementing global contextual information interaction modeling among transformers at multiple sites, the system pre-initializes and loads a first linear mapping network, a second linear mapping network, and a third linear mapping network in memory. The network structures of these three networks are all single-layer fully connected networks, and the learnable weight matrix size is set to [size missing]. The model embedding dimension And attention head projection dimension The first linear mapping network takes the initial site summary vector of the target site as input and outputs a query vector matrix Q. Before cross-site concatenation, the system aligns the initial site summary vectors of each site according to a unified sampling time window or a unified batch index, ensuring that positions in the same batch correspond to the same time window or the same training sample group. The second linear mapping network takes the global feature context tensor obtained by concatenating the initial site summary vectors of all sites as input and outputs a key vector matrix K. The third linear mapping network also takes the global feature context tensor as input and outputs a value vector matrix V.

[0035] For any target site processed in the parallel stream, the initial site summary vector of size B×512 generated for that target site is fed forward into the first linear mapping network to perform linear spatial projection, generating a query vector matrix Q of size B×512. Simultaneously, a memory scheduling mechanism extracts all N concurrent monitoring sites in the network. Assuming a network size of N=10, the initial site summary vectors are merged and concatenated along a specific site dimension, typically the first dimension of a tensor, to form a global feature context tensor of size B×N×512. This tensor is then fed as independent inputs into the second and third linear mapping networks, mapping out a key vector matrix K of size B×N×512 and a value vector matrix V of size B×N×512, preserving the state information of each site.

[0036] During the core cross-attention computation phase, the hardware accelerator utilizes high-speed batch matrix multiplication (BMM) to adjust the target site's query vector matrix Q to a B×1×512 form and perform an inner product operation with the transpose of the global key vector matrix K, B×512×N, to obtain the initial dot product score matrix. To prevent gradient underflow during backpropagation caused by extreme values, this dot product matrix is ​​divided by a fixed scalar scaling factor, theoretically set to the square root of the feature dimension, i.e., 22.627. The scaled dot product score matrix is ​​then input into the Softmax activation function, where a discrete probability distribution is calculated along the site dimension for exponential normalization, resulting in an attention weight matrix of shape B×1×N, representing the contribution of other sites to the temporal feature information of the current target site. The probability-normalized attention weight matrix is ​​then multiplied and added with the value vector matrix V representing global features using a tensor multiplication-addition operation. By weighted summation of the distribution weights, the observation representations of different sites are aggregated, outputting a globally context-aware site vector with a dimension restored to B×512, such as... Figure 3 As shown.

[0037] In one embodiment, feeding the vector back to the next round of in-station self-attention input features for feature modulation includes:

[0038] The global context-aware site vector is input into a parameter generation network containing two independent fully connected layers, which respectively map and output a scaling factor vector for feature scaling and a bias factor vector for feature translation.

[0039] Before the next round of internal self-attention calculation, a single-dimensional axis is inserted according to the actual dimensions of the scaling factor vector and the bias factor vector, and broadcasting is performed at least in the time step dimension; when the scaling factor vector and the bias factor vector do not contain the batch dimension, broadcasting is performed simultaneously in the batch dimension to align its dimension with the input feature tensor.

[0040] The input feature tensor is multiplied element-wise with the broadcast scaling coefficient vector, and the result of the multiplication is added element-wise with the bias coefficient vector that has undergone the same broadcast, thus completing the feature affine transformation modulation based on the global context signal.

[0041] To feed back the high-dimensional aggregated global macroscopic fault prior information to the fine-grained spatiotemporal sequence modeling tasks at each site, a parameter generation network module dedicated to feature modulation was constructed, similar to the generation component in conditional layer normalization. The parameter generation network structure consists of two parallel fully connected layers with independent activation functions. The output dimension of each fully connected layer is constrained and aligned with the channel dimension of the internal hidden states, set as follows: The parameter generation network takes a globally context-aware site vector as input and outputs a scaling factor vector for feature scaling and a bias factor vector for feature translation, respectively. A globally context-aware site vector of shape B×512 is input to this network branch. Thus, the first independent fully connected branch maps and generates a scaling factor vector for performing signal amplitude scaling. The second independent fully connected branch synchronously maps the output of the bias coefficient vector used for the translation of the feature center position. .

[0042] Before the engine triggers the next round of internal self-attention forward computation of the time series data at the current site, the memory bus first acquires the initial input feature tensor of size B×T×512 pre-allocated to the site. To meet the dimensional constraints of element-wise tensor operations, the framework utilizes the underlying built-in tensor expansion and reshaping mechanisms, such as the BroadcastTo semantic instruction, to insert a single-dimensional axis on the missing time step length T dimension. The two B×512 scaling coefficient vectors and bias coefficient vectors are then copied across time and uniformly broadcast and aligned to expand to the same B×T×512 three-dimensional shape as the target input feature tensor. The conditional affine transformation modulation process then proceeds: the original input feature tensor is first multiplied by the scaling coefficient vector after dimensional broadcasting (i.e., element-wise multiplication). The temporary scaled feature result obtained from the multiplication is then added element-wise to the bias coefficient vector expanded by the same broadcasting mechanism, performing an arithmetic bias operation. This enables the transformer sequence features, which originally only had local information, to be fully integrated into and respond to the feedback modulation of macroscopic abnormal signals in the global context through deep affine transformation processing. The input feature tensor after the affine transformation then enters the next round of internal self-attention layer, and the next round of query matrix, key matrix, and value matrix are generated from this input feature tensor.

[0043] S2, during training, the model weight tensor is partitioned, and the tensor activity score is determined based on the normalized result of the average L1 norm of the gradient of each partition in the previous training batch and the weight density. After backpropagation, the parameter update of the corresponding partition is skipped with a probability inversely proportional to the tensor activity score. For partitions that are not skipped, the learning rate is adjusted by a dynamic factor proportional to the tensor activity score. When the tensor activity score is lower than the unloading threshold, the corresponding partition is unloaded from the graphics processor to the central processing unit and reloaded when needed in subsequent calculations.

[0044] A tensor-parallel algorithm using the MegatronLM framework is employed. The `torchchunk` function proportionally divides the massive weight tensors of the fully connected and attention layers along the hidden layer dimension into multiple sub-partitions, which are then distributed across different GPUs within the cluster using an NVIDIA NCCL communication backend. In training batches other than the first, the system reads the parameter gradient tensor corresponding to the current partition after backpropagation of the previous batch. In the first training batch, since there is no parameter gradient tensor from the previous batch, the system sets the initial activity score of each tensor partition to a preset default value, or forces all tensor partitions to perform a complete gradient calculation to obtain the initial average parameter gradient statistics, which serve as the basis for calculating the tensor activity score in subsequent training batches. The average parameter gradient L1 norm is obtained by calling the `torchabs` and `torchmean` functions to calculate the average absolute value of all gradient elements in the parameter gradient tensor corresponding to the current partition. Before weighted combination, the average parameter gradient L1 norm is normalized according to a preset gradient reference upper limit or historical moving average, mapping it to the interval between 0 and 1. The number of non-zero elements with absolute values ​​greater than a minimum threshold in the current weight matrix is ​​counted, and divided by the total number of elements in the matrix to obtain the weight density parameter. Using an addition operator, the normalized gradient activity multiplied by a preset decay coefficient is added to the weight density multiplied by a compensation coefficient to obtain the scalar tensor activity score for that partition.

[0045] During the backpropagation phase of the current batch, uniformly distributed random numbers between 0 and 1 are generated. The basic skip constant is divided by the sum of the tensor activity score and the smoothing term constant, and the result is truncated between 0 and a preset maximum skip probability, where the preset maximum skip probability is less than 1, to obtain the skip probability. If the generated random number is less than this skip probability, the weight writing of that sub-partition is skipped or masked when the optimizer performs parameter updates, but the complete gradient calculation and error propagation process of that partition in this backpropagation is retained. If the generated random number is greater than or equal to the skip probability, the parameter update is performed normally. For tensor partitions that are not skipped, the global basic learning rate is extracted, multiplied by a dynamic factor obtained by scaling the square root of the tensor activity score, and the scaled local learning rate is passed into the torchoptimAdamW optimizer object to independently update the weights of that partition. In terms of memory management, a fixed floating-point unloading threshold is set, and logical judgment statements are used for comparison.

[0046] When the tensor activity score of a partition falls below the offload threshold, the DeepSpeed ​​framework's ZeroOffload mechanism is triggered. This calls the `torchTensorto` function with the device parameter set to `cpu`, and enables the non-blocking asynchronous flag. Using the PCIe bus, the data and optimizer state of the tensor partition are asynchronously migrated from the GPU memory to the host CPU's main memory, freeing up GPU memory space. Before subsequent backpropagation at the network layer or the next round of forward propagation reaches this level, the CUDA stream listener mechanism uses the `forward` static method to preemptively call the `torchTensorto` function with the device parameter set to `cuda`. This re-copy and loads the required tensor data into the corresponding GPU memory, thus hiding communication latency.

[0047] In one embodiment, determining the tensor activity score based on the normalized result of the average L1 norm of the gradients of each partition in the previous training batch and the weight density includes:

[0048] Obtain the parametric gradient tensor generated in the previous training batch for the current tensor partition, calculate the sum of the absolute values ​​of all elements of the parametric gradient tensor and divide it by the total number of elements of the parametric gradient tensor to obtain the average parametric gradient L1 norm.

[0049] The number of weight parameters in the current tensor partition whose absolute value is less than a preset near-zero threshold is counted. The ratio of the number of these parameters to the total number of weight parameters in the partition is taken as the weight sparsity. The weight density is obtained by subtracting the weight sparsity from 1.

[0050] The average parameter gradient L1 norm is normalized and multiplied by the first weight coefficient. The weight density is then multiplied by the second weight coefficient. The two product terms are added together to obtain the activity score of the tensor partition.

[0051] When initiating back-update scheduling for a Transformer model for power transformer fault diagnosis constructed with massive parameters based on an improved hybrid parallel strategy, the evaluation engine system represents the actual gradient update efficiency of the model weight tensor partitions at each layer. The engine automatically identifies the parameter gradient tensor copies generated in the previous training batch for the currently examined target tensor partition using a gradient recognition interface pre-built at the bottom layer of the backpropagation computation graph. At the hardware operator level, the absolute values ​​of all floating-point elements within the parameter gradient tensor are calculated and a global summation is performed. The summation result is then divided by the total number of elements in the parameter gradient tensor; for example, assuming a linear subpartition of a projection layer contains a total of... These parameters are used to calculate the average parameter gradient L1 norm, which represents the activity level of the gradient signal. For example, a typical order of magnitude distribution of this norm can be obtained through testing, approximately [value missing]. Nearby. Before weighted fusion with the weight density, the system normalizes the average parameter gradient L1 norm according to a preset gradient reference upper limit or historical moving average to obtain the normalized gradient activity within the closed interval of 0.0 to 1.0.

[0052] A density check mechanism is applied to the numerical distribution of the weight parameters of the current tensor partition. The system loads a preset near-zero threshold, and the empirically optimized value is... Count all absolute values ​​within the current target tensor block that are less than 1. The number of weight parameters is calculated, and the weight sparsity is obtained by dividing this number by the total number of weight parameters in the tensor partition. Subtracting this weight sparsity from 1.0 yields the weight density factor, which fully identifies the actual parameter carrying capacity of the current subdomain of the model; the result is within a closed interval of 0.0 to 1.0. The scheduling system calls the first weight coefficient, for example, configured as follows: The second weighting coefficient, for example, is configured as To ensure the sum of the two coefficients is 1.0, the control unit multiplies the calculated normalized gradient activity by the first weight coefficient and multiplies the calculated weight density by the second weight coefficient. An accumulator algebraically adds the two independent product terms to achieve linear information fusion, outputting a uniquely determined scalar value. This value serves as the activity score for the next stage scheduling algorithm to determine the priority of model sub-partitions for cross-device transfer between different computing cores, such as CPU and GPU architectures.

[0053] In one embodiment, after the backpropagation is completed, the corresponding partition parameter update is skipped with a probability inversely proportional to the tensor activity score, including:

[0054] Set a base skip constant, divide the base skip constant by the sum of the activity score of the current tensor partition and the smoothing term constant, and truncate the calculation result between 0 and the preset maximum skip probability, where the preset maximum skip probability is less than 1, to obtain the skip probability;

[0055] Generate a random test floating-point number that follows a uniform distribution from 0 to 1 within the system;

[0056] The random test floating-point number is compared with the skip probability. If the random test floating-point number is less than the skip probability, a freeze command is triggered. During the optimizer update phase of the current training step, the update of the tensor partition parameters is stopped, but the partition retains its complete gradient calculation and error propagation functions in backpropagation.

[0057] To implement memory fragmentation and intelligently allocate computing resources in a fine-grained probabilistic triggering mode, the system resource controller pre-constructs two hyperparameter constants in the global configuration file: a basic skip constant that serves as the upper limit of the degradation of the skip algorithm, for example, set to c=0.08, and a numerical smoothing term constant to prevent division-by-zero anomalies, for example, set to c=0.08 for FP32 precision. When the reverse computation sequence starts, the engine reads the active score of the current tensor partition, and first uses the underlying floating-point adder to combine the active score with the smoothing term constant. The denominator is constructed by safe accumulation, and the mathematical derivation is performed by dividing the basic skip constant c by this denominator. Hardware instructions are used to truncate and lock the calculated value between 0.0 and a preset maximum skip probability, where the preset maximum skip probability is less than 1, resulting in a high-order skip probability parameter with inversion dependency characteristics for the active state of the same partition. Once the reverse propagation calculation process is initiated, a hardware-level high-speed pseudo-random number generator generates a temporary random test floating-point number uniformly distributed between 0 and 1 at the underlying level, for example, 0.573 in a certain sample. The logic control unit in the computation graph instantly compares this random test floating-point number with the skip probability. If the random test floating-point number is less than the skip probability, the system immediately issues a freeze interception command, temporarily disabling the computation enable switch of the current parameter node in the backpropagation computation graph (e.g., setting it to False), thus stopping the computational power consumption required for parameter updates to the tensor partition. To maintain the mathematical topological completeness of the multi-level network structure, the graph optimizer uses the tensor block as an intermediate buffer node to preserve its channel transmission attributes. This prevents the tensor block from being cut or destroyed and allows it to continue the process of error propagation to shallower network layers. It only skips time-consuming periodic update operations such as parameter accumulation, physical writing, and refreshing for the currently selected tensor partition. For tensor partitions whose parameter updates were skipped in the previous training batch, the system uses the most recent effective average parameter gradient statistics for that partition and updates it according to a preset decay coefficient. When the number of times the same tensor partition is skipped consecutively reaches a preset threshold, the tensor partition is forced to participate in a complete gradient calculation in the next training batch to refresh its activity score and avoid long-term freezing.

[0058] S3, after backpropagation, decompose the gradient vector of each site relative to the global average gradient into consensus components and novel components; weight the consensus components according to the global rarity of fault type, and dynamically weight the novel components according to the global sum of activity scores of each region and the contribution of the novel components of each site; when the sum of the norms of the novel components is lower than the preset norm threshold, differentiated site exploration weights are introduced, and the weight amplitude is limited by smoothing terms and exploration coefficient ranges; when the global sum of activity scores is low, the weight of the novel components is enhanced, and the weighted consensus components and novel components are aggregated to update the global model.

[0059] After backpropagation is completed on all computing nodes to obtain local gradients, for each station, the system collects the local gradient sub-vectors of each tensor partition according to the preset partitioning order of the model weight tensor. For tensor partitions that have not skipped parameter updates, their actual parameter gradient sub-vectors are read. For tensor partitions whose parameter updates have been skipped, zero-value placeholder vectors are filled in the corresponding partition positions or the gradient sub-vectors from the previous training batch are used, thus ensuring that the local gradient vectors formed by different stations have the same partitioning order and the same total dimension. By calling the torchdistributedallreduce function and specifying the operation enumeration value as ReduceOpSUM, the one-dimensional tiling gradient vectors of each station are summed; then, the global average gradient vector is calculated by dividing by the total number of stations participating in the training.

[0060] The system calculates the inner product of the local gradient vector and the global average gradient vector of the current site, and calculates the squared L2 norm of the global average gradient vector. The inner product result is divided by the sum of the squared L2 norm and a preset smoothing term to obtain the projection scalar coefficients. When the L2 norm of the global average gradient vector is lower than a preset norm threshold, the projection calculation is performed using the global average gradient direction vector from the previous training batch; otherwise, the projection calculation is performed based on the current global average gradient vector. A consensus component vector parallel to the adopted gradient direction is obtained based on the projection calculation result. The consensus component vector is subtracted from the original local gradient vector using a tensor subtraction operator to separate novel component vectors orthogonal to the global direction. Based on a historical transformer equipment fault log database, the inverse document frequency (IDF) algorithm is used to calculate the reciprocal frequency index of each specific fault category in all samples, which is used as the global rarity value. Since the current training batch may contain both normal data and multiple fault type data, the system further reads the fault type label of each sample in the current batch and calculates the percentage of samples of each fault type in the batch; or calculates the percentage of loss contribution of each fault type in the loss function of the batch. The sample size ratio or loss contribution ratio is used as the batch mapping weight. The global rarity values ​​of each fault type are weighted and summed to obtain the batch rarity weight of the current training batch. For the normal category, its global rarity can be set as a preset baseline weight to avoid undefined weights when normal samples participate in batch training.

[0061] The batch rarity weight is used as a weight scalar and multiplied by the consensus component vector to complete consensus weighting. Simultaneously, tensor activity scores recorded on all GPU devices are collected and summed to obtain a global sum scalar. Using an inverse proportional mapping function, a preset constant is divided by the sum of the global sum of activity scores and the smoothing term, and truncated within a preset interval to obtain a dynamic exploration scaling factor. The proportion of novel component norms for each site is calculated to obtain site-level novelty weights. When the sum of the norms of all site novel components is lower than a preset norm threshold, differentiated site exploration weights are generated using historical effective novelty weights, fault sample loss contribution weights, or data reliability weights, and these weights replace the unified coefficients in subsequent weighting calculations. The dynamic exploration scaling factor is multiplied by the site-level novelty weights to obtain the final weighting coefficients for each site's novel components. The final weighting coefficients are multiplied by the corresponding site's novel component vector to complete dynamic weighting. The weighted consensus component vector and the weighted novel component vector are added element-wise. For each site, the weighted consensus component vector and the weighted novel component vector are added element-wise to generate the corresponding site's corrected gradient vector. The master node performs a weighted average of the corrected gradient vectors for all sites according to the number of site samples, site data reliability weights, or a preset equal weighting rule, to obtain a unique corrected global gradient tensor. Through slicing operations, this corrected global gradient tensor is reshaped back into the multidimensional tensor shape corresponding to each network layer, covering the grad attribute space assigned to each parameter of the model. The optimizer instance's step method is called to execute the gradient descent algorithm based on the corrected first and second moment estimates, completing the iterative update of the global weight parameters of the transformer fault diagnosis model.

[0062] In one embodiment, the step of decomposing the gradient vectors of each site into consensus components and novel components relative to the global average gradient includes:

[0063] Collect the local gradient sub-vectors calculated by each station on each tensor partition in the current batch, and concatenate them according to the preset partition order to form the local gradient vector of each station; for tensor partitions that have skipped parameter updates in the current training batch, use the valid gradient sub-vectors that were not skipped in the previous training batch at the corresponding partition position to keep the dimension of the local gradient vector of each station consistent.

[0064] Calculate the arithmetic mean of the local gradient vectors of all stations to obtain the global average gradient vector;

[0065] Calculate the projection vector of the local gradient vector of each site onto the direction of the global average gradient vector, and use the projection vector as the consensus component of the site;

[0066] Subtract the consensus component corresponding to each site from the local gradient vector of each site to obtain a residual vector orthogonal to the global average gradient vector. Use the residual vector as the novel component of the site.

[0067] In the distributed hybrid Transformer model training architecture, during the backpropagation computation at the end of the model and the generation of gradient values ​​for each node within the model, concurrent aggregation is performed before synchronous parameter updates. Communication nodes invoke underlying high-concurrency multicast parallel primitives, such as the All-Reduce and All-Gather communication kernel instructions, to collect the local gradient sub-vectors formed by each station for each tensor partition in the current batch, and concatenate them according to a preset partitioning order to form the local gradient vectors for each station. For tensor partitions in the current training batch where parameter updates have been skipped, the valid gradient sub-vectors not skipped in the previous training batch are used at the corresponding partition positions. After the above processing, the local gradient vectors of each station have a uniform vector length and a uniform partitioning order. The arithmetic mean of the local gradient vectors of each station is calculated using the averaging operator, thereby constructing a global average gradient vector in Euclidean space that can represent the common optimization direction of the batch data.

[0068] Calculate the dot product of the current local gradient vector and the global average gradient vector. Divide this dot product by the sum of the squared L2 norm of the global average gradient vector and a preset smoothing term to obtain the projected scalar coefficients. When the L2 norm of the global average gradient vector is lower than a preset norm threshold, recalculate the projected scalar coefficients using the global average gradient direction vector of the previous training batch. Multiply the recalculated projected scalar coefficients by the global average gradient direction vector of the previous training batch to obtain the consensus component of the site. Otherwise, multiply the projected scalar coefficients by the current global average gradient vector to obtain the consensus component of the site. Subtract the consensus component from the current local gradient vector to obtain the residual vector. Use the residual vector as the unique novel component of the site for subsequent steps to perform refined independent weighted modulation.

[0069] In one embodiment, the dynamic weighting of the novel component based on the global sum of the activity scores by region and the contribution of each site's novel component includes:

[0070] In the current training iteration step, the activity scores of all tensor partitions in the model are summed to obtain the global sum of activity scores.

[0071] Calculate the norm of each site's novelty component, and divide the norm of each site's novelty component by the sum of the norms of all sites' novelty components and the sum of the preset smoothing term to obtain the site-level novelty weight.

[0072] When the sum of the norms of all novel components of all sites is lower than the preset norm threshold, differentiated site exploration weights are generated based on the historical effective novel weights, the contribution of fault sample loss, or the data reliability weights of each site. The differentiated site exploration weights are then normalized so that the novel component weighting coefficients of at least two sites are different, so as to avoid the novel components of each site canceling each other out during aggregation.

[0073] Set an exploration base constant, divide the exploration base constant by the sum of the global sum of the activity scores and the preset smoothing term, and truncate the calculation result between the preset minimum exploration coefficient and the preset maximum exploration coefficient to obtain the dynamic exploration scaling factor;

[0074] The dynamic exploration scaling factor is multiplied by the site-level novelty weight to obtain the final weighting coefficient of the novelty component of each site, and the novelty component of each site is weighted using the final weighting coefficient.

[0075] To implement exploratory knowledge control and gradient scaling strategies for global parameter network update states, during a specific training iteration cycle, the underlying scheduler monitors traverse all currently defined parallel tensor partitions in the model's memory pool. It calls the hardware high-speed reduce-accumulate operator ReduceSum to continuously perform a global floating-point summation operation on the real-time activity score of each tensor partition, merging them to obtain a global sum of activity scores representing the current macro-state of the model's update signal-to-noise ratio and activity level. After completing the statistics, the control system loads a predefined fixed exploration cardinality constant, which can be set as follows: or The division pipeline divides the exploration cardinality constant by the sum of the global sum of activity scores and the preset smoothing term. A truncation function then limits the resulting exploration scaling factor between a preset minimum and a preset maximum exploration coefficient to prevent abnormal amplification of novel components when the global sum of activity scores is too low. During the parameter synthesis and modification execution phase, the matrix calculation unit extracts the independently separated novel components for each site, calculates the norm proportion of each site's novel components to obtain site-level novel weights, and multiplies the dynamic exploration scaling factor by these site-level novel weights before applying it to the novel components of the corresponding site. When the overall model activity enters a prolonged stagnation phase (i.e., the global sum of activity scores is at a low point), a larger scaling factor drives the multiplication amplification factor. This, combined with the historical effective novel weights of each site, the contribution of faulty sample loss, or the data reliability weights, forms differentiated site exploration weights. This ensures that the novel components of different sites have different amplification ratios during the aggregation phase, preventing orthogonal residual vectors from canceling each other out during equal-weight summation. This completes the task of adjusting the weights of all novel components containing unseen pattern data exploration features in the global gradient update phase before aggregation, as needed.

[0076] Example 2

[0077] Embodiment 2 of the present invention proposes a collaborative training system for a multi-site transformer fault diagnosis model, comprising the following modules:

[0078] The generation module is used to acquire multi-site transformer operation data sequences and perform iterative attention processing during forward propagation: perform intra-site self-attention on each site sequence to generate initial site summary vectors; perform cross-site cross-attention based on each initial site summary vector to obtain global context-aware site vectors; and feed this vector back to the input features of the next round of intra-site self-attention for feature modulation.

[0079] The computation module is used to partition the model weight tensor during training. It determines the tensor activity score based on the normalized result of the average L1 norm of the gradient of each partition in the previous training batch and the weight density. After backpropagation, the corresponding partition parameter update is skipped with a probability inversely proportional to the tensor activity score. For partitions that are not skipped, the learning rate is adjusted by a dynamic factor proportional to the tensor activity score. When the tensor activity score is lower than the unloading threshold, the corresponding partition is unloaded from the graphics processor to the central processing unit and reloaded when needed in subsequent calculations.

[0080] The aggregation module, after backpropagation, decomposes the gradient vector of each site relative to the global average gradient into consensus components and novel components. It weights the consensus components according to the global rarity of fault types and dynamically weights the novel components according to the global sum of activity scores in each region and the contribution of each site's novel components. When the sum of the norms of the novel components is lower than a preset norm threshold, it introduces differentiated site exploration weights and uses smoothing terms and exploration coefficient ranges to limit the weight amplitude. When the global sum of activity scores is low, it enhances the weight of the novel components and aggregates the weighted consensus components and novel components to update the global model.

[0081] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

Claims

1. A collaborative training method for a multi-site transformer fault diagnosis model, characterized in that, Includes the following steps: Obtain the transformer operation data sequence from multiple sites and perform iterative attention processing during forward propagation: perform intra-site self-attention on each site sequence to generate an initial site summary vector; Cross-site cross-attention is performed based on the initial site summary vectors to obtain a global context-aware site vector; this vector is then fed back to the next round of in-site self-attention input features for feature modulation. During training, the model weight tensor is partitioned, and the tensor activity score is determined based on the normalized result of the average L1 norm of the gradient of each partition in the previous training batch and the weight density. After backpropagation, the parameter update of the corresponding partition is skipped with a probability inversely proportional to the tensor activity score. For partitions that are not skipped, the learning rate is adjusted by a dynamic factor proportional to the tensor activity score. When the tensor activity score is lower than the unloading threshold, the corresponding partition is unloaded from the graphics processor to the central processing unit and reloaded when needed in subsequent calculations. After backpropagation, the gradient vector of each site relative to the global average gradient is decomposed into consensus components and novel components. The consensus components are weighted according to the global rarity of fault type, and the novel components are dynamically weighted according to the global sum of activity scores of each region and the contribution of the novel components of each site. When the sum of the norms of the novel components is lower than the preset norm threshold, differentiated site exploration weights are introduced, and the weight amplitude is limited by smoothing terms and exploration coefficient ranges. When the global sum of activity scores is low, the weight of the novel components is enhanced, and the weighted consensus components and novel components are aggregated to update the global model.

2. The method according to claim 1, characterized in that, The step of performing intra-site self-attention on each site sequence to generate an initial site summary vector includes: The output feature matrix of the self-attention within each site is input into a feedforward neural network with GeLU activation function to perform nonlinear mapping of the multidimensional vector space and obtain the mapped feature tensor. A global average pooling operation is performed on the mapped feature tensor along the time step dimension to compress the time step dimension, retain the feature dimension, and extract the one-dimensional site summary vector corresponding to each batch of samples as the initial site summary vector.

3. The method according to claim 1 or 2, characterized in that, The process of performing cross-site cross-attention based on the initial site summary vectors to obtain global context-aware site vectors includes: The initial site summary vector of the target site is converted into a query vector matrix through a first linear mapping network; The initial site summary vectors of all sites are concatenated and then converted into key vector matrices and value vector matrices through the second and third linear mapping networks, respectively. Calculate the dot product of the query vector matrix and the key vector matrix, divide by the scaling factor, and then process using the Softmax function to obtain the attention weight matrix; Multiplying the attention weight matrix with the value vector matrix yields the global context-aware site vector.

4. The method according to claim 1, characterized in that, The step of feeding the vector back to the next round of in-station self-attention input features for feature modulation includes: The global context-aware site vector is input into a parameter generation network containing two independent fully connected layers, which respectively map and output a scaling factor vector for feature scaling and a bias factor vector for feature translation. Before the next round of internal self-attention calculation, a single-dimensional axis is inserted according to the actual dimensions of the scaling factor vector and the bias factor vector, and broadcasting is performed at least in the time step dimension; when the scaling factor vector and the bias factor vector do not contain the batch dimension, broadcasting is performed simultaneously in the batch dimension to align its dimension with the input feature tensor. The input feature tensor is multiplied element-wise with the broadcast scaling coefficient vector, and the result of the multiplication is added element-wise with the bias coefficient vector that has undergone the same broadcast, thus completing the feature affine transformation modulation based on the global context signal.

5. The method according to claim 1, characterized in that, The determination of the tensor activity score based on the normalized result of the average L1 norm of the gradients of each partition in the previous training batch and the weight density includes: Obtain the parametric gradient tensor generated in the previous training batch for the current tensor partition, calculate the sum of the absolute values ​​of all elements of the parametric gradient tensor and divide it by the total number of elements of the parametric gradient tensor to obtain the average parametric gradient L1 norm. The number of weight parameters in the current tensor partition whose absolute value is less than a preset near-zero threshold is counted. The ratio of the number of these parameters to the total number of weight parameters in the partition is taken as the weight sparsity. The weight density is obtained by subtracting the weight sparsity from 1. The average parameter gradient L1 norm is normalized and multiplied by the first weight coefficient. The weight density is then multiplied by the second weight coefficient. The two product terms are added together to obtain the activity score of the tensor partition.

6. The method according to claim 1, characterized in that, After the backpropagation is completed, the corresponding partition parameter update is skipped with a probability inversely proportional to the tensor activity score, including: Set a base skip constant, divide the base skip constant by the sum of the activity score of the current tensor partition and the smoothing term constant, and truncate the calculation result between 0 and the preset maximum skip probability, where the preset maximum skip probability is less than 1, to obtain the skip probability; Generate a random test floating-point number that follows a uniform distribution from 0 to 1 within the system; The random test floating-point number is compared with the skip probability. If the random test floating-point number is less than the skip probability, a freeze command is triggered. During the optimizer update phase of the current training step, the update of the tensor partition parameters is stopped, but the partition retains its complete gradient calculation and error propagation functions in backpropagation.

7. The method according to claim 1, characterized in that, The process of decomposing the gradient vector of each site relative to the global average gradient into consensus components and novel components includes: Collect the local gradient sub-vectors calculated by each station on each tensor partition in the current batch, and concatenate them according to the preset partition order to form the local gradient vector of each station; for tensor partitions that have skipped parameter updates in the current training batch, use the valid gradient sub-vectors that were not skipped in the previous training batch at the corresponding partition position to keep the dimension of the local gradient vector of each station consistent. Calculate the arithmetic mean of the local gradient vectors of all stations to obtain the global average gradient vector; Calculate the projection vector of the local gradient vector of each site onto the direction of the global average gradient vector, and use the projection vector as the consensus component of the site; Subtract the consensus component corresponding to each site from the local gradient vector of each site to obtain a residual vector orthogonal to the global average gradient vector. Use the residual vector as the novel component of the site.

8. The method according to claim 1, characterized in that, The dynamic weighting of novel components based on the global sum of activity scores by region and the contribution of each site's novel component includes: In the current training iteration step, the activity scores of all tensor partitions in the model are summed to obtain the global sum of activity scores. Calculate the norm of each site's novelty component, and divide the norm of each site's novelty component by the sum of the norms of all sites' novelty components and the sum of the preset smoothing term to obtain the site-level novelty weight. When the sum of the norms of all novel components of all sites is lower than the preset norm threshold, differentiated site exploration weights are generated based on the historical effective novel weights, the contribution of fault sample loss, or the data reliability weights of each site. The differentiated site exploration weights are then normalized so that the novel component weighting coefficients of at least two sites are different, so as to avoid the novel components of each site canceling each other out during aggregation. Set an exploration base constant, divide the exploration base constant by the sum of the global sum of the activity scores and the preset smoothing term, and truncate the calculation result between the preset minimum exploration coefficient and the preset maximum exploration coefficient to obtain the dynamic exploration scaling factor; The dynamic exploration scaling factor is multiplied by the site-level novelty weight to obtain the final weighting coefficient of the novelty component of each site, and the novelty component of each site is weighted using the final weighting coefficient.

9. A collaborative training system for a multi-site transformer fault diagnosis model, characterized in that, Includes the following modules: The generation module is used to acquire the operating data sequence of transformers at multiple sites and perform iterative attention processing during forward propagation: perform intra-site self-attention on each site sequence to generate an initial site summary vector; Cross-site cross-attention is performed based on the initial site summary vectors to obtain a global context-aware site vector; this vector is then fed back to the next round of in-site self-attention input features for feature modulation. The computation module is used to partition the model weight tensor during training. It determines the tensor activity score based on the normalized result of the average L1 norm of the gradient of each partition in the previous training batch and the weight density. After backpropagation, the corresponding partition parameter update is skipped with a probability inversely proportional to the tensor activity score. For partitions that are not skipped, the learning rate is adjusted by a dynamic factor proportional to the tensor activity score. When the tensor activity score is lower than the unloading threshold, the corresponding partition is unloaded from the graphics processor to the central processing unit and reloaded when needed in subsequent calculations. The aggregation module, after backpropagation, decomposes the gradient vector of each site relative to the global average gradient into consensus components and novel components. It weights the consensus components according to the global rarity of fault types and dynamically weights the novel components according to the global sum of activity scores in each region and the contribution of each site's novel components. When the sum of the norms of the novel components is lower than a preset norm threshold, it introduces differentiated site exploration weights and uses smoothing terms and exploration coefficient ranges to limit the weight amplitude. When the global sum of activity scores is low, it enhances the weight of the novel components and aggregates the weighted consensus components and novel components to update the global model.

10. The system according to claim 9, characterized in that, The step of performing intra-site self-attention on each site sequence to generate an initial site summary vector includes: The output feature matrix of the self-attention within each site is input into a feedforward neural network with GeLU activation function to perform nonlinear mapping of the multidimensional vector space and obtain the mapped feature tensor. A global average pooling operation is performed on the mapped feature tensor along the time step dimension to compress the time step dimension, retain the feature dimension, and extract the one-dimensional site summary vector corresponding to each batch of samples as the initial site summary vector.

Images