Method for constructing heterogeneous multi-modal image coding and cross-domain fusion network for on-load tap changer fault diagnosis

Abstract

The application discloses a kind of heterogeneous multi-modal image coding and cross-domain fusion network construction methods for on-load tap-changer fault diagnosis, including the construction of time-frequency dual-domain cross-modal energy statistics hilbert coding module, generate unified scale time-domain cross-modal coding image and frequency-domain cross-modal coding image;Based on parallel time-domain stream branch and frequency-domain stream branch, build double-flow pseudo-twin feature extraction network, output time-domain multi-scale feature map and frequency-domain multi-scale feature map through each branch;Double-branch feature enhancement module is constructed, and time-domain and frequency-domain multi-scale feature map are independently carried out adaptive recalibration, and enhanced feature map, time-domain enhanced feature map and frequency-domain enhanced feature map are obtained;Time-domain enhanced feature map and frequency-domain enhanced feature map are sent into cross-domain feature fusion module based on bidirectional multi-head cross attention, and cross-domain fusion feature representation is obtained;The cross-domain fusion feature representation is input into two-stage dimension reduction classification decision structure, and the probability distribution of each fault category is output.The application provides a high-precision, strong robustness and engineering feasible solution for intelligent diagnosis of on-load tap-changer mechanical fault under complex working conditions.

Description

Technical Field

[0001] This invention relates to a heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis, belonging to the field of intelligent operation and maintenance and fault diagnosis technology of power equipment. Background Technology

[0002] With the accelerated construction of new power systems and the continued growth in demand for long-distance, large-capacity power transmission, high-voltage direct current (HVDC) transmission, with its advantages of large transmission capacity, low loss, flexible regulation, and adaptability to large-scale grid integration of new energy sources, is becoming an important direction for future power grid development. The stable operation of HVDC transmission systems essentially depends on the efficient conversion and reliable transmission of AC and DC power. As a key piece of equipment in this process, the reliability of the converter transformer directly affects the safety and stability of the power grid. The on-load tap changer (OLTC), as the core component of the converter transformer responsible for on-load voltage regulation, operates under high voltage, high current, and strong mechanical impact environments for extended periods, making it prone to mechanical failures such as contact wear, mechanism jamming, contact burning, and loosening of transmission components.

[0003] In recent years, OLTC (On-Load Tap Changer) faults have accounted for approximately 27% of all on-load tap changing transformer (OTTC) faults, with mechanical faults being one of the most prevalent types. Therefore, conducting research on efficient and accurate mechanical fault diagnosis of OLTCs is of great significance for ensuring the safe and reliable operation of converter transformers and improving the stability of the power grid. Summary of the Invention

[0004] To address the technical challenges of on-load tap changer fault diagnosis, this invention provides a heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis. This method constructs a heterogeneous multimodal image coding and cross-domain fusion network model based on a time-frequency dual-domain cross-modal energy statistical Hilbert coding module, a dual-stream pseudo-twin feature extraction network, a dual-branch feature enhancement module, a cross-domain feature fusion module based on bidirectional multi-head cross-attention, and a two-stage dimensionality reduction classification decision structure. This provides a high-precision, robust, and engineering-feasible solution for on-load tap changer fault diagnosis.

[0005] The technical solution of this invention is:

[0006] Firstly, a method for constructing heterogeneous multimodal image coding and cross-domain fusion networks for on-load tap changer fault diagnosis is provided, including:

[0007] A time-frequency dual-domain cross-modal energy statistical Hilbert coding module is constructed. The time-frequency dual-domain cross-modal energy statistical Hilbert coding module takes the heterogeneous four-channel modal signals of the on-load tap changer as input, extracts the normalized sequence in the time domain and frequency domain respectively, reconstructs it into a two-dimensional feature tensor through Hilbert bijective mapping, and uses color mapping and image coding to realize four-channel collaborative mapping to generate time-domain cross-modal coded images and frequency-domain cross-modal coded images of the same scale.

[0008] A dual-stream pseudo-twin feature extraction network is constructed based on parallel temporal and frequency-domain flow branches. Temporal cross-modal encoded images and frequency-domain cross-modal encoded images are fed into the topologically consistent and parameter-independent temporal and frequency-domain flow branches, respectively. Both branches are based on a multi-scale residual encoder based on depthwise separable convolution, and output temporal multi-scale feature maps and frequency-domain multi-scale feature maps through each branch.

[0009] A dual-branch feature enhancement module is constructed, wherein a convolutional block attention module is introduced into the output of the temporal flow branch and the frequency flow branch, and the temporal and frequency multi-scale feature maps are independently adaptively recalibrated through channel attention and spatial attention extraction to obtain enhanced feature maps: temporal enhanced feature map and frequency enhanced feature map;

[0010] The time-domain enhanced feature map and the frequency-domain enhanced feature map are fed into the cross-domain feature fusion module based on bidirectional multi-head cross-attention. The cross-domain feature fusion module based on bidirectional multi-head cross-attention constructs two parallel attention paths: time-domain query frequency domain and frequency-domain query time domain. It globally and explicitly models the deep coupling and complementary features of the time and frequency domains. After reorganization, the cross-domain fused feature representation is obtained by splicing in the channel dimension.

[0011] The cross-domain fusion feature representation is input into the two-stage dimensionality reduction classification decision structure, and the probability distribution of each fault category is output.

[0012] Furthermore, the heterogeneous four-mode signals specifically refer to the vibration, sound, current, and angle four-mode signals of the on-load tap changer.

[0013] Furthermore, the time-frequency dual-domain cross-modal energy statistical Hilbert coding module includes:

[0014] (1) Extraction of time-frequency dual-domain normalized sequences:

[0015] Z-Score normalization is performed on each modal signal to obtain a normalized signal;

[0016] For the time-domain branch, first calculate the first-order difference of the standardized signal to obtain the difference sequence, and then perform a length-wise division on the square of the difference sequence. The moving average is used to obtain the local power estimate; based on the local power estimate, the time-domain standardized sequence is obtained. ;

[0017] For the frequency domain branch, the frequency domain energy distribution of the normalized signal is characterized using Welch power spectral density estimation, resulting in the Welch power spectral density estimate; based on the Welch power spectral density estimate, the frequency domain normalized sequence is obtained. ;

[0018] (2) Hilbert bijective mapping

[0019] We uniformly employ Hilbert bijective mapping to convert time-domain normalized sequences and frequency-domain normalized sequences into two-dimensional feature tensors: time-domain feature tensor and frequency-domain feature tensor, respectively.

[0020] (3) Four-modal CMYK color mapping and image encoding

[0021] The time-domain feature tensors corresponding to the four modal signals are randomly mapped to the cyan, magenta, yellow, and black channels without repetition, forming a four-channel time-domain feature stack. ; express Channel time-domain feature stack: The frequency domain feature tensors corresponding to the four modal signals are mapped to the cyan, magenta, yellow, and black channels, respectively, forming a four-channel frequency domain feature stack. ; express Channel frequency domain feature stack;

[0022] Subsequently, the four-channel temporal feature stack and the four-channel frequency feature stack are converted into RGB images based on the subtractive color model: temporal cross-modal encoded image and frequency cross-modal encoded image.

[0023] Each branch of the dual-stream pseudo-twin feature extraction network includes a Stem layer and four hierarchical feature extraction stages in sequence. The Stem layer is used to extract the shallow texture response of the input image. Then, four hierarchical feature extraction stages are set: the first hierarchical feature extraction stage contains one depthwise separable convolutional multi-scale residual block, and the remaining three stages each contain two depthwise separable convolutional multi-scale residual blocks. The first residual block of the latter three stages uses a convolution operation with a stride of 2 to achieve spatial downsampling.

[0024] Furthermore, the depthwise separable convolution-based multi-scale residual encoder uses a depthwise separable convolution multi-scale residual block as its core. This depthwise separable convolution multi-scale residual block contains two parallel convolutional branches: one employing... Depthwise separable convolution is used to extract local fine-grained texture information; another one uses... Depthwise separable convolutions are used to capture regional structural features within a larger receptive field. The outputs of the two branches are first concatenated along the channel dimension, then non-linearly integrated using GN and GELU activation functions. The integrated features are then processed through a single layer. Depthwise separable convolution further refines and reshapes spatial information, and after reshaping, it undergoes nonlinear mapping via GN and GELU activation functions.

[0025] Furthermore, the depthwise separable convolutional multi-scale residual block introduces a residual branch. When the input and output have different numbers of channels, the residual branch... Convolution and GN implement projection mapping; when the input and output have the same number of channels, the output of the main path of the residual block is concatenated with the residual and element-wise addition is performed.

[0026] Furthermore, the cross-domain feature fusion module based on bidirectional multi-head cross-attention specifically comprises:

[0027] Flatten the time-domain enhanced feature map and the frequency-domain enhanced feature map into sequence representations respectively;

[0028] For a time-domain query with a frequency-domain branch, using the time-domain sequence as the query term and the frequency-domain sequence as the key and value, then the... The output of each attention head Represented as:

[0029] ;

[0030] in, Indicates the first The time-domain query matrix in the head. Indicates the first The frequency domain key matrix in the head. Indicates the first The frequency domain value matrix in the head. For the feature dimensions of a single attention head, Indicates transpose;

[0031] By concatenating the outputs of all attention heads and performing a linear transformation, a cross-domain enhanced representation in the time-domain query frequency-domain direction is obtained. ;

[0032] For the time-domain branch of a frequency-domain query, using the frequency-domain sequence as the query term and the time-domain sequence as the key and value, a cross-domain enhanced representation in the time-domain direction of the frequency-domain query can be obtained. ;

[0033] Enhance the cross-domain representation in the time domain direction of the frequency domain query. Adding the enhanced feature map to the temporal domain, and then passing it through residual connections and layer normalization, yields the cross-domain enhanced temporal sequence; the cross-domain enhanced representation in the temporal domain direction is then queried from the frequency domain. The frequency domain enhanced feature map is added to the frequency domain enhanced feature map, and then subjected to residual connections and layer normalization to obtain the cross-domain enhanced frequency domain sequence. The cross-domain enhanced time domain sequence and the cross-domain enhanced frequency domain sequence are reshaped into two-dimensional feature maps according to the original spatial index order and then concatenated in the channel dimension to obtain the cross-domain fused feature representation. .

[0034] Furthermore, the two-stage dimensionality reduction classification decision module specifically involves: firstly, through a single layer... Convolutional, CN, and GELU activation functions are used to perform the first stage of dimensionality reduction in the channel dimension, resulting in a pre-dimensionality-reduced feature map. Subsequently, global average pooling is used to aggregate spatial information to obtain a global feature vector. On this basis, a second stage of dimensionality reduction is completed by a fully connected mapping with GELU activation combined with Dropout, resulting in high-level discriminative features. Finally, a fully connected mapping with GELU activation and Dropout regularization are introduced, and the posterior probability distribution of each fault category is output through a Softmax classification layer.

[0035] In a second aspect, a processor is provided for running a program, wherein the program, when running, executes the steps of the heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis as described above.

[0036] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, the device where the computer-readable storage medium is located executes the steps of the heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis as described above.

[0037] The beneficial effects of this invention are:

[0038] This invention comprehensively considers four aspects of signals: vibration, sound, current, and angle. It designs a time-frequency dual-domain cross-modal energy statistical Hilbert coding module to convert one-dimensional sensor signals into two-dimensional time-frequency image tensors at a unified scale. Based on this, a dual-stream pseudo-twin feature extraction network is designed to independently learn multi-scale fault features in the time and frequency domains. A dual-branch feature enhancement module is designed to perform independent adaptive recalibration and dynamic enhancement of the dual-branch features. A cross-domain feature fusion module is designed to achieve deep information interaction and complementary fusion of multi-modal features in the time and frequency domains. Combined with a two-stage dimensionality reduction classification decision structure and an internal and external dual-layer parameter optimization strategy, the final on-load tap changer fault diagnosis result is output. This provides a high-precision, robust, and engineering-feasible solution for intelligent diagnosis of mechanical faults in on-load tap changers under complex operating conditions. Attached Figure Description

[0039] Figure 1This is a block diagram of the diagnostic method of the present invention;

[0040] Figure 2 It is a two-layer parameter optimization process. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0042] Example 1: As Figures 1-2 As shown, according to a first aspect of the present invention, a method for constructing a heterogeneous multimodal image coding and cross-domain fusion network for on-load tap changer fault diagnosis is provided, comprising:

[0043] A time-frequency dual-domain cross-modal energy statistical Hilbert coding module is constructed. The time-frequency dual-domain cross-modal energy statistical Hilbert coding module takes four modal signals of vibration, sound, current and angle of on-load tap changer as input, extracts standardized sequences in the time domain and frequency domain respectively, reconstructs them into two-dimensional feature tensors through Hilbert bijective mapping, and uses color mapping and image coding to realize four-channel collaborative mapping to generate time-domain cross-modal coded images and frequency-domain cross-modal coded images of the same scale.

[0044] A dual-stream pseudo-Twin feature extraction network is constructed based on parallel temporal and frequency-domain stream branches. Temporal cross-modal encoded images and frequency-domain cross-modal encoded images are fed into the topologically consistent and parameter-independent temporal and frequency-domain stream branches, respectively. Both branches are based on a multi-scale residual encoder based on depthwise separable convolution, and output temporal multi-scale feature maps through each branch. With frequency domain multi-scale feature map ;

[0045] A dual-branch feature enhancement module is constructed, which introduces convolutional block attention modules at the output ends of the time-domain flow branch and the frequency-domain flow branch, and extracts them sequentially through channel attention and spatial attention. The time-domain and frequency-domain multi-scale feature maps are independently adaptively recalibrated to highlight fault-sensitive features and suppress background noise, resulting in enhanced feature maps: time-domain enhanced feature map and frequency-domain enhanced feature map.

[0046] The time-domain enhanced feature map and the frequency-domain enhanced feature map are fed into the cross-domain feature fusion module based on bidirectional multi-head cross-attention. The cross-domain feature fusion module based on bidirectional multi-head cross-attention constructs two parallel attention paths: time-domain query frequency domain and frequency-domain query time domain. It globally and explicitly models the deep coupling and complementary features of the time and frequency domains. After reorganization, the cross-domain fused feature representation is obtained by splicing in the channel dimension.

[0047] The cross-domain fusion feature representation is input into a two-stage dimensionality reduction classification decision structure. Dimensionality reduction is performed step by step through channel pre-compression and global spatial information aggregation. The probability distribution of each fault category is output through a fully connected hidden layer and a Softmax classification layer. At the same time, the internal and external double-layer parameters are optimized.

[0048] Furthermore, the time-frequency dual-domain cross-modal energy statistical Hilbert coding module includes:

[0049] (1) Extraction of time-frequency dual-domain normalized sequences:

[0050] For the first Road mode signal Z-score standardization is performed to eliminate dimensional differences. The calculation formula is as follows:

[0051] ;

[0052] in, For the first Road standardization signals and They represent the first Mean and standard deviation of the road mode signal; This indicates the number of sampling points for each modal signal;

[0053] For the time-domain branch, first calculate the first-order difference of the standardized signal to obtain the difference sequence, and then perform a length-wise division on the square of the difference sequence. The moving average is used to obtain the local power estimate; based on the local power estimate, the time-domain standardized sequence is obtained. ;

[0054] For the frequency domain branch, the frequency domain energy distribution of the normalized signal is characterized using Welch power spectral density estimation, resulting in the Welch power spectral density estimate; based on the Welch power spectral density estimate, the frequency domain normalized sequence is obtained. .

[0055] (2) Hilbert bijective mapping

[0056] We uniformly adopt Hilbert bijective mapping to convert time-domain normalized sequences and frequency-domain normalized sequences into two-dimensional feature tensors: time-domain feature tensor and frequency-domain feature tensor, respectively.

[0057] (3) Four-modal CMYK color mapping and image encoding

[0058] The time-domain feature tensors corresponding to the four modal signals are randomly mapped without repetition to the cyan, magenta, yellow, and key channels (i.e., each hue channel is assigned once, corresponding to the time-domain feature tensor of one modal signal), forming a four-channel time-domain feature stack. ; express Channel temporal feature stack; for example, Channels 1, 2, 3, and 4 correspond to the cyan, magenta, yellow, and black channels, respectively; that is, the temporal feature tensor corresponding to the vibration signal is mapped to the cyan channel, forming a cyan channel temporal feature stack. The temporal feature tensor corresponding to the sound signal is mapped to the magenta channel, forming a magenta channel temporal feature stack. The time-domain feature tensor corresponding to the current signal is mapped to the yellow channel, forming the time-domain feature stack of the yellow channel. The temporal feature tensor corresponding to the angle signal is mapped to the black channel, forming a temporal feature stack of the black channel. It should be noted that, Choosing 1, 2, 3, and 4 can be a four-hue combination that ensures each hue channel is assigned once. For example, they can also correspond to the magenta, cyan, black, and yellow channels respectively.

[0059] The frequency domain feature tensors corresponding to the four modal signals are mapped to the cyan, magenta, yellow, and key channels, respectively, to form a four-channel frequency domain feature stack. ; express Channel frequency domain feature stack; that is, to obtain The corresponding relationship is the same as that of the four-channel time-domain feature stack;

[0060] Subsequently, the four-channel temporal feature stack and the four-channel frequency feature stack are converted into RGB images based on the subtractive color model: temporal cross-modal encoded image and frequency cross-modal encoded image.

[0061] Furthermore, the depthwise separable convolution-based multi-scale residual encoder uses a depthwise separable convolution multi-scale residual block as its core. This depthwise separable convolution multi-scale residual block contains two parallel convolutional branches: one employing... Depthwise separable convolution is used to extract local fine-grained texture information; another one uses... Depthwise separable convolutions are used to capture regional structural features within a larger receptive field. The outputs of the two branches are first concatenated along the channel dimension, then non-linearly integrated using GN and GELU activation functions. The integrated features are then processed through a single layer. Depthwise separable convolution further refines and reshapes spatial information, and after reshaping, it undergoes nonlinear mapping via GN and GELU activation functions.

[0062] Furthermore, the depthwise separable convolutional multi-scale residual block introduces a residual branch. When the input and output have different numbers of channels, the residual branch... Convolution and GN implement projection mapping; when the input and output have the same number of channels, the output of the main path of the residual block is concatenated with the residual and element-wise addition is performed.

[0063] Furthermore, the cross-domain feature fusion module based on bidirectional multi-head cross-attention specifically comprises:

[0064] Flatten the time-domain enhanced feature map and the frequency-domain enhanced feature map into sequence representations respectively;

[0065] For a time-domain query with a frequency-domain branch, using the time-domain sequence as the query term and the frequency-domain sequence as the key and value, then the... The output of each attention head Represented as:

[0066] ;

[0067] in, Indicates the first The time-domain query matrix in the head. Indicates the first The frequency domain key matrix in the head. Indicates the first The frequency domain value matrix in the head. For the feature dimensions of a single attention head, This indicates transpose.

[0068] By concatenating the outputs of all attention heads and performing a linear transformation, a cross-domain enhanced representation in the time-domain query frequency-domain direction is obtained. ;

[0069] For the time-domain branch of a frequency-domain query, using the frequency-domain sequence as the query term and the time-domain sequence as the key and value, a cross-domain enhanced representation in the time-domain direction of the frequency-domain query can be obtained. ;

[0070] Enhance the cross-domain representation in the time domain direction of the frequency domain query. Adding the enhanced feature map to the temporal domain, and then passing it through residual connections and layer normalization, yields the cross-domain enhanced temporal sequence; the cross-domain enhanced representation in the temporal domain direction is then queried from the frequency domain. The frequency domain enhanced feature map is added to the frequency domain enhanced feature map, and then subjected to residual connections and layer normalization to obtain the cross-domain enhanced frequency domain sequence. The cross-domain enhanced time domain sequence and the cross-domain enhanced frequency domain sequence are reshaped into two-dimensional feature maps according to the original spatial index order and then concatenated in the channel dimension to obtain the cross-domain fused feature representation. .

[0071] Furthermore, the two-stage dimensionality reduction classification decision module specifically involves: firstly, through a single layer... Convolutional, CN, and GELU activation functions are used to perform the first stage of dimensionality reduction in the channel dimension, resulting in a pre-dimensionality-reduced feature map. Subsequently, global average pooling is used to aggregate spatial information to obtain a global feature vector. On this basis, a second stage of dimensionality reduction is completed by a fully connected mapping with GELU activation combined with Dropout, resulting in high-level discriminative features. Finally, a fully connected mapping with GELU activation and Dropout regularization are introduced, and the posterior probability distribution of each fault category is output through a Softmax classification layer.

[0072] Furthermore, a two-layer parameter optimization strategy is adopted, with the inner layer focusing on learning network weights and the outer layer focusing on searching for key hyperparameters. Specifically, the inner layer updates network weights based on a network using AdamW and cosine annealing, while the outer layer employs a Bayesian optimization algorithm based on a tree-structured Parzen estimator to jointly optimize the key hyperparameters of the model.

[0073] According to a second aspect of the present invention, a processor is provided, the processor being configured to run a program, wherein the program, when running, executes the steps of the heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis as described above.

[0074] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, the device where the computer-readable storage medium is located executes the steps of the heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis as described above.

[0075] Example 2: In an optional implementation, a method for constructing a heterogeneous multimodal image coding and cross-domain fusion network for on-load tap changer fault diagnosis includes:

[0076] I. Time-Frequency Dual-Domain Cross-Modal Energy Statistical Hilbert Encoding Module

[0077] To simultaneously characterize the local abrupt response of fault signals in the time domain and the spectral energy redistribution characteristics in the frequency domain, this paper constructs a Time–Frequency Dual-Domain Cross-Modal Energy-Statistical Hilbert Encoding (TFD-ESH) module. This module takes four modal signals—vibration, sound, current, and angle—as input, extracts one-dimensional statistical sequences of local energy in both the time and frequency domains, reconstructs them into a two-dimensional feature tensor of uniform scale using Hilbert space-filling mapping, and obtains the TFD-ESH tensor through four-channel collaborative encoding via CMYK color mapping. Specifically, the time-domain branch focuses on characterizing the transient energy accumulation features during local signal changes, while the frequency-domain branch focuses on characterizing the stable spectral energy distribution pattern induced by the fault.

[0078] (1) Time-frequency dual-domain data processing

[0079] Let the first The road mode signal is To eliminate the dimensional differences between the various mode signals and suppress the influence of DC bias, Z-score normalization is first performed on each mode signal, resulting in:

[0080] ;

[0081] in, For the first Road standardization signals and They represent the first Mean and standard deviation of the road mode signal; This indicates the number of sampling points for each modal signal.

[0082] For the time-domain branch, the first-order difference of the standardized signal is first calculated to highlight fault impacts, abrupt edge changes, and local fluctuations. The difference sequence is then denoted. for:

[0083] ;

[0084] Subsequently, the length of the squared difference sequence is executed. The moving average is used to obtain the local power estimate. :

[0085] ;

[0086] The local power is estimated by taking the square root to obtain the local root mean square response, which is then logarithmically compressed to obtain the time-domain energy sequence. Next, bicubic interpolation is used to resample this time-domain energy sequence to a fixed length. And map it to a linear normalized form. The interval is used to obtain the time-domain standardized sequence. .

[0087] In the frequency domain branch, Welch power spectral density estimation is used to characterize the frequency domain energy distribution of the normalized signal. Specifically, the normalized signal is segmented, windowed, and overlapped. A Fast Fourier Transform is performed on each segment, and the power spectrum is averaged to obtain a stable power spectral density estimation result. Let the window length be... The overlap length is , for the Segmented standard signal Power spectrum estimation It can be represented as:

[0088] ;

[0089] in, represents an imaginary number, This represents the Hann window function. This represents the frequency point index. The Welch power spectral density estimate is obtained by averaging the power spectral estimates of all M1 segments.

[0090] ;

[0091] To further compress the dynamic range of spectral amplitude and enhance the separability of weak features, the Welch power spectral density estimation results were also logarithmically compressed, and then bicubic interpolation was used to resample them to a fixed length. And map it to a linear normalized form. The interval is used to obtain the frequency domain normalized sequence. Through the above processing, the time-domain and frequency-domain inputs are unified in scale and numerical range, providing consistent input constraints for subsequent two-dimensional coding.

[0092] (2) Hilbert bijective mapping

[0093] Hilbert curves are a typical type of space-filling curve, capable of mapping adjacent sampling points in a one-dimensional sequence to neighboring pixels in a two-dimensional plane, thus preserving the local topological relationships of the original modal sequence. Based on this, this paper uniformly employs Hilbert bijective mapping to convert time-domain normalized sequences and frequency-domain normalized sequences into two-dimensional feature tensors, respectively. For the target feature map size... Construct the corresponding order The Hilbert curve is obtained, and a one-dimensional index and two-dimensional coordinates are established. bijective relationship between Based on the above, the time-domain normalized sequence is... and frequency domain normalized sequence Fill the two-dimensional plane point by point to obtain the temporal feature tensor. and frequency domain feature tensor ,Right now:

[0094] ;

[0095] ;

[0096] in, This is the inverse mapping function. After the above mapping, the temporal neighborhood relationships in the one-dimensional reconstructed signal are transformed into a two-dimensional texture structure, thereby realizing the transformation from the physical signal space to the visual representation space.

[0097] (3) Four-modal CMYK color mapping and image encoding

[0098] To simultaneously carry four channels of sensing information in a single image, this paper adopts a multi-channel mapping strategy based on the CMYK color space. Unlike the traditional RGB three-channel representation, CMYK has four independent color components, making it more suitable for synchronous encoding of four modal signals. Specifically, the four channels are mapped to the cyan, magenta, yellow, and key channels, forming a four-channel feature stack: a four-channel temporal feature stack and a four-channel frequency domain feature stack. After CMYK color space mapping, two sets of feature stacks fusing the collaborative information of the four channels are obtained, respectively characterizing the temporal impulse aggregation characteristics and the frequency domain abrupt change correlation characteristics.

[0099] Subsequently, based on the subtractive color model, the four-channel temporal feature stack and the four-channel frequency feature stack are converted into an RGB image that the network can directly input: a temporal cross-modal encoded image. Frequency domain cross-modal coded image For any pixel position of the four-channel temporal feature stack and the four-channel frequency feature stack. The synthesis result is defined as:

[0100] ;

[0101] In the formula, This represents the numerical value of the two-dimensional feature tensor in the corresponding channel (if used for a four-channel time-domain feature stack). correspond If used for a four-channel frequency domain feature stack, correspond ); , , This represents the "residual reflectance" of the corresponding hue channel to light, and controls the intensity of the complementary colors of red, green, and blue in the image, respectively. The global brightness attenuation coefficient for the black channel is used as a base to adjust the overall brightness.

[0102] II. Dual-stream pseudo-twin feature extraction network

[0103] Considering temporal cross-modal coded images With frequency domain cross-modal coded images Significant differences exist in texture morphology, local structure, and physical representation. This paper constructs a two-stream pseudo-twin feature extraction network. The two-stream pseudo-twin feature extraction network employs two branches with consistent network topology and independent parameters, thus avoiding the constraints imposed by shared weights on heterogeneous modality discrimination representation and enhancing the model's adaptability to dual-domain differences. Each branch uses a depthwise separable convolution-based multi-scale residual encoder (DSC-MSRN) as its backbone. Specifically, a single branch consists of a Stem layer and four sequential feature extraction stages. The Stem layer uses a well-known method... Depthwise separable convolution, group normalization (GN), and GELU activation function are used to extract shallow texture responses from the input image. Four hierarchical feature extraction stages are then set up: Stage 1 contains one depthwise separable convolutional multi-scale residual block, and the remaining three stages (Stage 2–Stage 4) each contain two depthwise separable convolutional multi-scale residual blocks. The first residual block in the latter three stages uses a convolution operation with a stride of 2 to achieve spatial downsampling, gradually expanding the receptive field and improving high-level semantic representation capabilities.

[0104] The core unit of DSC-MSRN is the Depthwise Separable Convolution-based Multi-Scale Residual Block (DSC-MSRB). This DSC-MSRB contains two parallel convolutional branches: one employing... Depthwise separable convolution is used to extract local fine-grained texture information; another one uses... Depthwise separable convolutions are used to capture regional structural features within a larger receptive field. The outputs of the two branches are first concatenated along the channel dimension, then nonlinearly integrated using GN and GELU, and finally a layer is applied to the integrated features. Depthwise separable convolutions are used to further refine and reorganize spatial information; the output of this convolution is then passed through GN and GELU activation functions to form a complete main path feature transformation. To alleviate the degradation problem in deep network training, the module also introduces residual connections. Therefore, DSC-MSRB can be expressed as:

[0105] ;

[0106] In the formula, , They represent , Depth-separable convolution, This indicates a channel splicing operation. This is a residual connection. When the input and output differ in the number of channels or spatial resolution, the residual branch connects through... Convolution and GN implement projection mapping.

[0107] Each branch ultimately outputs a multi-scale feature map in the temporal domain. With frequency domain multi-scale feature map ; Indicates width, Indicates the number of channels.

[0108] III. CBAM-based dual-branch feature enhancement module

[0109] After completing the dual-stream pseudo-twin feature extraction, the model has obtained multi-scale feature maps in the temporal domain. With frequency domain multi-scale feature map Although both types of features can characterize the local texture structure and regional semantic information within their respective domains, they may still contain redundant responses, background noise, and weakly correlated patterns unrelated to fault detection. Considering the differences in physical connotation and distribution statistics between temporal and frequency domain features, this paper introduces a Convolutional Block Attention Module (CBAM) at the multi-scale feature output of each branch to perform independent adaptive recalibration on the temporal and frequency domain multi-scale feature maps, thereby further highlighting fault-sensitive information within their respective domains and suppressing irrelevant interference.

[0110] First, in the channel attention stage, global average pooling and global max pooling are performed on the input feature maps to obtain two sets of channel description vectors. Then, the two vectors are input into a multilayer perceptron with shared weights, and a non-linear mapping is applied. The outputs are then summed element-wise and passed through a sigmoid function to generate a channel attention weight map. This process effectively uncovers the dependencies between channels, enhances the channel responses related to fault modes, and suppresses redundant background information. For any branch of input features... (Right now or The channel-recalibrated features are obtained by multiplying the input features element-wise with the channel attention. It can be represented as:

[0111] ;

[0112] in, This represents the channel attention mapping function. This indicates an element-wise multiplication operation.

[0113] Building upon channel-based filtering, a spatial attention mechanism is further introduced to enhance the ability to locate critical fault regions. Specifically, average pooling and max pooling are performed along the channel dimension on the channel-weighted features to generate two sets of two-dimensional spatial response maps; these maps are then concatenated and input into the database. Convolutional layers aggregate spatial information, which is then processed by a sigmoid function to obtain a spatial attention map. This attention map adaptively adjusts the feature weights at different spatial locations, thereby highlighting areas where faulty textures are concentrated and reducing interference from irrelevant regions. Finally, the spatial attention map... Element-wise multiplication with the input features yields the enhanced feature map for this branch:

[0114] ;

[0115] in, This represents the spatial attention mapping function.

[0116] Through the above processing, the model can adaptively enhance the key channel responses and significant spatial regions in the time domain branch and the frequency domain branch respectively. While retaining the effective representation information in their respective domains, it further suppresses redundant noise and background interference, thereby improving the feature representation of the two branches.

[0117] IV. Cross-domain Feature Fusion Module Based on BMCM

[0118] Considering the significant complementary correlation between temporal and frequency domain features, this paper constructs a cross-domain feature fusion module based on a Bidirectional Multi-Head Cross-Attention Mechanism (BMCM). This module takes enhanced temporal and frequency domain feature maps (both enhanced by CBAM) as input and establishes cross-domain dependencies between temporal and frequency domain features globally through a bidirectional "Query-Key-Value" interaction mechanism. Compared to shallow fusion methods such as simple concatenation or element-wise weighting, this module can adaptively allocate attention weights based on cross-domain response relationships, thereby achieving a deep collaborative representation of time-frequency dual-domain information.

[0119] First, the temporal and frequency domain augmented feature maps are flattened into sequence representations. During cross-domain interaction, this paper constructs two parallel attention paths: "temporal domain query frequency domain" and "frequency domain query temporal domain." For the temporal domain query frequency domain branch, the temporal sequence is used as the query term, and the frequency domain sequence is used as the key and value. The output of an attention head can be represented as:

[0120] ;

[0121] in, Indicates the first The time-domain query matrix in the head. Indicates the first The frequency domain key matrix in the head. Indicates the first The frequency domain value matrix in the head. This represents the feature dimension of a single attention head. By concatenating the outputs of all attention heads and performing a linear transformation, a cross-domain augmented representation of the time-domain query in the frequency domain can be obtained. :

[0122] ;

[0123] in, Indicates the number of attention heads; Indicates a splicing operation; This indicates the output projection matrix.

[0124] Similarly, by using the frequency domain sequence as the query term and the time domain sequence as the key and value, a cross-domain enhanced representation of the frequency domain query in the time domain direction can be obtained. .

[0125] The bidirectional cross-attention mechanism establishes a global correlation between time-domain and frequency-domain features, enabling deep interaction and complementary fusion of information from both domains. Based on this, it enhances the cross-domain representation of the time-domain query in the frequency-domain direction. The enhanced temporal feature map is added to the CBAM-enhanced feature map, and training stability is improved while preserving the original discriminative information through residual connections and layer normalization to obtain the cross-domain enhanced temporal sequence. The cross-domain enhanced representation in the temporal direction is then queried from the frequency domain. The frequency domain enhanced feature map is added to the CBAM-enhanced feature map, and training stability is enhanced while preserving the original discriminative information through residual connections and layer normalization to obtain the cross-domain enhanced frequency domain sequence. Finally, based on the inherent characteristic that cross-attention does not change the sequence length, the cross-domain enhanced time domain sequence and the cross-domain enhanced frequency domain sequence are reshaped into two-dimensional feature maps in the original spatial index order and concatenated in the channel dimension to obtain the cross-domain fused feature representation. .

[0126] V. Two-stage dimensionality reduction classification decision and two-level parameter optimization

[0127] (1) Two-stage dimensionality reduction classification decision module

[0128] To suppress feature redundancy and reduce classification computation overhead while ensuring the discriminative power of high-level fusion features, this paper designs a two-stage dimensionality reduction strategy in the classification decision stage. First, a single-stage dimensionality reduction layer... Convolution, group normalization (GN), and GELU activation function are used to perform the first stage of dimensionality reduction for the channel dimension, resulting in a pre-dimensionality-reduced feature map. Subsequently, global average pooling (GAP) is used to aggregate spatial information to obtain a global feature vector. Based on this, the number of global feature vectors is reduced in the second stage by a fully connected layer with GELU activation combined with Dropout, resulting in high-level discriminative features. Finally, a fully connected layer with GELU activation (hidden layer) and Dropout regularization are introduced, and the posterior probability distribution of each fault category is output through a Softmax classification layer.

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] In the formula, This represents the pre-dimensionality-reduced feature map after the first stage of dimensionality reduction. This represents the global feature vector after global average pooling. This represents the high-level discriminative features after the second-stage dimensionality reduction. This represents the output of the classification hidden layer. This represents the predicted probability vector of the model.

[0135] (2) Two-layer parameter optimization strategy

[0136] To achieve coordinated optimization of model structure parameters and training parameters, this paper adopts a nested optimization strategy with inner and outer layers. The inner layer optimizes network weight learning, while the outer layer optimizes key hyperparameter search; together, they constitute a complete two-layer parameter optimization framework. For example... Figure 2 As shown, the inner layer updates the model parameters on the training set through gradient descent; the outer layer, with the validation set performance as the objective, jointly optimizes the model structure parameters, regularization parameters, and training parameters, thereby achieving overall coordination between the network structure and training dynamics.

[0137] Inner layer: Network weight update based on AdamW and cosine annealing. The inner layer optimization stage uses the classification output branch as the sole supervision objective, employing a multi-class cross-entropy loss function combined with label smoothing for parameter updates. Let the batch size be... The number of fault categories is The smoothed true label is The model predicts the probability as The classification cross-entropy loss can then be expressed as:

[0138] ;

[0139] In optimizing the model, this invention employs the AdamW algorithm to implement parameter updates with weight decay, enhancing regularization during training. Simultaneously, a learning rate scheduling strategy of "linear warm-up + cosine annealing" is introduced. This involves gradually and linearly increasing the learning rate over the first 10 epochs to stabilize the initial training process, followed by a smooth decrease in the learning rate according to a cosine decay law, thus balancing convergence stability in the early stages of training with fine-grained optimization capabilities in the later stages. From the 11th epoch onwards, a decision-making phase begins, comparing the best validation accuracy of the current trial up to that epoch with the median of completed trials in the same training epochs. If the accuracy is lower than the median, pruning is performed; otherwise, training continues. During training, model saving and model selection are both based on validation set classification accuracy.

[0140] Outer Layer: Bayesian Hyperparameter Optimization Based on TPE. Due to the complex nonlinear coupling between each hyperparameter and the training-related parameters, traditional grid search is not only computationally expensive but also struggles to obtain optimal configurations within a limited number of trials. Therefore, this paper employs a Bayesian optimization algorithm based on the Tree-structured Parzen Estimator (TPE) to jointly optimize the key hyperparameters of the model. For ease of subsequent mathematical expression, all key hyperparameters to be optimized are uniformly denoted as vector θ, and the specific candidate parameter configuration in the k-th trial is represented as θk. k Meanwhile, the highest accuracy achieved by the model on the validation set is used as the evaluation metric, which is defined as the objective function J(θ). k This is used to quantify the performance of each set of hyperparameter configurations. The specific process is as follows:

[0141] 1. Construct the hyperparameter space:

[0142] The search space incorporates the following parameters: convolutional encoding base width, number of encoding output channels, number of cross-attention heads, number of channels in the first-stage dimensionality reduction, dimension of the second-stage dimensionality reduction, CBAM compression ratio, size of the classification hidden layer, Dropout ratio, learning rate, weight decay coefficient, and label smoothing coefficient.

[0143] 2. Proxy Model Construction:

[0144] Based on existing experimental results, TPE estimates the probability density distribution of high-performance and low-performance regions respectively, and dynamically adjusts the subsequent sampling direction in the non-convex search space accordingly, thereby improving the probability of candidate combinations hitting high-quality parameter regions.

[0145] 3. Evaluation of candidate combinations:

[0146] For each set of candidate hyperparameters, the model is rebuilt and trained, and the optimal validation accuracy of the experiment is used as the target value to feed back to the optimizer.

[0147] 4. Pruning Strategies:

[0148] To reduce the computational overhead of low-potential trials, this paper introduces an automatic pruning mechanism during the optimization process. Specifically, a median-based pruner is used, combined with a validation accuracy monitoring callback, to terminate candidate combinations that significantly lag behind in performance in the early stages of training. Simultaneously, to avoid premature model elimination during the initial convergence phase, the first few training rounds are used only for observation without executing early stopping or pruning decisions.

[0149] 5. Iterative optimization:

[0150] The outer layer optimization iterates continuously according to a closed-loop process of "sampling-training-evaluation-feedback" until the preset number of trials is reached. After the search is completed, the optimal hyperparameter combination is determined based on the validation performance, and the final diagnostic model is reconstructed accordingly.

[0151] Specific implementation results:

[0152] This experiment uses the TensorFlow deep learning framework and Python as the programming language. The experimental subject is a UCG-type OLTC prototype manufactured by ABB.

[0153] To verify the effectiveness of the proposed method, this paper selects seven different diagnostic models for comparative experiments and conducts a comprehensive evaluation from three aspects: classification performance, model complexity, and inference efficiency. The comparison objects include CNN-BiGRU, CNN-BiLSTM, CNN-Transformer, MobileNetV3, and ResNet18, all based on the same four-modal signal TFD-ESH encoding input, to compare the impact of different diagnostic network structures on the recognition results. Simultaneously, the StarNet-EJCD-YOLO model based on vibration signal single-modal CWT encoding, and the proposed model based on vibration signal single-modal GASF-GADF encoding and four-modal signal (TFD-ESH) encoding, are selected to further analyze the impact of input representation methods on fault recognition performance. Specific experimental comparison results are shown in Table 1 below:

[0154] Table 1

[0155]

[0156] In Table 1, Accuracy refers to the proportion of correctly classified samples out of the total number of samples; Precision refers to the proportion of correctly classified samples among those predicted as fault samples; Recall refers to the proportion of correctly identified samples among actual fault samples; F1 score is the harmonic mean of Precision and Recall; ROC-AUC is used to measure the model's ability to distinguish fault categories under different discrimination thresholds. As can be seen from Table 1, the model of this invention achieved the highest overall recognition performance, with Accuracy, Precision, Recall, F1-score, and ROC-AUC reaching 99.60%, 99.61%, 99.60%, 99.60%, and 100.00%, respectively.

[0157] The actual operating environment of the converter transformer OLTC is quite complex. Different modal signals can be affected by external interference. Specifically, CH1 (vibration) and CH5 (sound) signals are susceptible to substation wind noise and acoustic interference from surrounding equipment; CH6 (current) may be affected by grid harmonics and electromagnetic coupling; and while CH8 (angle) is relatively stable, it can still be affected by quantization noise and measurement disturbances. To comprehensively evaluate the model's noise robustness under four modal channel data, this invention independently superimposes additive Gaussian white noise (AWGN) onto each of the four modal channels (CH1, CH5, CH6, and CH8) to simulate the complex noise environment in the field. This invention sets three signal-to-noise ratio levels: 6dB, 0dB, and -6dB, corresponding to mild interference, strong interference, and extreme interference noise scenarios. The data in Table 2 are obtained by conducting experiments on the model of this invention and other models under noisy conditions.

[0158] Table 2

[0159]

[0160] As shown in Table 2, the model based on four-modal TFD-ESH encoding in this invention achieves accuracy of 98.41%, 97.82%, and 94.44% under SNR conditions of 6 dB, 0 dB, and −6 dB, respectively, all of which are optimal results.

[0161] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for constructing heterogeneous multimodal image coding and cross-domain fusion networks for on-load tap changer fault diagnosis, characterized in that, include: A time-frequency dual-domain cross-modal energy statistical Hilbert coding module is constructed. The time-frequency dual-domain cross-modal energy statistical Hilbert coding module takes the heterogeneous four-channel modal signals of the on-load tap changer as input, extracts the normalized sequence in the time domain and frequency domain respectively, reconstructs it into a two-dimensional feature tensor through Hilbert bijective mapping, and uses color mapping and image coding to realize four-channel collaborative mapping to generate time-domain cross-modal coded images and frequency-domain cross-modal coded images of the same scale. A dual-stream pseudo-twin feature extraction network is constructed based on parallel temporal and frequency-domain flow branches. Temporal cross-modal encoded images and frequency-domain cross-modal encoded images are fed into the topologically consistent and parameter-independent temporal and frequency-domain flow branches, respectively. Both branches are based on a multi-scale residual encoder based on depthwise separable convolution, and output temporal multi-scale feature maps and frequency-domain multi-scale feature maps through each branch. A dual-branch feature enhancement module is constructed, wherein a convolutional block attention module is introduced into the output of the temporal flow branch and the frequency flow branch, and the temporal and frequency multi-scale feature maps are independently adaptively recalibrated through channel attention and spatial attention extraction to obtain enhanced feature maps: temporal enhanced feature map and frequency enhanced feature map; The time-domain enhanced feature map and the frequency-domain enhanced feature map are fed into the cross-domain feature fusion module based on bidirectional multi-head cross-attention. The cross-domain feature fusion module based on bidirectional multi-head cross-attention constructs two parallel attention paths: time-domain query frequency domain and frequency-domain query time domain. It globally and explicitly models the deep coupling and complementary features of the time and frequency domains. After reorganization, the cross-domain fused feature representation is obtained by splicing in the channel dimension. The cross-domain fusion feature representation is input into the two-stage dimensionality reduction classification decision structure, and the probability distribution of each fault category is output.

2. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 1, characterized in that, The heterogeneous four-mode signals specifically refer to the vibration, sound, current, and angle modes of the on-load tap changer.

3. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 1, characterized in that, The time-frequency dual-domain cross-modal energy statistical Hilbert coding module includes: (1) Extraction of time-frequency dual-domain normalized sequences: Z-Score normalization is performed on each modal signal to obtain a normalized signal; For the time-domain branch, first calculate the first-order difference of the standardized signal to obtain the difference sequence, and then perform a length-wise division on the square of the difference sequence. The moving average is used to obtain the local power estimate; based on the local power estimate, the time-domain standardized sequence is obtained. ; For the frequency domain branch, the frequency domain energy distribution of the normalized signal is characterized using Welch power spectral density estimation, resulting in the Welch power spectral density estimate; based on the Welch power spectral density estimate, the frequency domain normalized sequence is obtained. ; (2) Hilbert bijective mapping We uniformly employ Hilbert bijective mapping to convert time-domain normalized sequences and frequency-domain normalized sequences into two-dimensional feature tensors: time-domain feature tensor and frequency-domain feature tensor, respectively. (3) Four-modal CMYK color mapping and image encoding The time-domain feature tensors corresponding to the four modal signals are randomly mapped to the cyan, magenta, yellow, and black channels without repetition, forming a four-channel time-domain feature stack. ; express Channel time-domain feature stack: The frequency domain feature tensors corresponding to the four modal signals are mapped to the cyan, magenta, yellow, and black channels, respectively, forming a four-channel frequency domain feature stack. ; express Channel frequency domain feature stack; Subsequently, the four-channel temporal feature stack and the four-channel frequency feature stack are converted into RGB images based on the subtractive color model: temporal cross-modal encoded image and frequency cross-modal encoded image.

4. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 1, characterized in that, Each branch of the dual-stream pseudo-twin feature extraction network includes a Stem layer and four hierarchical feature extraction stages in sequence. The Stem layer is used to extract the shallow texture response of the input image. Then, four hierarchical feature extraction stages are set: the first hierarchical feature extraction stage contains one depthwise separable convolutional multi-scale residual block, and the remaining three stages each contain two depthwise separable convolutional multi-scale residual blocks. The first residual block of the latter three stages uses a convolution operation with a stride of 2 to achieve spatial downsampling.

5. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 1 or 4, characterized in that, The depthwise separable convolution-based multi-scale residual encoder uses a depthwise separable convolution multi-scale residual block as its core. This block contains two parallel convolutional branches: one employing... Depthwise separable convolution is used to extract local fine-grained texture information; another one uses... Depthwise separable convolutions are used to capture regional structural features within a larger receptive field. The outputs of the two branches are first concatenated along the channel dimension, then non-linearly integrated using GN and GELU activation functions. The integrated features are then processed through a single layer. Depthwise separable convolution further refines and reshapes spatial information, and after reshaping, it undergoes nonlinear mapping via GN and GELU activation functions.

6. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 5, characterized in that, The depthwise separable convolutional multi-scale residual block introduces a residual branch. When the input and output have different numbers of channels, the residual branch... Convolution and GN implement projection mapping; when the input and output have the same number of channels, the output of the main path of the residual block is concatenated with the residual and element-wise addition is performed.

7. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 1, characterized in that, The cross-domain feature fusion module based on bidirectional multi-head cross-attention is specifically as follows: Flatten the time-domain enhanced feature map and the frequency-domain enhanced feature map into sequence representations respectively; For a time-domain query with a frequency-domain branch, using the time-domain sequence as the query term and the frequency-domain sequence as the key and value, then the... The output of each attention head Represented as: ； in, Indicates the first The time-domain query matrix in the head. Indicates the first The frequency domain key matrix in the head. Indicates the first The frequency domain value matrix in the head. For the feature dimensions of a single attention head, Indicates transpose; By concatenating the outputs of all attention heads and performing a linear transformation, a cross-domain enhanced representation in the time-domain query frequency-domain direction is obtained. ; For the time-domain branch of a frequency-domain query, using the frequency-domain sequence as the query term and the time-domain sequence as the key and value, a cross-domain enhanced representation in the time-domain direction of the frequency-domain query can be obtained. ; Enhance the cross-domain representation in the time domain direction of the frequency domain query. Adding the enhanced feature map to the temporal domain, and then passing it through residual connections and layer normalization, yields the cross-domain enhanced temporal sequence; the cross-domain enhanced representation in the temporal domain direction is then queried from the frequency domain. The frequency domain enhanced feature map is added to the frequency domain enhanced feature map, and then the result is obtained by residual connection and layer normalization. The cross-domain enhanced time domain sequence and the cross-domain enhanced frequency domain sequence are reshaped into two-dimensional feature maps according to the original spatial index order and spliced ​​in the channel dimension to obtain the cross-domain fusion feature representation.

8. The heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis according to claim 1, characterized in that, The two-stage dimensionality reduction classification decision module specifically involves: firstly, passing through a layer... Convolution, CN, and GELU activation functions perform the first stage of dimensionality reduction in the channel dimension to obtain the pre-dimensionality-reduced feature map; Subsequently, global average pooling is used to aggregate spatial information to obtain global feature vectors; Based on this, a second-stage dimensionality reduction is completed by using a fully connected mapping with GELU activation and Dropout to obtain high-level discriminative features. Finally, a fully connected mapping with GELU activation and Dropout regularization are introduced, and the posterior probability distribution of each fault category is output through a Softmax classification layer.

9. A processor, characterized in that, The processor is used to run a program, wherein the program executes the steps of the heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the steps of the heterogeneous multimodal image coding and cross-domain fusion network construction method for on-load tap changer fault diagnosis as described in any one of claims 1-8.

Images