A Transformer Multi-Source Modeling Method and System Based on Hierarchical Features and Cross-Modal Fusion

Abstract

This invention belongs to the field of electrical data processing technology. It proposes a multi-source modeling method and system for transformers based on hierarchical features and cross-modal fusion. The method acquires maintenance text data, inspection image data, and acoustic signature data from the transformer operation process, and performs semantic feature extraction, spatial structure feature extraction, and time-frequency domain feature extraction on these data respectively. This generates a set of text entity relationship triples, image classification labels, and acoustic signature classification labels, which are then mapped to a unified semantic vector space. The cross-modal similarity index between each modality is calculated. This index comprehensively considers the distance between semantic vectors, classification probability distribution confidence, and information entropy. Based on this, fusion weights are dynamically allocated and weighted fusion is performed, ultimately generating a multi-modal joint representation vector characterizing the transformer's operating state. This achieves effective collaboration and accurate representation of multi-source heterogeneous information.

Description

Technical Field

[0001] This invention relates to the field of electrical data processing technology, specifically to a transformer multi-source modeling method and system based on hierarchical features and cross-modal fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous improvement of the digitalization and intelligence of power grids, power equipment condition monitoring technology is developing towards multi-source sensing and intelligent analysis. Transformers, as core equipment for power transmission and transformation, accumulate multimodal heterogeneous data during their operation, including maintenance records, inspection images, and acoustic signatures. These data carry information on the equipment's operating status in different dimensions, such as semantic description, visual representation, and acoustic characteristics, providing a rich data foundation for constructing comprehensive and accurate condition representation models. Utilizing artificial intelligence technology to collaboratively model the aforementioned multi-source information has become an important research trend for improving transformer condition sensing capabilities.

[0004] However, existing modeling methods struggle to effectively integrate the aforementioned multimodal information to form a unified state representation. The main problem lies in the significant differences in feature space, semantic structure, and confidence levels among the data from different modalities, while current technologies lack a mechanism to dynamically measure and integrate the reliability of each modality. This prevents the adaptive allocation of fusion weights based on consistency, classification confidence, and prediction reliability among modalities when constructing the joint representation vector. Consequently, it hinders the realization of the complementary advantages of multi-source information and makes it difficult to generate robust and accurate transformer operating state representations. Summary of the Invention

[0005] To address the issues of insufficient feature representation and inadequate cross-modal information fusion in existing multi-source heterogeneous data modeling of transformer operating status, this invention provides a transformer multi-source modeling method and system based on hierarchical features and cross-modal fusion. By extracting and fusing multi-modal features hierarchically, the method effectively improves the information complementarity and representation consistency among heterogeneous data, thereby solving the problems of insufficient single-modal representation, weak cross-modal correlation modeling ability, and insufficient feature utilization in existing methods. This enhances the completeness and robustness of transformer multi-source information modeling.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a transformer multi-source modeling method based on hierarchical features and cross-modal fusion.

[0007] A transformer multi-source modeling method based on hierarchical features and cross-modal fusion includes the following process: Acquire maintenance text data, inspection image data, and acoustic signal data generated during transformer operation; Semantic features are extracted from operation and maintenance text data to generate a set of text entity relation triples; spatial structure features are extracted from patrol image data to generate image classification labels; and time-frequency domain features are extracted from voiceprint signal data to generate voiceprint classification labels. The text entity relation triplet set, image classification label and voiceprint classification label are mapped to a unified semantic vector space to obtain text semantic vector, image semantic vector and voiceprint semantic vector respectively. Based on text semantic vectors, image semantic vectors, and voiceprint semantic vectors, cross-modal similarity indices are calculated between each pair of modalities. The cross-modal similarity index comprehensively considers the distance between semantic vectors, the probability distribution reliability of the corresponding classification results, and the information entropy of the probability distribution. Based on the cross-modal similarity index, dynamic fusion weights are assigned to the text semantic vector, image semantic vector and voiceprint semantic vector, and the three are weighted and fused to generate a multimodal joint representation vector for characterizing the transformer's operating state.

[0008] In one implementation of the first aspect of the present invention, semantic features are extracted from the operation and maintenance text data to generate a set of text entity relation triples, including: The operation and maintenance text data is jointly encoded with the preset task instruction information to construct a structured prompt input; The structured prompts are fed into a pre-trained language model for semantic reasoning, generating a structured output containing entity and relation information. The structured output is parsed and formatted, and the subject entity, relation type, and object entity are extracted and organized into a set of text entity relation triples.

[0009] In one implementation of the first aspect of the present invention, spatial structure feature extraction is performed on the patrol image data to generate image classification labels, including: The inspection image data is input into a convolutional neural network backbone pre-trained on a large-scale image dataset; Spatial structural features of the patrol image data are extracted using a convolutional neural network backbone, and then classified using a task classification head network to output image classification labels.

[0010] In one implementation of the first aspect of the present invention, time-frequency domain feature extraction is performed on the voiceprint signal data to generate a voiceprint classification label, including: The voiceprint signal data is framed, windowed, and subjected to short-time Fourier transform to obtain the time spectrum; The time spectrum is filtered using the Mel filter bank, and the filtered results are subjected to logarithmic compression and discrete cosine transform to obtain the MFCC cepstral feature sequence. The MFCC cepstral feature sequence is input into a deep acoustic embedding network for high-dimensional feature modeling to obtain a deep acoustic embedding feature vector; based on the deep acoustic embedding feature vector, a classifier outputs a voiceprint classification label.

[0011] As a further limitation of the first aspect of the present invention, the deep acoustic embedding network includes a convolutional residual front-end, a Dense-TDNN backbone network with an introduced channel attention mechanism, and an embedding layer. The high-dimensional feature modeling process includes: The local time-frequency structure features of the MFCC cepstral feature sequence are extracted through the convolutional residual front-end; the local time-frequency structure features are input into the Dense-TDNN backbone network to learn long-term temporal context dependencies, and important feature responses related to abnormal voiceprint discrimination are adaptively enhanced through the channel attention mechanism; and deep acoustic embedding feature vectors are output through the embedding layer.

[0012] In one implementation of the first aspect of the present invention, the text entity relation triplet set, image classification labels, and voiceprint classification labels are respectively mapped to a unified semantic vector space, including: The relationships in the set of text entity relationship triples are mapped to text semantic vectors through a preset entity relationship encoding function; image classification labels are mapped to image semantic vectors through a preset image label embedding function; and voiceprint classification labels are mapped to voiceprint semantic vectors through a preset voiceprint label embedding function.

[0013] In one implementation of the first aspect of the present invention, calculating the cross-modal similarity index between each pair of modes includes: For any two modalities, calculate the distance between their corresponding two semantic vectors; obtain the probability distribution vectors of the classification results of each of the two modalities, and calculate the information entropy of each probability distribution vector; based on the distance, probability distribution vectors, and information entropy, calculate the cross-modal similarity index between the two modalities. The calculation of the cross-modal similarity index includes: using distance as the basic term, introducing a confidence decay coefficient related to the probability distribution vector, and a reliability calibration term related to information entropy, which together constitute the cross-modal similarity index.

[0014] In one implementation of the first aspect of the present invention, dynamic fusion weights are assigned to the text semantic vector, image semantic vector, and voiceprint semantic vector based on a cross-modal similarity index, including: The cross-modal similarity index between any two modalities is compared with a preset similarity threshold. If the cross-modal similarity index is higher than the similarity threshold, the corresponding modal pair is assigned a first fusion weight; otherwise, a second fusion weight is assigned. The first fusion weight is greater than the second fusion weight.

[0015] Secondly, the present invention provides a transformer multi-source modeling system based on hierarchical features and cross-modal fusion.

[0016] A transformer multi-source modeling system based on hierarchical features and cross-modal fusion includes: The multi-source data acquisition unit is configured to acquire maintenance text data, inspection image data, and acoustic signal data generated during transformer operation. The hierarchical feature extraction unit is configured to: extract semantic features from maintenance text data to generate a set of text entity relation triples; extract spatial structure features from patrol image data to generate image classification labels; and extract time-frequency domain features from voiceprint signal data to generate voiceprint classification labels. The semantic vector mapping unit is configured to map the set of text entity relation triples, image classification labels and voiceprint classification labels to a unified semantic vector space to obtain text semantic vectors, image semantic vectors and voiceprint semantic vectors. The cross-modal similarity calculation unit is configured to: calculate the cross-modal similarity index between each pair of text semantic vectors, image semantic vectors and voiceprint semantic vectors. The cross-modal similarity index comprehensively considers the distance between semantic vectors, the probability distribution reliability of the corresponding classification results and the information entropy of the probability distribution. The dynamic fusion generation unit is configured to: assign dynamic fusion weights to the text semantic vector, image semantic vector and voiceprint semantic vector according to the cross-modal similarity index, and perform weighted fusion of the three to generate a multimodal joint representation vector for characterizing the transformer's operating state.

[0017] Thirdly, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the transformer multi-source modeling method based on hierarchical features and cross-modal fusion, which is the first aspect of the present invention.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention effectively solves the technical problem of the difficulty in uniformly representing multimodal information through hierarchical feature extraction and cross-modal fusion mechanisms. First, targeted semantic feature extraction, spatial structure feature extraction, and time-frequency domain feature extraction are performed on operation and maintenance text data, inspection image data, and voiceprint signal data, respectively, generating a set of text entity relationship triples, image classification labels, and voiceprint classification labels, achieving a unified representation of different modal data at the structured level. Second, the above three types of labels are mapped to a unified semantic vector space, eliminating the feature space differences between modalities and laying the foundation for cross-modal fusion. Crucially, this invention introduces a cross-modal similarity index, which comprehensively considers the distance between semantic vectors, the probability distribution confidence of classification results, and information entropy, achieving a quantitative assessment of the reliability and consistency of each modality. Based on this index, fusion weights are dynamically allocated, allowing features with high confidence, low information entropy, and good intermodal consistency to receive greater weight during the weighted fusion process. This results in a multimodal joint representation vector that adaptively highlights the contribution of reliable modalities and suppresses the interference of unreliable modalities, significantly improving the accuracy and robustness of transformer operating status representation.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 A schematic diagram of the framework of a transformer multi-source modeling method based on hierarchical features and cross-modal fusion provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of a text knowledge extraction model based on joint semantic modeling provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram of a transformer inspection image classification model based on depth vision perception provided as an exemplary embodiment of the present invention; Figure 4 A schematic diagram of a transformer acoustic signature classification method based on time-frequency acoustic characterization provided as an exemplary embodiment of the present invention; Figure 5 A schematic diagram of a transformer multi-source modeling system based on hierarchical features and cross-modal fusion, provided as an exemplary embodiment of the present invention; Figure 6 A schematic diagram of a computer device provided for an exemplary embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] The accelerating digital and intelligent transformation of power grids places higher demands on the refined modeling and intelligent analysis capabilities of power equipment operating status. As a key core device in the power system for voltage transformation and energy transmission, the operating status of transformers directly affects the safety and stability of the power grid. Faults can lead to equipment performance degradation or even power outages, causing significant economic losses. Therefore, high-precision, multi-dimensional modeling and analysis of transformer operating status has become an important direction in current research on intelligent power equipment. Currently, transformer operating status analysis and modeling mainly rely on traditional single data sources or experience-driven methods, such as statistical modeling and feature analysis methods based on electrical quantities, temperature, or oil chromatography data. These methods typically rely on manual experience for feature selection and model construction, making it difficult to fully explore the potential correlations between multiple information sources. They suffer from problems such as limited modeling dimensions, limited feature expression capabilities, and insufficient adaptability to complex operating conditions. Furthermore, different types of transformers differ in their operating environment and structural characteristics, making it difficult for modeling methods based on fixed rules or single-modal data to achieve unified and generalized state representation.

[0025] With the development of intelligent sensing technology, the Industrial Internet of Things (IIoT), and multi-source data acquisition methods, transformers have gradually accumulated multimodal heterogeneous data during operation, including maintenance text records, image monitoring data, and acoustic vibration signals. These data differ significantly in their expression, time scale, and semantic structure, but they can reflect the equipment's operating status from different perspectives, possessing strong complementarity and modeling value. However, effectively extracting features from multimodal data and constructing a unified fusion modeling representation space to achieve collaborative expression and consistent modeling of cross-modal information remains a significant challenge. Existing methods attempt to jointly model and represent features from multi-source heterogeneous data, but these methods often rely on a single feature space or fixed rules, making it difficult to adapt to the dynamic changes in data distribution and the complex coupling of multimodal information during transformer operation. On the one hand, traditional modeling methods have limited feature representation capabilities, making it difficult to depict the deep relationships between different modes; on the other hand, some data-driven methods still suffer from insufficient interaction and weak structural constraints during cross-modal alignment and fusion. Therefore, existing methods still have certain limitations in the unified feature modeling and fusion representation of multi-source heterogeneous data.

[0026] In view of the problems existing in the existing solutions, this invention proposes a transformer multi-source modeling method based on hierarchical features and cross-modal fusion, such as... Figure 1 As shown, by performing unified feature modeling and collaborative fusion of multi-source data such as text, images, and voiceprints, a multi-dimensional information representation and joint modeling of transformer operating status is achieved. This invention mainly consists of three parts: a multimodal data acquisition module, a feature extraction module, and a fusion modeling module. First, the multimodal data acquisition module collects maintenance text data, image monitoring data, and voiceprint signal data generated during transformer operation, and cleans and standardizes the raw data. Second, the feature extraction module extracts semantic features from the text data, spatial structure features from the image data, and time-frequency domain features from the voiceprint data, thereby obtaining basic representation vectors for different modalities. Then, based on feature alignment, the fusion modeling module introduces a cross-modal interaction mechanism to correlate and model the features of different modalities, and realizes the fusion expression of multi-source information in a unified representation space, thus obtaining the fused multimodal feature representation. This invention achieves unified representation and collaborative modeling of multi-source heterogeneous data such as text, images, and voiceprints through multi-modal feature hierarchical extraction and fusion modeling. It effectively improves the information complementarity and expression consistency between different modalities, thereby specifically solving the problems of insufficient single-modal expression capability and insufficient cross-modal fusion in existing methods, and improving the integrity and robustness of transformer multi-source information modeling.

[0027] To achieve the above objectives, the specific implementation methods adopted by the present invention are as follows: S101: Multimodal data acquisition module.

[0028] The data quality of the multimodal data acquisition module directly affects the subsequent feature extraction and fusion modeling results. To ensure the integrity and consistency of multi-source heterogeneous data, this invention adopts a combination of multi-source acquisition and data preprocessing to construct the original multimodal dataset, including maintenance text data, image monitoring data, and acoustic signature data generated during transformer operation. The text data mainly comes from unstructured and semi-structured text information such as power equipment maintenance records, defect analysis reports, and inspection logs; the image data mainly comes from visible light inspection images; and the acoustic signature data mainly comes from partial discharge signals, equipment vibration signals, and abnormal operation acoustic signals.

[0029] S101-1: Text data normalization processing.

[0030] To address the issues of inconsistent expression and excessive redundant information in text data, the original text data is standardized. First, a specialized dictionary for the power equipment field is constructed to uniformly map professional terms and abbreviations, such as standardizing the expression of "PT abnormality" and "voltage transformer abnormality." Second, a word segmentation tool is used to segment the text, and a stop word list is used to filter out invalid words to reduce the impact of redundant information on subsequent semantic feature extraction.

[0031] S101-2: Image data preprocessing.

[0032] To address issues such as inconsistent resolution, lighting differences, and noise interference in image data, the acquired raw images undergo standardization processing. First, the images are resized and normalized to meet model input requirements and improve data consistency. Second, noise reduction and key region enhancement are performed to reduce environmental interference and highlight key components of the equipment.

[0033] S101-3: Voiceprint data preprocessing.

[0034] To address issues such as inconsistent sampling, noise interference, and non-stationary characteristics in voiceprint signals, the original voiceprint data is standardized. First, voiceprint signals with different sampling frequencies are uniformly processed to ensure consistency across the time scale. Second, the signals undergo denoising and feature transformation to reduce environmental interference and obtain stable time-frequency feature representations.

[0035] S102: Feature extraction module.

[0036] S102-1: Text knowledge extraction model based on joint semantic modeling.

[0037] This invention proposes a prompt-driven entity relation joint extraction method based on a pre-trained language model, used to achieve integrated modeling of entity recognition and relation extraction in transformer operation and maintenance texts, such as... Figure 2 The image shows the text feature extraction model of this invention. This invention improves the accuracy and consistency of extracting complex industrial text information by constructing structured task prompts, introducing a pre-trained language model for semantic reasoning, and combining an output constraint mechanism to achieve structured knowledge generation.

[0038] S102-1.1: Task prompt construction and input reconstruction.

[0039] First, the transformer operation and maintenance text to be processed is jointly encoded with the task instruction information to construct a structured prompt input, so that the model can clearly understand the current task objective and the form of output constraints, thereby transforming the unstructured text analysis task into a controllable generative reasoning task.

[0040] (1); in, This represents the one-hot encoded vector matrix of the words in the input sentence. This represents task instruction information, used to define extraction targets and output format constraints. This represents the structured prompt input after construction, which is used as the unified input for the pre-trained language model.

[0041] S102-1.2: Semantic reasoning using pre-trained language models.

[0042] After receiving structured prompts, the input is fed into a pre-trained language model for semantic understanding and relational reasoning. Through the model's internal context modeling capabilities, entity information and potential relationships in the text are jointly parsed to generate structured semantic output results. (2); Wherein, LLM() represents the semantic reasoning process of a pre-trained language model, which is based on large-scale corpus pre-training and has the ability of context understanding and implicit logical reasoning. Y This represents the model output, which includes entity information, relational information, or structured descriptive information.

[0043] S102-1.3: Structured parsing and triple generation.

[0044] Since the output of pre-trained language models is usually in a freely generated form, to ensure the structural consistency and usability of the extracted results, this invention further performs structured parsing and normalization on the model output, converting it into a unified entity relation triple representation: (3); Here, Format() represents the structure parsing and constraint function, used to extract the main entity s and relation types from the model output. r and object entity o And organize them according to a unified format; T represents the final set of entity relation triples generated.

[0045] S102-2: Transformer inspection image classification model based on depth visual perception.

[0046] For transformer inspection image data collected during transformer operation and maintenance, a transformer inspection image classification model based on deep transfer representation learning is constructed, such as... Figure 3 As shown, this model uses a ResNet-50 network pre-trained on a large-scale natural image dataset as its backbone feature extraction structure. It transfers knowledge from the source domain to the target task through transfer learning, thereby improving the model's feature representation and generalization capabilities in small-sample scenarios.

[0047] S102-2.1: Feature modeling method based on migration initialization.

[0048] For power transformer fault image data, data augmentation is first performed by randomly cropping, flipping, and scaling the training samples. The processed samples are then normalized. The ResNet50 pre-trained model on ImageNet is used as the image feature processor, and the model structure is as follows.

[0049] (4); in, x This represents the input image of a transformer fault. This represents a feature extraction backbone network based on ResNet-50, where... These are the base parameters obtained from pre-training; This represents the task classification head network, where is a learnable parameter; f(x) represents the final classification output.

[0050] S102-2.2: Model Training and Loss Optimization.

[0051] During model training, a full fine-tuning strategy is employed to update the network parameters. Simultaneously, L2-SP regularization is introduced to limit the deviation of model parameters relative to pre-trained parameters, thereby improving the stability and generalization ability of transfer learning. The joint loss function is defined as follows: (5); in, Denotes the overall optimization objective function; Represents the cross-entropy loss function; The regularization coefficient is used. This represents the set of all parameters involved in the optimization of the model; p This represents the model parameters at the current training stage; This represents the initial parameters corresponding to the pre-trained model.

[0052] S102-3: Transformer acoustic signature classification method based on time-frequency acoustic characterization.

[0053] To address the problems of strong non-stationarity, complex noise interference, and unclear fault mode differences in abnormal acoustic signature signals generated during power transformer operation, a transformer acoustic signature classification method based on MFCC and deep acoustic embedding networks is proposed, such as... Figure 4As shown, firstly, a Short-Time Fourier Transform (STFT) is performed on the original acoustic signature signal to obtain its time-frequency representation. Then, the spectrum is filtered using a Mel filter bank, and logarithmic compression and Discrete Cosine Transform (DCT) are combined to extract MFCC cepstral features to characterize the spectral envelope properties and local time-frequency structure of the acoustic signature signal. Based on this, a CAMPPLus deep acoustic embedding network is introduced to learn the high-dimensional representation of the MFCC features. Abnormal acoustic signature patterns are modeled through multi-layer nonlinear mapping, thereby obtaining an acoustic embedding representation with stronger discriminative capabilities. Finally, a classifier is used to determine the fault category of the embedded features, achieving automatic identification and output of abnormal transformer operating states.

[0054] First, the acquired raw voiceprint signal is framed and windowed, and its time-spectral representation is obtained using Short Time Fourier Transform (STFT): (6); (7); in, This represents the raw acoustic signature signal collected during transformer operation. For window functions, N This indicates the number of sampling points per frame. t and k These are the time frame index and the frequency index, respectively. This is the time-frequency representation obtained after the original voiceprint signal undergoes a short-time Fourier transform. Let represent the frequency response function of the r-th Mel filter. Indicates the first t Frame in r The filter bank energy in each Mel frequency band is used to characterize the time-frequency energy distribution of transformer anomalous acoustic signatures.

[0055] Subsequently, logarithmic compression is applied to the Mel band energy to reduce the spectral dynamic range and enhance the characteristics of the low-energy region: (8); in, Let represent the logarithmic energy feature corresponding to the r-th Mel frequency band. Based on this, a Discrete Cosine Transform (DCT) is introduced to decorrelate the logarithmic Mel energy, thereby extracting the MFCC cepstral features: (9); in, Indicates the first n MFCC cepstral coefficients; R Indicates the number of Mel filters; Indicates the first r Logarithmic energy values ​​corresponding to each Mel frequency band This represents the total number of Mel filters. Through the above process, an MFCC feature sequence that can effectively characterize the spectral envelope structure of abnormal acoustic signatures in transformers can be obtained.

[0056] The extracted MFCC feature sequences are input into the CAMPPLus deep acoustic embedding network for high-dimensional feature modeling. This network consists of an FCM convolutional residual front-end, a Dense-TDNN backbone network incorporating CAM channel attention, and embedding layers. It is used to learn unified representations of anomalous speaker information under different time scales and frequency distributions. The feature extraction process is as follows: (10); in, This represents the MFCC input feature corresponding to frame t; This represents the convolutional feature mapping operation, used to extract local time-frequency structural features; This represents the Dense-TDNN feature modeling module, used to learn long-term temporal context dependencies; This indicates the channel attention enhancement module, which is used to adaptively enhance important feature responses related to abnormal voiceprint detection; This represents the final output deep acoustic embedding feature vector.

[0057] After obtaining the deep acoustic embedding features, a Softmax classifier is used to distinguish the transformer fault categories. The calculation process is as follows: (11); in, Represents the classification layer weight matrix. b For bias terms, This is a normalization function used to output the probability distribution of each fault category. Labels for the final predicted transformer fault categories.

[0058] S103: A multimodal fusion method based on semantic similarity.

[0059] This invention proposes a semantic similarity-driven multimodal fusion method to achieve adaptive information fusion between text entity relationships, image classification labels, and voiceprint classification labels. This method calculates the similarity relationships between multiple modalities in a unified semantic space and constructs a dynamic weighting mechanism based on similarity to achieve weighted fusion and collaborative decision-making of multi-source heterogeneous information.

[0060] S103-1: Multimodal semantic construction.

[0061] To achieve comparability between data from different modalities, a unified semantic representation is first applied to text entity relationships, image classification labels, and voiceprint classification labels, mapping them to the same vector space. Text entity relationships are then converted into vector representations using a semantic encoding function, the calculation process of which is as follows: (12); (13); (14); in, Semantic vectors of text entity relations For relationships between entities, and The classification labels are for images and voiceprints, respectively. These are entity relation encoding function, image label embedding function, and voiceprint label embedding function, respectively.

[0062] S103-2: PARCS cross-modal similarity calculation.

[0063] To comprehensively measure semantic consistency, classification confidence, and prediction reliability among different modalities, this invention proposes a cross-modal similarity index, PARCS, based on probability-aware reliability calibration. The calculation process is as follows: (15); (16); (17); in, For modality i With mode j Cross-modal fusion metrics: the distance between semantic vectors of different modalities. The confidence decay coefficient is... These are the category probability distribution vectors corresponding to different modalities. For modal reliability calibration items, The information entropy corresponding to the mode. It is an exponential function. Representative mode , Semantic vector distance and Representative mode and modality The semantic vector.

[0064] After obtaining the PARC cross-modal similarity index among different modalities, dynamic fusion weights are generated based on semantic consistency, classification confidence, and reliability levels among the different modalities. A cross-modal similarity threshold τ is set to filter modal associations. When the PARC index between different modalities is higher than the threshold, they are considered to have strong semantic association and are assigned a higher fusion weight (i.e., the first fusion weight); otherwise, the contribution of the corresponding modality to the fusion result is reduced, thereby reducing the interference of low-reliability information on the overall modeling result (i.e., a second fusion weight is assigned, which is less than the first fusion weight). Subsequently, text features, image features, and voiceprint features are weighted and fused to construct a unified multimodal joint representation vector to achieve collaborative expression and semantic alignment between different modal information. Through the above adaptive fusion mechanism, the information complementarity and fusion consistency between multi-source heterogeneous data can be effectively enhanced, improving the accuracy, stability, and robustness of transformer multi-source information modeling under complex operating environments.

[0065] In summary, this invention proposes a transformer multi-source information modeling framework based on hierarchical feature extraction and cross-modal fusion. It constructs a hierarchical feature extraction method for text, image, and voiceprint data, which can extract semantic features, visual features, and time-frequency acoustic features respectively, effectively improving the feature representation and discrimination capabilities of different modal information. Furthermore, it proposes a cross-modal fusion method based on semantic similarity, which realizes information association and collaborative fusion between different modalities through a unified semantic space and dynamic weight mechanism, thereby improving the accuracy and consistency of multi-source heterogeneous data fusion modeling.

[0066] Figure 5 This paper presents a transformer multi-source modeling system based on hierarchical features and cross-modal fusion, including: The multi-source data acquisition unit 501 is configured to acquire maintenance text data, inspection image data and acoustic signal data generated during transformer operation. The hierarchical feature extraction unit 502 is configured to: extract semantic features from operation and maintenance text data to generate a set of text entity relation triples; extract spatial structure features from patrol image data to generate image classification labels; and extract time-frequency domain features from voiceprint signal data to generate voiceprint classification labels. The semantic vector mapping unit 503 is configured to map the set of text entity relation triples, image classification labels and voiceprint classification labels to a unified semantic vector space to obtain text semantic vectors, image semantic vectors and voiceprint semantic vectors. The cross-modal similarity calculation unit 504 is configured to: calculate the cross-modal similarity index between each pair of text semantic vectors, image semantic vectors and voiceprint semantic vectors. The cross-modal similarity index comprehensively considers the distance between semantic vectors, the probability distribution reliability of the corresponding classification results and the information entropy of the probability distribution. The dynamic fusion generation unit 505 is configured to: assign dynamic fusion weights to the text semantic vector, image semantic vector and voiceprint semantic vector according to the cross-modal similarity index, and perform weighted fusion of the three to generate a multimodal joint representation vector for characterizing the transformer's operating state.

[0067] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0068] According to another embodiment of the present invention, the system of this embodiment can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method of the present invention on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0069] Figure 6 A computer device is shown, which includes a processor 601, a communication interface 602, and a computer-readable storage medium 603. The processor 601, communication interface 602, and computer-readable storage medium 603 can be connected via a bus or other means.

[0070] The communication interface 602 is used to receive and send data. The computer-readable storage medium 603 can be stored in the memory of the electronic device. The computer-readable storage medium 603 is used to store computer programs, which include program instructions. The processor 601 is used to execute the program instructions stored in the computer-readable storage medium 603.

[0071] The processor 601 is the computing and control core of the electronic device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve the corresponding method flow or corresponding function.

[0072] Processor 1001 is configured to perform the following procedure: Acquire maintenance text data, inspection image data, and acoustic signal data generated during transformer operation; Semantic features are extracted from operation and maintenance text data to generate a set of text entity relation triples; spatial structure features are extracted from patrol image data to generate image classification labels; and time-frequency domain features are extracted from voiceprint signal data to generate voiceprint classification labels. The text entity relation triplet set, image classification label and voiceprint classification label are mapped to a unified semantic vector space to obtain text semantic vector, image semantic vector and voiceprint semantic vector respectively. Based on text semantic vectors, image semantic vectors, and voiceprint semantic vectors, cross-modal similarity indices are calculated between each pair of modalities. The cross-modal similarity index comprehensively considers the distance between semantic vectors, the probability distribution reliability of the corresponding classification results, and the information entropy of the probability distribution. Based on the cross-modal similarity index, dynamic fusion weights are assigned to the text semantic vector, image semantic vector and voiceprint semantic vector, and the three are weighted and fused to generate a multimodal joint representation vector for characterizing the transformer's operating state.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0074] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital cable) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A transformer multi-source modeling method based on hierarchical features and cross-modal fusion, characterized in that, The process includes the following: Acquire maintenance text data, inspection image data, and acoustic signal data generated during transformer operation; Semantic features are extracted from operation and maintenance text data to generate a set of text entity relation triples; spatial structure features are extracted from patrol image data to generate image classification labels; and time-frequency domain features are extracted from voiceprint signal data to generate voiceprint classification labels. The text entity relation triplet set, image classification label and voiceprint classification label are mapped to a unified semantic vector space to obtain text semantic vector, image semantic vector and voiceprint semantic vector respectively. Based on text semantic vectors, image semantic vectors, and voiceprint semantic vectors, cross-modal similarity indices are calculated between each pair of modalities. The cross-modal similarity index comprehensively considers the distance between semantic vectors, the probability distribution reliability of the corresponding classification results, and the information entropy of the probability distribution. Based on the cross-modal similarity index, dynamic fusion weights are assigned to the text semantic vector, image semantic vector and voiceprint semantic vector, and the three are weighted and fused to generate a multimodal joint representation vector for characterizing the transformer's operating state.

2. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 1, characterized in that, Semantic feature extraction is performed on the operation and maintenance text data to generate a set of text entity relation triples, including: The operation and maintenance text data is jointly encoded with the preset task instruction information to construct a structured prompt input; The structured prompts are fed into a pre-trained language model for semantic reasoning, generating a structured output containing entity and relation information. The structured output is parsed and formatted, and the subject entity, relation type, and object entity are extracted and organized into a set of text entity relation triples.

3. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 1, characterized in that, Spatial structure features are extracted from the patrol image data to generate image classification labels, including: The inspection image data is input into a convolutional neural network backbone pre-trained on a large-scale image dataset; Spatial structural features of the patrol image data are extracted using a convolutional neural network backbone, and then classified using a task classification head network to output image classification labels.

4. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 1, characterized in that, Time-frequency domain feature extraction is performed on the voiceprint signal data to generate voiceprint classification labels, including: The voiceprint signal data is framed, windowed, and subjected to short-time Fourier transform to obtain the time spectrum; The time spectrum is filtered using the Mel filter bank, and the filtered results are subjected to logarithmic compression and discrete cosine transform to obtain the MFCC cepstral feature sequence. The MFCC cepstral feature sequence is input into a deep acoustic embedding network for high-dimensional feature modeling to obtain a deep acoustic embedding feature vector; based on the deep acoustic embedding feature vector, a classifier outputs a voiceprint classification label.

5. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 4, characterized in that, Deep acoustic embedding networks consist of a convolutional residual front-end, a Dense-TDNN backbone network with channel attention mechanism, and embedding layers. The high-dimensional feature modeling process includes: The local time-frequency structure features of the MFCC cepstral feature sequence are extracted through the convolutional residual front-end; the local time-frequency structure features are input into the Dense-TDNN backbone network to learn long-term temporal context dependencies, and important feature responses related to abnormal voiceprint discrimination are adaptively enhanced through the channel attention mechanism; and deep acoustic embedding feature vectors are output through the embedding layer.

6. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 1, characterized in that, The text entity relation triplet set, image classification labels, and voiceprint classification labels are mapped to a unified semantic vector space, including: The relationships in the set of text entity relationship triples are mapped to text semantic vectors through a preset entity relationship encoding function; image classification labels are mapped to image semantic vectors through a preset image label embedding function; and voiceprint classification labels are mapped to voiceprint semantic vectors through a preset voiceprint label embedding function.

7. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 1, characterized in that, Calculate the cross-modal similarity index between each pair of modes, including: For any two modalities, calculate the distance between their corresponding two semantic vectors; obtain the probability distribution vectors of the classification results of each of the two modalities, and calculate the information entropy of each probability distribution vector; based on the distance, probability distribution vectors, and information entropy, calculate the cross-modal similarity index between the two modalities. The calculation of the cross-modal similarity index includes: using distance as the basic term, introducing a confidence decay coefficient related to the probability distribution vector, and a reliability calibration term related to information entropy, which together constitute the cross-modal similarity index.

8. The transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in claim 1, characterized in that, Based on cross-modal similarity metrics, dynamic fusion weights are assigned to text semantic vectors, image semantic vectors, and voiceprint semantic vectors, including: The cross-modal similarity index between any two modalities is compared with a preset similarity threshold. If the cross-modal similarity index is higher than the similarity threshold, the corresponding modal pair is assigned a first fusion weight; otherwise, a second fusion weight is assigned. The first fusion weight is greater than the second fusion weight.

9. A transformer multi-source modeling system based on hierarchical features and cross-modal fusion, characterized in that, include: The multi-source data acquisition unit is configured to acquire maintenance text data, inspection image data, and acoustic signal data generated during transformer operation. The hierarchical feature extraction unit is configured to: extract semantic features from maintenance text data to generate a set of text entity relation triples; extract spatial structure features from patrol image data to generate image classification labels; and extract time-frequency domain features from voiceprint signal data to generate voiceprint classification labels. The semantic vector mapping unit is configured to map the set of text entity relation triples, image classification labels and voiceprint classification labels to a unified semantic vector space to obtain text semantic vectors, image semantic vectors and voiceprint semantic vectors. The cross-modal similarity calculation unit is configured to: calculate the cross-modal similarity index between each pair of text semantic vectors, image semantic vectors and voiceprint semantic vectors. The cross-modal similarity index comprehensively considers the distance between semantic vectors, the probability distribution reliability of the corresponding classification results and the information entropy of the probability distribution. The dynamic fusion generation unit is configured to: assign dynamic fusion weights to the text semantic vector, image semantic vector and voiceprint semantic vector according to the cross-modal similarity index, and perform weighted fusion of the three to generate a multimodal joint representation vector for characterizing the transformer's operating state.

10. A computer device, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the transformer multi-source modeling method based on hierarchical features and cross-modal fusion as described in any one of claims 1 to 8.

Images