Power equipment cross-modal sample data alignment method and system based on semantic anisotropic mapping

Abstract

The application discloses a power equipment cross-modal sample data alignment method and system based on semantic anisotropic mapping, and relates to the field of intelligent power information processing. The method comprises the following steps: acquiring a visible light image, an infrared image, sensor time series data and maintenance text of the same device instance, extracting original features and identifying component categories respectively; calling a semantic anisotropic mapping model based on the component categories, implementing non-equidistant scaling and rotating projection on the features of each mode by using a corresponding metric matrix, and obtaining anisotropic embedding in a unified semantic space; calculating anisotropic distance between modes, and introducing cross-modal attention as a weight penalty term to dynamically correct interaction weights to generate a multi-modal fusion representation; and inputting the fusion representation into a task decision model to output a device state evaluation result. Thus, the consistency of the alignment of heterogeneous modes and the fault sensitive representation capability are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent power information processing technology, and in particular to a method and system for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping. Background Technology

[0002] With the development of smart grids, the condition monitoring of power equipment has shifted from single-modality to multi-modal fusion. Traditional power equipment diagnosis mainly relies on visible light images or infrared thermal imaging. The image matching techniques used (such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), or conventional deep learning feature extraction) typically focus on solving rotation and scale consistency problems. However, when processing heterogeneous images (such as visible light and infrared), these methods often struggle to overcome significant radiometric differences and noise interference, and are mostly limited to single-modality data processing. They cannot effectively establish joint feature representations between images, text logs, and sensor time-series data, resulting in insufficient utilization of multi-source information.

[0003] In the field of cross-modal alignment and retrieval, existing technical solutions (such as image-text retrieval in remote sensing or general cross-modal attention models) introduce attention mechanisms to capture the correlation between modalities, but most rely on global feature calculations for similarity or only focus on coarse-grained spatial location correspondences. This isotropic approach ignores the complex internal semantic structure of power equipment. For example, the crack direction of a surge arrester or the surface contamination of an insulator have different importance weights and directional sensitivities in diagnosis, which cannot support the realization of refined fault diagnosis and condition early warning. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a method and system for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping. This method aims to at least solve the problem in the prior art that it is difficult to achieve semantic consistency alignment and effective fusion of heterogeneous data in multimodal condition monitoring of power equipment, which leads to insufficient utilization of key fault clues and limited diagnostic reliability.

[0005] A method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping, the method comprising:

[0006] Acquire multimodal sample data to be processed from power equipment; the multimodal sample data to be processed includes at least visible light images, infrared images, sensor time-series data, and maintenance text data for the same power equipment instance;

[0007] Feature extraction operations are performed on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality, and the component category to which the power equipment instance belongs is identified;

[0008] Based on the component category, a pre-built semantic anisotropic mapping model is invoked to perform projection processing on the original feature vectors of each modality; the semantic anisotropic mapping model is configured with a metric matrix defined for different component categories, which is used to perform non-equidistant scaling and rotation transformations on the original feature vectors according to the fault-sensitive direction of the component category, so as to obtain anisotropic semantic embedding vectors in a unified semantic space.

[0009] Based on the anisotropic semantic embedding vectors of each modality, the anisotropic distance between modalities is calculated, and the anisotropic distance is introduced as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance.

[0010] The multimodal fusion representation vector is input into a preset task decision model to output the equipment status assessment result for the power equipment instance.

[0011] Furthermore, the acquisition of the multimodal sample data to be processed from the power equipment includes performing the following spatiotemporal consistency preprocessing steps:

[0012] Perform multi-rate timing alignment, obtain the absolute timestamp of each modal data based on a unified timing reference, and use the modal data with the lowest sampling frequency as the anchor point. Use a sliding time window to extract visible light images, infrared images, sensor timing data and maintenance text data that overlap with its timing to construct a multimodal synchronization data frame.

[0013] A unified spatial index benchmark is established, and the infrared image is registered to the pixel coordinate system of the visible light image using a pre-calibrated extrinsic transformation matrix. Based on the installation coordinates of the physical sensor in three-dimensional space corresponding to the sensor time series data, the sensor data is projected and mapped to the pixel coordinate system, thereby giving the sensor data a spatial location attribute.

[0014] The text entity disambiguation association is performed, and the device entity descriptions in the maintenance text data are extracted using a pre-built power domain named entity recognition network. The device entity descriptions are then mapped to unique device instance identifiers using a device topology tree index to ensure that the maintenance text data matches the power equipment instances in the visible light image.

[0015] Further, the step of performing feature extraction operations on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality, and identifying the component category to which the power equipment instance belongs, includes:

[0016] For the visible light image, a visual coding network including a key component detection branch is constructed to output the component category label and location bounding box of the power equipment instance, and the component category label is used as the result of identifying the component category to which the power equipment instance belongs; at the same time, a region of interest alignment operation is performed based on the location bounding box to extract local visual features that retain the geometric structure information of the components.

[0017] For the infrared image, deep thermal image features are extracted using a visual Transformer, and pixel-level heat flow vector fields are calculated using a temperature gradient operator, thereby separating the temperature gradient direction features that characterize the direction of fault heat diffusion.

[0018] For the sensor time-series data, a time-domain trend feature is extracted using a time-series convolutional network, and a multi-scale wavelet packet decomposition technique is introduced to project the time-series signal onto different frequency band subspaces in order to extract multi-band spectral features that reflect the directional differences in the frequency domain energy distribution of the fault signal.

[0019] For the maintenance text data, a pre-trained BERT model fine-tuned with knowledge of the power field is used to perform structured semantic encoding to extract entity embedding vectors representing equipment identity and description embedding vectors representing fault states, respectively.

[0020] The local visual features, temperature gradient direction features, multi-band spectral features, and entity and description embedding vectors extracted from each of the above modalities are vectorized and concatenated or dimensionally aligned to generate the original feature vectors corresponding to each modality.

[0021] Furthermore, the semantic anisotropic mapping model is configured with a metric matrix defined for different component categories, used to perform non-equidistant scaling and rotation transformations on the original feature vector according to the fault-sensitive direction of the component category, to obtain anisotropic semantic embedding vectors in a unified semantic space, including:

[0022] For each component category Construct a learnable positive definite metric matrix And perform matrix decomposition on it to obtain the transformation matrix. Thus, a geometric transformation basis satisfying the following constraints is established:

[0023] ;

[0024] In the formula, Represents the transpose operation of a matrix or vector;

[0025] Using the transformation matrix The linear projection operator will... The original feature vectors of each modality Mapped to the anisotropic semantic embedding vector :

[0026] ;

[0027] in, For component categories In the Featured cluster centers under each modality are used to eliminate distribution bias between modalities;

[0028] The row vector direction represents the fault-sensitive direction in the feature space, and its norm represents the semantic importance weight along that direction. Through this mapping, the small differences in the fault-sensitive direction of the original feature vector are amplified in the unified semantic space, thereby making the anisotropic semantic embedding vector... The Euclidean distance is equivalent to the distance in the original space based on the metric matrix. Mahalanobis distance.

[0029] Furthermore, based on the anisotropic semantic embedding vectors of each modality, the anisotropic distance between modalities is calculated, and the anisotropic distance is introduced as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance, including:

[0030] Constructing the cross-modal distance matrix , of which Anisotropic semantic embedding vectors of each modality With the Anisotropic semantic embedding vectors of each modality Elements between Defined as the squared Euclidean distance between the two in the unified semantic space, it is used to quantify the semantic differences of different modal features in the fault-sensitive direction:

[0031] ;

[0032] Perform linear projection on the anisotropic semantic embedding vectors of each modality to generate the query matrix required by the attention mechanism. Key matrix AND-value matrix And an anisotropic attention model incorporating a distance penalty term is constructed:

[0033] ;

[0034] in, The statistical correlation between modes was characterized. Learnable physical constraint coefficients; distance penalty term This is used to forcibly suppress pseudo-matching modalities that have high statistical correlation but are semantically far apart in the fault-sensitive direction, thereby outputting a weighted result that conforms to physical laws as the multimodal fusion representation vector.

[0035] Furthermore, the method also includes sparsification enhancement of the cross-modal attention mechanism by introducing a hard threshold gating mechanism, specifically including:

[0036] In execution Before the operation, based on the distance matrix Constructing a semantic association mask matrix It sets an anisotropic distance threshold to represent the effective semantic association boundary. ;

[0037] When the anisotropic distance between cross-modal sample pairs Exceeding the threshold When this happens, the feature pair is determined to be an invalid association, and the mask matrix is... The corresponding element is assigned a value of negative infinity, otherwise it is assigned a value of zero;

[0038] The mask matrix The weighted penalty term is superimposed onto the attention score matrix, which already contains the weighted penalty term, to forcibly truncate long-tailed distribution interactions with excessively large anisotropic distances, thereby eliminating false matching interference that are statistically similar but semantically significantly different in fault-sensitive physical directions.

[0039] Furthermore, the semantic anisotropy mapping model is obtained by training by minimizing a multi-task joint loss function, which at least includes anisotropy consistency loss. This is used to explicitly constrain the consistency of projections of different modal features onto the fault-sensitive direction during model optimization.

[0040] ;

[0041] in, It is a set of cross-modal positive sample pairs consisting of different modal data for the same power equipment instance; and These are the anisotropic semantic embedding vectors of the positive sample pair in the unified semantic space, respectively. It represents the Hadamah accumulation. For component categories Category balance coefficient; Represents the square of the L2 norm;

[0042] The fault sensitivity weight vector is defined based on the physical characteristics of the component. It is assigned a high weight value on the dimension corresponding to the principal axis of the fault feature and a low weight value on the dimension corresponding to the environmental noise. This allows the model to prioritize minimizing the cross-modal semantic differences of the key fault dimensions during training, thereby achieving feature alignment driven by physical mechanisms.

[0043] Further, the step of inputting the multimodal fusion representation vector into a preset task decision model to output an equipment status assessment result for the power equipment instance includes:

[0044] The multimodal fusion representation vector is weighted and fused with the anisotropic semantic embedding vector of each modality through residual connection to generate enhanced device state features;

[0045] Depending on the specific task requirements, select one or more of the following output headers for processing:

[0046] Fault classification head: The fully connected layer is used to classify and map the device state features, and the mapping results are normalized and probability calculated to output the fault type probability distribution of the power equipment.

[0047] State regression head: Uses a regression network to process the state characteristics of the equipment and outputs the health index or remaining service life prediction value of the power equipment;

[0048] Based on one or more of the fault type probability distribution, the health index, and the remaining service life prediction value, determine the equipment status assessment result used to characterize the current physical operating status of the power equipment instance.

[0049] A cross-modal sample data alignment system for power equipment based on semantic anisotropic mapping includes:

[0050] The data acquisition unit is used to acquire multimodal sample data to be processed from power equipment; the multimodal sample data to be processed includes at least visible light images, infrared images, sensor time-series data, and maintenance text data for the same power equipment instance;

[0051] The feature and category recognition unit is used to perform feature extraction operations on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality and to identify the component category to which the power equipment instance belongs;

[0052] The semantic anisotropic mapping unit is used to call a pre-built semantic anisotropic mapping model to perform projection processing on the original feature vectors of each modality based on the component category. The semantic anisotropic mapping model is configured with a metric matrix defined for different component categories, which is used to perform non-equidistant scaling and rotation transformations on the original feature vectors according to the fault-sensitive direction of the component category, so as to obtain anisotropic semantic embedding vectors in a unified semantic space.

[0053] An anisotropic fusion unit is used to calculate the anisotropic distance between modalities based on the anisotropic semantic embedding vectors of each modality, and to introduce the anisotropic distance as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance.

[0054] The state assessment unit is used to input the multimodal fusion representation vector into a preset task decision model to output the equipment state assessment result for the power equipment instance.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] This invention, after extracting features from multimodal sample data, introduces the identification of component categories in power equipment instances. Based on these component categories, it invokes a semantic anisotropic mapping model configured with corresponding metric matrices to perform non-equidistant scaling and rotation transformations on the original feature vectors of each modality, forming anisotropic semantic embedding vectors in a unified semantic space. Because the metric matrix is ​​bound to the component category, the embedding space imposes differentiated geometric constraints on the fault-sensitive directions of different components. This makes it easier to aggregate key representations related to the state of the same component in different modalities within the unified space and easier to separate them in non-key directions. Therefore, cross-modal alignment no longer relies on a single global similarity but forms a more stable alignable representation constrained by component semantics, providing a more consistent semantic foundation for subsequent multimodal interactions.

[0057] After obtaining the anisotropic semantic embedding vectors of each modality, this invention further calculates the anisotropic distance between modalities and introduces this distance as a weight penalty term into the cross-modal attention mechanism to dynamically adjust the attention weights of cross-modal interactions, thereby generating a multimodal fusion representation vector. This enables the cross-modal fusion process to possess semantic consistency filtering capabilities: when different modal features are more consistent in the fault-sensitive direction of a component, the anisotropic distance is smaller, the penalty on the attention weights is weaker, and the corresponding cross-modal interaction is strengthened; conversely, when cross-modal features are only similar at the appearance level but deviate in the fault-sensitive direction, the anisotropic distance increases and generates a stronger penalty, suppressing ineffective interactions and noise propagation. Therefore, the generated fusion representation vector more centrally expresses the cross-modal evidence related to equipment state determination and can provide a more discriminative input representation for subsequent task decision models, ensuring that the state assessment output maintains consistency and reliability even with multi-source information redundancy, noise differences, or local missing information.

[0058] This invention, based on a semantic anisotropic mapping coupled with component categories and metric matrices, further achieves adaptive filtering of cross-modal interactions through attention weight correction using anisotropic distance penalties, forming a closed-loop logical chain from unified embedding to fusion decision-making. Thus, the implicit component-differentiated semantics and fault-sensitive directions in power equipment diagnosis are explicitly injected into the cross-modal alignment and fusion process, giving the fused representation vector stronger semantic consistency and directional sensitivity, thereby more effectively supporting the task decision model to output more reliable equipment condition assessment results. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping.

[0060] Figure 2 A flowchart illustrating the operation of outputting equipment status assessment results for power equipment instances based on a task decision model;

[0061] Figure 3 This is a schematic diagram illustrating the principle mechanism of a cross-modal sample data alignment method for power equipment based on semantic anisotropic mapping.

[0062] Figure 4 A comparison chart of F1 score distributions using different methods on five typical types of power equipment;

[0063] Figure 5 The three-dimensional F1 fractional response surface plots are shown under the coupling effects of different sensor noise intensities and anisotropy coefficients.

[0064] Figure 6 A comparison chart showing the trade-off between inference latency and retrieval accuracy for different methods;

[0065] Figure 7 This is a block diagram of a cross-modal sample data alignment system for power equipment based on semantic anisotropic mapping. Detailed Implementation

[0066] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0067] Example 1:

[0068] It should be noted that the data acquired in the daily monitoring and fault diagnosis of power equipment is becoming increasingly diverse, encompassing various modalities such as visible light images, infrared thermal images, sensor monitoring curves, and text maintenance records. How to fully utilize these multi-source heterogeneous data for joint analysis has become crucial for improving the intelligence level of the power grid.

[0069] Early image matching techniques, such as SIFT and SURF algorithms, achieved matching by detecting key points in images and constructing feature descriptors. These methods are robust to rotation and scale changes in images of the same modality, but their matching performance often drops significantly when dealing with images from different sources, such as visible light and infrared, due to the substantial differences in their radiometric characteristics and noise distribution. Although later template matching methods (such as Fast-Match and Deep-Matching) have improved noise tolerance to some extent, it is difficult to balance matching accuracy and computational speed in practical applications. In recent years, deep learning technology has achieved end-to-end feature learning through architectures such as Siamese networks, graph neural networks, and Transformers. While this has improved feature representation capabilities, it is still largely limited to single-modality data processing and struggles to achieve effective fusion of cross-modal information.

[0070] To address the nonlinear radiometric distortion problem in heterogeneous image matching, a matching method based on anisotropic weighted moments and absolute phase-coherence direction histograms has been proposed. This method utilizes anisotropic filtering for nonlinear diffusion, calculates phase-coherence moments and constructs weighted moment equations, and combines this with absolute phase-direction gradient histograms for corresponding point matching. Experiments show that this method can effectively increase the number of corresponding points and reduce matching errors in heterogeneous image matching. However, this method mainly focuses on geometric and radiometric correction at the image level and has not yet addressed semantic alignment between broader modalities such as images, text, and sensor data. Furthermore, its Euclidean distance-based metric is insufficient to capture complex semantic relationships.

[0071] In the field of cross-modal retrieval, some studies have begun to focus on the semantic alignment problem within and between modalities. For example, in cross-modal recipe retrieval tasks, some scholars have pointed out that focusing only on the alignment between different modalities while ignoring the semantic consistency within a modality can lead to matching ambiguities. By introducing specific loss functions to constrain the semantic structure within a modality, retrieval accuracy can be effectively improved. This approach suggests that when dealing with complex multimodal sample data, it is necessary to mine the deep semantic information within each modality. In addition, in Remote Sensing Image–Text Retrieval (RSITR), some methods attempt to enhance the model's ability to perceive spatial relationships by reconstructing cross-modal location information, in order to solve the redundancy and mismatch problems caused by relying solely on global features to calculate similarity. Although these methods have made progress in specific domains, most of them have not been optimized for the specific semantics of power equipment (such as component structure and fault types), nor have they covered modalities specific to industrial scenarios such as sensor time-series data.

[0072] Within the power industry, some technical solutions have attempted cross-modal recognition and dataset construction. For example, some solutions propose extracting image and text features separately and fusing them using a cross-attention mechanism to achieve cross-modal target recognition. Simultaneously, some solutions involve the simultaneous acquisition and alignment of infrared and visible light images, as well as the generation of temperature statistics. However, these existing solutions typically employ global or coarse-grained attention fusion strategies, lacking refined modeling of the semantic anisotropy characteristics exhibited by key components of power equipment (such as surge arresters and insulators) under different modalities. In other words, current technologies fail to fully consider the different directional sensitivities and feature weights of different components in fault diagnosis, making it difficult to achieve high-precision condition assessment and fault location when faced with complex fault mechanisms and multi-source data.

[0073] In summary, although current technologies have made some progress in single-modal feature extraction, heterogeneous image matching, and preliminary cross-modal fusion, there is still a lack of a cross-modal alignment framework that can uniformly process multimodal sample data of power equipment and effectively model the semantic anisotropy of components. This makes it difficult to fully exploit the complementary advantages of multi-source data in practical applications, thus limiting the further improvement of the intelligent diagnostic level of power equipment.

[0074] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision and disclosure of users' personal information comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0075] Figure 1 A flowchart illustrating an example of a method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping according to an embodiment of the present invention is shown.

[0076] Regarding the execution subject of the method in this embodiment of the invention, it can be any controller or processor with computing or processing capabilities, such as a power equipment intelligent operation and maintenance analysis system platform (which can be deployed on a station-side edge computing gateway, an integrated online monitoring host, or a cloud computing node). Using semantic anisotropy measurement as a unified constraint, it runs through two layers of unified representation and cross-modal fusion: First, based on the component category of the power equipment instance, the semantic anisotropy mapping model is called to perform non-equidistant transformation on the original features of each modality, establishing a unified semantic space oriented towards the fault-sensitive direction; then, based on the anisotropic distance in the unified semantic space, weight penalty and dynamic correction are applied to the cross-modal attention interaction, so that the cross-modal fusion changes from generalized similarity aggregation to consistent aggregation oriented towards diagnostic semantics, thereby generating a multimodal fusion representation that better meets the needs of equipment status discrimination and supports subsequent status assessment and early warning output.

[0077] In some examples, it can be integrated into electronic devices or terminals through software, hardware, or a combination of both. The types of terminals or electronic devices can be diverse, including but not limited to integrated online monitoring devices at the station, inspection robot control terminals, handheld terminals for maintenance, edge computing servers, or cloud analytics servers.

[0078] like Figure 1 As shown, in step S110, the multimodal sample data to be processed of the power equipment is obtained. The multimodal sample data to be processed includes at least visible light images, infrared images, sensor time series data and maintenance text data for the same power equipment instance.

[0079] Here, the system constructs a multi-source data perception layer for specific power equipment (such as transformers, circuit breakers, or insulator strings) by connecting to edge acquisition terminals of the power Internet of Things and enterprise-level management systems. Specifically, the system uses a time synchronization protocol to acquire image data reflecting the external structure of the equipment from the visible light camera of the substation inspection robot, thermal image data reflecting the thermal field distribution of the equipment from the infrared thermal imager, high-frequency time-series waveforms reflecting the internal state from online monitoring devices (such as UHF sensors or vibration accelerometers), and retrieves historical maintenance records and defect log text of the equipment from the production management system (PMS). In this way, the acquisition of the above data is strictly based on the unique identifier (ID) of the same equipment instance, ensuring the consistency of the data source in the physical object for subsequent processing, thereby overcoming the misdiagnosis problem often caused by mixed data sources and providing an accurate data foundation for the deep fusion of multimodal features.

[0080] In step S120, feature extraction operations are performed on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality and to identify the component category to which the power equipment instance belongs.

[0081] Here, the aim is to transform heterogeneous raw data into high-dimensional feature representations that can be processed by computers and to determine the physical properties of devices. Specifically, the system runs multiple modality-specific feature encoding networks in parallel: convolutional neural networks are used to process visual and thermal imaging data to extract texture and shape features, recurrent neural networks or temporal convolutional networks are used to process sensor data to extract trend and frequency domain features, and natural language processing models are used to transform unstructured text into semantic vectors.

[0082] In particular, the system can identify the specific component category of the power equipment instance (e.g., zinc oxide surge arrester or composite insulator) based on visual features in visible light images using target detection or image classification algorithms. This identification result then acts as a routing controller for subsequent processing, determining which physical prior knowledge the system should use to guide subsequent semantic mapping, thereby avoiding accuracy loss when using a general model to process devices with different physical mechanisms.

[0083] In step S130, based on the component category, a pre-built semantic anisotropic mapping model is invoked to perform projection processing on the original feature vectors of each modality.

[0084] Here, the semantic anisotropic mapping model is configured with a metric matrix defined for different component categories. This matrix is ​​used to perform non-equidistant scaling and rotation transformations on the original feature vectors according to the fault-sensitive direction of the component category, so as to obtain anisotropic semantic embedding vectors in a unified semantic space.

[0085] Specifically, after determining the component category, the system enters the core semantic mapping stage. Based on the identified component category, the system calls the corresponding semantic anisotropy mapping model from a pre-set parameter library. This model is configured with a metric matrix specifically defined for the physical characteristics of that type of component (such as the longitudinal potential distribution of a surge arrester and the creepage distance of an insulator's skirt).

[0086] Using this metric matrix, the system performs a non-uniform linear transformation (including spatial rotation and axial scaling) on ​​the original feature vectors of each modality, projecting the originally chaotic features into a direction-sensitive unified semantic subspace. Within this subspace, feature dimensions strongly correlated with the fault mechanism (such as gradients along the crack propagation direction) are stretched and amplified, while irrelevant dimensions related to the environmental background are compressed and suppressed. This processing mechanism endows the feature vectors with physical semantics, making the data from different modalities no longer merely a numerical set, but comparable in the direction of physical faults. This effectively solves the technical challenge of traditional isotropic feature spaces being unable to distinguish between weak faults and background noise.

[0087] In step S140, based on the anisotropic semantic embedding vectors of each modality, the anisotropic distance between modalities is calculated, and the anisotropic distance is introduced as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating a multimodal fusion representation vector of the power equipment instance.

[0088] In this embodiment, physical constraint-based cross-modal feature fusion is performed. The system calculates the anisotropic distance between the mapped modal embedding vectors in a unified subspace, which essentially quantifies the semantic differences between different modal data in the fault-sensitive direction. Subsequently, this distance is introduced as a penalty term into the calculation of the cross-modal attention mechanism (e.g., subtracted before calculating the Softmax normalized weights). This means that even if the features of two modalities appear statistically similar (e.g., similar colors), if they are far apart in the fault-sensitive direction (i.e., physically and semantically mismatched), their attention weights will be forcibly reduced.

[0089] Through this dynamic correction mechanism, the system can suppress spurious cross-modal correlations, focus on feature combinations that are truly consistent in terms of physical laws, and finally generate a high-confidence multimodal fusion representation vector by weighted summation, which significantly improves the robustness of fusion features to complex working conditions.

[0090] In step S150, the multimodal fusion representation vector is input into a preset task decision model to output the equipment status assessment result for the power equipment instance.

[0091] In this embodiment, the system inputs the fused multimodal representation vector into a decision model trained for a specific task. This decision model may include a multilayer perceptron (MLP) or a regression network to decode the high-dimensional fused features into user-understandable diagnostic results. Depending on the operational needs, the model can output discrete fault categories for power equipment (such as "normal," "general overheating," or "serious defect"), or continuous state assessment indicators (such as health index and remaining life prediction). Thanks to the anisotropic mapping and physical constraint fusion of the input features, the assessment results can keenly capture early, subtle fault characteristics while significantly reducing the false alarm rate caused by environmental interference, providing accurate and intuitive data support for power grid maintenance personnel to formulate maintenance strategies.

[0092] To facilitate a more intuitive understanding of the inventive spirit of the embodiments of this invention by the public, the following exemplary description is provided using a specific power equipment fault diagnosis scenario: a zinc oxide surge arrester.

[0093] Suppose that the system collected a set of multimodal sample data for a certain zinc oxide surge arrester in step S110. In the visible light image, due to direct sunlight at noon, a bright reflective area (strong noise) appears on the surface of the porcelain bushing of the surge arrester; while in the infrared thermal image, the inside of the surge arrester shows a weak temperature rise band distributed along the axial direction due to moisture aging of the valve plate (true fault feature). If the traditional isotropic alignment method is used, the model is easily misled by the bright reflective area in the visible light image, forcibly aligning it with the high-temperature area in the infrared image, because both show "high values" in pixel intensity, thus leading to incorrect feature fusion and false alarms.

[0094] However, applying the method of this invention, after identifying the device as a "surge arrester" in step S120, step S130 automatically invokes the metric matrix configured for the physical characteristics of the surge arrester. This matrix knows that the core fault feature of the surge arrester is the temperature gradient along the axial direction. Therefore, when performing semantic anisotropic mapping, the model explicitly stretches the gradient feature dimension distributed along the axial direction in a unified semantic space, while compressing or rotating the light spot feature dimension that is isotropically distributed. After this non-equidistant transformation, the weak temperature rise band in the infrared image becomes a highly significant principal feature in the semantic space, while the bright reflection in the visible light is reduced in dimension and suppressed into background noise, and the directions of the feature vectors of the two are distinguished in the semantic space.

[0095] Furthermore, when calculating the intermodal distance in step S140, due to the significant difference between reflection (isotropic) and moisture (anisotropic) in the fault-sensitive direction of the semantic space, the anisotropic distance calculated by the system is... The distance becomes very large, and this large distance is introduced as a penalty term into the attention mechanism, forcibly suppressing the weight of visible light reflective features. Ultimately, the fused representation vector generated in step S150 almost completely eliminates illumination interference, highly focusing on the axial temperature rise characteristics of the infrared thermogram and the leakage current trend in the sensor data. Therefore, the task decision model can accurately output the assessment result of internal moisture in the surge arrester based on this pure fused feature, without being misled by illumination interference leading to abnormal appearances.

[0096] Regarding the implementation details of obtaining the multimodal sample data to be processed from the power equipment in step S110, in some examples of the embodiments of the present invention, a spatiotemporal consistency preprocessing operation can be adopted.

[0097] More specifically, firstly, multi-rate timing alignment is performed. The absolute timestamps of each modal data are obtained based on a unified timing reference. The modal data with the lowest sampling frequency is used as the anchor point. Visible light images, infrared images, sensor timing data, and maintenance text data that overlap with its timing are extracted using a sliding time window to construct a multi-modal synchronization data frame.

[0098] In the steps of this embodiment of the invention, considering the significant differences in sampling frequencies among the various acquisition terminals in the power Internet of Things system (for example, ultrasonic sensors may reach up to 100kHz, while infrared inspection cameras are typically only 1-30Hz), direct correlation would lead to data misalignment. Therefore, the system first assigns an absolute timestamp with nanosecond-level precision to all access data based on a unified timing reference (such as BeiDou / GPS satellite timing or the IEEE 1588 PTP protocol). Subsequently, a low-frequency anchoring strategy is adopted to timestamp the modal data with the lowest sampling frequency (usually visible light image frames or infrared image frames). As an anchor point, the system constructs a dynamically sliding time window. ,in This is a tolerance threshold (e.g., 50ms). All sensor time-series data segments and maintenance text logs falling within this window are considered state descriptions generated at the same physical instant as the image frame. This alignment mechanism effectively solves the problem of asynchronous heterogeneous data in the time dimension, ensuring the correctness of causal logic during subsequent feature fusion.

[0099] Then, a unified spatial index benchmark is established, and the infrared image is registered to the pixel coordinate system of the visible light image using a pre-calibrated extrinsic transformation matrix. Based on the installation coordinates of the physical sensor in three-dimensional space corresponding to the sensor time series data, the sensor data is projected and mapped to the pixel coordinate system, thereby giving the sensor data spatial location attributes.

[0100] In the steps of this embodiment of the invention, strict spatial registration is performed to establish a spatial correspondence between abstract sensor readings and intuitive image pixels. For infrared and visible light images, the system uses a pre-calibrated homography matrix to perform perspective transformation, eliminating image distortion caused by differences in viewing angle.

[0101] For example, for sensor data, the system utilizes photogrammetry principles to convert the installation coordinates of the physical sensors in the three-dimensional space of the substation into a homogeneous form of two-dimensional pixel coordinates in a visible light image. The specific mapping process can be expressed using the following formula for a pinhole camera projection model:

[0102] ;

[0103] In the formula, This represents the sensor's three-dimensional physical coordinates in the world coordinate system; and These are the rotation matrix and translation vector (extrinsic parameters) that transform the 3D physical coordinates in the world coordinate system to the camera coordinate system. This is the intrinsic parameter matrix of the camera; These are the non-homogeneous pixel coordinates on the image plane after mapping; The scaling factor is defined using the formula above. Each sensor's data stream is assigned a specific pixel anchor point, which not only achieves spatial normalization of multi-source data but also provides the necessary geometric orientation reference for subsequent steps in calculating anisotropic distances (e.g., determining the physical orientation of the sensor relative to the device center). Here, [u,v,1] on the left-hand side of the formula... T Let [u, v] be the homogeneous form of the pixel coordinates. T Let u and v represent the corresponding non-homogeneous pixel coordinates, where u and v represent the x and y coordinates of the pixel, respectively, and the constant 1 represents the homogeneous coordinate components.

[0104] Then, text entity disambiguation association is performed, and a pre-built power domain named entity recognition network is used to extract equipment entity descriptions from the maintenance text data. The equipment entity descriptions are then mapped to unique equipment instance identifiers through the equipment topology tree index to ensure that the maintenance text data matches the power equipment instances in the visible light image.

[0105] It should be noted that, addressing the common issue of ambiguous descriptions in power maintenance texts (e.g., "main transformer phase A" may refer to multiple devices at different voltage levels), this embodiment performs entity disambiguation based on a knowledge graph. Specifically, the system loads a Named Entity Recognition (NER) network fine-tuned for the power sector to extract key entity descriptions such as "equipment name," "voltage level," and "phase" from unstructured operation and maintenance logs. Subsequently, the system invokes a pre-built power grid equipment topology tree, which stores the hierarchical relationships and unique asset IDs of all equipment within the substation. The system performs path matching between the extracted entity descriptions and topology tree nodes, mapping the text descriptions to unique device instance IDs. Only when the device ID in the text completely matches the device ID in the image metadata is the text included in the multimodal sample data set to be processed, thereby effectively filtering irrelevant text noise and ensuring the consistency of semantically understood objects.

[0106] Through the spatiotemporal consistency preprocessing of this invention, a high-purity, high-precision multimodal synchronous data cube is constructed before feature extraction by employing multi-rate anchoring sequence alignment, heterogeneous spatial mapping based on projection matrices, and text entity disambiguation based on topology trees. This eliminates temporal delays, spatial misalignments, and semantic ambiguities in heterogeneous data sources, providing strictly aligned physical coordinates and semantic objects for subsequent semantic anisotropy mapping. This ensures that the directional features learned by the subsequent model truly reflect the physical state of the device, rather than being spurious correlations stemming from data acquisition errors, thereby significantly improving the robustness of cross-modal alignment and the reliability of diagnostic results.

[0107] Regarding the implementation details of extracting the original feature vector and identifying the component category in step S120, in some examples of embodiments of the present invention, for visible light images, a visual coding network containing a key component detection branch is constructed to output the component category label and location bounding box of the power equipment instance, and the component category label is used as the result of identifying the component category to which the power equipment instance belongs; at the same time, a region of interest alignment operation is performed based on the location bounding box to extract local visual features that retain the geometric structure information of the component.

[0108] In some implementations, a parallel detection and feature extraction strategy can be adopted for visible light images. Specifically, a visual encoding network with a shared backbone (such as ResNet-50) is constructed, with its backend connected to a Region Proposal Network (RPN) and a detection head. First, the key component detection branch is used to locate power equipment instances, outputting category labels containing equipment components (such as insulator strings and equipotential rings). And the precise location bounding box, whose category label directly serves as the index for subsequent calls to specific metric matrices.

[0109] It should be noted that, to address the feature misalignment issue caused by positional quantization in traditional convolution operations, this embodiment performs Region of Interest Alignment (RoI Align) based on the positional bounding box. Unlike the coarse quantization of RoI Pooling, RoI Align calculates feature values ​​on continuous coordinates through bilinear interpolation, thereby generating a fixed-size (e.g., The feature map of the component. This allows for the preservation of the component's geometric structure information (such as the fine edges of cracks and the distribution of dirt textures), generating local visual features. This not only removes background interference but also provides a high-fidelity structural benchmark for subsequent calculation of geometric similarity in a unified space.

[0110] Then, for the infrared image, the deep thermal image features are extracted using a visual Transformer, and the pixel-level heat flow vector field is calculated using a temperature gradient operator, thereby separating the temperature gradient direction features that characterize the direction of fault thermal diffusion.

[0111] In this embodiment, for infrared images, in order to capture the unique thermal diffusion physical characteristics of power equipment faults, a dual-stream extraction mechanism is adopted. On the one hand, a visual Transformer (ViT) is used to capture long-range dependencies and extract deep global thermal image features; on the other hand, a temperature gradient operator is introduced to calculate the pixel-level heat flow vector field to explicitly separate the features characterizing the direction of fault thermal diffusion.

[0112] Specifically, for infrared images The temperature gradient components in the horizontal and vertical directions can be calculated using the Sobel operator or first-order difference. and This leads to the construction of a heat flow vector field. :

[0113] ;

[0114] In the formula, the modulus It characterizes the severity of the temperature rise, the amplitude. This temperature gradient direction characterizes the main direction of heat diffusion. This allows the model to transform simple temperature values ​​into physically directional vector features, enabling it to distinguish the essential differences between point-like overheating (isotropic diffusion) and band-like overheating (anisotropic diffusion). Here, x and y represent the horizontal and vertical coordinates of a pixel in the infrared image, respectively. This represents the infrared image temperature value at pixel (x,y); and These represent the temperature gradient components of the pixel in the horizontal and vertical directions, respectively. This represents the heat flow vector at this pixel, consisting of the magnitude and angle of the temperature gradient; the superscript T indicates the transpose operation. This represents the temperature gradient direction feature formed by the heat flow vectors of each pixel.

[0115] In addition, for sensor time-series data, a time-domain trend feature is extracted using a time-series convolutional network, and a multi-scale wavelet packet decomposition technique is introduced to project the time-series signal onto different frequency band subspaces in order to extract multi-band spectral features that reflect the directional differences in the frequency domain energy distribution of the fault signal.

[0116] Here, for sensor time-series data (such as partial discharge UHF signals or transformer vibration signals), joint time-frequency domain anisotropic feature decomposition aims to extract "frequency domain directionality" that reflects the fault mechanism. First, a temporal convolutional network (TCN) is used to extract and capture long-term dependent temporal trend features. Then, multi-scale wavelet packet decomposition (WPD) is introduced to project the one-dimensional time-series signal onto different frequency band subspaces.

[0117] Let the original signal from the sensor be... The first level is obtained through multi-level decomposition. Layer coefficients of each node And calculate the energy value of the node in that frequency band. As a feature component:

[0118] ;

[0119] Thus, the constructed multi-band spectral characteristics Essentially, it describes the distribution of signal energy along the frequency domain axis. Different fault types (such as low-frequency vibrations caused by insulation aging and high-frequency pulses caused by partial discharge) exhibit significant directional differences in their frequency domain energy distribution, providing a physical orthogonal feature for distinguishing different fault sources in the semantic space. Here, S(t) represents the original sensor signal at time t, where t represents time; j represents the wavelet packet decomposition level, and n represents the node index of the j-th level. Let represent the coefficient of the k-th wavelet packet of the n-th node in the j-th layer, where k represents the coefficient index within that node; |·| represents the modulo operation. This represents the summation of all coefficients within that node; This represents the energy value of the nth frequency band node in the j-th layer; This represents a multi-band spectral feature vector composed of the energy values ​​of each frequency band node in the j-th layer. This represents the total number of frequency band nodes in the j-th layer.

[0120] In addition, for maintenance text data, a pre-trained BERT model fine-tuned with knowledge of the power field is used to perform structured semantic encoding, extracting entity embedding vectors representing equipment identity and description embedding vectors representing fault states, respectively.

[0121] It should be noted that for unstructured maintenance text, directly using a general language model often fails to distinguish between the equipment entity and the fault phenomenon. Therefore, structured semantic encoding based on domain knowledge can be employed to utilize maintenance text data. This embodiment utilizes a pre-trained BERT model fine-tuned on an electrical corpus to perform structured semantic encoding. By designing a hierarchical head at the model output, the text features are decoupled into two parts: one is the entity embedding vector representing the equipment identity. (e.g., "phase A of the high-voltage side bushing of the main transformer"), used to assist in cross-modal entity alignment; secondly, as a descriptive embedding vector characterizing the fault state. (e.g., "oil level too low" or "discharge sound"), used to provide prior information about fault semantics. This decoupling ensures that the text modality can both participate in object consistency verification and provide an independent semantic dimension for state assessment.

[0122] Then, the local visual features, temperature gradient direction features, multi-band spectral features, and entity and description embedding vectors extracted from each modality are vectorized and concatenated or dimensionally aligned to generate the original feature vectors corresponding to each modality.

[0123] Specifically, to input the aforementioned heterogeneous features into a unified mapping model, the vectorization of the multimodal features is aligned to generate the original features. The system then extracts the specialized features (local visual features) from each modality. Temperature gradient characteristics Multi-band spectral characteristics and text embedding Projected to the same dimension through a fully connected layer (FC) or a multilayer perceptron (MLP). Based on the modality type label, the original feature vectors corresponding to each modality are generated. :

[0124] ;

[0125] Here, instead of performing deep cross-modal fusion, the most original physical and semantic properties of each modality are preserved, providing a high-purity, high-information input basis for subsequent refined anisotropic transformation using metric matrices. In the formula, m represents the modality index; This represents the original feature vector of the m-th mode; This represents the multilayer perceptron corresponding to the m-th modality; Concat(·) represents the vector concatenation operation; The specific feature representing the m-th mode; D represents the global feature extracted by the encoding network for the m-th modality; D represents the original feature vector. Dimensions.

[0126] Through the embodiments of this invention, a physical mechanism-driven heterogeneous feature extraction framework is constructed, significantly improving the representation depth of multimodal sample data. RoI Align is used to solve the spatial alignment accuracy problem of visual features; the temperature gradient operator is used to elevate infrared thermal images from a scalar temperature field to a vector heat flow field, endowing thermal features with physical directionality; wavelet packet decomposition is used to extract the energy anisotropy distribution of sensor signals in the frequency domain. Personalized extraction strategies for different modalities are adopted to ensure that the generated original feature vectors not only contain statistical information but also imply physical prior knowledge such as the unique geometric structure, thermodynamic diffusion, and frequency domain energy distribution of power equipment. This effectively overcomes the technical shortcomings of traditional general feature extraction methods in complex power operating conditions, such as semantic ambiguity and unclear physical directionality, laying a solid data foundation for subsequent high-precision semantic anisotropy mapping.

[0127] Regarding the implementation details of determining the anisotropic semantic embedding vector through the semantic anisotropic mapping model in step S130, in some examples of embodiments of the present invention, for each component category... Construct a learnable positive definite metric matrix And perform matrix decomposition on it to obtain the transformation matrix. This allows for the establishment of a geometric transformation basis that satisfies geometric constraints.

[0128] In this embodiment, considering that different components of power equipment (such as surge arresters and insulators) have drastically different fault physical characteristics, a uniform mapping parameter cannot be used. Therefore, the system first targets each component category... (For example, category ID=1 represents zinc oxide surge arresters), independently construct a learnable positive definite metric matrix. To ensure the positive definiteness of the metric matrix throughout model training, thereby guaranteeing the non-negativity and geometric validity of subsequent distance calculations, direct optimization can be avoided. Instead, it performs Cholesky decomposition or a similar matrix decomposition operation on it, parameterizing it into a lower triangular matrix. The product of its transpose and its inverse.

[0129] Specifically, this constraint can be expressed by the following formula:

[0130] ;

[0131] In the formula, This represents the transpose operation of a matrix or vector. Indicates the category of components A defined positive definite metric matrix is ​​used to characterize the semantic distribution of this type of component in the feature space; This indicates that the transformation matrix obtained through decomposition (usually a lower triangular matrix) constitutes the geometric basis of the feature transformation. Using the decomposition strategy described above, the network only needs to update [the following information] during backpropagation. The parameters can automatically guarantee the reconstructed result. It always satisfies the geometric constraints of the metric space, thus avoiding numerical instability.

[0132] Furthermore, using a transformation matrix-based approach... The linear projection operator will... The original feature vectors of each modality Mapped to anisotropic semantic embedding vectors .

[0133] Here, the aim is to use linear projection mapping based on centralized correction. After establishing the transformation basis, the system utilizes a transformation matrix-based approach. The system employs a centralized linear projection operator to map the original feature vectors of each modality to a unified semantic subspace. To eliminate the inherent numerical biases between different modalities (such as the RGB feature distribution of visible light images versus the temperature feature distribution of infrared images), the system introduces feature clustering centers. Perform centralization correction.

[0134] The specific mapping calculation logic is shown in the following formula:

[0135] ;

[0136] in, Indicates the first Anisotropic semantic embedding vectors after modality mapping; Indicates the first The original feature vectors of each modality; For component categories In the The feature cluster centers under each modality (which can be obtained statistically from the training set or used as learnable parameters) are used to debias the original features to eliminate distribution bias between modalities.

[0137] The row vector direction represents the fault-sensitive direction in the feature space, and its norm represents the semantic importance weight along that direction. Through this mapping, the small differences in the fault-sensitive directions of the original feature vectors are amplified in the unified semantic space, thereby making the anisotropic semantic embedding vectors... The Euclidean distance is equivalent to the distance in the original space based on the metric matrix. Mahalanobis distance.

[0138] In the steps of this embodiment of the invention, a coordinate system translation was performed on the original feature space (by...). (Control) and rotation scaling (by) (Control), which enables the alignment of multimodal features in a unified space.

[0139] The above mapping process is not a simple mathematical transformation, but rather has profound physical semantics. Transformation matrix The direction of each row vector physically represents a fault-sensitive direction in the feature space (e.g., the direction of a sharp change in temperature gradient in an infrared image, or the direction of change of a specific harmonic component in the sensor spectrum); while the norm (length) of the row vector represents the semantic importance weight along that direction.

[0140] Using the mapping described above, the minute differences in the original feature vectors in these fault-sensitive directions will be... The weights are amplified, while differences in the direction of environmental noise are suppressed. By employing non-equidistant scaling and rotation transformations, the mapped anisotropic semantic embedding vectors are made more efficient. The simple Euclidean distance in the unified semantic space is equivalent to the distance based on the metric matrix in the original feature space. The Mahalanobis distance. This means that subsequent steps only need to calculate the simple Euclidean distance to capture the deep semantic differences that conform to the physical fault patterns of power equipment, significantly improving the model's ability to identify weak fault features.

[0141] This invention constructs a physics-semantic-driven metric space mapping mechanism. Through decomposition strategies and projection calculations, the abstract concept of anisotropy is concretized into learnable matrix operations. This breaks the limitation of the isotropic nature of traditional feature mapping, enabling adaptive stretching of the fault feature dimension and compression of the noise dimension based on the specific physical mechanisms of different power components (such as surge arresters and insulators). It not only achieves geometric alignment of heterogeneous modal data but also realizes semantic isometric expression at the fault sensitivity level, allowing the model to ignore background interference and focus on the key physical features that determine the health status of equipment.

[0142] Regarding the implementation details of generating multimodal fusion representation vectors through a cross-modal attention mechanism in step S140, in some examples of embodiments of the present invention, in order to quantify the semantic differences of different modal features in the fault-sensitive direction of power equipment, the system first constructs a fully connected cross-modal distance matrix based on the embedded vectors in the output unified semantic space. , among which, the Anisotropic semantic embedding vectors of each modality With the Anisotropic semantic embedding vectors of each modality Elements between Defined as the square of the Euclidean distance between the two in a unified semantic space, it is used to quantify the semantic differences of different modal features in the fault-sensitive direction.

[0143] ;

[0144] In the formula, Indicates the first The modality and the first Anisotropic distance between modes; and The first The and the first Anisotropic semantic embedding vectors for each modality; This represents the square operation of Euclidean distance.

[0145] Specifically, due to The vector has already passed the previous steps. The transformation, and the Euclidean distance calculated at this time. In mathematical terms, this is equivalent to the Mahalanobis distance in the original feature space. This means that if there is inconsistency between the data of two modes in a fault-sensitive direction (such as the potential gradient direction of an aging surge arrester valve), this distance... This will be significantly amplified, thus providing a hard metric for subsequent attention mechanisms to measure whether physical semantics match. Therefore, each element in the matrix... It is no longer a simple numerical difference, but represents the first... The modality and the first The semantic distance of a modality after correction for physical mechanisms (such as the direction of heat diffusion and the direction of crack propagation).

[0146] Then, a linear projection is performed on the anisotropic semantic embedding vectors of each modality to generate the query matrix required by the attention mechanism. Key matrix AND-value matrix Furthermore, an anisotropic attention model incorporating a distance penalty term was constructed.

[0147] Specifically, after obtaining the distance matrix, the system follows the standard paradigm of the self-attention mechanism, performing a linear projection operation on the anisotropic semantic embedding vectors of each modality. Specifically, the embedding vectors are mapped to a query matrix using three independent learnable weight matrices. (Query), Key Matrix (Key) and value matrix (Value), thereby transforming physical semantic features into a subspace representation suitable for correlation calculation.

[0148] In this embodiment, anisotropic attention calculation of the fusion distance penalty term is performed to execute cross-modal fusion. Traditional attention mechanisms only... Calculating dot product similarity can be easily misled by spurious correlations in the environmental background (such as the statistical similarity between tree shadows in visible light images and low-temperature regions in infrared images). To address this, this embodiment constructs an anisotropic attention model that incorporates a distance penalty term, introducing physical distance as negative feedback into the calculation of statistical correlation.

[0149] For example, the computational logic of this model is shown in the following equation:

[0150] ;

[0151] in, It characterizes the statistical correlation between modes; This is a scaling factor for the feature dimension, used to prevent the gradient from vanishing due to excessively large dot product results; This refers to the cross-modal anisotropic distance matrix constructed above; These are learnable physical constraint coefficients used to adaptively adjust the strength of the distance penalty. (This is achieved through the distance penalty term.) This is used to forcibly suppress pseudo-matching modalities that have high statistical correlation but are semantically far apart in the fault-sensitive direction, thereby outputting a weighted result that conforms to physical laws as a multimodal fusion representation vector.

[0152] Referring to the above technical logic, It characterizes the statistical correlation between modes, while This characterizes the physical semantic differences between modalities. Through the subtraction operation, the system forcibly suppresses modalities with high statistical correlation (large dot product) but large semantic distances in the fault-sensitive direction (…). The pseudo-matched mode pairs (large) are normalized by Softmax. The final weighted output is the multi-modal fusion representation vector that conforms to the physical laws of power equipment.

[0153] This invention constructs a physical distance-constrained attention correction mechanism, fundamentally solving the problem that correlation does not equal causation in traditional multimodal fusion. By utilizing a penalty term design, the anisotropic semantic distances extracted in the preceding steps are transformed into gating signals in the attention mechanism. This automatically eliminates false high-confidence matches caused by environmental noise or background clutter, forcing the model to focus its attention weights on key modal features that are highly consistent in both statistical characteristics and physical mechanisms. This significantly improves the anti-interference capability of the fused features in complex electromagnetic environments.

[0154] As a further preferred embodiment of the present invention, the method further includes enhancing the sparsity of the cross-modal attention mechanism by introducing a hard threshold gating mechanism.

[0155] Specifically, in execution Before the operation, based on the distance matrix Constructing a semantic association mask matrix It sets an anisotropic distance threshold to represent the effective semantic association boundary. .

[0156] Here, to further enhance the model's robustness against extremely complex backgrounds (such as a substation background with numerous metal structures of similar color to the equipment), a physical semantic gating is introduced before performing Softmax normalization. Specifically, the system uses the cross-modal distance matrix generated in the preceding steps... To construct a semantic association mask matrix of the same dimension. In this process, a key hyperparameter was set—the anisotropic distance threshold. This represents the physical boundary by which different modal features are judged to be validly associated in the fault-sensitive direction (for example, it can be set as the 95th percentile of the distance distribution of positive sample pairs in the training set). Any feature interaction that exceeds this physical boundary, even if there is some similarity in statistical features (such as texture and brightness), is considered an invalid interaction that violates physical laws.

[0157] When the anisotropic distance between cross-modal sample pairs Exceeding the threshold When this happens, the feature pair is determined to be an invalid association, and the mask matrix is... The corresponding element is assigned a value of negative infinity, otherwise it is assigned a value of zero.

[0158] Here, the system iterates through each element of the cross-modal distance matrix. The validity of the feature pair is determined using hard thresholding logic. This applies when the anisotropic distance between cross-modal sample pairs... Exceeding the preset threshold This means that although these two modes may both belong to high-frequency signals or bright regions, they have an essential conflict in the directionality of the fault mechanism (such as one being the direction of cracks and the other being the texture of background noise).

[0159] To mathematically sever this correlation completely, the system uses a mask matrix. The corresponding element Assigning a value of negative infinity ( Its logical expression is shown in the following formula:

[0160] ;

[0161] In the formula, For the mask matrix, the first Line 1 The elements of the column. This assignment strategy utilizes the exponential property of the Softmax function (i.e., This ensures that the probability weight of the masked element is strictly zero after normalization, thus achieving true sparsity.

[0162] Furthermore, the mask matrix Added to the already included weighted penalty term (i.e., physical soft penalty term) In the attention score matrix, long-tailed distribution interactions with excessively large anisotropic distances are forcibly truncated, thereby eliminating false matching interference that are statistically similar but semantically significantly different in fault-sensitive physical directions.

[0163] Thus, a dual noise reduction mechanism combining soft and hard methods is achieved: the soft penalty term reduces the weights of features with large differences in physical semantics, while the hard mask term directly truncates those long-tailed distribution interactions with excessive differences.

[0164] The revised formula for calculating sparsified attention is as follows:

[0165] ;

[0166] By introducing Able to forcibly filter out those in The mid-range significant pseudo-matching mode has extremely high practical value in power scenarios. For example, this mechanism can effectively eliminate the erroneous association between metallic reflections (visible light) caused by direct sunlight and hot spots (infrared) caused by internal overheating, because although the two are similar in image intensity, they are far apart in terms of anisotropic features of temperature gradient fields and will be directly truncated by hard thresholding.

[0167] Through the embodiments of the present invention, an anisotropic distance-driven sparse attention mechanism is constructed by introducing a hard threshold. and negative infinity mask matrix It superimposes hard semantic gating on top of soft physical constraints, solving the long-tail noise interference problem commonly found in multimodal fusion. Specifically, it completely eliminates spurious cross-modal interactions that are statistically deceptive (highly correlated) but completely mutually exclusive (large distance) in terms of physical fault mechanisms. Through sparsification, it not only significantly reduces the computational redundancy of the model but also greatly improves the signal-to-noise ratio and robustness of the fault diagnosis model in the complex environment of high-noise and strong-interference substations.

[0168] In some examples of embodiments of the present invention, the semantic anisotropy mapping model is obtained by training by minimizing a multi-task joint loss function, which includes at least anisotropy consistency loss. This is used to explicitly constrain the consistency of projection of different modal features onto the fault-sensitive direction during model optimization.

[0169] Here, in the data preparation phase of model training, the system first constructs a set of cross-modal positive sample pairs for calculating the consistency loss. Unlike the strategy of randomly selecting positive and negative samples in general contrastive learning, this embodiment strictly bases pairing on the physical entity identity of the power equipment. Specifically, the system filters out instances of the same power equipment (e.g., the transformer with ID #T502) from a spatiotemporally preprocessed data pool, but at different modes (e.g., at different times). Infrared thermal image and time The characteristic representation of the ultrasonic spectrum. These pairs constitute a set. This ensures that the differences in subsequent calculations stem only from the different representations between modes, rather than the differences in the physical objects themselves, thus providing the model with a correct benchmark for learning mode invariance.

[0170] To impart physical intuition to the model, this embodiment introduces a fault sensitivity weight vector defined based on the physical characteristics of the components. This vector is not randomly initialized based on data, but rather obtained based on prior expert knowledge or learned under supervised conditions. Its core logic lies in: in the feature space... Of these dimensions, some correspond to key physical quantities (such as the direction of the temperature gradient and the fundamental frequency energy of the vibration signal), while others correspond to environmental noise (such as background light intensity).

[0171] therefore, The dimensions corresponding to the principal axis of fault features are assigned high weight values ​​(e.g., 1.0), while the dimensions corresponding to environmental noise are assigned low weight values ​​(e.g., 0.1). This is equivalent to embedding a physical filter into the loss function, guiding the model on which dimensions' alignment accuracy should be prioritized during optimization.

[0172] During the forward propagation of training, the system calculates the anisotropy consistency loss. The loss function aims to explicitly constrain the consistency of projection of different modal features onto the fault-sensitive direction after semantic mapping.

[0173] The specific calculation logic is shown in the following formula:

[0174] ;

[0175] in, It is a set of cross-modal positive sample pairs consisting of different modal data for the same power equipment instance. Represents a pair of cross-modal indices in the set; and These are the anisotropic semantic embedding vectors of the positive sample pair in the unified semantic space, respectively. It represents the Hadamardi (or Hadama) stack; For component categories The class balance coefficient is used to dynamically increase the contribution of scarce fault class samples to the total loss when the training data is imbalanced (e.g., there are far fewer fault samples than normal samples). This represents the square of the L2 norm, used to calculate the weighted Euclidean distance.

[0176] The fault sensitivity weight vector is defined based on the physical characteristics of the component. It is assigned a high weight value on the dimension corresponding to the principal axis of the fault feature and a low weight value on the dimension corresponding to the environmental noise. This allows the model to prioritize minimizing the cross-modal semantic differences of the key fault dimensions during training, thereby achieving feature alignment driven by physical mechanisms.

[0177] Specifically, the calculation yielded Then, the system weights and sums the total loss with other task losses (such as classification loss and regression loss) to obtain the total loss, and then executes the backpropagation algorithm. Because Hadama volume was introduced in China. When calculating gradients, feature dimensions with larger alignment errors in fault-sensitive directions will produce larger gradient values, thereby driving the semantic anisotropic mapping model (i.e., the aforementioned transformation matrix). The parameters are updated in a way that prioritizes reducing the differences in fault dimensions. In this way, through backpropagation and parameter updates driven by physical mechanisms, the model no longer pursues minimizing the global statistical average error during training, but instead prioritizes achieving cross-modal semantic alignment of key fault dimensions.

[0178] Through the embodiments of the present invention, a loss constraint paradigm driven by physical mechanisms is constructed, and a fault sensitivity weight vector is introduced. and category balance coefficient By combining the above formula, the physical fault mechanism of power equipment is successfully embedded into the optimization objective of deep learning, changing the traditional black-box fitting logic of the model and forcing the model to have the ability to identify anisotropy when aligning features. As a result, it can have zero tolerance for differences in fault-related physical features, while being tolerant of differences in noise caused by environmental background. This makes the trained model not only perform well in statistical indicators, but also achieve a qualitative leap in physical interpretability and sensitivity to minor faults.

[0179] Figure 2 A flowchart illustrating an example of an operation that outputs equipment status assessment results for an instance of power equipment based on a task decision model according to an embodiment of the present invention is provided.

[0180] like Figure 2 As shown, in step S210, the multimodal fusion representation vector is weighted and fused with the anisotropic semantic embedding vector of each modality through residual connection to generate enhanced device state features.

[0181] It should be noted that although the preceding cross-modal attention mechanism successfully captures the global correlation between modalities, subtle but crucial anisotropic features of certain single modalities (such as extremely small temperature gradients in infrared images) may be diluted due to smoothing effects during the hierarchical propagation of deep networks. To address this, this embodiment constructs a feature recalibration module, which utilizes a residual connection mechanism to perform weighted aggregation of the multimodal fusion representation vector output by the attention mechanism with the original anisotropic semantic embedding vectors of each modality.

[0182] For example, this feature enhancement process can be expressed by the following formula:

[0183] ;

[0184] In the formula, For the generated enhanced device state characteristics; The multimodal fusion representation vector output by the attention mechanism; For the first Anisotropic semantic embedding vectors for each modality; These are learnable channel weighting coefficients used to adaptively adjust the retention ratio of the original features; For layer normalization operation; This represents the total number of modalities participating in the weighted fusion. By employing complementary fusion of global context and local directional features, the system not only preserves cross-modal reasoning logic but also recovers the original physical details of each modality in the fault-sensitive direction, effectively solving the information loss problem in deep feature extraction.

[0185] In step S220, one or more of the following output heads are selected for processing according to specific task requirements: fault classification head and status regression head.

[0186] Specifically, regarding the details of the fault classification head, a fully connected layer is used to classify and map the device state features, and the mapping results are normalized and probability calculated to output the probability distribution of fault types of the power equipment. Regarding the details of the state regression head, a regression network is used to process the device state features and output the health index or remaining service life prediction value of the power equipment.

[0187] For example, a multi-task learning architecture is adopted, which uses different decoding heads to process the enhanced equipment status features in parallel according to the actual needs of power operation and maintenance.

[0188] The fault classification head aims to address qualitative diagnostic problems. The system utilizes fully connected layers (FC) to map high-dimensional features to the fault category space and calculates normalized probabilities using the Softmax function.

[0189] ;

[0190] In the formula, The output fault type probability distribution (e.g., [Normal: 0.05, General overheating: 0.1, Severe overheating: 0.85]) enables the system to not only determine whether the equipment is faulty, but also to provide the confidence level of the fault type; and These are the classification weight matrix and classification bias vector of the fault classification head, respectively;

[0191] The state regression head aims to solve quantitative assessment problems. The system utilizes a multilayer perceptron regression network to process device state characteristics and outputs predictions of continuous values.

[0192] ;

[0193] In the formula, It can be instantiated as a normalized health index (0~1) or a predicted remaining useful life (RUL) in hours / days. This branch provides maintenance personnel with a forward-looking quantitative reference by fitting the mapping relationship between features and equipment degradation curves. The output activation function of the state regression head; and These are the regression weight matrix and regression bias parameters of the state regression head, respectively.

[0194] In step S230, the equipment status assessment result used to characterize the current physical operating status of the power equipment instance is determined based on one or more of the fault type probability distribution, health index, and remaining service life prediction value.

[0195] Specifically, after obtaining the above classification and regression results, the system performs logical synthesis at the decision-making level. Unlike traditional single-indicator outputs, this embodiment constructs a multi-dimensional equipment status profile based on the fault type probability distribution, health index, and remaining service life prediction. For example, when the fault probability exceeds a threshold and the health index shows a rapid downward trend, the system will determine the current physical operating state as critical and automatically trigger a high-priority maintenance work order; conversely, if there are only slight probability fluctuations but the remaining service life prediction is stable, it is marked as a state of alert. By adopting a comprehensive evaluation mechanism, the algorithm output is transformed into an executable operation and maintenance strategy, achieving a comprehensive characterization of the current physical operating state of power equipment instances.

[0196] This invention constructs a residual-enhanced, multi-task collaborative diagnostic framework. By introducing residual connections, it achieves secondary reuse of semantically anisotropic features at the decision layer, ensuring that weak physical fault signals are not annihilated in deep networks. Simultaneously, through parallel classification and regression output heads, the model achieves a closed loop from qualitative judgment to quantitative prediction. Therefore, the final equipment status assessment result combines diagnostic accuracy (guaranteed by the classification head) with operational foresight (guaranteed by the regression head), significantly improving the refinement level of smart grid equipment lifecycle management.

[0197] Figure 3 A schematic diagram illustrating the principle mechanism of an example of a cross-modal sample data alignment method for power equipment based on semantic anisotropic mapping according to an embodiment of the present invention is shown. The schematic diagram fully illustrates the entire data flow and logical interaction from multi-source heterogeneous data input to final equipment status assessment.

[0198] like Figure 3As shown, firstly, the multimodal input layer acquires raw data covering visible light / infrared images, sensor time-series data, and maintenance log text. This data is then processed by a preprocessing and feature extraction module, which performs spatiotemporal alignment and multi-scale feature encoding to generate raw feature vectors for each modality. Subsequently, the core semantic anisotropic mapping and alignment module receives these features, projects them into physically orientation-sensitive semantic embedding vectors using a pre-built mapping model, and generates a unified fusion representation that effectively suppresses environmental noise through an anisotropic cross-modal attention mechanism that incorporates a distance penalty term.

[0199] also, Figure 3 The anisotropic consistency constraint module shown above reveals the training mechanism of the model. It drives the updating and optimization of the mapping model parameters by minimizing the projection differences of different modes in the fault-sensitive direction during backpropagation. Finally, the generated unified fusion representation is sent to the application output layer. Through the task interfaces such as fault diagnosis and cross-modal retrieval, it outputs high-precision state assessment results for power equipment instances, realizing cross-modal semantic depth alignment driven by physical mechanisms.

[0200] To verify the effectiveness of the proposed cross-modal alignment method based on semantic anisotropic mapping, a dedicated experimental dataset containing 2000 power equipment samples was constructed. Each sample in this dataset underwent rigorous spatiotemporal consistency verification and includes visible light images, infrared images, multi-channel sensor time-series data (covering five typical sensor curves), and maintenance text for the same equipment instance. The equipment types cover core components such as surge arresters, circuit breakers, disconnectors, insulators, and conductors, and evenly include normal operation and various typical fault states. The dataset was randomly divided into training, validation, and test sets in a 6:2:2 ratio.

[0201] The experiment set up two representative baseline models for comparison with the method of this invention: Baseline 1 (Cross-Attention Baseline) adopts a traditional cross-modal cross-attention mechanism based on global features, which only utilizes the statistical correlation between the overall image and text, without introducing anisotropic mapping; Baseline 2 (PR-CLIP Baseline) introduces a cross-modal position reconstruction strategy to enhance spatial alignment capability, but does not cover sensor time-series data and lacks semantic modeling for the physical mechanisms of power components. In contrast, the method of this invention fully implements the semantic anisotropic mapping model and the anisotropic cross-modal attention mechanism that introduces a distance penalty term, aiming to verify the advantages of physically semantically driven feature alignment in improving fault diagnosis accuracy.

[0202] Figure 4This diagram compares the F1 score distributions of different methods on five typical types of power equipment. The violin plot illustrates the performance probability density distribution based on 50 independent repeated experiments. The horizontal axis represents power equipment with five different physical structures (surge arresters, circuit breakers, disconnectors, insulators, and conductors), and the vertical axis represents the F1 evaluation score. Specifically, the dark blue violin plot represents the cross-attention baseline method using only global features, the blue-green violin plot represents the PR-CLIP-based location reconstruction baseline method, and the light green violin plot represents the semantic anisotropic mapping-based method proposed in this invention.

[0203] like Figure 4 As shown, significant differences in physical laws can be observed: for devices with complex geometries and specific physical fault directions (such as heat dissipation direction or creepage distance direction), such as surge arresters, insulators, and disconnectors, the median F1 score of the method of this invention (light green) is significantly higher than that of the two baseline methods, and its distribution is more compact (i.e., smaller variance). This proves that semantic anisotropic mapping effectively constrains the dispersion of features, making the model more robust and stable to viewpoint rotation and local occlusion. For relatively simple linear components such as conductors, the performance gap between the three methods shows a convergence trend, which is consistent with the theoretical expectation of this invention, that is, anisotropic mapping can bring higher marginal gains when dealing with devices with complex internal semantic structures, thus verifying the effectiveness of this method in addressing the semantic alignment pain point under complex power conditions.

[0204] Figure 5 The diagram illustrates the F1 fractional three-dimensional response surface plots of the method according to an embodiment of the present invention under different coupling effects of sensor noise intensity and anisotropy coefficient. This three-dimensional coordinate system visually demonstrates the coupling effect between the model's noise robustness and the anisotropy constraint strength, where the X-axis represents the sensor noise intensity (…). ), used to simulate signal interference under harsh working conditions; the Y-axis represents the anisotropy coefficient ( The Z-axis is used to characterize the strength of the semantic anisotropic mapping model's constraint on feature directionality; the Z-axis represents the F1 score of the model diagnosis.

[0205] like Figure 5 As shown in the curve trend, it can be seen that the overall model performance decreases with increasing sensor noise intensity across all configurations. However, when the anisotropy coefficient... Within a specific range of 0.5 to 0.7, the slope of performance degradation slows significantly, and the surface exhibits a distinct bulge in this region. This phenomenon verifies that the anisotropic mapping mechanism proposed in this invention acts like a semantic filter: by strengthening the feature weights on fault-sensitive principal axes (such as temperature gradients or spectral energy directions), it effectively suppresses isotropic random environmental noise, thereby maintaining high diagnostic reliability in low signal-to-noise ratio environments. Furthermore, when... The slight collapse of the surface when it approaches 1.0 indicates that excessively strong directional constraints may cause the model to overfit to a specific viewpoint. Therefore, reasonable adjustment of anisotropy intensity is crucial for balancing robustness and generalization ability.

[0206] Figure 6 A scatter plot comparing the trade-offs between inference latency and retrieval accuracy for different methods is presented. This plot visually illustrates the trade-off between computational efficiency and performance of different algorithm models. The horizontal axis represents single-sample inference latency (in milliseconds), used to measure the algorithm's time complexity; the vertical axis represents retrieval accuracy (mAP), used to measure the algorithm's accuracy; and the size of the bubbles in the plot represents the scale of the model's parameters. Specifically, it visually demonstrates a comparison of LinearBaseline, Cross-Attention, and PR-CLIP methods, showing that the proposed method has a significant advantage in overall retrieval accuracy.

[0207] like Figure 6 As shown, it illustrates the method of the present invention in different anisotropic mapping ranks. (i.e., transformation matrix) The performance trajectory under the number of columns or the dimension of the semantic subspace. When At this point, the model reached a peak mAP of 0.92. Although the inference time (approximately 45ms) increased compared to the linear baseline, a significant performance leap was achieved. Further comparison revealed that when the mapping rank... Increasing the number of model parameters from 4 to 8 significantly increases the number of parameters and inference latency (the bubble enlarges and shifts to the right), but the marginal gain in accuracy is extremely limited (only increasing to 0.925). This experimental result indicates that... It represents the optimal balance between computational cost and diagnostic accuracy under the current architecture, verifying the feasibility of deploying the method of this invention on edge inspection devices with limited computing power while maintaining high accuracy.

[0208] Experimental results first confirm the significant advantages of the proposed method in terms of stability and robustness. Compared to baseline methods, this scheme significantly narrows the confidence interval of performance indicators on devices with complex geometries such as surge arresters and insulators, physically verifying the effectiveness of semantic anisotropy mapping: it does not simply increase the feature dimension, but effectively resists interference caused by viewpoint changes and local occlusion by constraining the discrete direction of semantics in a unified space. Simultaneously, 3D response surface analysis shows that moderate anisotropy constraints construct an effective semantic filtering mechanism. By strengthening the weights of fault-sensitive principal axes (such as the temperature gradient direction), the model can effectively cancel omnidirectional environmental noise in low signal-to-noise ratio environments. Furthermore, evaluation of computational resources shows that model performance is related to the rank of the anisotropy mapping (i.e., the dimension of the transformation matrix). There exists an optimal balance point in engineering (e.g.) This indicates that the proposed method can achieve an optimal balance between accuracy and speed under the limited computing power of edge computing devices.

[0209] In summary, to address the common problem of semantic directionality loss in multimodal sample data fusion of power equipment, this invention proposes a cross-modal alignment method based on semantic anisotropic mapping. The substantial technical contributions of this invention are as follows: First, a semantic anisotropic mapping model is constructed, utilizing learnable metric matrix decomposition and linear projection operators for component categories to map heterogeneous modal features to a direction-sensitive unified semantic subspace, overcoming the shortcomings of traditional global mapping that ignores component geometry and physical mechanisms. Second, an anisotropic cross-modal attention mechanism with a fusion distance penalty term is designed, and combined with a hard threshold gating strategy, dynamic correction of cross-modal interaction weights and long-tail noise truncation are achieved, thus balancing statistical correlation and physical semantic consistency during feature alignment. Finally, a multi-task collaborative training strategy including anisotropic consistency loss is established. By explicitly constraining projection consistency in fault-sensitive directions, the dynamic balance between feature alignment and state assessment tasks is ensured, providing a physically interpretable solution for multi-source data fusion in smart grids.

[0210] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0211] Figure 7 A structural block diagram of an example of a cross-modal sample data alignment system for power equipment based on semantic anisotropic mapping according to an embodiment of the present invention is shown.

[0212] like Figure 7 As shown, the power equipment cross-modal sample data alignment system 700 based on semantic anisotropic mapping includes a data acquisition unit 710, a feature and category recognition unit 720, a semantic anisotropic mapping unit 730, an anisotropic fusion unit 740, and a state evaluation unit 750.

[0213] The data acquisition unit 710 is used to acquire multimodal sample data to be processed from the power equipment; the multimodal sample data to be processed includes at least visible light images, infrared images, sensor time-series data and maintenance text data for the same power equipment instance.

[0214] The feature and category recognition unit 720 is used to perform feature extraction operations on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality and to identify the component category to which the power equipment instance belongs.

[0215] The semantic anisotropic mapping unit 730 is used to call a pre-built semantic anisotropic mapping model to perform projection processing on the original feature vectors of each modality based on the component category. The semantic anisotropic mapping model is configured with a metric matrix defined for different component categories, which is used to perform non-equidistant scaling and rotation transformations on the original feature vectors according to the fault-sensitive direction of the component category, so as to obtain anisotropic semantic embedding vectors in a unified semantic space.

[0216] The anisotropic fusion unit 740 is used to calculate the anisotropic distance between modalities based on the anisotropic semantic embedding vectors of each modality, and introduce the anisotropic distance as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating a multimodal fusion representation vector of the power equipment instance.

[0217] The condition assessment unit 750 is used to input the multimodal fusion representation vector into a preset task decision model to output the equipment condition assessment result for the power equipment instance.

[0218] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions. The execution instructions can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform the steps of any of the above-described methods for aligning cross-modal sample data of power equipment based on semantic anisotropy mapping.

[0219] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of any of the above-described methods for aligning cross-modal sample data of power equipment based on semantic anisotropy mapping.

[0220] In some embodiments, the present invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a method for aligning cross-modal sample data of power equipment based on semantic anisotropy mapping.

[0221] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0222] The electronic devices of the present invention exist in various forms, including but not limited to: mobile communication devices, ultra-mobile personal computer devices, portable entertainment devices, or other airborne electronic devices with data interaction functions.

[0223] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0225] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping, characterized in that, The method includes: Acquire multimodal sample data to be processed from power equipment; the multimodal sample data to be processed includes at least visible light images, infrared images, sensor time-series data, and maintenance text data for the same power equipment instance; Feature extraction operations are performed on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality, and the component category to which the power equipment instance belongs is identified; Based on the component category, a pre-built semantic anisotropic mapping model is invoked to perform projection processing on the original feature vectors of each modality; the semantic anisotropic mapping model is configured with a metric matrix defined for different component categories, which is used to perform non-equidistant scaling and rotation transformations on the original feature vectors according to the fault-sensitive direction of the component category, so as to obtain anisotropic semantic embedding vectors in a unified semantic space. Based on the anisotropic semantic embedding vectors of each modality, the anisotropic distance between modalities is calculated, and the anisotropic distance is introduced as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance. The multimodal fusion representation vector is input into a preset task decision model to output the equipment status assessment result for the power equipment instance; The semantic anisotropic mapping model is configured with metric matrices defined for different component categories. These matrices are used to perform non-equidistant scaling and rotation transformations on the original feature vectors based on the fault-sensitive direction of the component category, to obtain anisotropic semantic embedding vectors in a unified semantic space, including: For each component category Construct a learnable positive definite metric matrix And perform matrix decomposition on it to obtain the transformation matrix. Thus, a geometric transformation basis satisfying the following constraints is established: ； In the formula, Represents the transpose operation of a matrix or vector; Using the transformation matrix The linear projection operator will... The original feature vectors of each modality Mapped to the anisotropic semantic embedding vector : ； in, For component categories In the Featured cluster centers under each modality are used to eliminate distribution bias between modalities; The row vector direction represents the fault-sensitive direction in the feature space, and its norm represents the semantic importance weight along that direction. Through this mapping, the small differences in the fault-sensitive direction of the original feature vector are amplified in the unified semantic space, thereby making the anisotropic semantic embedding vector... The Euclidean distance is equivalent to the distance in the original space based on the metric matrix. Mahalanobis distance; The anisotropic semantic embedding vectors based on each modality are used to calculate the anisotropic distance between modalities. This anisotropic distance is then used as a weight penalty term in the cross-modal attention mechanism to dynamically adjust the attention weights for cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance, including: Constructing the cross-modal distance matrix , of which Anisotropic semantic embedding vectors of each modality With the Anisotropic semantic embedding vectors of each modality Elements between Defined as the squared Euclidean distance between the two in the unified semantic space, it is used to quantify the semantic differences of different modal features in the fault-sensitive direction: ； Perform linear projection on the anisotropic semantic embedding vectors of each modality to generate the query matrix required by the attention mechanism. Key matrix AND-value matrix And an anisotropic attention model incorporating a distance penalty term is constructed: ； in, It characterizes the statistical correlation between modes; , which is the scaling factor for the feature dimension; Learnable physical constraint coefficients; distance penalty term This is used to forcibly suppress pseudo-matching modalities that have high statistical correlation but are semantically far apart in the fault-sensitive direction, thereby outputting a weighted result that conforms to physical laws as the multimodal fusion representation vector.

2. The method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping according to claim 1, characterized in that, The acquisition of the multimodal sample data to be processed from the power equipment includes performing the following spatiotemporal consistency preprocessing steps: Perform multi-rate timing alignment, obtain the absolute timestamp of each modal data based on a unified timing reference, and use the modal data with the lowest sampling frequency as the anchor point. Use a sliding time window to extract visible light images, infrared images, sensor timing data and maintenance text data that overlap with its timing to construct a multimodal synchronization data frame. A unified spatial index benchmark is established, and the infrared image is registered to the pixel coordinate system of the visible light image using a pre-calibrated extrinsic transformation matrix. Based on the installation coordinates of the physical sensor in three-dimensional space corresponding to the sensor time series data, the sensor data is projected and mapped to the pixel coordinate system, thereby giving the sensor data a spatial location attribute. The text entity disambiguation association is performed, and the device entity descriptions in the maintenance text data are extracted using a pre-built power domain named entity recognition network. The device entity descriptions are then mapped to unique device instance identifiers using a device topology tree index to ensure that the maintenance text data matches the power equipment instances in the visible light image.

3. The method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping according to claim 1, characterized in that, The step of performing feature extraction operations on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality and identifying the component category to which the power equipment instance belongs includes: For the visible light image, a visual coding network including a key component detection branch is constructed to output the component category label and location bounding box of the power equipment instance, and the component category label is used as the result of identifying the component category to which the power equipment instance belongs; at the same time, a region of interest alignment operation is performed based on the location bounding box to extract local visual features that retain the geometric structure information of the components. For the infrared image, deep thermal image features are extracted using a visual Transformer, and pixel-level heat flow vector fields are calculated using a temperature gradient operator, thereby separating the temperature gradient direction features that characterize the direction of fault heat diffusion. For the sensor time-series data, a time-domain trend feature is extracted using a time-series convolutional network, and a multi-scale wavelet packet decomposition technique is introduced to project the time-series signal onto different frequency band subspaces in order to extract multi-band spectral features that reflect the directional differences in the frequency domain energy distribution of the fault signal. For the maintenance text data, a pre-trained BERT model fine-tuned with knowledge of the power field is used to perform structured semantic encoding to extract entity embedding vectors representing equipment identity and description embedding vectors representing fault states, respectively. The local visual features, temperature gradient direction features, multi-band spectral features, and entity and description embedding vectors extracted from each of the above modalities are vectorized and concatenated or dimensionally aligned to generate the original feature vectors corresponding to each modality.

4. The method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping according to claim 1, characterized in that, The method further includes sparsification enhancement of the cross-modal attention mechanism by introducing a hard threshold gating mechanism, specifically including: In execution Before the operation, based on the distance matrix Constructing a semantic association mask matrix It sets an anisotropic distance threshold to represent the effective semantic association boundary. ; When the anisotropic distance between cross-modal sample pairs Exceeding the threshold When this happens, the feature pair is determined to be an invalid association, and the mask matrix is... The corresponding element is assigned a value of negative infinity, otherwise it is assigned a value of zero; The mask matrix The weighted penalty term is superimposed onto the attention score matrix, which already contains the weighted penalty term, to forcibly truncate long-tailed distribution interactions with excessively large anisotropic distances, thereby eliminating false matching interference that are statistically similar but semantically significantly different in fault-sensitive physical directions.

5. The method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping according to claim 1, characterized in that, The semantic anisotropy mapping model is obtained by training by minimizing a multi-task joint loss function, which includes at least anisotropy consistency loss. This is used to explicitly constrain the consistency of projections of different modal features onto the fault-sensitive direction during model optimization. ； in, It is a set of cross-modal positive sample pairs consisting of different modal data for the same power equipment instance; and These are the anisotropic semantic embedding vectors of the positive sample pair in the unified semantic space, respectively. It represents the Hadamah accumulation. For component categories Category balance coefficient; Represents the square of the L2 norm; The fault sensitivity weight vector is defined based on the physical characteristics of the component. It is assigned a high weight value on the dimension corresponding to the principal axis of the fault feature and a low weight value on the dimension corresponding to the environmental noise. This allows the model to prioritize minimizing the cross-modal semantic differences of the key fault dimensions during training, thereby achieving feature alignment driven by physical mechanisms.

6. The method for aligning cross-modal sample data of power equipment based on semantic anisotropic mapping according to claim 1, characterized in that, The step of inputting the multimodal fusion representation vector into a preset task decision model to output the equipment status assessment result for the power equipment instance includes: The multimodal fusion representation vector is weighted and fused with the anisotropic semantic embedding vector of each modality through residual connection to generate enhanced device state features; Depending on the specific task requirements, select one or more of the following output headers for processing: Fault classification head: The fully connected layer is used to classify and map the device state features, and the mapping results are normalized and probability calculated to output the fault type probability distribution of the power equipment. State regression head: Uses a regression network to process the state characteristics of the equipment and outputs the health index or remaining service life prediction value of the power equipment; Based on one or more of the fault type probability distribution, the health index, and the remaining service life prediction value, determine the equipment status assessment result used to characterize the current physical operating status of the power equipment instance.

7. A cross-modal sample data alignment system for power equipment based on semantic anisotropic mapping, characterized in that, include: The data acquisition unit is used to acquire multimodal sample data to be processed from power equipment. The multimodal sample data to be processed includes at least visible light images, infrared images, sensor time-series data, and maintenance text data for the same power equipment instance; The feature and category recognition unit is used to perform feature extraction operations on the multimodal sample data to be processed to obtain the original feature vectors corresponding to each modality and to identify the component category to which the power equipment instance belongs; The semantic anisotropic mapping unit is used to call a pre-built semantic anisotropic mapping model to perform projection processing on the original feature vectors of each modality based on the component category. The semantic anisotropic mapping model is configured with a metric matrix defined for different component categories, which is used to perform non-equidistant scaling and rotation transformations on the original feature vectors according to the fault-sensitive direction of the component category, so as to obtain anisotropic semantic embedding vectors in a unified semantic space. An anisotropic fusion unit is used to calculate the anisotropic distance between modalities based on the anisotropic semantic embedding vectors of each modality, and to introduce the anisotropic distance as a weight penalty term into the cross-modal attention mechanism to dynamically correct the attention weights of cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance. The state assessment unit is used to input the multimodal fusion representation vector into a preset task decision model to output the equipment state assessment result for the power equipment instance; The semantic anisotropic mapping model is configured with metric matrices defined for different component categories. These matrices are used to perform non-equidistant scaling and rotation transformations on the original feature vectors based on the fault-sensitive direction of the component category, to obtain anisotropic semantic embedding vectors in a unified semantic space, including: For each component category Construct a learnable positive definite metric matrix And perform matrix decomposition on it to obtain the transformation matrix. Thus, a geometric transformation basis satisfying the following constraints is established: ； In the formula, Represents the transpose operation of a matrix or vector; Using the transformation matrix The linear projection operator will... The original feature vectors of each modality Mapped to the anisotropic semantic embedding vector : ； in, For component categories In the Featured cluster centers under each modality are used to eliminate distribution bias between modalities; The row vector direction represents the fault-sensitive direction in the feature space, and its norm represents the semantic importance weight along that direction. Through this mapping, the small differences in the fault-sensitive direction of the original feature vector are amplified in the unified semantic space, thereby making the anisotropic semantic embedding vector... The Euclidean distance is equivalent to the distance in the original space based on the metric matrix. Mahalanobis distance; The anisotropic semantic embedding vectors based on each modality are used to calculate the anisotropic distance between modalities. This anisotropic distance is then used as a weight penalty term in the cross-modal attention mechanism to dynamically adjust the attention weights for cross-modal interactions, thereby generating the multimodal fusion representation vector of the power equipment instance, including: Constructing the cross-modal distance matrix , of which Anisotropic semantic embedding vectors of each modality With the Anisotropic semantic embedding vectors of each modality Elements between Defined as the squared Euclidean distance between the two in the unified semantic space, it is used to quantify the semantic differences of different modal features in the fault-sensitive direction: ； Perform linear projection on the anisotropic semantic embedding vectors of each modality to generate the query matrix required by the attention mechanism. Key matrix AND-value matrix And an anisotropic attention model incorporating a distance penalty term is constructed: ； in, It characterizes the statistical correlation between modes; , which is the scaling factor for the feature dimension; Learnable physical constraint coefficients; distance penalty term This is used to forcibly suppress pseudo-matching modalities that have high statistical correlation but are semantically far apart in the fault-sensitive direction, thereby outputting a weighted result that conforms to physical laws as the multimodal fusion representation vector.

Images