Power grid multi-modal data fusion processing method and system

Abstract

The application provides a kind of power grid multi-modal data fusion processing method and system, comprising: collecting power grid system multi-modal original data and using momentum distillation technology to clean;Based on the clean data stream, using double-level network structure model to carry out feature enhancement processing, obtain enhanced multi-modal feature vector;Based on the enhanced multi-modal feature vector, using multi-scale decomposition technology is separated and using semantic guiding technology is aligned, obtain the feature after semantic alignment;The feature after semantic alignment is optimized using Transform model and community discovery algorithm, the optimized feature is input into joint representation layer to process timing dependence relationship, obtain semantic representation vector.The application improves the efficiency of power artificial intelligence model knowledge fusion.

Description

Technical Field

[0001] This invention relates to the field of data fusion processing technology, specifically to a method and system for multimodal data fusion processing of power grids. Background Technology

[0002] During operation, power grid systems generate data in multiple modalities, such as electrical parameters monitored by sensors, user load curves recorded by smart meters, and environmental information provided by meteorological satellites. These data sources vary and have diverse formats, and individual analysis may not be sufficient to fully depict the dynamics of the power grid. Multimodal data fusion technology integrates heterogeneous data streams and applies machine learning methods and statistical models for feature extraction and correlation analysis, thereby constructing a unified representation of the power grid's state.

[0003] Existing power grid multimodal data fusion technologies suffer from the following technical challenges: Multimodal data from different sources and of different types, such as equipment inspection images, operational log texts, and time-series sensor readings, inherently exhibit semantic gaps in their original feature spaces. The lack of unified high-level semantic alignment between image visual features and textual descriptive features makes it difficult for AI models to establish deep cross-modal associations. In power grid safety monitoring scenarios, an image showing oil leakage on a transformer surface and a text log recording recent abnormal temperature rises in the transformer should both point to a potential fault due to insulation aging. However, the semantic gap prevents power AI models from automatically and accurately associating and cross-validating visual cues in images with key descriptions in text at the semantic level. This results in inefficient knowledge fusion processes, limiting the accuracy and timeliness of fault diagnosis.

[0004] Therefore, how to eliminate the cross-modal semantic gap in the original multimodal data of the power grid is an important problem that urgently needs to be solved. Summary of the Invention

[0005] To address the problem that existing technologies cannot eliminate the cross-modal semantic gap in raw multimodal power grid data, this invention proposes a multimodal power grid data fusion processing method, comprising the following steps: Collect timestamp-aligned multimodal raw data within the power grid system; the multimodal raw data includes image data and text data. Based on the original multimodal data, momentum distillation is used to clean the data to obtain a clean data stream; based on the clean data stream, a two-level network structure model is used for feature enhancement to obtain enhanced multimodal feature vectors. Based on the enhanced multimodal feature vector, multi-scale decomposition technology is used for separation and semantic guidance technology is used for semantic alignment to obtain semantically aligned features; The semantically aligned features are input into the Transformer model to obtain fused features; the fused features are optimized using a community detection algorithm to obtain optimized features; the optimized features are input into a joint representation layer to process temporal dependencies and obtain semantic representation vectors.

[0006] Optionally, based on the clean data stream, feature enhancement processing is performed using a two-level network structure model to obtain enhanced multimodal feature vectors. Specific steps include: Based on the clean data stream, image encoders and text encoders are used to extract image patch feature vectors and text semantic feature vectors, respectively. Based on the image patch feature vector and the text semantic feature vector, a cross-attention mechanism is used to fuse them to obtain an enhanced multimodal feature vector.

[0007] Optionally, the enhanced multimodal feature vector is obtained by fusing the image patch feature vector and the text semantic feature vector using a cross-attention mechanism. Specific steps include: Random linear interpolation transformation is performed on batches of the image patch feature vectors to obtain enhanced image patch feature vectors; The text sequence of the text semantic feature vector is concatenated and recombined to obtain the enhanced text semantic feature vector; Based on the enhanced image patch feature vector and the enhanced text semantic feature vector, a cross-attention mechanism is used for alignment and integration to obtain the enhanced multimodal feature vector.

[0008] Optionally, based on the enhanced multimodal feature vector, the feature vector is separated using multi-scale decomposition technology and semantically aligned using semantic guidance technology to obtain semantically aligned features. Specific steps include: Based on the enhanced multimodal feature vector, the image feature vector is separated using multi-scale decomposition technology to obtain low-frequency approximate components and high-frequency detail components; Based on the low-frequency approximation component and the high-frequency detail component, adaptive weights are used for fusion to obtain reconstructed image features; Based on the enhanced multimodal feature vector, the text feature vector is extracted using a pre-trained language model to obtain contextual features; Based on the reconstructed image features and the context features, a cross-modal attention mechanism is used for alignment to obtain semantically aligned features.

[0009] Optionally, the step of fusing the low-frequency approximation component and the high-frequency detail component using adaptive weights to obtain reconstructed image features includes the following steps: Based on the low-frequency approximation component, Gaussian smoothing filtering is used for noise cancellation to obtain an enhanced low-frequency approximation component; the information entropy of the enhanced low-frequency approximation component is calculated to construct an information entropy matrix. Based on the high-frequency detail components, the Laplacian operator is used for sharpening to obtain enhanced high-frequency detail components. The gradient variance of the enhanced high-frequency detail components is then calculated to construct the gradient variance matrix. Based on the information entropy matrix and the gradient variance matrix, a single-layer convolutional neural network and the activation function Softmax are used to process them respectively to obtain a low-frequency adaptive weight matrix and a high-frequency adaptive weight matrix. The low-frequency adaptive weight matrix is ​​multiplied element-wise with the enhanced low-frequency component to obtain the low-frequency multiplication result; The high-frequency adaptive weight matrix is ​​multiplied element-wise with the enhanced high-frequency component to obtain the high-frequency multiplication result; The low-frequency multiplication result and the high-frequency multiplication result are fused by matrix addition to obtain the reconstructed image features.

[0010] Optionally, the pre-training process of the language model includes: The training data is input from a general domain text corpus, power sector-specific operation log text, and maintenance report text. The output uses power sector contextual features as training data. Based on the input and output of the training data, a deep bidirectional Transformer text model is trained using the masked language model task and the next sentence prediction task as pre-training tasks.

[0011] Optionally, the step of inputting the semantically aligned features into the Transformer model to obtain the fused features includes the following steps: Based on the image and text features in the semantically aligned features, a self-attention mechanism is used for filtering to obtain image hidden representations and text hidden representations. Based on the image and text hiding representations, a cross-attention layer is used to align cross-modal information interaction, resulting in aligned image and text feature vectors. Based on the aligned image feature vector and text feature vector, respectively, an orthogonal projection model is used to map them to the same common subspace to obtain image projection vector and text projection vector; Calculate the cosine similarity between the image projection vector and the text projection vector, and use the cosine similarity values ​​to generate a correlation matrix; Based on the numerical value of the correlation matrix, the fusion weights for the image projection vector and the text projection vector are dynamically adjusted; Based on the fusion weights, the image projection vector and the text projection vector are weighted and summed to obtain the fused features.

[0012] Optionally, the step of optimizing the fused features using a community detection algorithm to obtain optimized features includes the following specific steps: Based on the fused features, a feature map is constructed using a community detection algorithm; wherein, the nodes of the feature map represent feature vectors, and the edges of the feature map represent the similarity between features; The nodes of the feature map are clustered using a modularity optimization algorithm to identify multiple communities with modularity values ​​greater than a preset threshold, and the required feature vectors are extracted from each community to generate semantic summary feature representations. The semantic summary features are used as the optimized features.

[0013] Optionally, the step of inputting the optimized features into the joint representation layer to process temporal dependencies and obtain semantic representation vectors includes the following specific steps: A joint representation layer is obtained by embedding a residual connection structure in a multi-layer Transformer encoder. Based on the optimized features, the joint representation layer is used to analyze the temporal series dependencies between features and output a semantic representation vector with a unified dimension.

[0014] Furthermore, this application also provides a power grid multi-modal data fusion processing system, including: The data acquisition module is used to collect timestamp-aligned multimodal raw data within the power grid system, including image data and text data. The multimodal feature fusion module is used to clean the original multimodal data using momentum distillation technology to obtain a clean data stream; and to perform feature enhancement processing using a two-level network structure model based on the clean data stream to obtain an enhanced multimodal feature vector. The semantic alignment module is used to separate the enhanced multimodal feature vectors using multi-scale decomposition technology and perform semantic alignment using semantic guidance technology to obtain semantically aligned features. The feature fusion optimization module is used to input the semantically aligned features into the Transformer model to obtain fused features; optimize the fused features using a community detection algorithm to obtain optimized features; and input the optimized features into the joint representation layer to process temporal dependencies to obtain a semantic representation vector.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method and system for multimodal data fusion processing in power grids. Based on timestamp-aligned original multimodal data within the power grid system, momentum distillation is used for cleaning. Based on the clean data stream, a two-level network structure model is used for feature enhancement to obtain enhanced multimodal feature vectors. These enhanced feature vectors are then separated using multi-scale decomposition and semantically aligned using semantic guidance techniques to obtain semantically aligned features. Finally, a Transformer model, community detection algorithm, and joint representation layer are sequentially used to process the data, obtaining semantic representation vectors. This progressive approach employs different algorithms and mechanisms to gradually bridge the cross-modal semantic gap in the original multimodal data of the power grid, thereby providing training data for power AI models to eliminate the cross-modal semantic gap and improving the data fusion efficiency of power AI models. Attached Figure Description

[0016] Figure 1 This is a flowchart of the power grid multi-modal data fusion processing method of the present invention; Figure 2 This is a schematic diagram illustrating the composition of the original multimodal data in the power grid multimodal data fusion processing method of the present invention; Figure 3 This is a schematic diagram of the structure of the power grid multimodal data fusion processing system of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] Example 1: This invention provides a method for multi-modal data fusion processing of power grids, the flowchart of which is shown below. Figure 1 As shown: Step 1: Collect timestamp-aligned multimodal raw data from the power grid system. The multimodal raw data includes electrical parameters, load curves, equipment inspection images, and operation log text.

[0019] Specifically, the power grid multimodal data fusion processing method first collects timestamp-aligned raw multimodal data from the power grid system. This data includes electrical parameters, load curves, equipment inspection images, and operation log text, such as... Figure 2As shown, the data acquisition process receives electrical parameter data streams, load curve data streams, equipment inspection image data streams, and operation log text data streams in parallel through a distributed data interface. A timestamp is added to each data stream to create a time-series index. Based on the time-series index, the multimodal data streams are aligned to a time window to ensure time series consistency. The time-aligned data is then converted into a standardized feature representation format to provide a consistent data foundation for subsequent processing.

[0020] Step 2: Use momentum distillation technology to clean the multimodal raw data and generate a clean data stream.

[0021] Specifically, momentum distillation constructs a momentum model with the same structure as the main model. The momentum model updates parameters using an exponential moving average strategy, smooths the current model weights using historical weights, and uses the updated momentum model to smooth the current prediction output, reducing noise and outliers in the data. It also generates pseudo-labels as additional signals, which provide stable supervision information to assist in the training of the main model, thereby reducing noise and outliers in the data and improving data quality and cleanliness.

[0022] The main model is a deep neural network model for handling multimodal classification tasks, specifically including a visual encoder module and a text encoder module. The momentum distillation cleaning process is as follows: The original multimodal data is input into the main model to obtain the predicted probability distribution; a momentum model with a structure completely identical to the main model is constructed, and the weight parameters of the momentum model are smoothly updated using an exponential moving average strategy based on the historical weight parameters of the main model, resulting in an updated momentum model; the original multimodal data is input into the updated momentum model for forward propagation prediction, outputting a smoothed prediction result; the smoothed prediction result is used as a pseudo-label, which is combined with the original data labels to provide supervision information for the original multimodal data and retrain the main model to reduce noise and outliers in the original multimodal data, outputting a clean data stream.

[0023] Step 3: Input the clean data stream into the two-level network structure model for processing to generate enhanced multimodal feature vectors.

[0024] The execution steps include: (1) using an image encoder and a text encoder to extract image patch feature vectors and text semantic feature vectors based on a clean data stream; (2) performing random linear interpolation transformation on batches of image patch feature vectors to obtain enhanced image patch feature vectors; (3) splicing and recombining the text sequence of text semantic feature vectors to obtain enhanced text semantic feature vectors; (4) using a cross-attention mechanism to align and integrate the enhanced image patch feature vectors and enhanced text semantic feature vectors to obtain enhanced multimodal feature vectors.

[0025] Specifically, a two-layer network structure integrates an image encoder and a text encoder. The image encoder uses a visual Transformer architecture to extract image patch features, while the text encoder generates word embeddings based on a pre-trained language model to capture text semantic features. Random linear interpolation transformations are applied to batches of images, and text sequences are concatenated and recombined to generate enhanced samples, thereby improving the data's representational power and diversity. A cross-attention mechanism integrates image and text information to achieve intermodal feature interaction.

[0026] Step 4: Perform multi-scale decomposition and semantic guidance processing on the enhanced multimodal feature vectors to obtain semantically aligned features.

[0027] The execution steps include: (1) using multi-scale decomposition technology to separate the image feature vector based on the enhanced multimodal feature vector to obtain low-frequency approximation components and high-frequency detail components; (2) using adaptive weights to fuse the low-frequency approximation components and high-frequency detail components to obtain reconstructed image features; (3) using a pre-trained language model to extract the text feature vector based on the enhanced multimodal feature vector to obtain context features; and (4) using a cross-modal attention mechanism to align the reconstructed image features and context features to obtain semantically aligned features.

[0028] Step (2) includes: <1> Based on the low-frequency approximation component, Gaussian smoothing filtering is used for noise cancellation to obtain the enhanced low-frequency approximation component; the information entropy of the enhanced low-frequency approximation component is calculated to construct the information entropy matrix. <2> Based on the high-frequency detail components, the Laplacian operator is used for sharpening to obtain enhanced high-frequency detail components. The gradient variance of the enhanced high-frequency detail components is calculated to construct the gradient variance matrix. <3> Based on the information entropy matrix and the gradient variance matrix, a single-layer convolutional neural network and the activation function Softmax are used to process them respectively to obtain the low-frequency adaptive weight matrix and the high-frequency adaptive weight matrix. <4> The low-frequency adaptive weight matrix is ​​multiplied element-wise with the enhanced low-frequency component to obtain the low-frequency multiplication result. <5> The high-frequency adaptive weight matrix is ​​multiplied element-wise with the enhanced high-frequency components to obtain the high-frequency multiplication result; <6> The low-frequency multiplication result is fused with the high-frequency multiplication result by matrix addition to obtain the reconstructed image features.

[0029] The language model pre-training process in step (3) includes: <1> The training data is input from a general domain text corpus, power sector-specific operation log text, and maintenance report text. <2> The output uses power sector contextual features as training data. <3> Based on the input and output of the training data, a deep bidirectional Transformer text model is trained using the masked language model task and the next sentence prediction task as pre-training tasks.

[0030] Specifically, multi-scale decomposition selects wavelet basis functions to decompose the image, separating low-frequency approximation components and high-frequency detail components. Gaussian smoothing filtering is applied to the low-frequency approximation components to highlight the main structure, while Laplacian sharpening is used to enhance edge textures in the high-frequency detail components. Image features are reconstructed through adaptive weight fusion to balance the contributions of different components. Simultaneously, a language model is used to extract word-level and sentence-level contextual features to obtain a deep semantic representation of the text. A cross-modal attention mechanism is used to calculate the relevance weights between image region features and text word vectors, gradually narrowing the semantic gap between modalities.

[0031] The adaptive weight fusion process dynamically determines weights by calculating the information entropy or spatial gradient variance of the low-frequency approximation component and the high-frequency detail component. Specifically, it involves calculating the information entropy matrix of the smoothed low-frequency approximation component and the gradient variance matrix of the sharpened high-frequency detail component; inputting these matrices into a convolutional neural network and processing them through a Softmax activation function to output a low-frequency adaptive weight matrix and a high-frequency adaptive weight matrix; performing element-wise multiplication of the low-frequency adaptive weight matrix with the smoothed low-frequency approximation component and the high-frequency adaptive weight matrix with the sharpened high-frequency detail component; and finally, fusing the two multiplied components using matrix addition to output the reconstructed image features.

[0032] This method uses a language model to extract word-level and sentence-level contextual features. The language model refers to a deep bidirectional Transformer text model, similar to BERT or RoBERTa architectures. The language model is pre-trained. The pre-training process is as follows: First, a large-scale general-domain text corpus, as well as power industry-specific operation logs and maintenance reports, are collected as training sets. Then, a masking language model task is used to randomly mask some words in the training set, requiring the model to predict the masked words. Simultaneously, a next-sentence prediction task is used to determine whether two texts have contextual continuity. The language model parameters are continuously updated by optimizing the loss functions of the masking and next-sentence prediction tasks, ultimately outputting a pre-trained language model capable of extracting contextual features from the power industry.

[0033] Step 5: Input the semantically aligned features into the Transformer architecture for fusion processing to obtain the fused features.

[0034] The execution steps include: (1) using self-attention mechanism to filter the image part features and text part features in the semantically aligned features to obtain image hidden representation and text hidden representation; (2) using cross-attention layer to perform cross-modal information interaction alignment based on image hidden representation and text hidden representation to obtain aligned image feature vector and text feature vector; (3) using orthogonal projection model to map the aligned image feature vector and text feature vector to the same common subspace based on the aligned image feature vector and text feature vector to obtain image projection vector and text projection vector; (4) calculating the cosine similarity between image projection vector and text projection vector, and using cosine similarity numerical combination to generate a correlation matrix; (5) dynamically adjusting the fusion weight for image projection vector and text projection vector based on the numerical value of the correlation matrix; (6) weighted summing of image projection vector and text projection vector based on fusion weight to obtain fused features.

[0035] Specifically, the Transformer architecture computes query-key-value pair weights within each modality through a self-attention mechanism, aggregating key features and filtering redundant information. It also aligns image and text hidden representations through cross-attention layers, facilitating information exchange between modalities. An orthogonal projection model is constructed to map features from different modalities to a common subspace, eliminating modal differences. Based on cosine similarity, an intermodal correlation matrix is ​​calculated, and fusion weights are dynamically adjusted to optimize feature fusion performance, enabling deep cross-modal interaction.

[0036] The dynamic adjustment of fusion weights is based on proportional allocation of similarity values ​​in the relevance matrix. Specifically, the adjustment mechanism is as follows: using the obtained relevance matrix, the cosine similarity score parameters between each image projection vector and text projection vector are extracted; these parameters are then processed using a Softmax normalization function, transforming them into a weight probability distribution vector with a sum of one; the elements of this weight probability distribution vector are used as fusion weight parameters, assigning larger fusion weight parameters to modal feature matching pairs with higher cosine similarity scores in the relevance matrix, and vice versa; finally, the calculated fusion weight parameters are multiplied by the corresponding image projection vector and text projection vector respectively, completing the dynamic adjustment of weights and multimodal feature fusion.

[0037] Step 6: Apply a community detection algorithm to the fused features to obtain optimized features. Input the optimized features into the joint representation layer for further processing, and output a semantic representation vector. Finally, input the semantic representation vector into the power artificial intelligence model for training and inference.

[0038] The execution steps include: (1) constructing a feature map using a community detection algorithm based on the fused features; wherein, the nodes of the feature map represent feature vectors and the edges of the feature map represent the similarity between features; (2) performing cluster analysis on the nodes of the feature map using a modularity optimization algorithm to identify multiple communities with modularity values ​​greater than a preset threshold, and extracting the required feature vectors from each community to generate a semantic summary feature representation; (3) using the semantic summary feature representation as the optimized features; (4) embedding a residual connection structure in a multi-layer Transformer encoder to obtain a joint representation layer; (5) using the joint representation layer to analyze the temporal series dependencies between features based on the optimized features, and outputting a semantic representation vector with a unified dimension.

[0039] Specifically, the community detection algorithm constructs a feature map from the fused features, where nodes represent feature vectors and edges represent the similarity between features. A modularity optimization algorithm is applied to cluster the nodes in the feature map, identifying high-cohesion communities with modularity values ​​greater than a preset threshold. Representative feature vectors are extracted from each high-cohesion community to capture key information. Semantic summary feature representations are generated based on these extracted representative feature vectors, condensing feature information for easier subsequent processing. The optimized features are then input into a joint representation layer for processing. This joint representation layer integrates a multi-layer Transformer encoder and embeds a residual connection structure to prevent gradient vanishing and stabilize the training process. A feedforward network is used to analyze the temporal dependencies between features, capturing dynamic evolution patterns in power system data and outputting semantic representation vectors with uniform dimensions, providing standardized input for model training.

[0040] Here, the edge represents the similarity between features, which refers to the distance measurement between multiple fused feature vectors output after the fusion process. It calculates the internal similarity association score between each feature node after the multimodal fusion operation has been completed. Usually, the weight of the edge in the feature map can be determined by calculating the Euclidean distance between two fused feature vectors or recalculating the vector inner product.

[0041] Finally, the semantic representation vectors are input into the power artificial intelligence model for training and inference. Specifically, the semantic representation vectors are used as the input feature set to train the fault diagnosis classification model. During the training process, the model parameters are optimized through the backpropagation algorithm. The optimized fault diagnosis model is used to perform inference analysis on real-time power grid data and output a fault prediction report including fault type identification results and confidence levels. This bridges the cross-modal semantic gap and improves the efficiency of knowledge fusion.

[0042] The specific effects of this invention are as follows: The training and inference of the power AI model uses semantic representation vectors as input feature sets to train a fault diagnosis classification model. During training, the model parameters are optimized using the backpropagation algorithm to improve model accuracy. The optimized fault diagnosis model is then used to perform inference analysis on real-time power grid data, outputting a fault prediction report including fault type identification results and confidence levels, supporting power grid fault diagnosis decisions.

[0043] This invention addresses the inefficiency of knowledge fusion caused by the cross-modal semantic gap through a multimodal data fusion processing method. First, it collects timestamp-aligned multimodal raw data from the power grid system, including electrical parameters, load curves, equipment inspection images, and operation log text. Time windows are aligned and formats are standardized using a distributed data interface to provide a consistent data foundation for subsequent processing. Momentum distillation is used to clean the multimodal raw data, constructing a momentum model consistent with the main model structure. Parameters are updated using an exponential moving average strategy, and pseudo-labels are generated as additional signals to reduce data noise and outliers, improving data quality. The clean data stream is then input into a two-level network structure for processing. A visual Transformer architecture is used to extract image patch features, word embeddings are generated based on a pre-trained language model, random linear interpolation is applied to batches of images, and text sequences are concatenated and recombined to generate enhanced samples, improving data representation capabilities. Finally, a cross-attention mechanism integrates image and text information. The enhanced multimodal feature vectors undergo multi-scale decomposition and semantic guidance processing. Wavelet basis functions are selected for multi-scale image decomposition, separating low-frequency approximation components and high-frequency detail components. Gaussian smoothing filtering is applied to the low-frequency components, while Laplacian sharpening is used for the high-frequency components. Image features are reconstructed through adaptive weight fusion. Simultaneously, a language model is used to extract word-level and sentence-level contextual features, and a cross-modal attention mechanism is used to calculate the relevance weights between image region features and text word vectors, gradually reducing the semantic gap between modalities. The semantically aligned features are input into the Transformer architecture for fusion processing. A self-attention mechanism is used to calculate query-key-value pair weights within each modality, aggregating key features. Image and text hidden representations are aligned through a cross-attention layer, and an orthogonal projection model is constructed to map features from different modalities to a common subspace. The intermodal relevance matrix is ​​calculated based on cosine similarity, and the fusion weights are dynamically adjusted to achieve deep cross-modal interaction. The fused features are processed using a community discovery algorithm to construct a feature map. A modularity optimization algorithm is then applied to cluster the nodes, identifying highly cohesive communities and extracting representative feature vectors to generate semantic summary feature representations, further optimizing the features. The optimized features are then input into a joint representation layer for processing. A multi-layer Transformer encoder is integrated with a residual connection structure, and a feedforward network is used to analyze time-series dependencies, capturing dynamic evolution patterns and outputting a semantic representation vector with a unified dimension. Finally, the semantic representation vector is input into a power AI model for training and inference. A fault diagnosis classification model is trained, and parameters are optimized using a backpropagation algorithm. The optimized model is then used to perform inference analysis on real-time power grid data, outputting a fault prediction report. This invention's processing flow, through data cleaning, feature enhancement, semantic alignment, fusion optimization, and joint representation, gradually bridges the cross-modal semantic gap, enabling the power AI model to effectively connect multimodal data and improve knowledge fusion efficiency.

[0044] This invention effectively addresses the problem of low knowledge fusion efficiency in power AI models caused by the cross-modal semantic gap through a multimodal data fusion processing method for power grids. The method begins with the collection of timestamp-aligned multimodal raw data, followed by time window alignment and format standardization via a distributed data interface, providing a consistent data foundation for subsequent processing. Momentum distillation data cleaning utilizes momentum models to generate pseudo-labels as additional signals, reducing data noise and outliers and improving data quality. A two-level network structure processing method extracts features through a visual Transformer architecture and a pre-trained language model, combined with cross-attention mechanisms and sample augmentation techniques to enhance data representation capabilities and diversity. Multi-scale decomposition and semantically guided processing employ wavelet transform to separate image components, combined with Gaussian smoothing. Laplacian sharpening enhances details, and a cross-modal attention mechanism is used to calculate relevance weights, gradually narrowing the semantic gap between modalities. The Transformer architecture fusion achieves feature interaction between modalities through self-attention and cross-attention layers. The orthogonal projection model maps to a common subspace, dynamically adjusting fusion weights to optimize feature fusion effects. The community detection algorithm constructs feature maps for cluster analysis, extracts representative features to generate semantic summaries, and strengthens semantic associations. The joint representation layer integrates multi-layer Transformer encoders and residual connection structures to capture dynamic evolution patterns of time series and output a unified-dimensional semantic representation vector. Finally, through fault diagnosis model training and inference, efficient fusion of multimodal data is achieved, improving knowledge fusion efficiency and fault diagnosis accuracy.

[0045] Example 2: Based on the same inventive concept, this invention also provides a power grid multi-modal data fusion processing system, the structure of which is as follows: Figure 3 As shown, it includes: The data acquisition module is used to collect timestamp-aligned multimodal raw data within the power grid system. The multimodal raw data includes image data and text data.

[0046] The multimodal feature fusion module is used to clean the original multimodal data using momentum distillation technology to obtain a clean data stream; based on the clean data stream, a two-level network structure model is used for feature enhancement processing to obtain the enhanced multimodal feature vector.

[0047] The semantic alignment module is used to separate the enhanced multimodal feature vectors using multi-scale decomposition techniques and perform semantic alignment using semantic guidance techniques to obtain semantically aligned features.

[0048] The feature fusion and optimization module is used to input semantically aligned features into the Transformer model to obtain fused features; optimize the fused features using a community detection algorithm to obtain optimized features; and input the optimized features into the joint representation layer to process temporal dependencies to obtain semantic representation vectors.

[0049] The multimodal feature fusion module is specifically used for: (1) cleaning the original multimodal data using momentum distillation technology to obtain a clean data stream; (2) extracting the image block feature vector and text semantic feature vector using an image encoder and a text encoder respectively based on the clean data stream; and (3) fusing the image block feature vector and text semantic feature vector using a cross-attention mechanism to obtain an enhanced multimodal feature vector.

[0050] The semantic alignment module is specifically used for: (1) separating the image feature vector based on the enhanced multimodal feature vector using multi-scale decomposition technology to obtain low-frequency approximation components and high-frequency detail components; (2) fusing the low-frequency approximation components and high-frequency detail components using adaptive weights to obtain reconstructed image features; (3) extracting the text feature vector based on the enhanced multimodal feature vector using a pre-trained language model to obtain context features; and (4) aligning the reconstructed image features and context features using a cross-modal attention mechanism to obtain semantically aligned features.

[0051] The feature fusion optimization module is specifically used for: (1) filtering the image and text features in the semantically aligned features using a self-attention mechanism to obtain image hidden representations and text hidden representations; (2) aligning cross-modal information interaction using a cross-attention layer based on the image and text hidden representations to obtain aligned image feature vectors and text feature vectors; (3) mapping the aligned image and text feature vectors to the same common subspace using an orthogonal projection model to obtain image projection vectors and text projection vectors; (4) calculating the cosine similarity between the image projection vectors and text projection vectors, and generating a correlation matrix using the cosine similarity values; (5) dynamically adjusting the fusion weights for the image projection vectors and text projection vectors based on the magnitude of the correlation matrix. (6) Based on the fusion weight, the image projection vector and the text projection vector are weighted and summed to obtain the fused features; (7) Based on the fused features, a feature map is constructed using a community detection algorithm; where the nodes of the feature map represent feature vectors and the edges of the feature map represent the similarity between features; (8) The nodes of the feature map are clustered using a modularity optimization algorithm to identify multiple communities with modularity values ​​greater than a preset threshold, and the required feature vectors are extracted from each community to generate a semantic summary feature representation; (9) The semantic summary feature representation is used as the optimized features; (10) A residual connection structure is embedded in a multi-layer Transformer encoder to obtain a joint representation layer; (11) Based on the optimized features, the joint representation layer is used to analyze the temporal series dependencies between features and output a semantic representation vector with a unified dimension.

[0052] Example 3: like Figure 4 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0053] The processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to implement the corresponding method flow or corresponding function, so as to realize the steps of the power grid multi-modal data fusion processing method in the above embodiments.

[0054] Example 4: Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of the power grid multimodal data fusion processing method in the above embodiments.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A method for multi-modal data fusion processing of power grids, characterized in that, Includes the following steps: Collect timestamp-aligned multimodal raw data within the power grid system; the multimodal raw data includes image data and text data. Based on the aforementioned multimodal raw data, momentum distillation technology is used for cleaning to obtain a clean data stream; Based on the clean data stream, a two-level network structure model is used for feature enhancement processing to obtain enhanced multimodal feature vectors. Based on the enhanced multimodal feature vector, multi-scale decomposition technology is used for separation and semantic guidance technology is used for semantic alignment to obtain semantically aligned features; The semantically aligned features are input into the Transformer model to obtain the fused features. The fused features are optimized using a community detection algorithm to obtain optimized features; the optimized features are then input into a joint representation layer to process temporal dependencies and obtain a semantic representation vector.

2. The method as described in claim 1, characterized in that, Based on the clean data stream, a two-level network structure model is used for feature enhancement processing to obtain enhanced multimodal feature vectors. The specific steps include: Based on the clean data stream, image encoders and text encoders are used to extract image patch feature vectors and text semantic feature vectors, respectively. Based on the image patch feature vector and the text semantic feature vector, a cross-attention mechanism is used to fuse them to obtain an enhanced multimodal feature vector.

3. The method as described in claim 2, characterized in that, The enhanced multimodal feature vector is obtained by fusing the image patch feature vector and the text semantic feature vector using a cross-attention mechanism. Specific steps include: Random linear interpolation transformation is performed on batches of the image patch feature vectors to obtain enhanced image patch feature vectors; The text sequence of the text semantic feature vector is concatenated and recombined to obtain the enhanced text semantic feature vector; Based on the enhanced image patch feature vector and the enhanced text semantic feature vector, a cross-attention mechanism is used for alignment and integration to obtain the enhanced multimodal feature vector.

4. The method as described in claim 1, characterized in that, The enhanced multimodal feature vector is used for separation using multi-scale decomposition technology and semantic alignment using semantic guidance technology to obtain semantically aligned features. Specific steps include: Based on the enhanced multimodal feature vector, the image feature vector is separated using multi-scale decomposition technology to obtain low-frequency approximate components and high-frequency detail components; Based on the low-frequency approximation component and the high-frequency detail component, adaptive weights are used for fusion to obtain reconstructed image features; Based on the enhanced multimodal feature vector, the text feature vector is extracted using a pre-trained language model to obtain contextual features; Based on the reconstructed image features and the context features, a cross-modal attention mechanism is used for alignment to obtain semantically aligned features.

5. The method as described in claim 4, characterized in that, The process of fusing the low-frequency approximation component and the high-frequency detail component using adaptive weights to obtain reconstructed image features includes the following steps: Based on the low-frequency approximation component, Gaussian smoothing filtering is used for noise cancellation to obtain an enhanced low-frequency approximation component; the information entropy of the enhanced low-frequency approximation component is calculated to construct an information entropy matrix. Based on the high-frequency detail components, the Laplacian operator is used for sharpening to obtain enhanced high-frequency detail components. The gradient variance of the enhanced high-frequency detail components is then calculated to construct the gradient variance matrix. Based on the information entropy matrix and the gradient variance matrix, a single-layer convolutional neural network and the activation function Softmax are used to process them respectively to obtain a low-frequency adaptive weight matrix and a high-frequency adaptive weight matrix. The low-frequency adaptive weight matrix is ​​multiplied element-wise with the enhanced low-frequency component to obtain the low-frequency multiplication result; The high-frequency adaptive weight matrix is ​​multiplied element-wise with the enhanced high-frequency component to obtain the high-frequency multiplication result; The low-frequency multiplication result and the high-frequency multiplication result are fused by matrix addition to obtain the reconstructed image features.

6. The method as described in claim 4, characterized in that, The pre-training process of the language model includes: The training data is input from a general domain text corpus, power sector-specific operation log text, and maintenance report text. The output uses power sector contextual features as training data. Based on the input and output of the training data, a deep bidirectional Transformer text model is trained using the masked language model task and the next sentence prediction task as pre-training tasks.

7. The method as described in claim 1, characterized in that, The specific steps for inputting the semantically aligned features into the Transformer model to obtain the fused features include: Based on the image and text features in the semantically aligned features, a self-attention mechanism is used for filtering to obtain image hidden representations and text hidden representations. Based on the image and text hiding representations, a cross-attention layer is used to align cross-modal information interaction, resulting in aligned image and text feature vectors. Based on the aligned image feature vector and text feature vector, respectively, an orthogonal projection model is used to map them to the same common subspace to obtain image projection vector and text projection vector; Calculate the cosine similarity between the image projection vector and the text projection vector, and use the cosine similarity values ​​to generate a correlation matrix; Based on the numerical value of the correlation matrix, the fusion weights for the image projection vector and the text projection vector are dynamically adjusted. Based on the fusion weights, the image projection vector and the text projection vector are weighted and summed to obtain the fused features.

8. The method as described in claim 1, characterized in that, The process of optimizing the fused features using a community detection algorithm to obtain optimized features includes the following steps: Based on the fused features, a feature map is constructed using a community detection algorithm; wherein, the nodes of the feature map represent feature vectors, and the edges of the feature map represent the similarity between features; The nodes of the feature map are clustered using a modularity optimization algorithm to identify multiple communities with modularity values ​​greater than a preset threshold, and the required feature vectors are extracted from each community to generate semantic summary feature representations. The semantic summary features are used as the optimized features.

9. The method as described in claim 1, characterized in that, The step of inputting the optimized features into the joint representation layer to process temporal dependencies and obtain semantic representation vectors includes: A joint representation layer is obtained by embedding a residual connection structure in a multi-layer Transformer encoder. Based on the optimized features, the joint representation layer is used to analyze the temporal series dependencies between features and output a semantic representation vector with a unified dimension.

10. A power grid multi-modal data fusion processing system, characterized in that, include: The data acquisition module is used to collect timestamp-aligned multimodal raw data within the power grid system, including image data and text data. The multimodal feature fusion module is used to clean the original multimodal data using momentum distillation technology to obtain a clean data stream; and to perform feature enhancement processing using a two-level network structure model based on the clean data stream to obtain an enhanced multimodal feature vector. The semantic alignment module is used to separate the enhanced multimodal feature vectors using multi-scale decomposition technology and perform semantic alignment using semantic guidance technology to obtain semantically aligned features. The feature fusion optimization module is used to input the semantically aligned features into the Transformer model to obtain the fused features. The fused features are optimized using a community detection algorithm to obtain optimized features; the optimized features are then input into a joint representation layer to process temporal dependencies and obtain a semantic representation vector.

Images