Multi-modal feature alignment and fusion method for electric power image-text data

Abstract

The invention discloses a multi-modal feature alignment and fusion method for electric power image-text data, and the method comprises the following steps: obtaining image data of electric power equipment and corresponding text data, and carrying out the standardization preprocessing of the image data and the text data; performing feature extraction on the image data to obtain image features; performing feature extraction on the text data to obtain semantic features; converting the image features and the semantic features into image feature vectors and semantic feature vectors respectively; mapping the image feature vector and the semantic feature vector to a unified semantic vector space by adopting a cross-modal contrast learning method, and realizing feature alignment by optimizing a triple loss module; and splicing the aligned image feature vector and semantic feature vector to generate a fusion feature vector. According to the method, the aligned image feature vectors and semantic feature vectors are spliced and fused, so that the comprehensive state of the power equipment can be described more comprehensively and accurately.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for multimodal feature alignment and fusion of power grid graphic data. Background Technology

[0002] With the deepening of smart grid construction, the power system has generated massive amounts of unstructured data. Among this, image and text data from power equipment constitute the core multimodal information for equipment status perception and fault analysis. How to effectively correlate and integrate these heterogeneous image and text data to achieve comprehensive and accurate perception of the operating status of power equipment has become crucial for improving the level of intelligent operation and maintenance of the power grid.

[0003] Existing methods typically perform isolated analysis of images and text, such as using images only for deep learning-based visual classification and text only for natural language processing tasks. This approach fails to establish cross-modal semantic connections, leading to the phenomenon of "information silos." Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for multimodal feature alignment and fusion of power grid graphic data.

[0005] This invention provides a method for multimodal feature alignment and fusion of power grid graphic data, comprising the following steps: Acquire image data and corresponding text data of power equipment, and perform standardized preprocessing on the image data and text data; A pre-trained CNN model is used to extract features from the image data to obtain image features; a pre-trained BERT language model is used to extract features from the text data to obtain semantic features. The image features and semantic features are respectively converted into fixed-dimensional image feature vectors and semantic feature vectors; A cross-modal contrastive learning method is adopted to map the image feature vector and semantic feature vector to a unified semantic vector space, and feature alignment is achieved by optimizing the triplet loss module; The aligned image feature vector and semantic feature vector are concatenated to generate a fused feature vector.

[0006] Preferably, image data and corresponding text data of the power equipment are acquired, and the image data and text data are subjected to standardized preprocessing; specifically including: Image data of power equipment and its semantically associated text descriptions are obtained from historical operation and maintenance documents to form image-text matching samples; wherein, the image data includes visible light images and infrared thermal images of the equipment, and the text data includes equipment nameplate parameters, inspection records and fault analysis reports; The images are scaled to a preset size and the pixel values ​​are normalized. The text data is processed by word segmentation, stop word filtering and special character cleaning. The pre-trained language model's word segmenter converts the text into a fixed-length word index sequence, and padding or truncation operations are used to ensure that the length of the input sequence is consistent.

[0007] Preferably, a pre-trained CNN model is used to extract features from the image data to obtain image features; specifically, this includes... A pre-trained CNN model is used as the image feature extractor; Remove the top fully connected layer of the pre-trained CNN model, and retain its convolutional and pooling layers as the core feature extraction module; The preprocessed image data is input into the core feature extraction module, and the feature map output by the last convolutional layer or pooling layer is obtained through forward propagation. The feature map is subjected to global average pooling to convert it into a one-dimensional feature vector, which is used as the image feature.

[0008] Preferably, a pre-trained BERT language model is used to extract features from the text data to obtain semantic features; specifically, this includes... A pre-trained BERT model was used as the text feature extractor; The preprocessed word index sequence is input into the BERT model, and the last hidden state of the entire sequence is obtained as the context feature of each word. The last hidden state is aggregated to generate a fixed-dimensional overall text representation vector, which serves as the semantic feature; wherein, the aggregation operation uses the vector corresponding to the first character [CLS] of the sequence.

[0009] Preferably, the image features and semantic features are converted into fixed-dimensional image feature vectors and semantic feature vectors, respectively; specifically, this includes: The image features are projected onto a preset common dimension through a projection layer to obtain the fixed-dimensional image feature vector; The semantic features are projected onto the same preset common dimension through a projection layer to obtain the fixed-dimensional semantic feature vector; wherein the image feature vector and the semantic feature vector have the same dimension.

[0010] Preferably, a cross-modal contrastive learning method is employed to map the image feature vectors and semantic feature vectors to a unified semantic vector space, and feature alignment is achieved by optimizing the triplet loss module; specifically, this includes... Cross-modal triples are constructed based on the fixed-dimensional image feature vectors and semantic feature vectors. Each triple includes an anchor sample, a positive sample, and a negative sample. The triples include two types: those anchored by image feature vectors and those anchored by semantic feature vectors. When the image feature vector is used as the anchor, its paired semantic feature vector is a positive sample, and its unpaired semantic feature vector is a negative sample. When the semantic feature vector is used as the anchor, its paired image feature vector is a positive sample, and its unpaired image feature vector is a negative sample. Based on the training triples, a triple loss function is calculated; the loss function is constrained to be that, in the unified semantic vector space, the feature distance between the anchor sample and the positive sample is less than the feature distance between the anchor sample and the negative sample, and the difference between the feature distance between the anchor sample and the positive sample and the feature distance between the anchor sample and the negative sample is greater than a preset boundary value. By minimizing the triplet loss function using the backpropagation algorithm, the parameters of the projection layer are optimized, thereby achieving alignment of image features and text features in a unified semantic space.

[0011] Preferably, the aligned image feature vector and semantic feature vector are concatenated to generate a fused feature vector, specifically including: Obtain the image feature vector and semantic feature vector after feature alignment; The image feature vector and the semantic feature vector are concatenated to obtain a joint feature vector. The concatenation operation involves connecting the image feature vector and the semantic feature vector end to end along the feature dimension to form the fused feature vector. The dimension of the fused feature vector is equal to the sum of the dimensions of the image feature vector and the semantic feature vector.

[0012] Preferably, the method further includes the step of evaluating the state of the power equipment based on the fused feature vector, specifically including: The fused feature vector is input into a pre-trained state classifier, which employs a random forest algorithm. The state classifier outputs the state type of the power equipment, including three classification results: normal state, attention state, and abnormal state. When the classification result is an abnormal state, a device warning message is generated. The warning message includes the device status, the corresponding original image data, and text data.

[0013] This invention discloses a multimodal feature alignment and fusion method for power grid graphic data, which has the following beneficial effects: By using cross-modal contrastive learning and triplet loss constraints, image features and text features are mapped to a unified semantic space, explicitly optimizing the similarity of paired samples and pushing away unpaired samples, which significantly enhances the consistency and discriminative power of cross-modal association between images and text. By concatenating and fusing aligned image feature vectors with semantic feature vectors, the resulting joint feature vector retains both visual appearance details and semantic information of the equipment. This deep fusion of features can more comprehensively and accurately describe the overall condition of power equipment. By inputting the fused feature vectors into a random forest classifier, accurate end-to-end discrimination of device status is achieved from raw image and text data. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention are provided below.

[0015] Figure 1 The flowchart illustrates the method for multimodal feature alignment and fusion of power grid graphic data provided by this invention. Detailed Implementation

[0016] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0017] refer to Figure 1 The multimodal feature alignment and fusion method for power grid graphic data provided by this invention includes the following steps: S1: Acquire image data and corresponding text data of power equipment, and perform standardized preprocessing on the image data and text data; S2: A pre-trained CNN model is used to extract features from image data to obtain image features; a pre-trained BERT language model is used to extract features from text data to obtain semantic features. S3: Convert image features and semantic features into fixed-dimensional image feature vectors and semantic feature vectors, respectively; S4: Employ a cross-modal contrastive learning method to map image feature vectors and semantic feature vectors to a unified semantic vector space, and achieve feature alignment by optimizing the triplet loss module; S5: Concatenate the aligned image feature vector and semantic feature vector to generate a fused feature vector.

[0018] In a preferred embodiment, S1: Acquire image data and corresponding text data of the power equipment, and perform standardized preprocessing on the image data and text data; specifically including: Image data of power equipment and its semantically associated text descriptions are obtained from historical operation and maintenance documents to form image-text matching samples; the image data includes visible light images and infrared thermal images of the equipment, and the text data includes equipment nameplate parameters, inspection records and fault analysis reports. Uniformly scale the image to a preset size and normalize the pixel values of the image; Perform word segmentation, stop word filtering, and special character cleaning on the text data. Use the tokenizer of the pre-trained language model to convert the text into a fixed-length word index sequence, and ensure that the input sequence lengths are consistent through padding or truncation operations.

[0019] Specifically, systematically collect image and text data from the historical operation and maintenance documents of the power system. These data include three typical power equipment images: visible light images and infrared thermal images. At the same time, collect the corresponding text description materials, including equipment nameplate parameters (such as model, rated value), inspection records (including operator observation records), and fault analysis reports (detailed descriptions of equipment anomalies). All these image and text data are paired to ensure that each image sample has a corresponding semantic-related text description, forming a structured training sample set.

[0020] Use OpenCV to standardize the collected images. First, uniformly scale all images to a fixed size of 224×224 pixels, normalize the pixel values of the images, and convert the original pixel value range of 0-255 to a floating-point range of 0-1; Use Jieba to perform word segmentation on the original text, and then remove meaningless stop words (such as "de", "le", etc.) and special characters based on the power domain dictionary. Use the tokenizer dedicated to the BERT model to convert the cleaned text into a fixed-length word index sequence, and set the maximum sequence length to 128 tokens. For text sequences with insufficient length, pad them with zeros, and for overly long sequences, perform truncation processing to ensure that all text inputs have a unified format and dimension.

[0021] In the preferred embodiment, S2: Use a pre-trained CNN model to extract features from the image data to obtain image features. Specifically, Use a pre-trained CNN model as an image feature extractor; Remove the top fully connected layer of the pre-trained CNN model and retain its convolutional layer and pooling layer as the core feature extraction module; Input the preprocessed image data into the core feature extraction module and obtain the feature map output by the last convolutional layer or pooling layer through forward propagation; Perform global average pooling on the feature map to convert it into a one-dimensional feature vector as the image feature. The extracted features include features such as edges, textures, local overheating points, and abnormally low temperature points.

[0022] Among them, the training process of the CNN model is: A historical power equipment image dataset is obtained. Based on the supervision labels in the image dataset, including edges, textures, local hot spots, and abnormally low temperature points, the cross-entropy loss function is used to train the initialized CNN model. The gradient descent algorithm is used to optimize the model to a convergent state, enabling the model to obtain general image feature extraction capabilities, thus forming a pre-trained CNN model.

[0023] First, the pre-trained ResNet-50 model is structurally modified by removing the top global average pooling layer and the subsequent fully connected classification layer, while retaining all convolutional and pooling layers from the initial convolutional layer to the last convolutional layer (usually a conv5_x block), which constitute the core feature extraction module.

[0024] The standardized preprocessed images of power equipment are batched and input into the core feature extraction module. The image data is passed sequentially through all retained convolutional and pooling layers using a forward propagation method. Finally, a feature map of size [B, 2048, 7, 7] is obtained from the output of the last convolutional layer (where B is the batch size, 2048 is the number of channels, and 7×7 is the spatial dimension of the feature map). Next, a global average pooling operation is applied to this feature map. Specifically, for each channel of the feature map (a total of 2048 channels), the average of the feature values ​​at all 7×7 spatial locations is calculated. After this operation, a four-dimensional feature map of [B, 2048, 7, 7] is compressed into a two-dimensional feature vector of [B, 2048]. This 2048-dimensional one-dimensional vector is the image feature representation extracted from the original image.

[0025] In a preferred embodiment, S3: Using a pre-trained BERT language model, features are extracted from the text data to obtain semantic features; specifically including, A pre-trained BERT model was used as the text feature extractor; The preprocessed word index sequence is input into the BERT model, and the last hidden state of the entire sequence is used as the contextual feature of each word. The last hidden state is aggregated to generate a fixed-dimensional overall text representation vector, which serves as the semantic feature; the aggregation operation uses the vector corresponding to the first character [CLS] of the sequence.

[0026] Specifically, the BERT-base model, pre-trained on a large Chinese corpus, was selected as the basic text feature extractor. This model has deeply mastered the grammatical structure, lexical semantics, and contextual knowledge of Chinese through self-supervised learning (such as masked language modeling tasks), and its powerful contextual modeling ability can accurately understand the professional descriptions in power-related texts.

[0027] A preprocessed, fixed-length 128-word index sequence (format: [CLS], token1, token2, ..., token127, [SEP], 0, 0, ..., padded with zeros) is input into the BERT model. Before input, this sequence is converted into three corresponding tensors: a word index tensor, a paragraph index tensor, and an attention mask tensor. The attention mask tensor is used to distinguish between real and padded words. The model performs deep bidirectional encoding on each word in the sequence through forward propagation.

[0028] After the model's forward propagation is complete, the hidden state of the entire sequence is extracted from the output of its last layer (i.e., the 12th layer) of the Transformer encoder. This hidden state is a tensor of dimension [B, 128, 768] (where B is the batch size, 128 is the sequence length, and 768 is the hidden layer dimension of the BERT-base model). Each vector in this tensor (e.g., [B, i, 768], where i ranges from 1 to 127) contains rich semantic information about the corresponding word in the context of the entire text.

[0029] The last hidden state is aggregated to generate a fixed-dimensional vector that represents the global semantics of the entire text. Specifically, the vector corresponding to the first character [CLS] of the sequence is extracted. [CLS] is a special classification token added at the beginning of the BERT input sequence, optimized during pre-training to aggregate the semantic information of the entire sequence for use in downstream classification tasks. The hidden state vector with dimensions [B, 768] corresponding to this [CLS] token is extracted. This 768-dimensional vector represents the semantic features extracted from the original text data, encapsulating device nameplate parameters, inspection records, or fault reports.

[0030] In a preferred embodiment, S3: converting image features and semantic features into fixed-dimensional image feature vectors and semantic feature vectors, respectively; specifically including, Image features are projected onto a preset common dimension through a projection layer to obtain a fixed-dimensional image feature vector. The semantic features are projected onto the same pre-defined common dimension through projection layers to obtain a fixed-dimensional semantic feature vector. The image feature vector and the semantic feature vector have the same dimension. Both projection layers use the ReLU activation function to introduce non-linear transformation capabilities. Through the above processing, the originally dimensionally heterogeneous image features and text features are standardized into two feature vectors with the same dimension.

[0031] In a preferred embodiment, S4: A cross-modal contrastive learning method is employed to map image feature vectors and semantic feature vectors to a unified semantic vector space, and feature alignment is achieved by optimizing the triplet loss module; specifically including, Cross-modal triples are constructed based on fixed-dimensional image feature vectors and semantic feature vectors. Each triple includes an anchor sample, a positive sample, and a negative sample. Triples include two types: those anchored by image feature vectors and those anchored by semantic feature vectors. When an image feature vector is used as the anchor, the paired semantic feature vector is the positive sample, and the unpaired semantic feature vector is the negative sample. When a semantic feature vector is used as the anchor, the paired image feature vector is the positive sample, and the unpaired image feature vector is the negative sample. Specifically, a cross-modal triple dataset is constructed for training. From a training batch, for each image-text paired sample (e.g., a visible light image of a circuit breaker and its corresponding inspection record of "B-phase terminal overheating"), two types of triples are constructed: Image anchor triplet: The 2048-dimensional feature vector of the image is used as the anchor, the 768-dimensional feature vector of the paired text is used as the positive sample, and the feature vector of another unrelated text (such as the text describing the disconnect switch) randomly selected from the same batch is used as the negative sample.

[0032] Text anchor triples: The 768-dimensional feature vector of the text is used as the anchor, the paired image feature vector is used as the positive sample, and another unrelated image feature vector randomly selected from the same batch is used as the negative sample.

[0033] Based on triples, the triple loss function is calculated. The loss function is constrained to be that, in the unified semantic vector space, the feature distance between the anchor sample and the positive sample is less than the feature distance between the anchor sample and the negative sample, and the difference between the feature distance between the anchor sample and the positive sample and the feature distance between the anchor sample and the negative sample is greater than a preset boundary value. The specific formula for calculating the triplet loss function is as follows:

[0034] : Represents the final calculated triplet loss value. is the objective that the model needs to minimize.

[0035] N: Represents the total number of triples in a training batch.

[0036] i: represents the i-th triplet currently being calculated.

[0037] : Represents the feature distance between the anchor sample and the positive sample in the i-th triplet.

[0038] : Represents the feature distance between the anchor sample and the negative sample in the i-th triplet.

[0039] margin: Represents a preset boundary value, which is a constant greater than 0, used to force a minimum distance difference between positive and negative samples.

[0040] []+: is the function to "take a positive value".

[0041] By minimizing the triplet loss function using the backpropagation algorithm and optimizing the parameters of the projection layer, image features and text features are aligned in a unified semantic space.

[0042] In a preferred embodiment, S5: concatenating the aligned image feature vector and semantic feature vector to generate a fused feature vector, specifically including, Obtain the image feature vector and semantic feature vector after feature alignment; The image feature vector and the semantic feature vector are concatenated to obtain a joint feature vector. The concatenation operation is to connect the image feature vector and the semantic feature vector end to end along the feature dimension to form a fused feature vector. The dimension of the fused feature vector is equal to the sum of the dimensions of the image feature vector and the semantic feature vector.

[0043] In a preferred embodiment, the method further includes step S6: evaluating the state of the power equipment based on the fused feature vector, specifically including: The fused feature vector is input into a pre-trained state classifier, which employs a random forest algorithm. The fused feature vector generated from the training dataset and its corresponding device state labels (e.g., normal, attentive, abnormal) are used as training samples to train the random forest classifier offline. The state classifier outputs the state type of the power equipment, including three classification results: normal state, attention state, and abnormal state. When the classification result is an abnormal state, a device warning message is generated. The warning message includes the device status, the corresponding original image data, and text data.

Claims

1. A method for multimodal feature alignment and fusion of power grid graphic data, characterized in that, Includes the following steps: Acquire image data and corresponding text data of power equipment, and perform standardized preprocessing on the image data and text data; A pre-trained CNN model is used to extract features from the image data to obtain image features; a pre-trained BERT language model is used to extract features from the text data to obtain semantic features. The image features and semantic features are respectively converted into fixed-dimensional image feature vectors and semantic feature vectors; A cross-modal contrastive learning method is adopted to map the image feature vector and semantic feature vector to a unified semantic vector space, and feature alignment is achieved by optimizing the triplet loss module; The aligned image feature vector and semantic feature vector are concatenated to generate a fused feature vector.

2. The multimodal feature alignment and fusion method for power grid graphic data according to claim 1, characterized in that, Acquire image data and corresponding text data of power equipment, and perform standardized preprocessing on the image data and text data; specifically including: Image data of power equipment and its semantically associated text descriptions are obtained from historical operation and maintenance documents to form image-text matching samples; wherein, the image data includes visible light images and infrared thermal images of the equipment, and the text data includes equipment nameplate parameters, inspection records and fault analysis reports; The images are scaled to a preset size and the pixel values ​​are normalized. The text data is processed by word segmentation, stop word filtering and special character cleaning. The pre-trained language model's word segmenter converts the text into a fixed-length word index sequence, and padding or truncation operations are used to ensure that the length of the input sequence is consistent.

3. The multimodal feature alignment and fusion method for power grid graphic data according to claim 1, characterized in that, A pre-trained CNN model is used to extract features from the image data to obtain image features; specifically, this includes... A pre-trained CNN model is used as the image feature extractor; Remove the top fully connected layer of the pre-trained CNN model, and retain its convolutional and pooling layers as the core feature extraction module; The preprocessed image data is input into the core feature extraction module, and the feature map output by the last convolutional layer or pooling layer is obtained through forward propagation. The feature map is subjected to global average pooling to convert it into a one-dimensional feature vector, which is used as the image feature.

4. The multimodal feature alignment and fusion method for power grid graphic data according to claim 1, characterized in that, A pre-trained BERT language model is used to extract features from text data to obtain semantic features; specifically, these include: A pre-trained BERT model was used as the text feature extractor; The preprocessed word index sequence is input into the BERT model, and the last hidden state of the entire sequence is obtained as the context feature of each word. The last hidden state is aggregated to generate a fixed-dimensional overall text representation vector, which serves as the semantic feature; wherein, the aggregation operation uses the vector corresponding to the first character [CLS] of the sequence.

5. The multimodal feature alignment and fusion method for power grid graphic data according to claim 1, characterized in that, The image features and semantic features are respectively converted into fixed-dimensional image feature vectors and semantic feature vectors; specifically, this includes, The image features are projected onto a preset common dimension through a projection layer to obtain the fixed-dimensional image feature vector; The semantic features are projected onto the same preset common dimension through a projection layer to obtain the fixed-dimensional semantic feature vector; wherein the image feature vector and the semantic feature vector have the same dimension.

6. The multimodal feature alignment and fusion method for power grid graphic data according to claim 5, characterized in that, A cross-modal contrastive learning method is employed to map the image feature vectors and semantic feature vectors to a unified semantic vector space. Feature alignment is achieved by optimizing the triplet loss module. Specifically, this includes... Cross-modal triples are constructed based on the fixed-dimensional image feature vectors and semantic feature vectors. Each triple includes an anchor sample, a positive sample, and a negative sample. The triples include two types: those anchored by image feature vectors and those anchored by semantic feature vectors. When the image feature vector is used as the anchor, its paired semantic feature vector is a positive sample, and its unpaired semantic feature vector is a negative sample. When the semantic feature vector is used as the anchor, its paired image feature vector is a positive sample, and its unpaired image feature vector is a negative sample. Based on the training triples, a triple loss function is calculated; the loss function is constrained to be that, in the unified semantic vector space, the feature distance between the anchor sample and the positive sample is less than the feature distance between the anchor sample and the negative sample, and the difference between the feature distance between the anchor sample and the positive sample and the feature distance between the anchor sample and the negative sample is greater than a preset boundary value. By minimizing the triplet loss function using the backpropagation algorithm, the parameters of the projection layer are optimized, thereby achieving alignment of image features and text features in a unified semantic space.

7. The multimodal feature alignment and fusion method for power grid graphic data according to claim 1, characterized in that, The aligned image feature vector and semantic feature vector are concatenated to generate a fused feature vector, specifically including... Obtain the image feature vector and semantic feature vector after feature alignment; The image feature vector and the semantic feature vector are concatenated to obtain a joint feature vector. The concatenation operation involves connecting the image feature vector and the semantic feature vector end to end along the feature dimension to form the fused feature vector. The dimension of the fused feature vector is equal to the sum of the dimensions of the image feature vector and the semantic feature vector.

8. The multimodal feature alignment and fusion method for power grid graphic data according to claim 1, characterized in that, The method further includes evaluating the state of power equipment based on the fused feature vector, specifically including: The fused feature vector is input into a pre-trained state classifier, which employs a random forest algorithm. The state classifier outputs the state type of the power equipment, including three classification results: normal state, attention state, and abnormal state. When the classification result is an abnormal state, a device warning message is generated. The warning message includes the device status, the corresponding original image data, and text data.

Images