A power multimodal feature-level fusion method considering data differences
By employing feature extraction, feature assimilation, and weight decision-making methods, the problem of semantic and perceptual capability differences in power multimodal data fusion was solved, achieving more efficient power multimodal data fusion and improving the data quality and recognition accuracy of the power system.
Patent Information
- Application Number
- CN202310445810.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-23
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2043-04-23
AI Technical Summary
Existing power multimodal data fusion methods fail to fully consider the semantic and perceptual differences of multimodal features, resulting in limited fusion effects and difficulty in dealing with data anomalies and noise issues, especially in the context of digital power systems.
Feature extraction is performed using convolutional neural networks and long short-term memory networks. Cosine loss is used to guide feature assimilation and cross loss entropy is used to evaluate perception capabilities. A weight decision module is constructed for weight allocation. Finally, the model is trained through the overall loss function to achieve feature-level fusion of multimodal data.
It effectively improves the perception effect of multimodal data fusion, reduces the decline in perception capability caused by data loss and noise, and enhances the data fusion quality of the power system.
Smart Images
Figure CN116738355B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of power operation and maintenance and safety, and specifically relates to a power multimodal feature-level fusion method that takes into account data differences. Background Technology
[0002] Power multimodal data fusion is an important technical means to fully utilize multimodal data from various aspects of the power grid and promote the digitalization of the power system. Multimodal data fusion includes data-level, feature-level, and decision-level fusion. Among them, feature-level fusion has broad application prospects due to its low information loss and applicability to power multimodal data fusion with different data formats and physical meanings. However, in power scenarios based on multimodal data-driven comprehensive decision-making, the differences between multimodal features limit the effectiveness of fusion perception.
[0003] The differences in multimodal features mainly include semantic differences and perceptual ability differences. Semantic differences refer to the feature differences caused by variations in data format, physical meaning, and their mechanisms of action in power target perception. Semantic differences give multimodal features different description spaces, and the description methods of features within each description space differ. Directly fusing them may limit the fusion effect due to the different feature description methods. Perceptual ability differences refer to the differences in perceptual effects caused by data quality, feature extraction networks, and mechanisms of action. Assigning the same level of trust to all features without considering perceptual ability differences may introduce noise and affect recognition performance. Therefore, fully considering the semantic and perceptual ability differences of multimodal features is of great significance in the multimodal feature fusion process.
[0004] To address the differences in perception capabilities during power multimodal feature fusion, existing research focuses on anomaly detection, cleaning, and correction of structured data. These operations are treated as preprocessing steps in data fusion, assuming the data is complete. However, preprocessing operations like cleaning and correction struggle to guarantee data quality; furthermore, these methods cannot handle inherent differences in perception capabilities caused by feature extraction methods and target perception mechanisms. Therefore, considering the differences between multimodal data during fusion remains a crucial issue. Meanwhile, existing power multimodal fusion methods primarily focus on structured data, with limited consideration for scenarios involving both structured and unstructured data, and between unstructured and unstructured data, limiting their adaptability to digital power system scenarios. Therefore, this invention proposes a power multimodal feature-level fusion method that simultaneously considers both semantic and representational differences in data, using power images and multi-parameter time series data as the fusion objects. Summary of the Invention
[0005] This invention provides a power multimodal feature-level fusion method that considers data differences, solving the problem that existing power multimodal data fusion methods do not adequately consider the semantic and perceptual differences between multimodal data, resulting in limited fusion effects and difficulty in dealing with data anomalies and noise.
[0006] This invention provides a power multimodal feature-level fusion method that considers data differences, comprising the following steps:
[0007] S1. Extract features from the multi-parameter time series of electricity to obtain the time series feature vector;
[0008] Further, in step S1, feature extraction is performed on the multi-parameter time series of electricity, specifically including:
[0009] S11. Arrange the power multi-parameter time series as X p×n Where p is the number of power parameters; n is the length of the time series.
[0010] S12. Feature extraction for multi-parameter time series data of electricity can be performed using convolutional neural networks and long short-term memory networks.
[0011] S13. Add a fully connected layer at the end of the feature extraction network and use it as the feature f of the power multi-parameter time series. t ,
[0012] S2. Extract features from the power image data to obtain image feature vectors;
[0013] Further, in step S2, feature extraction is performed on the power image data, specifically including:
[0014] S21. Perform feature extraction on the power image data using a convolutional neural network.
[0015] S22. Add a fully connected layer at the end of the feature extraction network and use it as the feature f of the power image. i ,
[0016] S3. Feature assimilation is performed on the power time series feature vector and the power image feature vector, specifically including:
[0017] As shown in the following equation, starting from the angle difference, a cosine loss is constructed to guide the search for the joint representation space, thereby performing feature assimilation:
[0018] l a =1-cos(f t ,f i (1)
[0019] Among them, l a For cosine loss, cos(f) t ,fi The cosine distance between the two types of features is represented by the equation f(x). Within the range [0, π], the larger the angle difference between the two features, the smaller the cosine value. To minimize the loss, the constructed loss function is shown above. After optimizing the two types of features using the above equation, f(x) = ... t and f i Since the angle difference is small, it can be assumed that after feature assimilation, the two types of features are mapped to the joint representation space and have the same feature description method.
[0020] S4. Evaluate the perceptual ability based on the two types of feature vectors respectively, and then perform fusion weight allocation based on this. The specific steps are as follows:
[0021] S41. Evaluate the perception capabilities of multi-parameter time series features and power image features respectively. If it is a classification problem, in f t A fully connected layer is then added, and the sensing capability is evaluated using cross-loss entropy, as shown in equation (2):
[0022]
[0023] Among them, l c P is the crossover loss entropy; i The model prediction result; P i * n represents the true label of the sample; n is the number of samples, P i The softmax method is used for calculation, as shown in equation (3).
[0024]
[0025] Where x = [x1, x2, ... x m ] is the input to the softmax function, determined by the network parameters preceding the softmax function; y represents the label category; m is the number of categories.
[0026] If it is a regression problem, a fully connected layer can be added, and iterative training can be performed with the goal of minimizing the mean squared error until convergence, as shown in equation (4).
[0027]
[0028] Where MSE is the mean square error function; y i The model prediction result; y *i Here are the true labels of the samples; n is the number of samples.
[0029] S42. Construct a weight decision module and weight loss to make weight decisions for binary modal features.
[0030] In terms of model structure, such as Figure 1As shown in the weighted decision module, the constructed weighted decision structure first concatenates the power multi-parameter time series features and image features, then passes them through two 2048-dimensional and one 2-dimensional fully connected layers respectively, and finally uses softmax to calculate the two-class weights w. t and w i ,
[0031] As shown in equation (3), the sum of the probabilities of each category calculated by softmax is 1. Therefore, using this method can make the sum of the fusion weights corresponding to the two types of data equal to 1.
[0032] Regarding the loss function, as shown in equation (5), the present invention constructs a loss function l w The guiding model makes fusion weight decisions based on power target perception capabilities. Among them, l s and l i The perceptual losses are the power multi-parameter time series and the power image, respectively. In order to minimize the loss, the model will make the two square terms in Equation (5) tend to 0, so that the branch with a larger cross loss entropy corresponds to a smaller weight, while the branch with a smaller loss corresponds to a larger weight. That is, the branch with a stronger power target perception ability corresponds to a larger fusion weight, and the branch with a weaker power target perception ability corresponds to a smaller fusion weight.
[0033]
[0034] S5. Based on weights, multimodal features are concatenated and fused, and further feature extraction and power target perception are performed, specifically including:
[0035] First, the two types of features are multiplied by their corresponding weights to obtain the weighted multi-parameter time series data features w of the power system. t f t and image features w i f i Then, the two types of features are concatenated to obtain 2048-dimensional features. Next, the fused features are processed through two 4096-dimensional layers, and corresponding fully connected layers and loss functions (cross loss entropy or mean squared error) are added.
[0036] S6. Construct the overall loss function and train the model, specifically including:
[0037] Taking into account feature extraction, feature assimilation, weight decision mechanism, and power target perception to construct loss function, and aiming to minimize overall loss, the model is trained.
[0038] l t =l main +αl a +βl w +γlt +δl i (6)
[0039] Wherein, the hyperparameters α, β, γ and δ are respectively l a l w l s and l i The corresponding weight, l main For power target perception loss based on fusion.
[0040] The beneficial effects of this invention are as follows: Based on two types of multimodal data, namely power multi-parameter time-series data and power image data, this invention proposes a feature assimilation method for the semantic differences of multimodal data, and a feature-level fusion method for the two types of data based on the differences in perception ability and weight decision mechanism. This solves the problem of limited fusion effect of previous multimodal data, and helps to avoid the problem of decreased perception ability caused by data loss, noise and other factors in power data fusion. Attached Figure Description
[0041] Figure 1 This is a diagram of the power multimodal feature-level fusion method that takes into account data differences, as described in this invention.
[0042] Figure 2 This is a graph illustrating the extraction and perception capability assessment of multi-parameter time series features of electricity in this embodiment of the invention.
[0043] Figure 3 This is an image for evaluating the power image feature extraction and perception capabilities in an embodiment of the present invention. Detailed Implementation
[0044] The invention will now be further described with reference to the accompanying drawings:
[0045] This invention provides a power multimodal feature-level fusion method that considers data differences. Based on two types of multimodal data, power multi-parameter time-series data and power image data, it performs fusion perception of power scenarios.
[0046] This invention first extracts features from power multi-parameter time-series data and power image data respectively; then, it constructs a cosine loss function for feature assimilation to address the semantic differences between the two types of data; next, it constructs separate perception capability assessment modules to evaluate the perception capabilities of the two modalities and calculates fusion weights based on this; finally, it concatenates and fuses the features of the two modalities based on the weights, and performs further feature extraction and power target perception. This invention fully considers the semantic and perception capability differences between power multi-modal data, effectively improving the fusion perception effect of power multi-modal data and effectively addressing problems such as poor data quality and noise.
[0047] This invention takes icing identification on transmission lines as an example. It fuses six types of time-series data, including temperature, humidity, instantaneous wind speed, maximum tension, wind deflection angle at maximum tension, and tilt angle at maximum tension, with icing image data, and then classifies the icing thickness. This embodiment comprehensively considers the degree of icing on the lines and insulators in the collected samples, as well as engineering experience, and abstracts the icing thickness level identification into a five-class classification problem, where category 1 is the case of no icing, and categories 2-5 represent progressively increasing icing thickness.
[0048] The training set used in this embodiment consists of 2875 pairs, and the test set consists of 450 pairs. The data distribution is shown in Table 1.
[0049] Table 1 Data Distribution
[0050]
[0051] The specific steps are as follows:
[0052] Step 1: Extract features from the multi-parameter time series of power data to obtain time series feature vectors, and evaluate their perceptual capabilities. The network used is as follows: Figure 2 As shown in the figure. First, the six types of multi-time series with a length of 20 are converted into multi-parameter recursive graphs of size 20×20 and dimension 20, as shown in Equation (7). Then, they are passed through a convolutional neural network of size 32×5×5 and a max pooling layer of size 2×2 respectively. Finally, they are connected to a fully connected layer of size 1024 and used as the feature f of the multi-parameter time series of electricity. t Next, the feature vector f t The input is fed into fully connected layers of sizes 1024 and 5, and then connected to a softmax layer to evaluate the ice thickness sensing capability of the above six types of data.
[0053]
[0054]
[0055] Among them, R ij The element in the i-th row and j-th column of the multi-parameter recursive graph represents the relationship between the i-th and j-th time points. The power multi-parameter sampled value represents the value at time i.
[0056] Step 2 involves extracting features from the icing image to obtain image feature vectors, and then evaluating their perceptual capabilities. The network used is as follows: Figure 3 As shown. This embodiment uses Faster R-CNN to extract features from icing image data, and uses the first fully connected layer after ROI Pooling as the icing image feature. After feature extraction, image data features f of size 1024 are obtained.i Next, the feature vector f i The input is fed into fully connected layers of sizes 1024 and 5, and then connected to a softmax layer to evaluate the ice thickness sensing capability of the above six types of data.
[0057] Step 3, for f t and f i The data are concatenated and fused according to weights, and then further feature extraction is performed on the fused features through two fully connected layers of size 4096 and one of size 5. Finally, softmax is used to classify the ice thickness.
[0058] Step 4: Set the weights of each loss term in equation (6) and train the network.
[0059] Step 5: Test the trained model.
[0060] In this implementation, precision (P), recall (R), average precision (AR), and average recall (AR) are used as evaluation metrics for model testing. To verify the effectiveness of this invention in addressing the variability of multimodal data, this embodiment compares models with and without feature assimilation, as well as models with and without weighted decision mechanisms. The comparison results are shown in Tables 2 and 3.
[0061] Table 2 Comparison of Feature Assimilation Perception Icing Recognition Performance
[0062]
[0063] Table 3 Comparison of Icing Recognition Performance of Fusion Weighted Decision Mechanism
[0064]
[0065] As shown in Table 2, the icing perception effect of the present invention is higher than that of the fusion perception effect without feature assimilation, verifying the effectiveness of the present invention in addressing semantic differences in power multimodal fusion. Meanwhile, the icing thickness recognition effects based on sensor data and image data in Table 2 are significantly lower than those based on the icing perception effect of two types of data fusion, further verifying the effectiveness of multimodal fusion in power scenarios.
[0066] To verify the effectiveness of this invention in sensing differences across different modalities of data, this invention processed noise in both sensor data and image data, and then directly spliced and fused them. This was compared with the weighted fusion method of this invention. Table 3 shows that the weighted fusion method of this invention achieved the best icing thickness recognition performance in scenarios without noise processing, with noise processing of sensor data, and with noise processing of image data. This indicates that this invention can better address the differences in sensing capabilities in power multimodal fusion.
[0067] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. A power multimodal feature-level fusion method considering data differences, characterized in that, Includes the following steps: S1. Extract features from the multi-parameter time series of electricity to obtain the time series feature vector; S2. Extract features from the power image data to obtain image feature vectors; S3. Assimilate the feature vectors of the power time series and the power image feature vectors; S4. Assess perception capabilities based on two types of feature vectors respectively, and then assign fusion weights accordingly. The specific steps are as follows: S41. Evaluate the perception capabilities of multi-parameter time series features and power image features respectively. If it is a classification problem, in f t A fully connected layer is then added, and the sensing capability is evaluated using cross-loss entropy, as shown in equation (2): Among them, l c P is the crossover loss entropy; i The model prediction result; P i * n represents the true label of the sample; n is the number of samples, P i The softmax method is used for calculation, as shown in equation (3). Where x = [x1, x2, ... x m ] is the input to the softmax function, determined by the network parameters preceding the softmax function; y represents the label category; m is the number of categories. If it is a regression problem, a fully connected layer can be added, and iterative training can be performed with the goal of minimizing the mean squared error until convergence, as shown in equation (4). Where MSE is the mean square error function; y i The model prediction result; y *i Here are the true labels of the samples; n is the number of samples. S42. Construct a weight decision module and weight loss to make weight decisions for binary modal features. In terms of model structure, the constructed weighted decision structure first concatenates the multi-parameter time series features of electricity and image features, then passes them through two 2048-dimensional and one 2-dimensional fully connected layers respectively, and finally uses softmax to calculate the binary weights w. t and w i , As shown in equation (3), the sum of the probabilities of each category calculated by softmax is 1. Therefore, using this method can make the sum of the fusion weights corresponding to the two types of data equal to 1. Regarding the loss function, as shown in equation (5), the present invention constructs a loss function l w The guidance model makes fusion weight decisions based on power target perception capabilities, where l s and l i The perceptual losses are the power multi-parameter time series and the power image, respectively. To minimize the loss, the model will make the two squared terms in equation (5) tend to 0, so that the branch with a larger cross-loss entropy corresponds to a smaller weight, while the branch with a smaller loss corresponds to a larger weight. That is, the branch with a stronger power target perception ability corresponds to a larger fusion weight, and vice versa. S5. Multimodal features are spliced and fused based on weights, and further feature extraction and power target perception are performed. S6. Construct the overall loss function and train the model.
2. The power multimodal feature-level fusion method considering data differences according to claim 1, characterized in that, In step S1, when extracting features from the multi-parameter power time series, the multi-parameter power time series is first arranged into a multi-parameter power time matrix X with p parameters and n time series length. p×n Then, feature extraction is performed on the multi-parameter time series of power data. Convolutional neural networks and long short-term memory networks can be used. Finally, a fully connected layer is added at the end of the feature extraction network, and this layer is used as the feature f of the multi-parameter time series of power data. t .
3. The power multimodal feature-level fusion method considering data differences according to claim 2, characterized in that, In step S2, a convolutional neural network is used for feature extraction, and a fully connected layer is added at the end of the feature extraction network, which is then used as the feature f of the power image data. i .
4. The power multimodal feature-level fusion method considering data differences according to claim 1, characterized in that, In step S3, the feature assimilation method for the power time series feature vector and the power image feature vector specifically includes the following: Construct a cosine loss as shown in equation (1) for feature assimilation. l a =1-cos(f t ,f i ) (1) Among them, l a For cosine loss, cos(f) t ,f i The cosine distance between the two types of features is represented.
5. The power multimodal feature-level fusion method considering data differences according to claim 1, characterized in that, In step S5, the two types of features are first multiplied by their corresponding weights to obtain the weighted power multi-parameter time series data features w. t f t and image features w i f i Then, the two types of features are concatenated to obtain 2048-dimensional features. Next, the fused features are processed through two 4096-dimensional layers, and corresponding fully connected layers and loss functions are added.
6. The power multimodal feature-level fusion method considering data differences according to claim 1, characterized in that, In step S6, considering feature extraction, feature assimilation, weight decision mechanism, and power target perception loss function, an overall loss function is constructed. l t =l main +αl a +βl w +γl t +δl i (6) Wherein, the hyperparameters α, β, γ and δ are respectively l a l w l s and l i The corresponding weight, l main For power target perception loss based on fusion.