Infrared image recognition method and system for power equipment based on feature fusion

Abstract

The application discloses a power equipment infrared spectrum recognition method and system based on feature fusion, collects multiple infrared images of different angles and constructs a data set; a deep learning network is used to extract features of the multi-angle infrared images, and features of the infrared images of different angles are fused; according to the definition and integrity of the target in the infrared images of different angles, the weight of each angle feature is adjusted, the gradient value of the infrared image is calculated based on a Sobel operator as a definition index, and the integrity index is obtained based on the target area ratio; the model is trained through the data set, and the center point coordinates of each component in the power equipment are predicted. The multi-angle image acquisition and adaptive feature fusion strategy is adopted, the image features of different angles are integrated through a feature pyramid network (FPN), and an adaptive weight distribution mechanism is designed, the weight of each angle is dynamically adjusted according to the target definition and integrity, and the information loss caused by the angle change is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of power equipment detection technology. It relates to a method and system for infrared spectrum identification of power equipment based on feature fusion. Background Technology

[0002] Infrared thermal imaging technology for power equipment is an important means of equipment condition monitoring. By detecting the surface temperature distribution of equipment, potential faults such as overheating can be detected in a timely manner. Traditional methods for processing infrared images of power equipment mainly include thresholding, edge detection, and region growing. In recent years, deep learning technology has made significant progress in the field of image processing, especially object detection and semantic segmentation algorithms based on convolutional neural networks (CNNs), which have been widely used in infrared image analysis. For example, the Mask R-CNN algorithm, as an advanced instance segmentation model, can simultaneously detect targets and generate accurate segmentation masks. The YOLO series of algorithms, on the other hand, performs excellently in real-time applications due to their fast detection speed.

[0003] Currently, the field of infrared image bounding box recognition for power equipment mainly includes the following:

[0004] An infrared image segmentation method based on an improved Mask R-CNN: This method replaces some residual modules in the ResNet feature extraction network with deformable convolutions, enhancing the model's adaptability to target geometric deformations; simultaneously, it introduces a parallel-connected attention mechanism module to improve feature extraction capabilities. The improved Mask R-CNN significantly improves the instance segmentation accuracy of substation equipment in infrared images, but it still has certain limitations when processing images with complex backgrounds and those taken from multiple angles.

[0005] Infrared target detection method based on key point detection: This method directly optimizes the coordinates of the target's center point and obtains multi-scale information of the target by designing a feature fusion module from low to high levels. This method has a high detection rate and a low false alarm rate, but it is mainly effective for small target detection and performs poorly in identifying bounding boxes of large power equipment.

[0006] A YOLOv5-based method for detecting power equipment in infrared images: This method improves the detection accuracy of power equipment in infrared images by modifying the YOLOv5 backbone network and loss function. However, this method mainly focuses on target detection and has limited ability to finely partition the internal regions of the equipment.

[0007] The existing technology mainly has the following problems:

[0008] (1) Insufficient adaptability to angle changes: The existing technology is not adaptable to changes in installation angle and shooting angle. When the above angles change, the relative position consistency of the generated partition frame is poor, which will have an adverse effect on the subsequent temperature analysis and fault diagnosis process.

[0009] (2) Failure to consider the positional correlation between regions: Existing technologies focus on the detection of a single region without considering the relative positional correlation between regions. When the device is in a complex occlusion scenario or the layout changes, it is easy to cause errors in the determination of the partition position, disordered order and missing information.

[0010] (3) Failure to utilize temperature features: Existing technologies rely heavily on the visual features of images and fail to fully explore the role of key information such as temperature gradients in infrared images in indicating the boundaries of partitions, thus limiting the improvement of partition accuracy. Summary of the Invention

[0011] The purpose of this invention is to provide a method and system for infrared image recognition of power equipment based on feature fusion. This invention adopts a multi-view image acquisition and adaptive feature fusion strategy, integrates image features from different angles through a feature pyramid network (FPN), and designs an adaptive weight allocation mechanism to dynamically adjust the weights of each viewpoint according to the clarity and integrity of the target, effectively reducing information loss caused by angle changes.

[0012] The technical solution to achieve the purpose of this invention is as follows:

[0013] An infrared image recognition method for power equipment based on feature fusion includes the following steps:

[0014] Collect infrared images from multiple different perspectives and construct a dataset;

[0015] Deep learning networks are used to extract features from multi-view infrared images and to fuse the features of infrared images from different viewpoints.

[0016] Based on the sharpness and integrity of the target in infrared images from different perspectives, the weights of the features of each perspective are adjusted, the gradient value of the infrared image is calculated based on the Sobel operator as a sharpness index, and the integrity index is obtained based on the proportion of the target area.

[0017] The model is trained using a dataset to predict the coordinates of the center points of various components in power equipment.

[0018] In the preferred technical solution, the following is included before feature extraction:

[0019] Data augmentation is performed on the original infrared image, preserving the single-channel grayscale information and discarding the pseudo-color mapping channel; the grayscale values ​​are mapped to the [0,1] interval, as shown in the following formula:

[0020]

[0021] in, This represents the infrared image pixel value mapped to the [0,1] interval after normalization. Represents the grayscale value of the original infrared image; Represents the maximum value in the grayscale of the original infrared image. This represents the maximum grayscale value in the original infrared image;

[0022] An adaptive median filter is used, with a small window for edge regions and a large window for flat regions.

[0023] In the preferred technical solution, a deep learning network is used to extract features from multi-view infrared images, and the features of infrared images from different viewpoints are fused, including:

[0024] ResNet50 was used as the backbone network for multi-view feature extraction to extract feature maps at different levels.

[0025] Through a top-down upsampling process, the high-level feature map is upsampled to the same resolution as the low-level feature map, and then fused with the low-level feature map through lateral connections to form a multi-scale feature representation.

[0026] Cross-view attention enhancement is applied to feature maps of different scales.

[0027] In the preferred technical solution, adjusting the weights of features from different viewing angles based on the sharpness and integrity of the target in infrared images from different perspectives includes:

[0028] The formula for calculating the sharpness index is:

[0029]

[0030] in, For the first Pixels in a view image gradient magnitude, This represents the total number of pixels in the image.

[0031] The formula for calculating the integrity index is:

[0032]

[0033] in, For the first The pixel area of ​​the target component in the view image. The total pixel area of ​​the image;

[0034] Using linear weighting to calculate the first The weight of the viewpoint :

[0035]

[0036] in, , Weighting coefficients.

[0037] In the preferred technical solution, after predicting the center point coordinates of each component in the power equipment, the method further includes using the inherent positional relationships between components to correct any missing or incorrect positions of each component in the equipment and optimize the component position coordinates.

[0038] In the preferred technical solution, optimizing the component position coordinates includes:

[0039] Construct a power equipment component relationship graph, defining nodes as information for each equipment component, and defining edges as spatial constraint information between components. If the adjacency matrix... The adjacency matrix indicates that there is a spatial constraint relationship between node i and node j. 0 indicates that there is no spatial constraint relationship between node i and node j;

[0040] Construct a graph convolutional neural network to learn the geometric relationships between components, predict the relative positional relationships between components, fill in missing nodes by the position of candidate nodes, and participate in the weight calculation of the graph convolutional neural network.

[0041] The loss between the predicted and actual values ​​is calculated, and the spatial distance loss and orientation constraint loss are used for backpropagation of the network. The corrected and optimized position coordinates of each component are obtained through the trained graph convolutional neural network model.

[0042] In the preferred embodiment, the graph convolutional neural network includes: an input layer, a first hidden layer, a second hidden layer, and an output layer;

[0043] The input layer consists of n-dimensional node feature information;

[0044] The first hidden layer performs graph convolution operations:

[0045]

[0046] in, This represents the feature representation of node i in the first hidden layer (the feature after graph convolution). It is an activation function used to introduce nonlinearity and solve the problem of "neuron death" in the negative region of ordinary ReLU.

[0047] The adjacency matrix element indicates whether there is an edge between nodes i and j (1 for an edge and 0 for no edge in an unweighted graph, and the weight of the edge in a weighted graph).

[0048] This represents the neighbor set of node i (all nodes directly connected to i).

[0049] This represents the degree of node i (the number of edges connected to i).

[0050] This represents the initial features of node j (the original features of the input layer).

[0051] This represents the weight matrix of the first hidden layer.

[0052] This represents the bias term of the first hidden layer, used to adjust the offset of features.

[0053] The second hidden layer introduces an attention mechanism:

[0054]

[0055]

[0056] in, The value represents the influence weight (attention score) of neighbor j on node i, reflecting the model's "attention level" to different neighbors.

[0057] This represents the attention weight matrix, used to learn the attention interaction patterns between nodes.

[0058] It is an activation function used to perform a non-linear transformation on the attention score.

[0059] It is in the form of a Softmax function used to normalize the attention score (the sum of the weights of all neighbors is 1).

[0060] This represents the feature representation of node i in the second hidden layer (features after attention-weighted aggregation).

[0061] This represents the weight matrix of the second hidden layer.

[0062] This represents the bias term of the second hidden layer.

[0063] This indicates that the features of neighbor j are weighted by attention. Weighting reflects the attention mechanism idea that "important neighbors contribute more information".

[0064] Output layer prediction position correction ;in This represents the X-coordinate offset of the position of key point i of the component relative to the input position. This represents the offset of the position of the component's key point i relative to the input position in the Y coordinate.

[0065] If only one of a certain type of component is detected in the initial point set (the coordinates of N key points of the input components), it is marked as a missing node. Based on the positions of the existing nodes and standard constraints (selecting a standard image for each type of device, setting the reference positions of each component under the standard image and the positional relationships between components), the positions of candidate nodes are generated:

[0066]

[0067] in, Indicates the coordinates of the candidate node position. Indicates the coordinates of the reference node position. This represents the standard distance between the reference node and the missing node. This represents the standard direction angle from the reference node to the missing node. Indicates the angle value;

[0068] Add the candidate node to the power equipment component relationship graph, calculate the sum of the edge weights between it and its neighboring nodes, and select the candidate node with the largest weight as the completion node.

[0069] In a preferred technical solution, utilizing the inherent positional relationships between components further includes determining component partition boundaries, wherein determining the component partition boundaries includes:

[0070] Temperature features in infrared images are extracted using a constructed thermal feature extraction network, resulting in multiple thermal feature maps.

[0071] By calculating the correlation coefficient between each feature map and the original temperature matrix, feature maps with a correlation coefficient greater than a threshold are selected as key thermal feature maps.

[0072] The selected key thermal feature maps are enhanced, and threshold segmentation is performed based on the enhanced key thermal feature maps. The segmentation threshold is determined by the maximum inter-class variance method, and the feature maps are divided into target candidate regions and background regions.

[0073] Based on the obtained coordinates of the equipment components, the initial component bounding boxes are calculated in combination with the size and image resolution of different equipment components. The proportion of the equipment region in the initial component bounding box is calculated. The candidate equipment component regions are adjusted according to the key thermal feature map. The rectangular candidate box with the highest equipment proportion is extracted. This rectangular candidate box is the optimal partition boundary of the power equipment component.

[0074] In a preferred embodiment, the constructed thermal feature extraction network includes three convolutional layers and two pooling layers.

[0075] The temperature matrix of the infrared image is obtained, and the temperature matrix is ​​input into the first convolutional layer for convolution operation to obtain the first feature map.

[0076] The first pooling layer is connected after the first convolutional layer to perform pooling on the resulting first feature map, thus preserving key features.

[0077] The output of the first pooling layer is input into the second convolutional layer for convolution operation to obtain the second feature map;

[0078] A second pooling layer is connected after the second convolutional layer to perform pooling processing on the second feature map;

[0079] The output of the second pooling layer is input into the third convolutional layer for convolution operation to obtain the thermal feature map.

[0080] This invention also discloses a feature fusion-based infrared image recognition system for power equipment, used to perform the feature fusion-based infrared image recognition method for power equipment as described in any of the preceding claims, comprising:

[0081] The dataset building module collects infrared images from multiple different perspectives and builds a dataset;

[0082] The multi-view feature extraction module uses a deep learning network to extract features from multi-view infrared images and fuses the features of infrared images from different viewpoints.

[0083] The adaptive weight allocation module adjusts the weights of features from different perspectives based on the sharpness and integrity of the target in infrared images from different perspectives. It calculates the gradient value of the infrared image based on the Sobel operator as a sharpness index and obtains the integrity index based on the proportion of the target area.

[0084] The model training and prediction module trains the model using a dataset and predicts the coordinates of the center points of each component in the power equipment.

[0085] The present invention also discloses a computer storage medium storing a computer program thereon, wherein when the computer executes the computer program, it implements the infrared spectrum recognition method for power equipment based on feature fusion as described above.

[0086] The present invention also discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program stored in the memory. When the computer program is executed, it implements the infrared spectrum recognition method for power equipment based on feature fusion as described above.

[0087] Compared with the prior art, the significant advantages of this invention are:

[0088] This invention addresses three key issues in existing infrared detection technologies for power equipment through three core technological innovations: multi-view feature fusion, graph convolutional network position constraints, and thermal feature enhancement, achieving significant technical results.

[0089] I. Significantly improves adaptability to angle changes and ensures consistency of partition positions.

[0090] Existing technologies suffer from significant errors in the relative position of bounding boxes when the device installation angle or shooting perspective changes. This invention employs a multi-view image acquisition and adaptive feature fusion strategy. It integrates image features from different angles through a Feature Pyramid Network (FPN) and designs an adaptive weight allocation mechanism to dynamically adjust the weights of each viewpoint based on target sharpness and integrity. This method effectively reduces information loss caused by angle changes. Experimental results show that within a ±30° viewing angle variation range, the bounding box position consistency error of this invention is reduced by 25% compared to traditional single-view methods, significantly outperforming the sensitivity of existing technologies to angle changes.

[0091] II. Solving the problem of accurate or incorrect partition locations in complex scenarios through spatial relationship modeling.

[0092] To address the shortcomings of existing technologies that neglect the positional relationships between partitions, this invention innovatively proposes a Graph Convolutional Network (GCN) method for constructing relative positional constraints. This method treats device partitions as nodes and spatial relationships as edges, learning the geometric relationships between partitions through GCN. This design enables the model to capture the topological structure between partitions. In complex scenarios with 30% occlusion, the error rate of partition position determination in this invention is reduced by 30% compared to traditional methods, and the phenomenon of disordered order is reduced by 55%, effectively solving the problem of isolated decision-making caused by the detection of a single region.

[0093] III. Integrating thermal and visual features significantly improves the accuracy of partition boundaries.

[0094] This invention designs a dedicated thermal feature extraction network that captures key information such as temperature gradients and hotspot distribution through a convolutional neural network, fully utilizing the unique temperature indication information of infrared images. Experimental data shows that after fusing thermal features, the pixel-level classification accuracy of partition boundaries is improved by 35%, especially in key heat-generating component areas such as busbar joints and circuit breakers, improving the accuracy of partition box boundaries for key components. Attached Figure Description

[0095] Figure 1 The flowchart shows a method for infrared image recognition of power equipment based on feature fusion.

[0096] Figure 2 The flowchart shows the position constraint method based on graph convolutional networks (GCN).

[0097] Figure 3Flowchart of the method for determining partition boundaries with enhanced thermal features. Detailed Implementation

[0098] The principle of this invention is as follows: This invention combines spatial-thermal feature fusion, and through multi-view feature fusion, explicit modeling of the relative positional relationship between device regions and the introduction of temperature gradient-driven thermal feature constraints, it aims to effectively improve the stability and reliability of partition box recognition in complex scenarios, and achieve efficient and accurate recognition of each partition of infrared power equipment.

[0099] Example:

[0100] like Figure 1 As shown, a method for infrared image recognition of power equipment based on feature fusion includes the following steps:

[0101] Collect infrared images from multiple different perspectives and construct a dataset;

[0102] Deep learning networks are used to extract features from multi-view infrared images and to fuse the features of infrared images from different viewpoints.

[0103] Based on the sharpness and integrity of the target in infrared images from different perspectives, the weights of the features of each perspective are adjusted, the gradient value of the infrared image is calculated based on the Sobel operator as a sharpness index, and the integrity index is obtained based on the proportion of the target area.

[0104] The model is trained using a dataset to predict the coordinates of the center points of various components in power equipment.

[0105] Specifically, the implementation steps include the following:

[0106] Step 1: Acquire multi-view infrared images and construct a dataset (S101)

[0107] Use a resolution of at least 640 The 512 infrared thermal imager takes the geometric center of the equipment (transformer, circuit breaker, etc.) as the origin and takes pictures in eight directions: 0° (front), 45°, 90° (side), 135°, 180° (back), 225°, 270° (side), and 315°.

[0108] Data augmentation of the original image mainly includes the following methods: 1) Generating random angles within a range of ±15° (step size 1°) to simulate a wider range of angular deviations; 2) Scaling at 0.9, 1.0, and 1.1 times the scale (while maintaining 640). 512 output), simulating shooting distance fluctuations. 3) Add random cropping and tilting to simulate the effects of handheld shaking.

[0109] The acquired images were divided into a training sample dataset and a detection dataset in a 7:3 ratio. The training sample dataset was used for model training, and the detection dataset was used for actual recognition and verification.

[0110] Step 2: Image data preprocessing (S102)

[0111] To preserve the single-channel grayscale information of the infrared image, and to adapt to the ResNet50 network structure, the input image resolution was scaled to 224. 224. Grayscale values ​​are positively correlated with temperature, so the pseudo-color mapping channel is discarded. Grayscale values ​​are mapped from [0,255] to the [0,1] range using the following formula:

[0112]

[0113] in, This represents the infrared image pixel value mapped to the [0,1] interval after normalization. Represents the grayscale value of the original infrared image; Represents the maximum value in the grayscale of the original infrared image. This represents the maximum grayscale value in the original infrared image;

[0114] Adaptive median filtering is used, with a window size of 3. 3~7 7. Maintain 3 for edge areas 3 small windows are used to protect details, and 7 is used for flat areas. Seven windows enhance noise reduction.

[0115] Step 3: Multi-view feature extraction based on FPN (S103)

[0116] This invention employs deep learning networks to extract features from multi-view images and uses a Feature Pyramid Network (FPN) for multi-scale feature fusion. Specifically:

[0117] ResNet50 is used as the backbone network for multi-view feature extraction. The main design is as follows: Shallow features C3 layer output (28 28 256), deep feature C5 layer output (7 7 2048).

[0118] Feature maps at different levels are fused using an FPN structure. Through a top-down path and lateral connections, high-level semantic features are fused with low-level detailed features to form a multi-scale feature representation.

[0119] We designed a multi-scale cross-view feature enhancement (MSCFE) module to enhance the cross-view attention of feature maps at different scales, thereby further improving the expressive power of the features.

[0120] Step 4: Adaptive weight allocation method (S104)

[0121] To avoid interference from low-quality viewpoints (such as blurred or occluded images) on the fusion results, an adaptive weight allocation method based on sharpness and integrity is designed:

[0122] Clarity metric: Calculated based on the Sobel gradient; a larger gradient indicates a sharper image. The formula is:

[0123]

[0124] in, For the first Pixels in a view image gradient magnitude, This represents the total number of pixels in the image.

[0125] (2) Integrity Index: Calculated based on the proportion of the target area. The more complete the target area (unobstructed, untrunculated), the higher the weight. The formula is:

[0126]

[0127] in, For the first The pixel area of ​​a target component (such as a transformer bushing) in a viewpoint image. This represents the total pixel area of ​​the image.

[0128] (3) Weight fusion: The first weight is calculated using linear weighting. The weight of the viewpoint The formula is:

[0129]

[0130] in, , 4.

[0131] Step 5: Model Training and Result Output (S105)

[0132] The model is trained using a training dataset to predict the coordinates of the center points of various components in the equipment (such as the transformer body, bushings, etc.). The output format is (label, x, y, conf), where label is the component label and conf is the confidence score.

[0133] In another embodiment, this method utilizes the inherent positional relationships between components to correct missing or incorrect positions of components in the device and optimize component position coordinates. For example... Figure 2 As shown.

[0134] Step 1: Constructing the component relationship diagram (S201)

[0135] First, the nodes and edges in the component relationships of each type of equipment are defined. Nodes are defined to represent the information of each equipment component, including component code, component position coordinates, etc. Edges are defined to represent the spatial constraint information between components, such as "Sleeve A phase is located above the body" and "Sleeve B phase is located to the right of sleeve A", where the up, down, left, and right directions are represented by "-y", "+y", "-x", and "+x".

[0136] Adjacency Matrix The adjacency matrix indicates that there is a spatial constraint relationship between node i and node j. 0 indicates that there is no spatial constraint relationship between node i and node j.

[0137] Step 2: Graph Convolutional Network (GCN) Structure Design (S202)

[0138] The network structure includes an input layer, hidden layers, and an output layer, specifically:

[0139] Input layer: n-dimensional node feature information.

[0140] The first hidden layer performs graph convolution operations:

[0141]

[0142] in, This represents the feature representation of node i in the first hidden layer (the feature after graph convolution). It is an activation function used to introduce nonlinearity and solve the problem of "neuron death" in the negative region of ordinary ReLU.

[0143] The adjacency matrix element indicates whether there is an edge between nodes i and j (1 for an edge and 0 for no edge in an unweighted graph, and the weight of the edge in a weighted graph).

[0144] This represents the neighbor set of node i (all nodes directly connected to i).

[0145] This represents the degree of node i (the number of edges connected to i).

[0146] This represents the initial features of node j (the original features of the input layer).

[0147] This represents the weight matrix of the first hidden layer.

[0148] This represents the bias term of the first hidden layer, used to adjust the offset of features.

[0149] The second hidden layer introduces an attention mechanism:

[0150]

[0151]

[0152] in, The value represents the influence weight (attention score) of neighbor j on node i, reflecting the model's "attention level" to different neighbors.

[0153] This represents the attention weight matrix, used to learn the attention interaction patterns between nodes.

[0154] It is an activation function used to perform a non-linear transformation on the attention score.

[0155] It is in the form of a Softmax function used to normalize the attention score (the sum of the weights of all neighbors is 1).

[0156] This represents the feature representation of node i in the second hidden layer (features after attention-weighted aggregation).

[0157] This represents the weight matrix of the second hidden layer.

[0158] This represents the bias term of the second hidden layer.

[0159] This indicates that the features of neighbor node j are weighted by attention. Weighting reflects the attention mechanism idea that "important neighbors contribute more information".

[0160] Output layer: 2D, predicted position correction amount ;in This represents the X-coordinate offset of the position of the critical node i of the component relative to the input position. This represents the offset of the position of the critical node i of the component relative to the input position in the Y coordinate.

[0161] If only one of a certain type of component is detected in the initial point set (the coordinates of N key nodes of the input components), it is marked as a missing node. Based on the position of the existing node and the standard constraint, the position of the candidate node is generated. The standard constraint is to select a standard image for each type of device and set the reference position of each component and the positional relationship between components under the standard image.

[0162] If only one instance of a certain type of critical component (such as the three bushings of a transformer) is detected in the initial point set, it is marked as a missing node. The candidate node generation strategy generates three candidate locations based on the positions of existing nodes (such as the ontology) and standard constraints:

[0163]

[0164] in Indicates the coordinates of the candidate node position. Indicates the reference position coordinates. This represents the standard distance between the reference node and the missing node. This represents the standard direction angle from the reference node to the missing node. This is to increase the diversity of candidate locations.

[0165] Then, the candidate node is added to the graph, the sum of the edge weights between it and its neighboring nodes is calculated, and the candidate node with the largest weight is selected as the completion node.

[0166] Step 3: Constraint Loss Function Optimization and Location Correction (S203)

[0167] Spatial distance loss is used:

[0168] Where N represents the total number of nodes. This represents the predicted distance between nodes i and j (the distance between nodes output by the model). This represents the standard distance between nodes i and j (the actual distance determined by the scene specification or the actual physical design). The scale factor is calculated from the shooting distance to ensure consistent physical dimensions.

[0169] Further utilize orientation constraint loss (using a sine function is more sensitive to small angular errors):

[0170]

[0171] in This represents the predicted orientation angle from node i to j (the orientation angle between nodes output by the model). This represents the standard orientation angle from node i to j (the actual orientation angle between nodes in the scene).

[0172] Simultaneously define the confidence consistency loss:

[0173]

[0174] in, This represents the true confidence level of node i (representing the degree of certainty of the node's existence, which can be understood as the label confidence level). This represents the node confidence level predicted by GCN (the model's "certainty" judgment on its own predictions).

[0175] Therefore, the total loss can be expressed as:

[0176]

[0177] Using the above loss function for backpropagation of the model network, the model output after training is a set of corrected and optimized coordinates of each component:

[0178] .

[0179] in, The X and Y coordinates represent the final position of component m.

[0180] This method designs a dedicated thermal feature extraction network, using a convolutional neural network to capture key information such as temperature gradients and hotspot distribution, thereby improving the accuracy of the partition box boundaries for critical components. For example... Figure 3 As shown.

[0181] Step 1: Hot Feature Extraction Network (S301)

[0182] The thermal feature extraction network is designed as follows:

[0183] A convolutional neural network with three convolutional layers and two pooling layers is constructed as a thermal feature extraction network. This network will retrieve temperature matrix information stored in infrared image files, containing 640... The temperature data is 512, and to adapt to the convolutional network structure, the temperature data matrix is ​​padded with 640. 640. Each element of this temperature matrix corresponds to the temperature value of the corresponding pixel in the infrared image, which serves as the input to the thermal feature extraction network.

[0184] First convolution operation: Set the parameters of the first convolutional layer to 32 3's. A 3x3 convolutional kernel with a stride of 1 and padding of 1 is used. The temperature matrix is ​​input into this convolutional layer for convolution, resulting in 32 feature maps.

[0185] First pooling operation: The first pooling layer is connected after the first convolutional layer, using 2... The max pooling method with a stride of 2 performs pooling on the 32 feature maps obtained from the first convolution operation, reducing the size of the feature maps and retaining key features.

[0186] Second convolution operation: Set the parameters of the second convolutional layer to 64 3's. A 3x3 convolutional kernel with a stride of 1 and padding of 1 is used. The output of the first pooling operation is input into this convolutional layer for convolution, resulting in 64 feature maps.

[0187] Second pooling operation: A second pooling layer is connected after the second convolutional layer, using 2... The max pooling method with a stride of 2 is used to pool the 64 feature maps obtained from the second convolution operation.

[0188] Third convolution operation: Set the parameters of the third convolutional layer to 128 3's. The convolutional kernel has a value of 3, a stride of 1, and padding of 1. The output of the second pooling operation is input into this convolutional layer for convolution, resulting in 128 heat maps, which are used as the output of the heat feature extraction network.

[0189] Step 2: Determining partition boundaries based on thermal feature maps (S302)

[0190] From 128 thermal feature maps, the feature maps with the highest correlation to temperature gradient changes and hotspot distribution are selected. By calculating the correlation coefficient between each feature map and the original temperature matrix, feature maps with a correlation coefficient greater than 0.85 are selected as key thermal feature maps.

[0191] The selected key thermal feature maps are enhanced by using an adaptive histogram equalization method to stretch the grayscale value range of the feature maps to [0, 255], thereby enhancing the discriminative power of temperature features.

[0192] Threshold segmentation is performed based on the enhanced key thermal feature map. The optimal segmentation threshold is determined by the maximum inter-class variance method, and the feature map is divided into target candidate region (device region) and background region.

[0193] Based on the list of device component location coordinates output by the aforementioned method, the initial component bounding box is calculated by combining the size and image resolution of different device components. The proportion of the device region in the initial component bounding box is calculated, and the proportion threshold is set to 0.70. The candidate device component regions are continuously adjusted by combining the key thermal feature map, and the rectangular candidate box with the highest device proportion is extracted. This rectangular candidate box is the optimal partition boundary of the power equipment component.

[0194] By fully exploring the role of key information such as temperature gradient in infrared images in indicating partition boundaries, and effectively integrating thermal and visual features, the accuracy of infrared image partitioning for power equipment is improved, providing a reliable foundation for subsequent temperature analysis and fault diagnosis.

[0195] In another embodiment, a feature fusion-based infrared image recognition system for power equipment is used to perform the feature fusion-based infrared image recognition method for power equipment as described in any of the preceding embodiments, comprising:

[0196] The dataset building module collects infrared images from multiple different perspectives and builds a dataset;

[0197] The multi-view feature extraction module uses a deep learning network to extract features from multi-view infrared images and fuses the features of infrared images from different viewpoints.

[0198] The adaptive weight allocation module adjusts the weights of features from different perspectives based on the sharpness and integrity of the target in infrared images from different perspectives. It calculates the gradient value of the infrared image based on the Sobel operator as a sharpness index and obtains the integrity index based on the proportion of the target area.

[0199] The model training and prediction module trains the model using a dataset and predicts the coordinates of the center points of each component in the power equipment.

[0200] The following example illustrates the workflow of a feature fusion-based infrared image recognition system for power equipment, including the following steps:

[0201] I. Multi-view image acquisition

[0202] Use a resolution of at least 640 The 512 infrared thermal imager takes the geometric center of the equipment (transformer, circuit breaker, etc.) as the origin and takes pictures in eight directions: 0° (front), 45°, 90° (side), 135°, 180° (back), 225°, 270° (side), and 315°.

[0203] II. Multi-view data enhancement

[0204] The following three methods are used to perform multi-view data augmentation on the original images:

[0205] Random angles are generated within a range of ±15° (step size 1°) to simulate a wider range of angle deviations;

[0206] Scale by 0.9, 1.0, and 1.1 times (maintaining 640). (512 outputs), simulating shooting distance fluctuations.

[0207] Add random device tilt at a scale of 0.05 to simulate the effect of hand-held shaking and improve data diversity.

[0208] III. Feature Fusion Methods

[0209] The feature fusion module uses a feature pyramid network (FPN) to fuse features from images from different perspectives and designs an adaptive weight allocation mechanism to achieve effective fusion of features from various perspectives.

[0210] The FPN consists of three parts: a bottom-up convolutional network, a top-down upsampling process, and lateral connections. For each viewpoint of the infrared image, the bottom-up convolutional network, using ResNet-50 as the base network, first extracts feature maps at different levels: C1 (1 / 2 resolution), C2 (1 / 4 resolution), C3 (1 / 8 resolution), C4 (1 / 16 resolution), and C5 (1 / 32 resolution). Then, through a top-down upsampling process, the higher-level feature maps (such as C5) are upsampled to the same resolution as the lower-level feature maps (such as C4), and then fused with the lower-level feature maps through lateral connections to obtain the fused feature map P4. The same method is used to obtain the feature maps P3, P2, and P1 sequentially.

[0211] Adaptive weight allocation mechanism

[0212] This mechanism automatically adjusts the weights of features from different viewpoints based on factors such as the sharpness and completeness of the target in the image. The specific implementation is as follows:

[0213] Sharpness assessment uses the Sobel operator to calculate the gradient values ​​of the image; a larger gradient value indicates a sharper image. For each viewpoint feature map, the mean gradient of the corresponding original image is calculated as a sharpness indicator. .

[0214] Integrity assessment is calculated based on the proportion of the target area. The more complete the target area (unobstructed and untruncated), the higher its weight, which serves as the integrity indicator. .

[0215] according to , The index is calculated using linear weighting. The weight of the viewpoint The formula is:

[0216]

[0217] in, , 4.

[0218] V. Relative Position Constraint Module

[0219] The relative position constraint module is built on a graph convolutional network (GCN) to learn the relative positional relationships between different partitions of power equipment and use it as supervision information to guide model training.

[0220] Each partition of the power equipment is considered as a node in a graph, and the node's features are the feature vectors of that partition in the fused feature graph. The spatial relationships between partitions are represented as edges.

[0221] GCN Network Structure: GCN uses two graph convolutional layers. The first graph convolutional layer takes node features and edge features as input and outputs node features after one convolution operation. The second graph convolutional layer takes the output of the first layer as input and further learns the deep relationships between nodes, outputting the final node features.

[0222] During model training, the node features output by GCN are used to predict the relative positional relationships between partitions, and the loss between the predicted and true values ​​is calculated using the mean squared error loss function. This loss is then backpropagated together with the object detection loss using a combination of FocalLoss and IoULoss losses to guide the model in learning the correct partition positions and order.

[0223] The relative position constraint module can effectively consider the relative positional relationships between different areas, avoiding problems such as incorrect partition position determination, disordered order, and missing information when dealing with complex occlusion scenarios or layout changes.

[0224] VI. Method for Determining Partition Boundaries with Enhanced Thermal Features

[0225] The thermal feature-enhanced partition boundary determination module is used to extract temperature features from infrared images and fuse them with visual features to improve the accuracy of partition boundaries.

[0226] This network is built on a convolutional neural network and consists of three convolutional layers and two pooling layers. The input is the temperature matrix of the infrared image. The first convolutional layer uses 32 three-dimensional pooling layers. A 3x3 convolutional kernel with a stride of 1 and padding of 1 outputs 32 feature maps; the first pooling layer uses a 2x3 kernel. Max pooling with a stride of 2; the second convolutional layer uses 64 3x3 pools. The first convolutional layer uses a 3x3 kernel with a stride of 1 and padding of 1, outputting 64 feature maps; the second pooling layer uses a 2x3 kernel. Max pooling with a stride of 2; the third convolutional layer uses 128 3x3 pools. A convolution kernel of 3, stride of 1, and padding of 1 output 128 hot feature maps.

[0227] Calculate the Pearson correlation coefficient between 128 thermal feature maps and the original temperature matrix, and select feature maps with an absolute correlation coefficient value ≥ 0.85 to form a key thermal feature map set (usually retaining 8-12 maps), with a focus on retaining feature maps that reflect abrupt changes in temperature gradient.

[0228] Adaptive histogram equalization was applied to the key thermal feature dataset, with a grid size of 8. 8. Enhance local temperature contrast while suppressing noise amplification.

[0229] The enhanced key thermal feature maps are fused at the pixel level to generate a single fused thermal feature map, highlighting the significant temperature features in each feature map. Based on the fused thermal feature map, the Otsu algorithm, improved by introducing a temperature weighting factor, is used to determine the segmentation threshold. The weighting factor is positively correlated with the pixel temperature value, making the segmentation of high-temperature areas more accurate, resulting in a binarized image (where the device target area is 1 and the background area is 0).

[0230] An edge detection algorithm is used, with a high threshold of 0.7 and a low threshold of 0.3, to extract the rectangular boundary of the candidate region of the device target. This boundary is the partition boundary of the power equipment.

[0231] In another embodiment, a computer storage medium stores a computer program thereon, wherein when a computer executes the computer program, it implements the infrared spectrum recognition method for power equipment based on feature fusion as described above.

[0232] In another embodiment, an electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program stored in the memory, wherein the computer program, when executed, implements the infrared spectrum recognition method for power equipment based on feature fusion as described above.

[0233] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for infrared image recognition of power equipment based on feature fusion, characterized in that, Includes the following steps: Collect infrared images from multiple different perspectives and construct a dataset; Deep learning networks are used to extract features from multi-view infrared images and to fuse the features of infrared images from different viewpoints. Based on the sharpness and integrity of the target in infrared images from different perspectives, the weights of the features of each perspective are adjusted, the gradient value of the infrared image is calculated based on the Sobel operator as a sharpness index, and the integrity index is obtained based on the proportion of the target area. The model is trained using a dataset to predict the coordinates of the center points of each component in a power equipment. By utilizing the inherent positional relationships between components, missing or incorrect positions of each component are corrected, and the component position coordinates are optimized. The utilization of these inherent positional relationships also includes determining component partition boundaries, which includes: Temperature features in infrared images are extracted using a constructed thermal feature extraction network, resulting in multiple thermal feature maps. By calculating the correlation coefficient between each feature map and the original temperature matrix, feature maps with a correlation coefficient greater than a threshold are selected as key thermal feature maps. The selected key thermal feature maps are enhanced, and threshold segmentation is performed based on the enhanced key thermal feature maps. The segmentation threshold is determined by the maximum inter-class variance method, and the feature maps are divided into target candidate regions and background regions. Based on the obtained coordinates of the equipment components, the initial component bounding boxes are calculated in combination with the size and image resolution of different equipment components. The proportion of the equipment region in the initial component bounding box is calculated. The candidate equipment component regions are adjusted according to the key thermal feature map. The rectangular candidate box with the highest equipment proportion is extracted. This rectangular candidate box is the optimal partition boundary of the power equipment component.

2. The infrared spectrum recognition method for power equipment based on feature fusion according to claim 1, characterized in that, Feature extraction is preceded by: Data augmentation is performed on the original infrared image, preserving the single-channel grayscale information and discarding the pseudo-color mapping channel; the grayscale values ​​are mapped to the [0,1] interval, as shown in the following formula: in, This represents the infrared image pixel value mapped to the [0,1] interval after normalization. Represents the grayscale value of the original infrared image; This represents the maximum value among the grayscale values ​​of the original infrared image. This represents the maximum grayscale value in the original infrared image; An adaptive median filter is used, with a small window for edge regions and a large window for flat regions.

3. The infrared spectrum recognition method for power equipment based on feature fusion according to claim 1, characterized in that, Deep learning networks are used to extract features from multi-view infrared images, and the features of infrared images from different viewpoints are fused, including: ResNet50 was used as the backbone network for multi-view feature extraction to extract feature maps at different levels. Through a top-down upsampling process, the high-level feature map is upsampled to the same resolution as the low-level feature map, and then fused with the low-level feature map through lateral connections to form a multi-scale feature representation. Cross-view attention enhancement is applied to feature maps of different scales.

4. The infrared spectrum recognition method for power equipment based on feature fusion according to claim 1, characterized in that, The weights of features from different perspectives are adjusted based on the sharpness and integrity of the target in infrared images from different viewpoints, including: The formula for calculating the sharpness index is: in, For the first Pixels in a view image gradient magnitude, This represents the total number of pixels in the image. The formula for calculating the integrity index is: in, For the first The pixel area of ​​the target component in the view image. The total pixel area of ​​the image; Using linear weighting to calculate the first The weight of the viewpoint : in, , Weighting coefficients.

5. The infrared spectrum recognition method for power equipment based on feature fusion according to claim 1, characterized in that, Optimizing component position coordinates includes: Construct a power equipment component relationship graph, defining nodes as information for each equipment component, and defining edges as spatial constraint information between components. If the adjacency matrix... The adjacency matrix indicates that there is a spatial constraint relationship between node i and node j. 0 indicates that there is no spatial constraint relationship between node i and node j; Construct a graph convolutional neural network to learn the geometric relationships between components, predict the relative positional relationships between components, fill in missing nodes by the position of candidate nodes, and participate in the weight calculation of the graph convolutional neural network. The loss between the predicted and actual values ​​is calculated, and the spatial distance loss and orientation constraint loss are used for backpropagation of the network. The corrected and optimized position coordinates of each component are obtained through the trained graph convolutional neural network model.

6. The infrared spectrum recognition method for power equipment based on feature fusion according to claim 5, characterized in that, A graph convolutional neural network consists of an input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer consists of n-dimensional node feature information; The first hidden layer performs graph convolution operations: in, This represents the feature representation of node i in the first hidden layer. It is the first activation function. The elements of the adjacency matrix represent whether there is an edge between nodes i and j. Describes the set of neighbors of node i. Indicates the degree of node i. This represents the initial characteristics of node j. This represents the weight matrix of the first hidden layer. This represents the bias term of the first hidden layer; The second hidden layer introduces an attention mechanism: in, This represents the influence weight of neighbor j on node i. Represents the attention weight matrix. It is the second activation function. It is in the form of a Softmax function, used to normalize the attention score. This represents the feature representation of node i in the second hidden layer. This represents the weight matrix of the second hidden layer. This represents the bias term of the second hidden layer. This indicates that the features of neighbor node j are weighted by attention. Weighted; Output layer prediction position correction ;in This represents the X-coordinate offset of the position of the critical node i of the component relative to the input position. This represents the offset of the position of the critical node i of the component relative to the input position in the Y coordinate. If only one instance of a certain type of component is detected in the initial set of critical component nodes, it is marked as a missing node. Based on the positions of existing nodes and standard constraints, the positions of candidate nodes are generated: in, Indicates the coordinates of the candidate node position. Indicates the coordinates of the reference node position. This represents the standard distance between the reference node and the missing node. This represents the standard direction angle from the reference node to the missing node. Indicates the angle value; Add the candidate node to the power equipment component relationship graph, calculate the sum of the edge weights between it and its neighboring nodes, and select the candidate node with the largest weight as the completion node.

7. The infrared spectrum recognition method for power equipment based on feature fusion according to claim 1, characterized in that, The constructed thermal feature extraction network includes 3 convolutional layers and 2 pooling layers; The temperature matrix of the infrared image is obtained, and the temperature matrix is ​​input into the first convolutional layer for convolution operation to obtain the first feature map. The first pooling layer is connected after the first convolutional layer to perform pooling on the resulting first feature map, thus preserving key features. The output of the first pooling layer is input into the second convolutional layer for convolution operation to obtain the second feature map; A second pooling layer is connected after the second convolutional layer to perform pooling processing on the second feature map; The output of the second pooling layer is input into the third convolutional layer for convolution operation to obtain the thermal feature map.

8. A feature fusion-based infrared image recognition system for power equipment, used to execute the feature fusion-based infrared image recognition method for power equipment as described in any one of claims 1-7, characterized in that, include: The dataset building module collects infrared images from multiple different perspectives and builds a dataset; The multi-view feature extraction module uses a deep learning network to extract features from multi-view infrared images and fuses the features of infrared images from different viewpoints. The adaptive weight allocation module adjusts the weights of features from different perspectives based on the sharpness and integrity of the target in infrared images from different perspectives. It calculates the gradient value of the infrared image based on the Sobel operator as a sharpness index and obtains the integrity index based on the proportion of the target area. The model training and prediction module trains the model using a dataset and predicts the coordinates of the center points of each component in the power equipment.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer executes the computer program, it implements the infrared spectrum recognition method for power equipment based on feature fusion as described in any one of claims 1-7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor runs the computer program stored in the memory. When the computer program is executed, it implements the infrared spectrum identification method for power equipment based on feature fusion as described in any one of claims 1-7.

Images