A power equipment RGB-T image semantic segmentation method based on cross-modal collaborative optimization, a terminal and a medium

Abstract

The application discloses a power equipment RGB-T image semantic segmentation method based on cross-modal collaborative optimization in the technical field of computer vision, a terminal and a medium, and the method comprises the following steps: obtaining a substation RGB image and an infrared image; inputting the registered image into an encoder to perform feature extraction to obtain multi-scale features; inputting the multi-scale features into a forward and reverse pyramid structure to perform semantic enhancement to obtain multi-scale features after semantic enhancement; inputting the multi-scale features after semantic enhancement into a high-level feature fusion module, a middle-level feature fusion module and a low-level feature fusion module to perform feature fusion to obtain fusion features; inputting the fusion features into the forward and reverse pyramid structure to perform semantic enhancement to obtain fusion features after semantic enhancement; and inputting the fusion features after semantic enhancement into a decoder to perform decoding to obtain a semantic segmentation result. The method improves the adaptability and performance of the power equipment segmentation task, and finally outputs an accurate semantic segmentation result.

Description

Technical Field

[0001] This invention relates to a semantic segmentation method, terminal, and medium for RGB-T images of power equipment based on cross-modal collaborative optimization, belonging to the field of computer vision technology. Background Technology

[0002] With the rapid development of the power industry, the scale and complexity of power equipment are constantly increasing, making efficient and accurate monitoring and maintenance crucial for ensuring the stable operation of the power system. Against this backdrop, image-based power equipment inspection technology has emerged, among which RGB visible light image and thermal infrared image fusion technology has demonstrated significant application value in the field of power equipment monitoring.

[0003] RGB images can clearly present information such as the appearance, texture, shape, and color of electrical equipment, allowing users to intuitively understand the basic condition of the equipment. For example, RGB images can reveal surface wear, deformation, and stains. Infrared images, on the other hand, have a unique advantage: they can sensitively capture temperature changes in equipment and promptly identify abnormal temperature areas caused by faults. During the operation of electrical equipment, localized overheating is often a significant indicator of potential faults, and infrared images can visually display these thermal anomalies, providing crucial clues for fault diagnosis.

[0004] However, traditional manual inspection and sensor monitoring methods are gradually revealing numerous problems when faced with increasingly complex power equipment systems. Manual inspection is not only inefficient, requiring a significant amount of manpower and time, but also limited by human subjective judgment and detection capabilities, making it prone to missed or false detections. While sensor monitoring can acquire some equipment data in real time, it suffers from limited monitoring range and high costs, making it difficult to comprehensively cover all parts of power equipment and complex operating conditions.

[0005] Despite the great potential of RGB-T image fusion technology, existing RGB-T image semantic segmentation methods still face many challenges in practical applications. In terms of feature extraction, existing methods struggle to fully exploit the complementary information between RGB and infrared images, resulting in incomplete and inaccurate extraction of power equipment features. Many methods simply stitch or weightedly fuse the two images, failing to deeply integrate their respective strengths. In complex environments, such as substations with numerous devices, cables, and complex lighting conditions, high-precision segmentation is difficult to achieve, hindering the accurate differentiation of power equipment from the background and impacting the accurate assessment of equipment status.

[0006] Furthermore, existing semantic segmentation models have shortcomings in handling the features of power equipment at different scales. Power equipment varies in size and location in images, ranging from small electronic components to large transformers. Existing models struggle to simultaneously take into account the features of equipment at different scales, resulting in lower segmentation accuracy for small equipment and a tendency to miss minor faults; while segmentation of large equipment may lead to problems such as blurred boundaries and loss of details.

[0007] In summary, current power equipment monitoring technology faces challenges in terms of efficiency, accuracy, and cost. Existing RGB-T image semantic segmentation methods have obvious limitations, and there is an urgent need for an innovative method to improve the accuracy and robustness of power equipment semantic segmentation in order to meet the needs of intelligent detection and maintenance of power equipment in complex scenarios. Summary of the Invention

[0008] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0009] In a first aspect, the present invention provides a semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization, comprising:

[0010] Acquire RGB images and corresponding infrared images of a substation scene;

[0011] The RGB image and the infrared image are registered, and the registered image is input into the encoder to extract features to obtain multi-scale features.

[0012] Multi-scale features are input into positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced multi-scale features.

[0013] The semantically enhanced multi-scale features are divided into three levels: high, medium, and low. They are then input into the high-level feature fusion module HLFM, the medium-level feature fusion module MLFM, and the low-level feature fusion module LLFM respectively to perform feature fusion and obtain the fused features.

[0014] The fused features are input into positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced fused features.

[0015] The semantically enhanced fused features are input into the decoder for layer-by-layer decoding to obtain the semantic segmentation result.

[0016] Furthermore, the multi-scale features are input into the forward and reverse pyramid structures for semantic enhancement, resulting in semantically enhanced multi-scale features, including:

[0017] Some multi-scale features are input from high resolution to low resolution into an inverted pyramid structure, and the semantics are enhanced by downsampling, convolution and lateral connection operations to obtain semantically enhanced multi-scale features.

[0018] Another part of the multi-scale features is input into the forward pyramid structure from low resolution to high resolution. The semantics are enhanced by upsampling, convolution and lateral connection operations to obtain the semantically enhanced multi-scale features.

[0019] Among them, the partial multi-scale features and the other partial multi-scale features have at least one layer of shared multi-scale features.

[0020] Furthermore, the semantically enhanced multi-scale features are input into the high-level feature fusion module HLFM, the mid-level feature fusion module MLFM, and the low-level feature fusion module LLFM, respectively, according to high, medium, and low levels, to perform feature fusion and obtain fused features, including:

[0021] The semantically enhanced multi-scale high-level features are input into HLFM. HLFM processes the high-level features through element-wise multiplication and addition, combined with spatial and channel attention modules, to obtain high-level fused features.

[0022] The semantically enhanced multi-scale mid-level features are input into MLFM, which uses convolution, pooling, and concatenation operations to process the mid-level features and obtain the mid-level fused features.

[0023] The semantically enhanced multi-scale low-level features are input into LLFM. LLFM uses dilated convolution to capture information at different scales to process low-level features and obtain low-level fused features.

[0024] Furthermore, the semantically enhanced multi-scale high-level features are input into HLFM. HLFM processes high-level features through element-wise multiplication and addition, combined with spatial and channel attention modules, including:

[0025] Semantically enhanced multi-scale high-level features include and Perform element-wise multiplication and addition operations on each of them to obtain the corresponding product characteristics. With summation characteristics As shown in the following formula:

[0026] ;

[0027] in, This represents the semantically enhanced features of a visible light image; This represents the semantically enhanced features of the infrared image; Indicates the number of feature layers; This represents element-wise multiplication. This represents element-wise addition.

[0028] The obtained product feature With summation characteristics The input spatial attention module performs weighted adjustments and then performs element-wise multiplication with... , By combining these, we obtain the weighted adjusted product characteristics. With summation characteristics As shown in the following formula:

[0029] ;

[0030] in, Represents spatial attention operations;

[0031] Weighted adjusted product characteristics With summation characteristics After combining elements-wise multiplication, the weight information is extracted by the channel attention module and then weighted in combination with the results of the original element-wise multiplication to obtain the weighted fused output features. As shown in the following formula:

[0032] ;

[0033] in, Including high-level integration features ; This indicates channel attention operation.

[0034] Furthermore, the semantically enhanced multi-scale mid-level features are input into MLFM. MLFM uses convolution, pooling, and concatenation operations to process the mid-level features, resulting in mid-level fused features, including:

[0035] Semantically enhanced multi-scale mid-level features include , , and The corresponding product characteristics are obtained by performing element-wise multiplication and addition on each other. With summation characteristics Then, after two consecutive 3×3 convolution operations, the data is concatenated, and then another 3×3 convolution operation is performed to obtain the intermediate features. As shown in the following formula:

[0036] ;

[0037] in, and Representing multi-scale features respectively Visible light image features after semantic enhancement using reverse pyramid structure and forward pyramid structure; and These are respectively represented as multi-scale features. Infrared image features enhanced semantically using reverse pyramid and forward pyramid structures. This represents the convolution operation. Indicates a splicing operation;

[0038] right Perform max pooling and 7×7 convolution operations to obtain intermediate features. As shown in the following formula:

[0039] ;

[0040] in, This indicates a convolution operation with a 7×7 kernel. This represents the max pooling operation;

[0041] Combining intermediate features and intermediate features The feature information is obtained by adjusting the detail representation of the features through element-wise multiplication to obtain intermediate features. As shown in the following formula:

[0042] ;

[0043] in, Indicates the activation function;

[0044] intermediate features and intermediate features After element-wise addition, the input channel attention module obtains the final mid-level feature fusion result. As shown in the following formula:

[0045] ;

[0046] in, Including mid-layer fusion features and , Indicates will and The mid-level fused features obtained after feature fusion. Indicates will and The mid-level fused features obtained after feature fusion. This indicates a channel attention operation.

[0047] Furthermore, the semantically enhanced multi-scale low-level features are input into LLFM. LLFM uses dilated convolution to capture information at different scales to process the low-level features, resulting in low-level fused features, including:

[0048] Semantically enhanced multi-scale low-level features include and The product characteristics are obtained by performing element-wise multiplication and addition on each other. With summation characteristics , product feature With summation characteristics Intermediate features are obtained by combining elements after convolution and then applying ReLU activation. As shown in the following formula:

[0049] ;

[0050] intermediate features The data is passed to four parallel dilated convolutional layers with different dilation rates. This expands the receptive field and captures feature information at different scales. The feature maps obtained through dilated convolution are... Represented as:

[0051] ;

[0052] in, Indicates the expansion rate; This indicates a convolution operation with a 3×3 kernel;

[0053] The output features of dilated convolutional layers with different dilation rates are concatenated along the channel dimension to fuse information at different scales, resulting in the final low-level fused features. .

[0054] Furthermore, the fused features are input into the positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced fused features, including:

[0055] Low-level fusion features and mid-layer fusion features , Inputting into the inverted pyramid structure yields semantically enhanced fusion features. , and ;

[0056] Mid-layer fusion features , Features of high-level integration Inputting into a forward pyramid structure yields semantically enhanced fusion features. , and .

[0057] Furthermore, the semantically enhanced fusion features are decoded layer by layer to obtain the semantic segmentation results, including:

[0058] semantically enhanced fusion features The input decoder is randomly deactivated, then the channel information of the features is extracted through two consecutive convolution operations, and the resolution of the features is restored through an upsampling operation. The decoding process is shown in the following formula:

[0059] ;

[0060] in, Indicates an upsampling operation; Indicates the convolution operation; Represents the random deactivation function;

[0061] By analyzing fusion features Decoding yields the highest-level features. Then the highest layer features and fusion features In the input decoder, the decoder output and the fused features are decoded layer by layer. The layer-by-layer decoding process is shown in the following equation:

[0062] ;

[0063] in, Indicates to With fusion features The features obtained after decoding; Indicates to Combination with fusion features and The features obtained after decoding; This indicates the key features generated after multiple decoding and feature fusion processes. Integrating from and fusion features Information; This represents the penultimate layer of features. Is Based on this, additional features were incorporated. Obtained after decoding;

[0064] The final semantic segmentation result is obtained. As shown in the following formula:

[0065] .

[0066] Secondly, the present invention also provides an electronic terminal, including a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the above-mentioned RGB-T image semantic segmentation method for power equipment based on cross-modal collaborative optimization are performed.

[0067] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the above-described method for semantic segmentation of RGB-T images of power equipment based on cross-modal collaborative optimization.

[0068] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0069] This invention provides a semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization. By acquiring RGB and infrared images of a substation scene and combining their complementary characteristics, it effectively improves the robustness and accuracy of semantic segmentation in complex environments. The registered images are encoded to extract multi-scale features, which are then semantically enhanced using forward and reverse pyramid structures. This fully leverages multi-scale features and contextual information, enhancing feature extraction performance. The semantically enhanced features are then fused into HLFM, MLFM, and LLFM modules at high, medium, and low levels, respectively, resulting in more accurate feature representation and improved boundary details and global consistency. In the decoder, the fused features are again semantically enhanced using forward and reverse pyramid structures and decoded layer by layer, improving the model's adaptability and performance for power equipment segmentation tasks. Ultimately, it outputs accurate semantic segmentation prediction results, meeting the needs of intelligent detection and maintenance of power equipment. Attached Figure Description

[0070] Figure 1 A flowchart illustrating the RGB-T image semantic segmentation method for power equipment based on cross-modal collaborative optimization provided by this invention;

[0071] Figure 2 A schematic diagram of the overall structure of the power equipment image semantic segmentation network based on cross-modal collaborative optimization provided by the present invention;

[0072] Figure 3 The forward and reverse pyramid structure diagrams provided for this invention;

[0073] Figure 4 The high-level feature fusion module (HLFM) structure diagram provided by this invention;

[0074] Figure 5 The structure diagram of the MLFM (Middle-Layer Feature Fusion Module) provided by this invention;

[0075] Figure 6 The structure diagram of the low-level feature fusion module LLFM provided by the present invention. Detailed Implementation

[0076] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0077] Example 1:

[0078] Figure 1 This is a flowchart of a semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization, as described in Embodiment 1 of the present invention. This flowchart merely illustrates the logical order of the method described in this embodiment. Without conflict, different methods may be used in other possible embodiments of the present invention. Figure 1 Perform the steps shown or described in the order indicated. See also Figure 1 The method implemented in this way specifically includes the following steps:

[0079] Acquire RGB images and corresponding infrared images of a substation scene;

[0080] The RGB image and the infrared image are registered, and the registered image is input into the encoder to extract features to obtain multi-scale features.

[0081] Multi-scale features are input into positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced multi-scale features.

[0082] The semantically enhanced multi-scale features are divided into three levels: high, medium, and low. They are then input into the high-level feature fusion module HLFM, the medium-level feature fusion module MLFM, and the low-level feature fusion module LLFM respectively to perform feature fusion and obtain the fused features.

[0083] The fused features are input into positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced fused features.

[0084] The semantically enhanced fused features are input into the decoder for layer-by-layer decoding to obtain the semantic segmentation result.

[0085] Specifically, the first step is to acquire RGB images and corresponding infrared images of the substation scene. This is the initial step of the entire method because RGB images can present information such as the appearance and texture of power equipment, while infrared images can reflect the temperature distribution of the equipment. The combination of the two provides a rich data foundation for subsequent accurate semantic segmentation.

[0086] Next, the registered image is input into the encoder for feature extraction to obtain multi-scale features. The registration operation ensures spatial consistency between the RGB and infrared images, enabling accurate correspondence of subsequently extracted features. For example... Figure 2As shown, the encoder used in this embodiment consists of two parallel ResNet-152 backbone networks. One network processes RGB images and contains five deep convolutional neural network layers, RStage1 to RStage5; the other network processes infrared images and also contains five deep convolutional neural network layers, TStage1 to TStage5. Leveraging its powerful feature extraction capabilities, features at different scales are obtained from the images. These multi-scale features contain rich information about the image, from details to the overall picture, providing crucial material for subsequent processing.

[0087] Next, multi-scale features are input into the forward and reverse pyramid structures for semantic enhancement, resulting in semantically enhanced multi-scale features. The reverse pyramid structure processes features from high to low resolution, expanding the receptive field to obtain global information through downsampling, convolution, and lateral connections; the forward pyramid structure processes features from low to high resolution, restoring resolution and enhancing detailed features through upsampling, convolution, and lateral connections. This structure fully utilizes multi-scale features and contextual information, significantly enhancing feature extraction performance and improving segmentation accuracy, making the segmentation results more consistent with the actual shape and location of power equipment. Specifically, this involves two processes: firstly, inputting multi-scale features into the reverse pyramid structure. For example... Figure 2 As shown, some multi-scale features , , , , , Inputting into the inverted pyramid structure yields semantically enhanced multi-scale features. , , , , , .like Figure 3 The inverted pyramid structure shown performs downsampling, convolution, and lateral connections layer by layer from high resolution to low resolution. Downsampling expands the receptive field, capturing more global information; convolution extracts key features; and lateral connections fuse feature information from different layers. This process can be expressed by the following formula:

[0088] ;

[0089] in, This represents the input features of the inverted pyramid structure, which encompasses multi-scale features. , , , , , ; This represents the output features of the inverted pyramid structure, which encompasses... , , , , , ; Indicates a downsampling operation; This indicates a convolution operation.

[0090] On the other hand, another set of multi-scale features is input into the forward pyramid structure. For example... Figure 2 As shown, multi-scale features , , , , , Inputting into a forward pyramid structure yields semantically enhanced multi-scale features. , , , , , .like Figure 3 The forward pyramid structure shown performs upsampling, convolution, and lateral connections layer by layer from low to high resolution. Upsampling restores resolution, allowing features to contain more detail; convolution and lateral connections further optimize feature representation. This process can be expressed by the following formula:

[0091] ;

[0092] in, This represents the input features of a forward pyramid structure, which encompasses... , , , , , ; This represents the output features of a positive pyramid structure, which encompasses the characteristics... , , , , , ; This indicates an upsampling operation.

[0093] Among them, the multi-scale features shared by some multi-scale features and another part of multi-scale features are the multi-scale features of the third layer. , ,Will , Inputting the inverted pyramid yields semantically enhanced multi-scale features. , ,Will , Inputting a forward pyramid yields semantically enhanced multi-scale features. , .

[0094] Then, the semantically enhanced multi-scale features are input into the HLFM, MLFM, and LLFM modules at three levels (high, medium, and low) for feature fusion. Different levels of modules employ specific computational methods to fully exploit the complementarity of cross-modal features. HLFM utilizes element-wise multiplication, addition, and attention modules; MLFM employs convolution, pooling, and concatenation operations; and LLFM uses dilated convolution to capture information at different scales. These operations make feature representation more accurate, effectively improve the boundary details of the segmentation results, make the outlines of power equipment clearer, and enhance global consistency, thereby improving the overall quality of the segmentation results.

[0095] For high-level features, element-wise multiplication and addition are used, combined with spatial and channel attention modules to process the high-level features, resulting in high-level fused features. Specifically, such as... Figure 4 As shown, the semantically enhanced multi-scale high-level features include and Perform element-wise multiplication and addition operations on each of them to obtain the corresponding product characteristics. With summation characteristics As shown in the following formula:

[0096] ;

[0097] in, This represents the semantically enhanced features of a visible light image; This represents the semantically enhanced features of the infrared image; Indicates the number of feature layers; This represents element-wise multiplication. This represents element-wise addition.

[0098] The characteristics of the obtained product and summation value , The input spatial attention module performs weighted adjustments, assigning weights based on feature importance to highlight key information. Then, element-wise multiplication is applied... , By combining these, we obtain the weighted adjusted product characteristics. With summation characteristics As shown in the following formula:

[0099] ;

[0100] in, Represents spatial attention operations;

[0101] Weighted adjusted product characteristics With summation characteristics After combining elements-wise multiplication, the weight information is extracted by the channel attention module and then weighted in combination with the results of the original element-wise multiplication to obtain the weighted fused output features. As shown in the following formula:

[0102] ;

[0103] in, Including high-level integration features ; This indicates channel attention operation.

[0104] For mid-level features, convolution, pooling, and concatenation operations are used to process them, resulting in mid-level fused features. Specifically, such as... Figure 5 As shown, the semantically enhanced multi-scale mid-level features include , , and The corresponding product characteristics are obtained by performing element-wise multiplication and addition on each other. With summation characteristics Then, after two consecutive 3×3 convolution operations, the data is concatenated, and then another 3×3 convolution operation is performed to obtain the intermediate features. As shown in the following formula:

[0105] ;

[0106] in, and Representing multi-scale features respectively Visible light image features after semantic enhancement using reverse pyramid structure and forward pyramid structure; and These are respectively represented as multi-scale features. Infrared image features enhanced semantically using reverse pyramid and forward pyramid structures. This represents the convolution operation. Indicates a splicing operation;

[0107] right Perform max pooling and 7×7 convolution operations to obtain intermediate features. As shown in the following formula:

[0108] ;

[0109] in, This indicates a convolution operation with a 7×7 kernel. This indicates the max pooling operation; max pooling can preserve the main features and reduce the amount of data. A 7×7 convolution kernel can obtain a wider range of feature information and obtain intermediate features.

[0110] Combining intermediate features and The feature information is obtained by adjusting the detail representation of the features through element-wise multiplication to obtain intermediate features. As shown in the following formula:

[0111] ;

[0112] in, Indicates the activation function;

[0113] intermediate features and After element-wise addition, the input channel attention module obtains the final mid-level feature fusion result. As shown in the following formula:

[0114] ;

[0115] in, Including mid-layer fusion features , , , , This indicates channel attention operations. Indicates will and The mid-level fused features obtained after feature fusion. Indicates will and The mid-level fused features obtained after feature fusion. This indicates a channel attention operation.

[0116] For low-level features, dilated convolution is used to capture information at different scales to process low-level features, resulting in low-level fused features. Specifically, such as... Figure 6 As shown, the semantically enhanced multi-scale low-level features include and The product characteristics are obtained by performing element-wise multiplication and addition on each other. With summation characteristics , product feature With summation characteristics Intermediate features are obtained by combining elements after convolution and then applying ReLU activation. As shown in the following formula:

[0117] ;

[0118] intermediate features The data is passed to four parallel dilated convolutional layers with different dilation rates. This expands the receptive field and captures feature information at different scales. The feature maps obtained through dilated convolution are... Represented as:

[0119] ;

[0120] in, Indicates the expansion rate; This indicates a convolution operation with a 3×3 kernel;

[0121] In this embodiment, the dilation rates of each convolutional layer are j=1, j=3, j=5, and j=7, respectively. The output features of convolutional layers with different dilation rates are concatenated along the channel dimension to fuse information at different scales, resulting in the final low-level feature fusion result. Represented as:

[0122] ;

[0123] in, Including low-level fusion features ; This indicates a splicing operation.

[0124] Next, the fused features are input into the decoder, and semantic enhancement is performed using a forward and reverse pyramid structure to obtain semantically enhanced fused features. For example... Figure 2 As shown, specifically, it involves fusing low-level features. and mid-layer fusion features , Inputting into the inverted pyramid structure yields semantically enhanced fusion features. , and ;Middle layer fusion features , Features of high-level integration Inputting into a forward pyramid structure yields semantically enhanced fusion features. , and The reuse of the forward and reverse pyramid structure in the decoder further optimizes the ability to express fused features. By combining random deactivation, convolution, and upsampling operations, high-resolution features are fused layer by layer, making the model more suitable for power equipment segmentation tasks, improving the model's performance, and enabling it to output more accurate semantic segmentation prediction results, thus meeting the needs of intelligent detection and maintenance of power equipment.

[0125] Finally, the semantically enhanced fused features are decoded layer by layer, and the prediction results are output. The input decoder undergoes random deactivation, followed by two consecutive convolutional operations to extract the channel information of the features. An upsampling operation then restores the feature resolution. Random deactivation prevents overfitting and improves the model's generalization ability. The convolutional operations then delve deeper into the channel dimensions of the features. Finally, upsampling restores the feature map's resolution to the scale of the original image, resulting in accurate segmentation.

[0126] decoder The decoding process is shown in the following formula:

[0127] ;

[0128] in, Indicates an upsampling operation; Indicates the convolution operation; Represents a random deactivation function. This is achieved through the analysis of fused features. Decoding yields the highest-level features. Then the highest layer features and fusion features In the input decoder, the decoder output and the fused features are decoded layer by layer. The layer-by-layer decoding process is shown in the following equation:

[0129] ;

[0130] in, Indicates to With fusion features The features obtained after decoding; Indicates to Combination with fusion features and The features obtained after decoding; This indicates the key features generated after multiple decoding and feature fusion processes. Integrating from and fusion features Information; This represents the penultimate layer of features. Is Based on this, additional features were incorporated. Obtained after decoding;

[0131] The final semantic segmentation result is obtained. As shown in the following formula:

[0132] .

[0133] Through this layer-by-layer decoding process, accurate semantic segmentation results are finally obtained, providing strong support for the intelligent detection and maintenance of power equipment.

[0134] This invention also provides a semantic segmentation model for RGB-T images of power equipment. This model is a concrete implementation of a cross-modal collaborative optimization method for semantic segmentation of RGB-T images of power equipment. The model constructs its architecture and undergoes training based on this method. During inference, it processes images according to the method's flow and outputs semantic segmentation results to achieve accurate segmentation of power equipment. The specific training process of the model is as follows:

[0135] During training, to more effectively guide model learning, a weighted cross-entropy function is used to calculate the cross-entropy loss value for both edge supervision and semantic supervision. The weighted cross-entropy function can assign weights according to the importance of different categories or regions, allowing the model to focus more on those parts that are more critical to the semantic segmentation task of power equipment, such as the edges and key component regions of power equipment, thereby improving the model's segmentation accuracy for complex power equipment scenes.

[0136] The training data comes from a self-built dataset, in which 80% of the RGB-T images are selected as the training set. The self-built dataset is specifically designed for RGB-T images of power equipment and contains a rich variety of image data from substation scenarios, covering RGB images and corresponding infrared images under different operating conditions, lighting conditions, and equipment states.

[0137] By selecting 80% of the data as the training set, we can ensure that the model has enough data to learn from, while reserving some data for subsequent validation and testing to evaluate the model's generalization ability.

[0138] Regarding training parameter settings, the initial learning rate was set to 0.00005. The learning rate determines the step size of parameter updates during model training; a suitable initial learning rate helps the model converge to the optimal solution quickly. Simultaneously, an exponential decay scheme with a base of 0.9 was used to reduce the learning rate. As training progresses, gradually decreasing the learning rate allows the model to quickly adjust parameters in the early stages of training, accelerating convergence, while preventing the model from oscillating around the optimal solution due to an excessively large learning rate in the later stages of training, thereby improving the model's training performance.

[0139] In addition, the batch size is set to 4. The batch size affects the number of samples used by the model each time the parameters are updated. Setting it to 4 can achieve a good balance between computing resources and training effect. It ensures that the model can make full use of the information of multiple samples to update parameters, and does not cause excessive memory consumption and reduced computing efficiency due to excessively large batch size.

[0140] To enhance the model's generalization ability, data augmentation operations are performed on the training set. Since the actual operating scenarios of power equipment are complex and varied, data augmentation techniques are used to expand the training data to enhance the model's generalization ability and enable it to adapt to different situations. Common data augmentation methods include, but are not limited to, image rotation, translation, scaling, flipping, and adding noise. These operations increase the diversity of the training data, allowing the model to encounter more different types of images during training, thereby learning a wider range of features and improving the model's adaptability to various complex scenarios. After data augmentation, the augmented training set is input into the power equipment RGB-T image semantic segmentation model to be trained.

[0141] The maximum number of iterations was set to 200. Based on the cross-entropy loss function, the Adam W optimizer was used to iteratively optimize the semantic segmentation model to be trained. The Adam W optimizer combines adaptive moment estimation (Adam) and weight decay techniques, effectively preventing overfitting while optimizing model parameters. During each iteration, the optimizer calculates the gradient based on the current model parameters and training data, and updates the model parameters based on the gradient and learning rate, continuously reducing the model's loss value. When the maximum number of iterations (200) is reached, the iteration process stops, resulting in a pre-trained weight model. This pre-trained weight model contains the feature representations and segmentation patterns of the RGB-T images of power equipment learned during training, providing strong support for subsequent semantic segmentation of power equipment in real-world scenarios.

[0142] Example 2:

[0143] This invention also provides an electronic terminal, including a processor and a memory connected to the processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, it performs the steps of the RGB-T image semantic segmentation method for power equipment based on cross-modal collaborative optimization described in Embodiment 1 above.

[0144] Example 3:

[0145] This invention also provides a computer-readable storage medium storing a computer program thereon, characterized in that, when executed by a processor, the program implements the steps of the RGB-T image semantic segmentation method for power equipment based on cross-modal collaborative optimization described in Embodiment 1.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization, characterized in that, include: Acquire RGB images and corresponding infrared images of a substation scene; The RGB image and the infrared image are registered, and the registered image is input into the encoder to extract features to obtain multi-scale features. Multi-scale features are input into positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced multi-scale features. The semantically enhanced multi-scale features are divided into three levels: high, medium, and low. They are then input into the high-level feature fusion module HLFM, the medium-level feature fusion module MLFM, and the low-level feature fusion module LLFM respectively to perform feature fusion and obtain the fused features. The fused features are input into positive and negative pyramid structures for semantic enhancement, resulting in semantically enhanced fused features; wherein, the fused features include low-level fused features. and mid-layer fusion features , ; The semantically enhanced fused features are input into the decoder for layer-by-layer decoding to obtain the semantic segmentation result; The semantically enhanced multi-scale high-level features are input into HLFM. HLFM processes high-level features through element-wise multiplication and addition, combined with spatial and channel attention modules, including: Semantically enhanced multi-scale high-level features include and Perform element-wise multiplication and addition operations on each of them to obtain the corresponding product characteristics. With summation characteristics As shown in the following formula: ； in, This represents the semantically enhanced features of a visible light image; This represents the semantically enhanced features of the infrared image; Indicates the number of feature layers; This represents element-wise multiplication. This represents element-wise addition. The obtained product feature With summation characteristics The input spatial attention module performs weighted adjustments and then performs element-wise multiplication with... , By combining these, we obtain the weighted adjusted product characteristics. With summation characteristics As shown in the following formula: ； in, Represents spatial attention operations; Weighted adjusted product characteristics With summation characteristics After combining elements-wise multiplication, the weight information is extracted by the channel attention module and then weighted in combination with the results of the original element-wise multiplication to obtain the weighted fused output features. As shown in the following formula: ； in, Including high-level integration features ; This represents channel attention operations; Low-level fusion features and mid-layer fusion features , Inputting into the inverted pyramid structure yields semantically enhanced fusion features. , and ; Mid-layer fusion features , Features of high-level integration Inputting into a forward pyramid structure yields semantically enhanced fusion features. , and .

2. The semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization according to claim 1, characterized in that, Multi-scale features are input into forward and reverse pyramid structures for semantic enhancement, resulting in semantically enhanced multi-scale features, including: Some multi-scale features are input from high resolution to low resolution into an inverted pyramid structure, and the semantics are enhanced by downsampling, convolution and lateral connection operations to obtain semantically enhanced multi-scale features. Another part of the multi-scale features is input into the forward pyramid structure from low resolution to high resolution. The semantics are enhanced by upsampling, convolution and lateral connection operations to obtain the semantically enhanced multi-scale features. Among them, the partial multi-scale features and the other partial multi-scale features have at least one layer of shared multi-scale features.

3. The semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization according to claim 1, characterized in that, The semantically enhanced multi-scale features are input into the high-level feature fusion module HLFM, the mid-level feature fusion module MLFM, and the low-level feature fusion module LLFM, respectively, according to high, medium, and low levels, to perform feature fusion and obtain fused features, including: The semantically enhanced multi-scale high-level features are input into HLFM. HLFM processes the high-level features through element-wise multiplication and addition, combined with spatial and channel attention modules, to obtain high-level fused features. The semantically enhanced multi-scale mid-level features are input into MLFM, which uses convolution, pooling, and concatenation operations to process the mid-level features and obtain the mid-level fused features. The semantically enhanced multi-scale low-level features are input into LLFM. LLFM uses dilated convolution to capture information at different scales to process low-level features and obtain low-level fused features.

4. The semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization according to claim 1, characterized in that, The semantically enhanced multi-scale mid-level features are input into MLFM. MLFM uses convolution, pooling, and concatenation operations to process the mid-level features, resulting in mid-level fused features, including: Semantically enhanced multi-scale mid-level features include , , and The corresponding product characteristics are obtained by performing element-wise multiplication and addition on each other. With summation characteristics Then, after two consecutive 3×3 convolution operations, the data is concatenated, and then another 3×3 convolution operation is performed to obtain the intermediate features. As shown in the following formula: ； in, and Representing multi-scale features respectively Visible light image features after semantic enhancement using reverse pyramid structure and forward pyramid structure; and These are respectively represented as multi-scale features. Infrared image features enhanced semantically using reverse pyramid and forward pyramid structures. This represents the convolution operation. Indicates a splicing operation; right Perform max pooling and 7×7 convolution operations to obtain intermediate features. As shown in the following formula: ； in, This indicates a convolution operation with a 7×7 kernel. This represents the max pooling operation; Combining intermediate features and intermediate features The feature information is obtained by adjusting the detail representation of the features through element-wise multiplication to obtain intermediate features. As shown in the following formula: ； in, Indicates the activation function; intermediate features and intermediate features After element-wise addition, the input channel attention module obtains the final mid-level feature fusion result. As shown in the following formula: ； in, Including mid-layer fusion features and , Indicates will and The mid-level fused features obtained after feature fusion. Indicates will and The mid-level fused features obtained after feature fusion. This indicates a channel attention operation.

5. The semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization according to claim 4, characterized in that, The semantically enhanced multi-scale low-level features are input into LLFM. LLFM uses dilated convolution to capture information at different scales to process the low-level features, resulting in low-level fused features, including: Semantically enhanced multi-scale low-level features include and The product characteristics are obtained by performing element-wise multiplication and addition on each other. With summation characteristics , product feature With summation characteristics Intermediate features are obtained by combining elements after convolution and then applying ReLU activation. As shown in the following formula: ； intermediate features The data is passed to four parallel dilated convolutional layers with different dilation rates. This expands the receptive field and captures feature information at different scales. The feature maps obtained through dilated convolution are... Represented as: ； in, Indicates the expansion rate; This indicates a convolution operation with a 3×3 kernel; The output features of dilated convolutional layers with different dilation rates are concatenated along the channel dimension to fuse information at different scales, resulting in the final low-level fused features. .

6. The semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization according to claim 1, characterized in that, The semantically enhanced fusion features are decoded layer by layer to obtain the semantic segmentation results, including: semantically enhanced fusion features The input decoder is randomly deactivated, then the channel information of the features is extracted through two consecutive convolution operations, and the resolution of the features is restored through an upsampling operation. The decoding process is shown in the following formula: ； in, Indicates an upsampling operation; Indicates the convolution operation; Represents the random deactivation function; By analyzing fusion features Decoding yields the highest-level features. Then the highest layer features and fusion features In the input decoder, the decoder output and the fused features are decoded layer by layer. The layer-by-layer decoding process is shown in the following equation: ； in, Indicates to With fusion features The features obtained after decoding; Indicates to Combination with fusion features and The features obtained after decoding; This indicates the key features generated after multiple decoding and feature fusion processes. Integrating from and fusion features Information; This represents the penultimate layer of features. Is Based on this, additional features were incorporated. Obtained after decoding; The final semantic segmentation result is obtained. As shown in the following formula: 。 7. An electronic terminal, characterized in that, The system includes a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the RGB-T image semantic segmentation method for power equipment based on cross-modal collaborative optimization as described in any one of claims 1 to 6 are performed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the semantic segmentation method for RGB-T images of power equipment based on cross-modal collaborative optimization as described in any one of claims 1 to 6.

Images