Multi-modal fusion power line segmentation method and model continuous learning method thereof

Abstract

The invention discloses a multi-modal fusion power line segmentation method and a model continuous learning method thereof. The continuous learning method comprises the following steps: initializing a teacher model and a student model in a current scene; for the power line segmentation task of the current scene, freezing the teacher model, calculating the output of training set data in the teacher model and the student model under the current scene, and calculating label loss and distillation loss; calculating the total gradient of the loss of the student model by dynamically adjusting the weights of the label loss gradient and the distillation loss gradient of the student model and combining the weights with the respective loss gradients, and further updating the parameters of the student model; and a power line segmentation model for the current scene is obtained through repeated iterative calculation of loss, weight adjustment, total gradient calculation and parameter updating, and is used for performing power line segmentation on the image in the current scene. According to the invention, plasticity and stability of the power line segmentation model in continuous learning can be balanced, and power line segmentation precision and parameter calculation amount can be balanced.

Description

Technical Field

[0001] This invention belongs to the field of image recognition, specifically relating to a multimodal fusion power line segmentation method and its model continuous learning method. Background Technology

[0002] Power lines, as a core component of modern industrial and urban energy systems, are crucial for grid security due to their stable operation. Traditional power line inspection methods, relying on manual inspections, suffer from significant drawbacks such as low efficiency, high cost, and high risk in complex terrain or extreme weather conditions. With the rapid development of power systems, traditional manual inspection methods for transmission lines can no longer meet the requirements of grid maintenance for line quality and reliability. In recent years, the rise of UAVs equipped with multi-sensor platforms has provided new technological pathways for power line inspection, especially power line segmentation technology based on computer vision, which has gradually become a research hotspot. With the development of deep learning, color images, due to their high resolution and strong intuitiveness, have a wide range of applications in power line inspection. However, using only color images cannot adapt to complex environmental scenarios such as nighttime, rain, fog, and haze, where power line inspection performance drops sharply. To overcome these challenges, a power line segmentation method based on the fusion of color and infrared images has been proposed, with the core objective of enhancing performance through multimodal information complementarity. Color images provide high-resolution texture and shape information, suitable for extracting geometric features of power lines; while infrared images are more resistant to environmental changes than color images and can significantly enhance the thermal contrast between the target and the background. Compared to using only single-modal images, multimodal power line segmentation models, by combining the advantages of visible light and thermal infrared data, can significantly improve detection performance in adverse weather and environments, and possess strong anti-interference capabilities. Although current research on power line segmentation has achieved some results, achieving accurate power line segmentation remains quite difficult due to the complex and variable backgrounds of aerial power line images and the small size of the power lines themselves, which occupy a very small percentage of pixels in the image. Therefore, it is essential to find a multimodal fusion method that can fully utilize the complementary information from color and infrared images.

[0003] On the other hand, in the real world, drone inspections often face varying lighting and weather conditions. When a drone travels to an environment significantly different from that in the training data, the accuracy of power line segmentation often drops sharply. However, due to issues such as data loss and the waste of computational resources to retrain the model using all the data, retraining the power line segmentation model with all the data is impractical. On the other hand, training with only new data can lead to catastrophic forgetting of the model, causing a sharp decline in segmentation accuracy in the original scene. Therefore, exploring a continuous learning method for the power line segmentation model—one that allows it to perform well in new environments without excessively forgetting the knowledge gained from training on older data—is particularly important. Summary of the Invention

[0004] This invention provides a multimodal fusion-based power line segmentation method and its continuous learning method, which can balance the plasticity and stability of the power line segmentation model in continuous learning, and effectively balance the power line segmentation accuracy and parameter computation.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: A continuous learning method for power line segmentation models, comprising: When a power line image for any scene is received for the first time to perform a power line segmentation task, the training set data of the power line image in that scene is used to train several power line segmentation models based on label loss, and the optimal power line segmentation model is selected to perform the power line segmentation task in that scene. When receiving power line images for other scenarios to perform power line segmentation tasks: The optimal power line segmentation model obtained from the previous scene training is used as the initial teacher model and student model for the current scene. For the power line segmentation task in the current scenario, freeze the teacher model, calculate the output of the teacher model and student model on the training set data in the current scenario, and calculate the label loss and distillation loss. The CAGrad algorithm is used to dynamically adjust the weights of the label loss gradient and distillation loss gradient of the student model; and based on the label loss gradient and distillation loss gradient and their respective weights, the total gradient of the student model loss is calculated and the parameters of the student model are updated accordingly. By iteratively calculating the loss, adjusting the weights, calculating the total gradient, and updating the parameters, a power line segmentation model for the current scene is obtained, which is used to segment images in the current scene using power lines.

[0006] Furthermore, the formula for calculating the label loss is: ; ; ; in, For label loss, For binary cross-entropy loss, Loss due to dice, Here, N represents the combined loss coefficient, and N is the number of pixels in the input electric field image. For the first The real label of each pixel Predicting the first power line segmentation model The pixel label is The predicted probability is given by H, where H represents the height of the input power line image and W represents the width of the input power line image.

[0007] Furthermore, the formula for calculating distillation loss is: ; In the formula, N is the number of pixels in the input power line image. For distillation losses, It is the first input power line image 1 pixel It is the output probability distribution of the teacher model. This is the output probability distribution of the student model; The temperature parameter introduced for knowledge distillation is used to soften the output probability distributions of the teacher and student models; Here, H represents the height of the input image, and W represents the width of the input image, which is the activation function.

[0008] Furthermore, the weights of the label loss gradient and distillation loss gradient of the student model are dynamically adjusted using the CAGrad algorithm, specifically including: (1) Initialize the weight list for the label loss gradient and the distillation loss gradient: ; In the formula, Represents a list of gradient weights. The weights representing the gradient of distillation loss. The weights represent the gradient of the label loss. (2) Calculate the inner product matrix of the gradient list. : ; In the formula, This represents a list of gradients consisting of the distillation loss gradient and the label loss gradient. (3) Calculate the norm of the average gradient : ; (4) Construct the objective function : ; ; In the formula, It is an intermediate variable, and its value is half the norm of the average gradient; (5) Solve for the optimal gradient weight list by minimizing the objective function: .

[0009] Furthermore, based on the label loss gradient and distillation loss gradient and their respective weights, the total gradient of the student model loss is calculated, including: (1) Using the optimal gradient weight list obtained by solving the problem The gradient is obtained by weighted summation of the distillation loss gradient and the label loss gradient. : ; In the formula, and These represent the gradients generated by the distillation loss and the gradients generated by the labeling loss, respectively. (2) Calculate the final optimized total gradient : ; ; In the formula, As an intermediate variable, used to measure the gradient Weighted.

[0010] Furthermore, the power line segmentation task in different scenarios refers to segmenting power lines in power line images obtained under different environmental conditions; different environments include: sunny days, cloudy days, foggy days, nighttime, and snowy days.

[0011] Furthermore, the power line segmentation model adopts the UNet architecture; In the UNet compression path, each layer includes two downsampling modules and one fusion module. The two downsampling modules downsample the input power line color feature map and infrared feature map, respectively. The fusion module first extracts color feature weights and infrared feature weights from the downsampled feature map, and then multiplies the downsampled color feature map and infrared feature map with the color feature weights and infrared feature weights, respectively, to obtain new color feature maps and infrared feature maps. Finally, the newly obtained color feature maps and infrared feature maps are used as the input of the next layer of the UNet compression path, and are also added together as the fusion feature map of the UNet compression path at this layer to be output to the corresponding layer of the UNet extension path.

[0012] Furthermore, the fusion module extracts color feature weights and infrared feature weights from the downsampled feature map, including: First, a 1×1 convolutional layer is used to process the color feature map and the infrared feature map respectively. Then, the two output feature maps are added together and passed through the ReLU activation function. Next, a convolutional layer is used to extract features from the upper feature map. After passing through a Softmax activation function, a dual-channel feature map is output. Finally, a channel dimension splitting layer is used to split the dual-channel feature map to obtain the color feature weights and infrared feature weights.

[0013] A multimodal fusion power line segmentation method, comprising: Acquire color and infrared images of power lines. Based on the image acquisition scenario, use the continuous learning method of the power line segmentation model described above to continuously learn the power line segmentation model for that scenario, and then use the power line segmentation model obtained through continuous learning to segment the power lines in the image.

[0014] Compared with the prior art, the technical effects of the present invention are as follows: (1) The present invention adopts the dynamic weight knowledge distillation continuous method to continuously learn the power line segmentation model under different scenarios. In the continuous learning training process of each scenario, the CAGrad algorithm is used to dynamically adjust the label gradient weight and distillation gradient weight to alleviate the gradient conflict between distillation loss and label loss, thereby balancing the plasticity and stability of the power line segmentation model in continuous learning.

[0015] (2) The power line segmentation model of this invention selects the Unet model as the benchmark model and integrates color image information and infrared image information to segment the power lines. The power line segmentation accuracy of the model of this invention and the benchmark model on the VITLD dataset is compared. It can be seen that the power line segmentation accuracy of the model of this invention is better than that of the benchmark model in all scenarios, and the number of parameters is much lower than that of the benchmark model, thus balancing high accuracy and low computational cost. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of the color feature and infrared feature fusion module proposed in this invention.

[0017] Figure 2 This is a flowchart of the continuous learning method proposed in this invention.

[0018] Figure 3 This is the power line segmentation model architecture of the present invention.

[0019] Figure 4 This is the downsampling module used in the power line segmentation model of this invention.

[0020] Figure 5This is the upsampling module used in the power line segmentation model of this invention. Detailed Implementation

[0021] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.

[0022] Example 1

[0023] This embodiment provides a continuous learning method for power line segmentation models, such as... Figure 1 As shown, it includes: Step 1: When a power line segmentation task for any scenario is received for the first time, the training set data for that scenario is used to train several power line segmentation models based on label loss. The optimal power line segmentation model is then selected to perform the power line segmentation task for that scenario.

[0024] In this embodiment, the power line segmentation model uses UNet as the baseline model, and a multimodal fusion module that mutually perceives bidirectional path multimodal information is added to the compressed path. The constructed model is as follows: Figure 2 As shown.

[0025] In the UNet compression path, each layer includes two downsampling modules and one fusion module, such as... Figure 3 As shown, two downsampling modules downsample the input color feature map and infrared feature map, respectively. The fusion module first extracts color feature weights and infrared feature weights from the downsampled feature map, then multiplies the downsampled color feature map and infrared feature map by the color feature weights and infrared feature weights, respectively, to obtain new color feature maps and infrared feature maps. Finally, the newly obtained color feature maps and infrared feature maps are used as the input to the next layer of the UNet compression path, and are also added together to form the fusion feature map of the UNet compression path at that layer, which is then output to the corresponding layer of the UNet extension path.

[0026] Most current multimodal fusion power line segmentation models still employ early fusion. However, early fusion merely stacks infrared and color images along the channel dimension, failing to fully utilize the complementary information between the color and infrared images, leading to information redundancy. The fusion module in this embodiment is as follows... Figure 3 As shown, color feature weights and infrared feature weights are extracted from the downsampled feature map, including: First, 1×1 convolutional layers are used to process the color feature map and the infrared feature map separately. Then, the two output feature maps are summed and passed through a ReLU activation function. Next, a convolutional layer is used to extract features from the upper-layer feature map, followed by a Softmax activation function to output a dual-channel feature map. Finally, a channel-dimensional splitting layer is used to separate the dual-channel feature map, obtaining color feature weights and infrared feature weights. This embodiment, through this dynamic spatial fusion method, accurately captures the spatial correlation between color and infrared features while suppressing background interference, making the fused multimodal information more focused on the power line region.

[0027] like Figure 4 As shown, the downsampling model in the power line segmentation model includes, in sequence, max pooling, a first convolutional layer, and a second convolutional layer. Figure 5 As shown, the upsampling module in the power line segmentation model first transposes and convolves the input low-resolution features, then merges the channels with the input high-resolution features, and then performs convolution, ReLU and batch normalization processing twice.

[0028] A multimodal power line segmentation dataset, VITLD, covering various scenarios including sunny, cloudy, foggy, dark, and snowy days, was acquired. This publicly available dataset contains 400 pairs of aligned RGB-IR images, each 256×256 pixels. The training, validation, and test sets each contain 280, 40, and 80 images respectively, randomly divided into two independent datasets. During training, this invention utilizes Python's OpenCV and imgaug libraries, employing image enhancement strategies for four typical weather scenarios: daytime, nighttime, fog, and snow. By adjusting HSV color space parameters, variations in light intensity and contrast at different times of day are simulated. Slight enhancement was achieved by reducing brightness to 60% and saturation to 70% and increasing green and blue channel values; heavy enhancement was achieved by significantly reducing brightness to 30% and saturation to 30% and increasing blue channel values; the Fogcorruption method from the imgaug library was used to simulate hazy weather conditions by adding natural fog effects, enhancing the model's adaptability to low visibility environments; and the Snow corruption method from the imgaug library was used to add snowflake textures and lighting effects to simulate the visual characteristics of snowfall.

[0029] In this embodiment, several power line segmentation models are trained using training set data corresponding to the scenario of the first power line segmentation task. A combination of binary cross-entropy (BCE) and dice loss (Dice) is used as the loss function. The former is used to optimize edge details and gradient stability, while the latter is used to solve class imbalance and global overlap, thereby improving the segmentation accuracy and robustness of power lines.

[0030] ; ; ; in, For label loss, For binary cross-entropy loss, Loss due to dice, Here, represents the combined loss coefficient, and N is the number of pixels in the input image. For the first The real label of each pixel Predicting the first power line segmentation model The pixel label is The predicted probability is given by H, where H represents the height of the input image and W represents the width of the input image.

[0031] In this embodiment, training requires an Ubuntu 18.04.5 LTS or later system, with Python 3.8.5 and Pytroch 1.10.1 or later. The hardware platform requires an NVIDIA GeForce RTX4090 (24GB) graphics card, at least 16GB of RAM, and a hard drive with at least 256GB of storage. The Rectified Adam learning rate was optimized for stable training, and 200 epochs were trained with an initial learning rate of 0.001, decreasing by 0.5 epochs every 45 epochs. The batch size was fixed at 5, and the weight decay was set to 5e-4. Other parameters were set to their default values.

[0032] The present invention uses the following indicators to measure and screen the accuracy of power line segmentation models.

[0033] This refers to the ratio of correctly predicted positive samples out of all samples predicted as positive.

[0034] ; Recall, also known as recall rate, refers to the ratio of correctly predicted positive samples to the total number of true positive samples.

[0035] ; IOU, or Intersection-Union Ratio, is the ratio of the intersection to the union of GT and Predict in power line segmentation.

[0036] ; Dice is a calculation result that takes into account both the model's precision and recall, and its value tends to favor the smaller value.

[0037] ; TP, FP, FN, and TN represent true positive, false positive, false negative, and true negative, respectively.

[0038] Step 2: When receiving power line segmentation tasks from other scenarios: use the best power line segmentation model trained in the previous scenario as the initial teacher and student models for the current scenario.

[0039] Step 3: For the power line segmentation task in the current scenario, freeze the teacher model, calculate the output of the training set data in the teacher model and student model in the current scenario, and calculate the label loss and distillation loss.

[0040] In this embodiment, five scenarios—sunny day, cloudy day, foggy day, night, and snowy day—were constructed through data augmentation and trained sequentially to build continuous learning.

[0041] When training with each new scene data, the best model trained in the previous scene is used as the teacher model and student model for this training. For example, when training with cloudy day data, the best model trained with sunny day data is used to initialize the student model and teacher model for this training. The teacher model parameters are frozen and not updated during training. The label loss during continuous learning is the same as in step 1, and the distillation loss is as follows.

[0042] ; in, It is the first input image 1 pixel This is the output probability distribution of the teacher model; The temperature parameter introduced for knowledge distillation is used to soften the output probability distributions of the teacher and student models; The student model operates with the same temperature parameters as the teacher model. The predicted output probability distribution is given below.

[0043] Step 4: After calculating the label loss and distillation loss, calculate the label gradient and distillation gradient respectively through backpropagation, and use the CAGrad algorithm to dynamically adjust the weights of the label loss gradient and distillation loss gradient of the student model; and calculate the total gradient that minimizes the conflict between the label gradient and distillation gradient of the student model based on the label loss gradient and distillation loss gradient and their respective weights, and use the total gradient to update the parameters of the student model.

[0044] Currently, continuous learning methods are categorized into regularization-based methods, knowledge replay-based methods, and knowledge distillation-based methods. This invention constructs a continuous learning method for power line segmentation models based on knowledge distillation. During training, by approximating the output distribution of the teacher model, the learning of the student model can be constrained, thereby significantly reducing the forgetting of past knowledge during continuous learning. Knowledge distillation can reduce the forgetting of old knowledge, and the label loss model can acquire the ability to segment power lines in new scenarios. Therefore, the gradients generated by distillation loss and label loss often conflict significantly. This invention proposes a continuous learning training method based on dynamic weight knowledge distillation, using the CAGrad algorithm to dynamically adjust the weights of the label gradient and the distillation gradient to minimize gradient conflicts. Specifically, it includes: (1) Initialize the weight list for the label loss gradient and the distillation loss gradient: ;

[0045] In the formula, Represents a list of gradient weights. The weights representing the gradient of distillation loss. The weights represent the gradient of the label loss.

[0046] (2) Calculate the inner product matrix of the gradient list. : .

[0047] In the formula, This represents a list of gradients consisting of the distillation loss gradient and the label loss gradient.

[0048] (3) Calculate the norm of the average gradient : .

[0049] (4) Construct the objective function : ; ; In the formula, It is an intermediate variable, and its value is half the norm of the average gradient.

[0050] (5) Solve for the optimal gradient weight list by minimizing the objective function: .

[0051] (6) Use the optimal gradient weight list obtained by solving the problem. The gradient is obtained by weighted summation of the distillation loss gradient and the label loss gradient. : ; In the formula, and These represent the gradients generated by distillation loss and label loss, respectively.

[0052] (7) Calculate the final optimized total gradient. : ; ; In the formula, As an intermediate variable, used to measure the gradient Weighted. This yields the total gradient. The magnitude of the gradient will not differ significantly from the magnitude of the total gradient before adjustment.

[0053] Step 5: By iteratively calculating the loss, adjusting the weights, calculating the total gradient, and updating the parameters, a power line segmentation model for the current scene is obtained, which is used to segment images in the current scene using power lines.

[0054] Table 1 shows a comparison of the accuracy of the power line segmentation model obtained through continuous learning in this invention and currently published segmentation models on the VITLD dataset. It can be seen that, compared to the baseline model, the multi-model power line segmentation model of this invention, which integrates color and infrared images, improves the Dice coefficient by 1.05%, recall by 2.25%, and Iou by 1.44%. Table 2 shows a comparison of the power line segmentation accuracy of the model of this invention and the baseline model in different scenarios on VITLD. It can be seen that the power line segmentation accuracy of the model of this invention is superior to the baseline model in all scenarios, and the number of parameters is much lower than that of the baseline model.

[0055] .

[0056] .

[0057] Example 2

[0058] This embodiment provides a multimodal fusion-based power line segmentation method. First, color and infrared images of power lines are acquired. Then, according to the image acquisition scenario, the power line segmentation model described in Embodiment 1 is continuously learned in that scenario. Finally, the power line segmentation model obtained through continuous learning is used to segment the power lines in the image.

[0059] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.

Claims

1. A continuous learning method for a power line segmentation model, characterized in that, Comprise: When the power line segmentation task is performed for the first time on the power line image of any scene, the power line image training set data in the scene is used to train several power line segmentation models based on the label loss, and the optimal power line segmentation model is selected from the models to perform the power line segmentation task in the scene; When the power line segmentation task is performed on the power line image of other scenes: The optimal power line segmentation model trained in the previous scene is used as the initialization teacher model and student model of the current scene; For the power line segmentation task of the current scene, the teacher model is frozen, the outputs of the teacher model and the student model for the training set data in the current scene are calculated, and the label loss and the distillation loss are calculated; The weight of the label loss gradient and the distillation loss gradient of the student model is dynamically adjusted by the CAGrad algorithm, and the total gradient of the student model loss is calculated based on the label loss gradient and the distillation loss gradient and their respective weights, and the parameters of the student model are updated; Through repeated iteration of loss calculation, weight adjustment, total gradient calculation and parameter update, the power line segmentation model for the current scene is obtained, which is used for power line segmentation on the image in the current scene. 2.The power line segmentation model's lifelong learning method of claim 1, wherein, The calculation formula of the label loss is: ； ； ； wherein, is a label loss, is a binary cross-entropy loss, is a dice loss, is a combination loss coefficient, N is the number of pixel points of the input power line image, is the true label of the th pixel point, is the predicted probability of the power line segmentation model that the th pixel point label is , H represents the height of the input power line image, and W represents the width of the input power line image. 3.The power line segmentation model's lifelong learning method of claim 1, wherein, The calculation formula of the distillation loss is: ； In the formula, N is the number of pixels in the input power line image. For distillation losses, It is the first input power line image 1 pixel It is the output probability distribution of the teacher model. It is the output probability distribution of the student model; The temperature parameter introduced for knowledge distillation is used to soften the output probability distributions of the teacher and student models; Here, H represents the height of the input image, and W represents the width of the input image, which is the activation function. 4.The power line segmentation model's continual learning method of claim 1, wherein, The weight of the label loss gradient and the distillation loss gradient of the student model is dynamically adjusted by the CAGrad algorithm, which specifically includes: (1) Initialize the weight list of the label loss gradient and the distillation loss gradient: ； wherein representing a list of gradient weights, representing a weight of a distillation loss gradient, representing a weight of a label loss gradient; (2) Compute the inner product matrix of the gradient list : ； In the formula, represents a gradient list consisting of a distillation loss gradient and a label loss gradient; (3) Compute norm of average gradient : ； (4) Constructing the objective function : ； ； wherein is an intermediate variable with value half the norm of the average gradient; (5) The optimal gradient weight list is solved by minimizing the objective function: 。 5.The power line segmentation model’s continual learning method of claim 4, wherein, Based on the label loss gradient and the distillation loss gradient and their respective weights, the total gradient of the student model loss is calculated, including: (1) using the optimal gradient weight list obtained by solving , the gradient is obtained by weighted sum of distillation loss gradient and label loss gradient : ； wherein and respectively represent the gradient produced by the distillation loss and the gradient produced by the label loss. (2) calculating the final optimized total gradient : ； ； In the formulae, is an intermediate variable used to weight the gradient . 6.The power line segmentation model’s continual learning method of claim 1, wherein, The power line segmentation task in different scenes refers to the segmentation of power lines in power line images obtained under different environmental conditions; different environments include sunny, cloudy, foggy, dark and snowy days. 7.The power line segmentation model’s continual learning method of claim 1, wherein, The power line segmentation model uses the UNet architecture; In the compression path of UNet, each layer includes two down-sampling modules and a fusion module; the two down-sampling modules respectively perform down-sampling processing on the input power line color feature map and infrared feature map; the fusion module first extracts color feature weights and infrared feature weights from the down-sampled feature map, then multiplies the down-sampled color feature map and infrared feature map with the color feature weights and infrared feature weights respectively to obtain new color feature map and infrared feature map, and finally adds the newly obtained color feature map and infrared feature map as the fusion feature map of the UNet compression path at this layer to output to the corresponding layer of the UNet expansion path. 8.The power line segmentation model’s continual learning method of claim 7, wherein, The fusion module extracts color feature weights and infrared feature weights from the down-sampled feature map, including: First, use a 1x1 convolutional layer to process the color feature map and the infrared feature map respectively, then add the outputs of the two feature maps after passing through the Relu activation function, then use a convolutional layer to extract features from the upper layer feature map, then pass through a Softmax activation function to output a double-channel feature map, and finally use a channel dimension splitting layer to split the double-channel feature map to obtain color feature weights and infrared feature weights.

9. A multi-modal fusion power line segmentation method, characterized in that, Comprise: Acquire the color image and the infrared image of the power line, and according to the acquisition scene of the image, use the continuous learning method of the power line segmentation model in any one of claims 1-8 to continuously learn the power line segmentation model in the scene, and then use the power line segmentation model obtained through the continuous learning to segment the power line in the image.

Images