A method, device, electronic device and storage medium for instance segmentation of infrared images of transmission lines

Abstract

The present invention discloses a method, device, electronic device, and storage medium for instance segmentation of infrared images of power transmission lines. The method comprises: training a first teacher model based on a visible light image and label information to generate a first trained teacher model; training a second teacher model based on the first infrared image and label information to generate a second trained teacher model; training a first student model based on the second infrared image and first pseudo-label information to generate a first trained student model; training the second student model based on the second infrared image and second pseudo-label information to generate a second trained student model; fine-tuning the first and second trained student models based on the second infrared image and third pseudo-label information to generate a second converged teacher model; and performing instance segmentation of the infrared image of the power transmission line using the second converged teacher model. The implementation of the present invention can improve the instance segmentation accuracy of infrared images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a method, device, electronic equipment and storage medium for segmenting infrared images of power transmission lines. Background Art

[0002] In modern power systems, transmission line inspections are crucial. Ensuring their safe and stable operation is fundamental to ensuring power supply. With the advancement of drone technology, drones equipped with visible light and infrared sensors have gained widespread application in inspections. Using infrared images for instance segmentation not only effectively identifies electrical targets such as insulators and wire clamps, but also accurately segments their contours, enabling early identification of overheating and defect detection. This method operates reliably in extreme environments, such as at night and in rain and fog, improving inspection efficiency and safety.

[0003] Current instance segmentation techniques face significant challenges when applied to infrared images, primarily due to data size and feature differences. First, the data size of infrared images is relatively small, and the label information obtained is limited, making it difficult for the model to fully learn and generalize. To compensate for this shortcoming, existing technologies hope to use abundant visible light image data to assist in fitting training of infrared image data. However, due to the drastic feature differences between infrared and visible light images, the difficulty of such cross-domain adaptation is significantly increased. This feature gap results in performance degradation when models trained in the source domain (visible light images) are transferred to the target domain (infrared images). Summary of the Invention

[0004] Embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for instance segmentation of power line infrared images. By implementing this method, it is possible to achieve transfer learning from visible light images to infrared images, which can effectively improve the accuracy of infrared image instance segmentation by using visible light image data to assist infrared image data fitting training due to significant inter-domain differences.

[0005] An embodiment of the present invention provides a method for segmenting transmission line infrared images, comprising:

[0006] Obtaining a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model having the same network architecture; the label information is used to characterize the boundaries of each instance in the image;

[0007] Performing supervised training on the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model; performing supervised training on the second teacher model based on the first infrared image and the corresponding label information to generate a second trained teacher model;

[0008] The first student model is trained based on the first pseudo-label information of the second infrared image and the second infrared image to generate a first trained student model; the second student model is trained based on the second pseudo-label information of the second infrared image and the second infrared image to generate a second trained student model; wherein the first pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo-label information is obtained by inputting the second infrared image into the second trained teacher model;

[0009] Fine-tune the first and second trained student models based on the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher models to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first and second trained teacher models, respectively, and performing weighted fusion on a first result of the first trained teacher model and a second result of the second trained teacher model;

[0010] The second converged teacher model is used to perform instance segmentation on infrared images of transmission lines.

[0011] Furthermore, before performing supervised training on the first teacher model based on the visible light image and the corresponding label information to generate the first trained teacher model, the method further includes:

[0012] performing a weak data augmentation operation on the visible light image to generate an updated visible light image;

[0013] performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image;

[0014] The weak data augmentation operation includes:

[0015] The current image is cropped to a random size and rotated at a random angle according to a preset probability, and the current image is updated based on the processed image.

[0016] Furthermore, before training the first student model according to the second infrared image and the first pseudo-label information of the second infrared image to generate the first trained student model, the method further includes:

[0017] performing a strong data augmentation operation on the second infrared image to generate an updated second infrared image;

[0018] The strong data augmentation operation includes:

[0019] Randomly adjust the grayscale of the current image and superimpose Gaussian noise, and update the current image according to the processed image.

[0020] Furthermore, the supervised training of the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model includes:

[0021] Repeat the loop process until a preset number of executions is reached, wherein the loop process includes:

[0022] Randomly dividing the visible light image into a plurality of batches of first training samples according to a preset number;

[0023] Inputting the batches of first training samples into the current first teacher model in sequence, so that each time a batch of first training samples is input, a supervised training operation is performed on the current first teacher model;

[0024] When the preset number of executions is reached, the current first teacher model is set as the first trained teacher model;

[0025] The supervised training operation includes:

[0026] According to the input first training sample, output the first instance mask corresponding to the first training sample through the current first teacher model;

[0027] Calculating a first loss function value using a cross entropy loss according to the first instance mask and the corresponding label information;

[0028] The optimizer is used to optimize the first teacher model according to the first loss function value, and the optimized model is updated as the current first teacher model.

[0029] Furthermore, the training of the first student model based on the second infrared image and the first pseudo-label information of the second infrared image to generate a first trained student model includes:

[0030] Repeat the loop process until a preset number of executions is reached, wherein the loop process includes:

[0031] randomly dividing the second infrared image into a plurality of batches of second training samples according to a preset number;

[0032] Inputting the batches of second training samples into the current first student model in sequence, so that a training operation is performed on the current first student model each time a batch of second training samples is input;

[0033] When the preset number of executions is reached, the current first student model is set as the first trained student model;

[0034] The training operation includes:

[0035] According to the input second training sample, output a second instance mask corresponding to the second training sample through the current first student model;

[0036] Calculating a second loss function value by cross entropy loss according to the second instance mask and the corresponding first pseudo-label information;

[0037] Utilizing an optimizer to optimize the first student model according to the second loss function value, and updating the optimized model as the current first student model;

[0038] According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated using the following formula:

[0039] θ t1 ←αθ t1 +(1-α)θ s1

[0040] Among them, θ t1 is the network parameter of the first trained teacher model; α is the preset smoothing coefficient; θ s1 are the network parameters of the first student model.

[0041] Furthermore, the first result of the first trained teacher model and the second result of the second trained teacher model are weightedly fused in the following manner:

[0042] w=Sigmoid(Conv(P t1 ,P t2 ))

[0043] P fuse =wP t1 +(1-w)P t2

[0044] Among them, is the predicted weight; Sigmoid is the activation function; Conv is the full convolution layer; P t1 is the first result; P t2 is the second result; P fuse is the third pseudo label information.

[0045] Based on the above method embodiments, the present invention provides corresponding device embodiments.

[0046] An embodiment of the present invention provides a transmission line infrared image instance segmentation device, comprising: a training image acquisition module, a teacher model training module, a student model training model, a student model fine-tuning module and an instance segmentation module.

[0047] The teacher model training module is used to perform supervised training on the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model; and to perform supervised training on the second teacher model based on the first infrared image and the corresponding label information to generate a second trained teacher model;

[0048] The student model training model is used to train the first student model based on the first pseudo-label information of the second infrared image and the second infrared image to generate a first trained student model; and to train the second student model based on the second pseudo-label information of the second infrared image and the second infrared image to generate a second trained student model; wherein the first pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo-label information is obtained by inputting the second infrared image into the second trained teacher model;

[0049] The student model fine-tuning module is configured to fine-tune the first and second trained student models based on the second infrared image and the third pseudo-label information of the second infrared image, and to update the network parameters of the corresponding trained teacher models to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first and second trained teacher models, respectively, and then performing weighted fusion on a first result of the first trained teacher model and a second result of the second trained teacher model;

[0050] The instance segmentation module is used to perform instance segmentation on the transmission line infrared image through the second converged teacher model.

[0051] Based on the above method embodiment, the present invention provides a corresponding electronic device embodiment.

[0052] An embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for instance segmentation of infrared images of transmission lines described in any one of the above-mentioned method embodiments can be implemented.

[0053] Based on the above method embodiment, the present invention provides a corresponding storage medium embodiment.

[0054] An embodiment of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for segmenting an instance of an infrared image of a power transmission line as described in any one of the above method embodiments can be implemented.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] Embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for instance segmentation of infrared images of power transmission lines. The method performs supervised training on respective teacher models based on a visible light image and a first infrared image, thereby generating a first trained teacher model and a second trained teacher model. Subsequently, the first student model and the second student model are trained based on the second infrared image, the first pseudo-label information of the second infrared image, and the second pseudo-label information of the second infrared image, thereby generating a first trained student model and a second trained student model. Furthermore, the first and second trained student models are fine-tuned based on the second infrared image and the third pseudo-label information of the second infrared image, and the network parameters of the corresponding trained teacher models are updated to generate a second converged teacher model. Finally, the second converged teacher model is used for instance segmentation of infrared images of power transmission lines.

[0057] The present invention trains a first teacher-student model using a large number of labeled visible light images and a large number of unlabeled second infrared images, enabling the model to learn knowledge features that are invariant between domains, thereby achieving effective alignment of knowledge features between different domains. Simultaneously, a second teacher-student model is trained using a small number of labeled first infrared images and a large number of unlabeled second infrared images, enabling it to learn knowledge features within the domain, thereby enhancing the perception of knowledge features within the same domain and reducing judgment errors. By weightedly fusing the two pseudo-label information of the second infrared image and fine-tuning the student model based on the fused pseudo-labels, the uncertainty of the pseudo-labels is effectively reduced, the inter-domain differences are narrowed, and the generalization ability of the model is enhanced, thereby improving the model's instance segmentation accuracy on infrared images. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The figure is a flow chart of a method for segmenting an infrared image of a power transmission line provided by an embodiment of the present invention.

[0059] Figure 2 The figure is a schematic structural diagram of a transmission line infrared image instance segmentation device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a method for segmenting an infrared image of a power transmission line, which includes at least the following steps:

[0062] Step S1: Acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model.

[0063] Specifically, the teacher-student model includes a teacher model and a student model having the same network architecture; the label information is used to represent the boundaries of each instance in the image;

[0064] The acquired visible light images must be labeled to facilitate subsequent training and evaluation. Label information is typically manually annotated using specialized labeling tools to ensure accuracy. These labels not only include the categories of objects in the image but also detail the boundaries of each object to achieve precise instance segmentation. Similarly, the first infrared image is annotated to ensure the model can learn the characteristics of different objects in the infrared image. The infrared image label information also includes the boundaries of each instance, which aids in subsequent model training and performance evaluation. A second infrared image is acquired without label information. These images are used to test the model's generalization ability and its ability to handle unseen data, especially in the absence of labels.

[0065] In an optional embodiment, the teacher model and student model described in the present invention have the same architecture and both use the Mask RCNN network. Among them, Mask RCNN includes an encoder ResNet-50 and a decoder FCN. During the training process, the student model uses the output of the teacher model as a pseudo label for learning through the knowledge distillation method. When processing labeled visible light images and infrared images, the teacher model provides high-quality features and classification results. By imitating the teacher's behavior, the student model gradually learns deeper feature representations. This process not only improves the learning efficiency of the student model, but also improves the model's generalization ability on unlabeled data. The label information contains the boundaries of each instance, forming a clear segmentation map. This is crucial for training the model because the boundary information can help the model identify the shape, size and relative position of the object. This information can significantly improve the accuracy and performance of the model in instance segmentation.

[0066] Step S2: Based on the visible light image and the corresponding label information, the first teacher model is supervised trained to generate a first trained teacher model; based on the first infrared image and the corresponding label information, the second teacher model is supervised trained to generate a second trained teacher model.

[0067] In a preferred embodiment, the supervised training of the first teacher model based on the visible light image and the corresponding label information to generate the first trained teacher model includes:

[0068] Repeat the loop process until a preset number of executions is reached, wherein the loop process includes:

[0069] Randomly dividing the visible light image into a plurality of batches of first training samples according to a preset number;

[0070] Inputting the batches of first training samples into the current first teacher model in sequence, so that each time a batch of first training samples is input, a supervised training operation is performed on the current first teacher model;

[0071] When the preset number of executions is reached, the current first teacher model is set as the first trained teacher model;

[0072] The supervised training operation includes:

[0073] According to the input first training sample, output the first instance mask corresponding to the first training sample through the current first teacher model;

[0074] Calculating a first loss function value using a cross entropy loss according to the first instance mask and the corresponding label information;

[0075] The optimizer is used to optimize the first teacher model according to the first loss function value, and the optimized model is updated as the current first teacher model.

[0076] It is important to explain that the visible light images are randomly divided into several batches, each containing a certain number of images. This random division can enhance the model's generalization ability and prevent overfitting. The first training sample from each batch is sequentially input into the current first teacher model. During the input process, the teacher model processes the images in each batch to generate corresponding instance masks. After reaching a preset number of executions, the current first teacher model is set as the first trained teacher model for subsequent use. At this point, the model should have strong feature extraction capabilities and be able to effectively infer unseen data. Each training operation in the loop involves: based on the input first training sample, the current first teacher model outputs a corresponding first instance mask. This mask describes the distribution of each instance in the image and provides a basis for subsequent loss calculations. The output first instance mask is compared with the corresponding label information, and the cross-entropy loss function is used to calculate the first loss function value. This loss function reflects the difference between the model's prediction and the true label; a smaller loss indicates a more accurate model. The current first teacher model is optimized using an optimizer (such as Adam or SGD) based on the calculated first loss function value. The optimizer calculates the gradient through the back-propagation algorithm and updates the model parameters to improve model performance. The updated model parameters will form the new current first teacher model to prepare for the next training cycle. After completing the preset number of executions, the first trained teacher model obtained should be evaluated on the validation set to determine its performance on unseen data. Evaluation indicators can include accuracy, F1-score, etc. to ensure that the model meets the expected performance standards. For the second teacher model, similar steps are used for supervised training based on the first infrared image and its corresponding label information to generate the second trained teacher model. By combining the teacher model training of visible light images and infrared images, effective learning of multimodal data can be achieved, further improving the overall performance of the model.

[0077] Step S3: train the first student model according to the second infrared image and the first pseudo-label information of the second infrared image to generate a first trained student model; train the second student model according to the second infrared image and the second pseudo-label information of the second infrared image to generate a second trained student model.

[0078] Specifically, the first pseudo label information is obtained by inputting the second infrared image into a first trained teacher model; the second pseudo label information is obtained by inputting the second infrared image into a second trained teacher model.

[0079] In an optional embodiment, the training of the first student model based on the second infrared image and the first pseudo-label information of the second infrared image to generate the first trained student model includes:

[0080] Repeat the loop process until a preset number of executions is reached, wherein the loop process includes:

[0081] randomly dividing the second infrared image into a plurality of batches of second training samples according to a preset number;

[0082] Inputting the batches of second training samples into the current first student model in sequence, so that a training operation is performed on the current first student model each time a batch of second training samples is input;

[0083] When the preset number of executions is reached, the current first student model is set as the first trained student model; the preset number of executions here can be 150;

[0084] The training operation includes:

[0085] According to the input second training sample, output a second instance mask corresponding to the second training sample through the current first student model;

[0086] Calculating a second loss function value by cross entropy loss according to the second instance mask and the corresponding first pseudo-label information;

[0087] Utilizing an optimizer to optimize the first student model according to the second loss function value, and updating the optimized model as the current first student model;

[0088] According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated using the following formula:

[0089] θ t1 ←αθ t1 +(1-α)θ s1

[0090] Among them, θ t1 is the network parameter of the first trained teacher model; α is the preset smoothing coefficient, which is 0.999 here; θ s1 are the network parameters of the first student model.

[0091] Similarly, the second trained student model can also be obtained by training through similar steps as above, which will not be repeated here.

[0092] In a preferred embodiment, when the present invention trains the network, the loss function adopted is the cross entropy loss function.

[0093] It should be noted that by inputting the second infrared image into the trained first teacher model, corresponding first pseudo labels can be generated. These pseudo labels provide the teacher model's predictions for each input image, indicating the possible location and category of each instance in the image. Similarly, by inputting the second infrared image into the trained second teacher model, second pseudo labels are generated. This method leverages the knowledge of the teacher model to help the student model learn richer features. Based on the network parameters of the current first student model, the network parameters of the first trained teacher model are updated using an exponential moving average. The teacher model uses the student model's exponential moving average to update its parameters, resulting in smoother changes in the teacher model parameters and enhancing the accuracy of instance mask prediction. In this process, the student model sees more infrared images, expanding its generalization, while the teacher model generates higher-quality pseudo labels through parameter updates. Using a similar method as above, the second student model is trained based on the second infrared image and second pseudo-label information to generate a second trained student model. In this way, the two student models can learn independently while maintaining a connection with their corresponding teacher model, achieving effective knowledge transfer.

[0094] Step S4: fine-tune the first trained student model and the second trained student model according to the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model.

[0095] Specifically, the third pseudo label information is obtained by inputting the second infrared image into the first trained teacher model and the second trained teacher model respectively, and then weightedly fusing the first result of the first trained teacher model and the second result of the second trained teacher model.

[0096] In a preferred embodiment, the first result of the first trained teacher model and the second result of the second trained teacher model are weightedly fused in the following manner:

[0097] w=Sigmoid(Conv(P t1 ,P t2 ))

[0098] P fuse =wP t1 +(1-w)P t2

[0099] Among them, is the predicted weight; Sigmoid is the activation function; Conv is the full convolution layer; P t1 is the first result; P t2 is the second result; P fuse is the third pseudo label information.

[0100] It is important to explain that, on the one hand, due to the significant feature differences between visible light and infrared images, the instance masks output by the first teacher model contain erroneous predictions. On the other hand, the small size of labeled infrared images makes it difficult for the neural network to learn, resulting in omissions in the instance masks output by the second teacher model. The pseudo-labels predicted in these situations will negatively impact the student model. Therefore, the instance masks output by the two teacher models are weighted and fused together to serve as the third pseudo-labels for the two teacher-student models, which are then fine-tuned in a few cycles. The third pseudo-labels are obtained by inputting the second infrared image into the first and second trained teacher models, respectively, and then using a weighted fusion approach. Weighted fusion effectively integrates the knowledge of the two teacher models, compensating for the shortcomings of a single model and effectively reducing the negative transfer interference of pseudo-labels on both teacher-student models, thereby generating more reliable third pseudo-label information. Using this third pseudo-label information, the first and second trained student models are fine-tuned, and the network parameters of the corresponding trained teacher models are updated to improve their recognition and classification accuracy for specific instances.

[0101] The network parameters of the corresponding trained teacher model are updated using the following formulas:

[0102] θ t1 ←αθ t1 +(1-α)θ′ s1

[0103] θ s2 ←βθ t2 +(1-β)θ′ s2

[0104] Among them, θ t2 is the network parameter of the second trained teacher model; β is the preset smoothing coefficient, which is 0.999 here; θ′ s2 is the network parameter of the second trained student model; θ′ s1 are the network parameters of the first trained student model.

[0105] It should be explained here that the fine-tuning training process here is similar to the embodiment of generating the first trained student model given in step S3. Fine-tuning training can be performed according to the above embodiment, and the preset number of cycles of fine-tuning training can be 15.

[0106] Step S5: Perform instance segmentation on the infrared image of the transmission line using the second converged teacher model.

[0107] During instance segmentation, the teacher model's predictions are typically more stable than those of the student model, providing relatively accurate pseudo-labels for unlabeled infrared images. The second teacher model predicts the unlabeled infrared images and outputs the corresponding instance masks, without requiring the involvement of the second student model or the first teacher-student model. Therefore, this method does not increase the computational complexity of the model when applied.

[0108] The first set of teacher-student models in this paper performs cross-domain learning, leveraging labeled visible light images and unlabeled infrared images to learn domain-invariant knowledge, such as the structural characteristics of targets, thereby achieving alignment between different domains. The second set of teacher-student models performs semi-supervised learning, leveraging labeled infrared images and unlabeled infrared images to learn domain-specific knowledge, such as the radiometric properties of targets, thereby promoting same-domain perception and reducing discrimination errors. These two complementary knowledge sets can propagate mutually, enabling accurate cross-domain semi-supervised instance segmentation of unlabeled infrared images.

[0109] In a preferred embodiment, to reduce the negative impact of erroneous pseudo-labels on the student model, pseudo-labels are filtered using mask confidence, with a confidence threshold set to 0.5. Specifically, the entropy of the pseudo-label is used to calculate the uncertainty of the pseudo-label, which is considered the confidence of the pseudo-label. For any pixel in a given pseudo-label, if its entropy value is above 0.5 after normalization, it is retained; otherwise, it is removed.

[0110] Another embodiment of the present invention provides a method for instance segmentation of infrared images of power transmission lines. In addition to steps S1 to S5 of the above embodiment, the method further includes the following steps before step S2:

[0111] Step S1.5: performing a weak data augmentation operation on the visible light image to generate an updated visible light image;

[0112] performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image;

[0113] The weak data augmentation operation includes:

[0114] The current image is cropped to a random size and rotated at a random angle according to a preset probability, and the current image is updated based on the processed image.

[0115] Before step S3, the following steps are also included:

[0116] Step S2.5: performing a strong data augmentation operation on the second infrared image to generate an updated second infrared image;

[0117] The strong data augmentation operation includes:

[0118] Randomly adjust the grayscale of the current image and superimpose Gaussian noise, and update the current image according to the processed image.

[0119] It should be explained here that in the present invention, weak data augmentation strategies and strong data augmentation strategies are used to transform images and labels (including pseudo labels). The image after the weak data augmentation strategy will be input into the first teacher model and the second teacher model to maintain the stability of the instance segmentation method, while the image after the strong data augmentation strategy will be input into the first student model and the second student model to improve the generalization of the instance segmentation method. Among them, the weak data augmentation strategy includes random cropping and random flipping with a probability of 0.5, etc., while the strong data augmentation strategy includes changing the grayscale of the image and applying Gaussian distribution noise.

[0120] Based on the above method embodiments, the present invention provides corresponding device embodiments.

[0121] like Figure 2 As shown, an embodiment of the present invention provides a transmission line infrared image instance segmentation device, including: a training image acquisition module, a teacher model training module, a student model training model, a student model fine-tuning module and an instance segmentation module.

[0122] The training image acquisition module is configured to acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model having the same network architecture; and the label information is used to characterize the boundaries of each instance in the image;

[0123] The teacher model training module is used to perform supervised training on the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model; and to perform supervised training on the second teacher model based on the first infrared image and the corresponding label information to generate a second trained teacher model;

[0124] The student model training model is used to train the first student model based on the first pseudo-label information of the second infrared image and the second infrared image to generate a first trained student model; and to train the second student model based on the second pseudo-label information of the second infrared image and the second infrared image to generate a second trained student model; wherein the first pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo-label information is obtained by inputting the second infrared image into the second trained teacher model;

[0125] The student model fine-tuning module is configured to fine-tune the first and second trained student models based on the second infrared image and the third pseudo-label information of the second infrared image, and to update the network parameters of the corresponding trained teacher models to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first and second trained teacher models, respectively, and then performing weighted fusion on a first result of the first trained teacher model and a second result of the second trained teacher model;

[0126] The instance segmentation module is used to perform instance segmentation on the transmission line infrared image through the second converged teacher model.

[0127] In a preferred embodiment, the above-mentioned transmission line infrared image instance segmentation device further comprises: a weak data augmentation module and a strong data augmentation module;

[0128] The weak data augmentation module is configured to perform a weak data augmentation operation on the visible light image to generate an updated visible light image; and perform a weak data augmentation operation on the first infrared image to generate an updated first infrared image; wherein the weak data augmentation operation includes: cropping the current image at a random size and rotating it at a random angle according to a preset probability, and updating the current image based on the processed image.

[0129] The strong data augmentation module is used to perform a strong data augmentation operation on the second infrared image to generate an updated second infrared image; wherein the strong data augmentation operation includes: randomly adjusting the grayscale of the current image and superimposing Gaussian distributed noise, and updating the current image according to the processed image.

[0130] It should be noted that the embodiments of the device described above correspond to the above-mentioned embodiments of the present invention, and can implement any of the methods described above in the present invention. In addition, the embodiments of the above-mentioned device are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the embodiment of the device provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0131] Based on the above method embodiment of the present invention, a corresponding electronic device embodiment is provided.

[0132] An embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for instance segmentation of infrared images of transmission lines described in any one of the present invention is implemented, or when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are implemented.

[0133] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0134] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0135] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0136] The memory can be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0137] Based on the above method embodiment, the present invention provides a corresponding storage medium embodiment;

[0138] Another embodiment of the present invention provides a storage medium, which includes a stored computer program. When the computer program is executed, the device where the storage medium is located is controlled to execute any of the above-mentioned power transmission line infrared image instance segmentation methods of the present invention.

[0139] The above-mentioned storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0140] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0141] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for instance segmentation of infrared images of power transmission lines, characterized in that: include: Obtaining a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model having the same network architecture; the label information is used to characterize the boundaries of each instance in the image; Performing supervised training on the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model; performing supervised training on the second teacher model based on the first infrared image and the corresponding label information to generate a second trained teacher model; The first student model is trained based on the first pseudo-label information of the second infrared image and the second infrared image to generate a first trained student model; the second student model is trained based on the second pseudo-label information of the second infrared image and the second infrared image to generate a second trained student model; wherein the first pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo-label information is obtained by inputting the second infrared image into the second trained teacher model; Fine-tune the first and second trained student models based on the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher models to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first and second trained teacher models, respectively, and performing weighted fusion on a first result of the first trained teacher model and a second result of the second trained teacher model; The second converged teacher model is used to perform instance segmentation on infrared images of transmission lines.

2. The method for instance segmentation of an infrared image of a power transmission line according to claim 1, wherein: Before performing supervised training on the first teacher model based on the visible light image and the corresponding label information to generate the first trained teacher model, the method further includes: performing a weak data augmentation operation on the visible light image to generate an updated visible light image; performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image; The weak data augmentation operation includes: The current image is cropped to a random size and rotated at a random angle according to a preset probability, and the current image is updated based on the processed image.

3. The method for segmenting transmission line infrared images according to claim 2, wherein: Before training the first student model according to the second infrared image and the first pseudo label information of the second infrared image to generate a first trained student model, the method further includes: performing a strong data augmentation operation on the second infrared image to generate an updated second infrared image; The strong data augmentation operation includes: Randomly adjust the grayscale of the current image and superimpose Gaussian noise, and update the current image according to the processed image.

4. The method for segmenting transmission line infrared images according to claim 3, wherein: The step of performing supervised training on the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model includes: Repeat the loop process until a preset number of executions is reached, wherein the loop process includes: Randomly dividing the visible light image into a plurality of batches of first training samples according to a preset number; Inputting the batches of first training samples into the current first teacher model in sequence, so that each time a batch of first training samples is input, a supervised training operation is performed on the current first teacher model; When the preset number of executions is reached, the current first teacher model is set as the first trained teacher model; The supervised training operation includes: According to the input first training sample, output the first instance mask corresponding to the first training sample through the current first teacher model; Calculating a first loss function value using a cross entropy loss according to the first instance mask and the corresponding label information; The optimizer is used to optimize the first teacher model according to the first loss function value, and the optimized model is updated as the current first teacher model.

5. The method for segmenting transmission line infrared images according to claim 4, wherein: The step of training the first student model according to the second infrared image and the first pseudo label information of the second infrared image to generate a first trained student model includes: Repeat the loop process until a preset number of executions is reached, wherein the loop process includes: randomly dividing the second infrared image into a plurality of batches of second training samples according to a preset number; Inputting the batches of second training samples into the current first student model in sequence, so that a training operation is performed on the current first student model each time a batch of second training samples is input; When the preset number of executions is reached, the current first student model is set as the first trained student model; The training operation includes: According to the input second training sample, output a second instance mask corresponding to the second training sample through the current first student model; Calculating a second loss function value by cross entropy loss according to the second instance mask and the corresponding first pseudo-label information; Utilizing an optimizer to optimize the first student model according to the second loss function value, and updating the optimized model as the current first student model; According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated using the following formula: i t1 ←th t1 +(1-α)θ s1 Among them, θ t1 is the network parameter of the first trained teacher model; α is the preset smoothing coefficient; θ s1 are the network parameters of the first student model.

6. The method for segmenting transmission line infrared images according to claim 5, wherein: The first result of the first trained teacher model is weightedly fused with the second result of the second trained teacher model in the following way: w=Sigmoid(Conv(P t1 ,P t2 )) P fuse =wP t1 +(1-in)P t2 Among them, w is the prediction weight; Sigmoid is the activation function; Conv is the full convolution layer; P t1 is the first result; P t2 is the second result; P fuse is the third pseudo label information.

7. A transmission line infrared image instance segmentation device, characterized in that: include: Training image acquisition module, teacher model training module, student model training module, student model fine-tuning module and instance segmentation module; The training image acquisition module is configured to acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model having the same network architecture; and the label information is used to characterize the boundaries of each instance in the image; The teacher model training module is used to perform supervised training on the first teacher model based on the visible light image and the corresponding label information to generate a first trained teacher model; and to perform supervised training on the second teacher model based on the first infrared image and the corresponding label information to generate a second trained teacher model; The student model training model is used to train the first student model based on the first pseudo-label information of the second infrared image and the second infrared image to generate a first trained student model; and to train the second student model based on the second pseudo-label information of the second infrared image and the second infrared image to generate a second trained student model; wherein the first pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo-label information is obtained by inputting the second infrared image into the second trained teacher model; The student model fine-tuning module is configured to fine-tune the first and second trained student models based on the second infrared image and the third pseudo-label information of the second infrared image, and to update the network parameters of the corresponding trained teacher models to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first and second trained teacher models, respectively, and then performing weighted fusion on a first result of the first trained teacher model and a second result of the second trained teacher model; The instance segmentation module is used to perform instance segmentation on the transmission line infrared image through the second converged teacher model.

8. The transmission line infrared image instance segmentation device according to claim 7, characterized in that: Also includes: Weak data augmentation module and strong data augmentation module; The weak data augmentation module is configured to perform a weak data augmentation operation on the visible light image to generate an updated visible light image; Performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image; wherein the weak data augmentation operation includes: cropping the current image to a random size and rotating the current image to a random angle according to a preset probability, and updating the current image based on the processed image; The strong data augmentation module is used to perform a strong data augmentation operation on the second infrared image to generate an updated second infrared image; wherein the strong data augmentation operation includes: randomly adjusting the grayscale of the current image and superimposing Gaussian distributed noise, and updating the current image according to the processed image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for segmenting transmission line infrared images as described in any one of claims 1 to 6 can be implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the method for segmenting transmission line infrared images as described in any one of claims 1 to 6.

Images