Power transmission line infrared image instance segmentation method and device, electronic equipment and storage medium

Abstract

The invention discloses a power transmission line infrared image instance segmentation method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the training of a first teacher model according to a visible light image and label information, and generating a first trained teacher model; according to the first infrared image and the label information, training a second teacher model to generate a second trained teacher model; according to the second infrared image and the first pseudo label information, training the first student model to generate a first trained student model; according to the second infrared image and the second pseudo label information, training the second student model to generate a second trained student model; according to the second infrared image and the third pseudo label information, performing fine tuning training on the first trained student model and the second trained student model to generate a second convergence teacher model; and performing instance segmentation on the infrared image of the power transmission line through the second convergence teacher model. According to the invention, the instance segmentation precision of the infrared image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a method, a device, an electronic device and a storage medium for segmenting an infrared image instance of a power transmission line. Background Art

[0002] In modern power systems, the inspection of transmission lines is crucial, and ensuring their safe and stable operation is the basis for ensuring power supply. With the development of drone technology, drones equipped with visible light and infrared sensors have been widely used in inspections. Using infrared images for instance segmentation can not only effectively identify electrical targets such as insulators and wire clamps, but also accurately segment their contours, thereby achieving early identification of overheating and defect detection. This method can operate stably in extreme environments such as night, rain and fog, improving the efficiency and safety of inspections.

[0003] Current instance segmentation techniques face significant challenges when applied to infrared images, mainly due to issues of data scale and feature differences. First, the data scale of infrared images is relatively small, and the acquired label information is limited, which makes it difficult for the model to fully learn and generalize. To make up for this shortcoming, existing technologies hope to assist the fitting training of infrared image data with abundant visible light image data. However, due to the drastic feature differences between infrared and visible light images, the difficulty of such cross-domain adaptation is significantly increased. This feature gap causes the performance of models trained on the source domain (visible light images) to often degrade when migrating to the target domain (infrared images). Summary of the invention

[0004] The embodiment of the present invention provides a method, device, electronic device and storage medium for instance segmentation of infrared images of power transmission lines. By implementing the present invention, the problem of poor effect of using visible light image data to assist infrared image data in fitting training due to large inter-domain differences can be solved when transferring learning from visible light images to infrared images, and the method is used for instance segmentation of infrared images of power transmission lines, thereby improving the instance segmentation accuracy of infrared images.

[0005] An embodiment of the present invention provides a method for segmenting an instance of an infrared image of a power transmission line, comprising:

[0006] Acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model with the same network architecture; the label information is used to characterize the boundary of each instance in the image;

[0007] According to the visible light image and the corresponding label information, supervised training is performed on the first teacher model to generate a first trained teacher model; according to the first infrared image and the corresponding label information, supervised training is performed on the second teacher model to generate a second trained teacher model;

[0008] According to the first pseudo label information of the second infrared image and the second infrared image, the first student model is trained to generate a first trained student model; according to the second pseudo label information of the second infrared image and the second infrared image, the second student model is trained to generate a second trained student model; wherein the first pseudo label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo label information is obtained by inputting the second infrared image into the second trained teacher model;

[0009] According to the second infrared image and the third pseudo-label information of the second infrared image, fine-tune the first trained student model and the second trained student model, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by respectively inputting the second infrared image into the first trained teacher model and the second trained teacher model, and then weightedly fusing the first result of the first trained teacher model and the second result of the second trained teacher model;

[0010] The second converged teacher model is used to perform instance segmentation on infrared images of transmission lines.

[0011] Furthermore, before the first teacher model is supervisedly trained according to the visible light image and the corresponding label information to generate the first trained teacher model, the method further includes:

[0012] Performing a weak data augmentation operation on the visible light image to generate an updated visible light image;

[0013] performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image;

[0014] The weak data augmentation operation includes:

[0015] The current image is cropped to a random size and rotated at a random angle according to a preset probability, and the current image is updated according to the processed image.

[0016] Furthermore, before training the first student model according to the second infrared image and the first pseudo-label information of the second infrared image to generate the first trained student model, the method further includes:

[0017] performing a strong data augmentation operation on the second infrared image to generate an updated second infrared image;

[0018] The strong data augmentation operation includes:

[0019] Randomly adjust the grayscale of the current image and superimpose Gaussian distribution noise, and update the current image according to the processed image.

[0020] Furthermore, the step of performing supervised training on the first teacher model according to the visible light image and the corresponding label information to generate a first trained teacher model includes:

[0021] Repeat the cycle process until a preset number of executions is reached, the cycle process comprising:

[0022] Randomly dividing the visible light image into a plurality of batches of first training samples according to a preset number;

[0023] Inputting the batches of first training samples into the current first teacher model in sequence, so that each time a batch of first training samples is input, a supervised training operation is performed on the current first teacher model;

[0024] When the preset number of executions is reached, the current first teacher model is set as the first trained teacher model;

[0025] Wherein, the supervised training operation includes:

[0026] According to the input first training sample, output a first instance mask corresponding to the first training sample through the current first teacher model;

[0027] Calculating a first loss function value by cross entropy loss according to the first instance mask and the corresponding label information;

[0028] The optimizer is used to optimize the first teacher model according to the first loss function value, and the optimized model is updated to the current first teacher model.

[0029] Furthermore, the step of training the first student model according to the second infrared image and the first pseudo-label information of the second infrared image to generate a first trained student model includes:

[0030] Repeat the cycle process until a preset number of executions is reached, the cycle process comprising:

[0031] randomly dividing the second infrared image into a plurality of batches of second training samples according to a preset number;

[0032] Inputting the batches of second training samples into the current first student model in sequence, so that each time a batch of second training samples is input, a training operation is performed on the current first student model;

[0033] When the preset number of executions is reached, the current first student model is set as the first trained student model;

[0034] The training operation includes:

[0035] According to the input second training sample, output a second instance mask corresponding to the second training sample through the current first student model;

[0036] Calculating a second loss function value by cross entropy loss according to the second instance mask and the corresponding first pseudo-label information;

[0037] Utilizing an optimizer to optimize the first student model according to the second loss function value, and updating the optimized model as the current first student model;

[0038] According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated by the following formula:

[0039] θ t1 ←αθ t1 +(1-α)θ s1

[0040] Among them, θ t1 is the network parameter of the first trained teacher model; α is the preset smoothing coefficient; θ s1 are the network parameters of the first student model.

[0041] Further, the first result of the first trained teacher model and the second result of the second trained teacher model are weightedly fused in the following manner:

[0042] w=Sigmoid(Conv(P t1 ,P t2 ))

[0043] P fuse =wP t1 +(1-w)P t2

[0044] Among them, is the predicted weight; Sigmoid is the activation function; Conv is the full convolution layer; P t1 is the first result; t2 is the second result; fuse is the third pseudo label information.

[0045] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0046] An embodiment of the present invention provides a transmission line infrared image instance segmentation device, including: a training image acquisition module, a teacher model training module, a student model training model, a student model fine-tuning module and an instance segmentation module.

[0047] The teacher model training module is used to perform supervised training on the first teacher model according to the visible light image and the corresponding label information to generate a first trained teacher model; and perform supervised training on the second teacher model according to the first infrared image and the corresponding label information to generate a second trained teacher model;

[0048] The student model training model is used to train the first student model according to the first pseudo label information of the second infrared image and the second infrared image to generate a first trained student model; train the second student model according to the second infrared image and the second pseudo label information of the second infrared image to generate a second trained student model; wherein the first pseudo label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo label information is obtained by inputting the second infrared image into the second trained teacher model;

[0049] The student model fine-tuning module is used to fine-tune the first trained student model and the second trained student model according to the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model and the second trained teacher model respectively, and then weightedly fusing the first result of the first trained teacher model and the second result of the second trained teacher model;

[0050] The instance segmentation module is used to perform instance segmentation on the infrared image of the transmission line through the second converged teacher model.

[0051] Based on the above method embodiment, the present invention provides a corresponding electronic device embodiment.

[0052] An embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for instance segmentation of infrared images of power transmission lines described in any one of the above method embodiments can be implemented.

[0053] Based on the above method item embodiments, the present invention provides a corresponding storage medium item embodiment.

[0054] An embodiment of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for segmenting an instance of an infrared image of a power transmission line described in any one of the above method embodiments can be implemented.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The embodiment of the present invention provides a method, device, electronic device and storage medium for instance segmentation of infrared images of power transmission lines. The method performs supervised training on respective teacher models according to a visible light image and a first infrared image, respectively, to generate a first trained teacher model and a second trained teacher model; then, the first student model and the second student model are trained according to the second infrared image, the first pseudo-label information of the second infrared image and the second pseudo-label information of the second infrared image, to generate a first trained student model and a second trained student model. In addition, the first trained student model and the second trained student model are fine-tuned according to the second infrared image and the third pseudo-label information of the second infrared image, and the network parameters of the corresponding trained teacher models are updated to generate a second converged teacher model. Finally, the second converged teacher model is used for instance segmentation of infrared images of power transmission lines.

[0057] The present invention trains the first teacher-student model by using a large number of labeled visible light images and a large number of unlabeled second infrared images, so that the model can learn knowledge features that are invariant between domains, thereby achieving effective alignment of knowledge features between different domains. At the same time, a small number of labeled first infrared images and a large number of unlabeled second infrared images are used to train the second teacher-student model, so that it can learn knowledge features within the domain, thereby enhancing the perception of knowledge features in the same domain and reducing judgment errors. By weighted fusion of the two pseudo-label information of the second infrared image and fine-tuning the student model based on the fused pseudo-label, the uncertainty of the pseudo-label is effectively reduced, the difference between domains is narrowed, and the generalization ability of the model is improved, thereby improving the instance segmentation accuracy of the model on infrared images. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a method for segmenting an infrared image instance of a power transmission line provided by an embodiment of the present invention.

[0059] Figure 2 The present invention is a schematic diagram of the structure of a transmission line infrared image instance segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a method for segmenting an instance of an infrared image of a power transmission line, which comprises at least the following steps:

[0062] Step S1, acquiring a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model.

[0063] Specifically, the teacher-student model includes a teacher model and a student model having the same network architecture; the label information is used to characterize the boundary of each instance in the image;

[0064] The acquired visible light image needs to have label information to facilitate subsequent training and evaluation. Label information is usually manually annotated using professional labeling tools to ensure accuracy. These labels include not only the category of the objects in the image, but also the boundaries of each object in detail to form accurate instance segmentation. Similarly, the first infrared image is annotated to ensure that the model can learn the characteristics of different objects in the infrared image. The label information of the infrared image also includes the boundaries of each instance, which is helpful for subsequent model training and performance evaluation. A second infrared image without label information is obtained. These images are used to test the model's generalization ability and ability to handle unseen data, especially in the absence of labels.

[0065] In an optional embodiment, the teacher model and the student model of the present invention have the same architecture, and both use the Mask RCNN network. Among them, Mask RCNN includes an encoder ResNet-50 and a decoder FCN. During the training process, the student model uses the output of the teacher model as a pseudo label for learning through the method of knowledge distillation. When processing visible light images and infrared images with labels, the teacher model provides high-quality features and classification results, and the student model gradually learns deeper feature representations by imitating the behavior of the teacher. This process not only improves the learning efficiency of the student model, but also improves the generalization ability of the model on unlabeled data. The label information contains the boundaries of each instance, forming a clear segmentation map. This is crucial for training models because boundary information can help the model identify the shape, size and relative position of objects. This information can significantly improve the accuracy and performance of the model in instance segmentation.

[0066] Step S2: Based on the visible light image and the corresponding label information, the first teacher model is supervised trained to generate a first trained teacher model; based on the first infrared image and the corresponding label information, the second teacher model is supervised trained to generate a second trained teacher model.

[0067] In a preferred embodiment, the step of performing supervised training on the first teacher model according to the visible light image and the corresponding label information to generate the first trained teacher model includes:

[0068] Repeat the cycle process until a preset number of executions is reached, the cycle process comprising:

[0069] Randomly dividing the visible light image into a plurality of batches of first training samples according to a preset number;

[0070] Inputting the batches of first training samples into the current first teacher model in sequence, so that each time a batch of first training samples is input, a supervised training operation is performed on the current first teacher model;

[0071] When the preset number of executions is reached, the current first teacher model is set as the first trained teacher model;

[0072] Wherein, the supervised training operation includes:

[0073] According to the input first training sample, output a first instance mask corresponding to the first training sample through the current first teacher model;

[0074] Calculating a first loss function value by cross entropy loss according to the first instance mask and the corresponding label information;

[0075] The optimizer is used to optimize the first teacher model according to the first loss function value, and the optimized model is updated to the current first teacher model.

[0076] It should be explained here that the visible light images are randomly divided into several batches, each batch containing a certain number of images. This random division can enhance the generalization ability of the model and avoid overfitting. The first training sample of each batch is sequentially input into the current first teacher model. During the input process, the teacher model processes the images of each batch to generate the corresponding instance mask. After reaching the preset number of executions, the current first teacher model is set as the first trained teacher model for subsequent use. At this time, the model should have a strong feature extraction capability and be able to effectively infer unseen data. Each training operation in the loop process includes: according to the input first training sample, the current first teacher model is used to output the corresponding first instance mask. This mask depicts the distribution of each instance in the image and provides a basis for subsequent loss calculation. By comparing the output first instance mask with the corresponding label information, the cross entropy loss function is used to calculate the first loss function value. The loss function reflects the difference between the model prediction and the true label. The smaller the value, the more accurate the model. The current first teacher model is optimized according to the calculated first loss function value using an optimizer (such as Adam, SGD, etc.). The optimizer calculates the gradient through the back-propagation algorithm and updates the model parameters to improve the model performance. The updated model parameters will form the new current first teacher model to prepare for the next training cycle. After completing the preset number of executions, the first trained teacher model obtained should be evaluated on the validation set to determine its performance on unseen data. Evaluation indicators may include accuracy, F1-score, etc. to ensure that the model meets the expected performance standards. For the second teacher model, similar steps are used for supervised training based on the first infrared image and its corresponding label information to generate the second trained teacher model. By combining the teacher model training of visible light images and infrared images, effective learning of multimodal data can be achieved, further improving the overall performance of the model.

[0077] Step S3: According to the second infrared image and the first pseudo-label information of the second infrared image, the first student model is trained to generate a first trained student model; according to the second infrared image and the second pseudo-label information of the second infrared image, the second student model is trained to generate a second trained student model.

[0078] Specifically, the first pseudo label information is obtained by inputting the second infrared image into a first trained teacher model; and the second pseudo label information is obtained by inputting the second infrared image into a second trained teacher model.

[0079] In an optional embodiment, the training of the first student model according to the second infrared image and the first pseudo-label information of the second infrared image to generate the first trained student model includes:

[0080] Repeat the cycle process until a preset number of executions is reached, the cycle process comprising:

[0081] randomly dividing the second infrared image into a plurality of batches of second training samples according to a preset number;

[0082] Inputting the batches of second training samples into the current first student model in sequence, so that each time a batch of second training samples is input, a training operation is performed on the current first student model;

[0083] When the preset number of executions is reached, the current first student model is set as the first trained student model; the preset number of executions here may be 150;

[0084] The training operation includes:

[0085] According to the input second training sample, output a second instance mask corresponding to the second training sample through the current first student model;

[0086] Calculating a second loss function value by cross entropy loss according to the second instance mask and the corresponding first pseudo-label information;

[0087] Utilizing an optimizer to optimize the first student model according to the second loss function value, and updating the optimized model as the current first student model;

[0088] According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated by the following formula:

[0089] θ t1 ←αθ t1 +(1-α)θ s1

[0090] Among them, θ t1 is the network parameter of the first trained teacher model; α is the preset smoothing coefficient, which is 0.999 here; θ s1 are the network parameters of the first student model.

[0091] Similarly, the second trained student model can also be obtained by training through similar steps as mentioned above, which will not be repeated here.

[0092] In a preferred embodiment, when the present invention trains the network, the loss function adopted is the cross entropy loss function.

[0093] It should be noted here that by inputting the second infrared image into the trained first teacher model, the corresponding first pseudo labels can be generated. These pseudo labels provide prediction results based on the teacher model for each input image, indicating the possible location and category of each instance in the image. Similarly, the second infrared image is input into the trained second teacher model to generate a second pseudo label. This method utilizes the knowledge of the teacher model to help the student model learn richer features. According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated by exponential moving average, and the teacher model uses the exponential moving average of the student model for parameter update, so that the change of the teacher model parameters is smoother, thereby enhancing the accuracy of instance mask prediction. In this process, the student model sees more infrared images and expands its generalization, while the teacher model generates higher quality pseudo labels through parameter update. Using a method similar to the above, the second student model is trained according to the second infrared image and the second pseudo label information to generate a second trained student model. In this way, the two student models can learn independently while maintaining contact with their corresponding teacher models to achieve effective knowledge transfer.

[0094] Step S4: fine-tune the first trained student model and the second trained student model according to the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model.

[0095] Specifically, the third pseudo label information is obtained by respectively inputting the second infrared image into the first trained teacher model and the second trained teacher model, and then weightedly fusing the first result of the first trained teacher model with the second result of the second trained teacher model.

[0096] In a preferred embodiment, the first result of the first trained teacher model and the second result of the second trained teacher model are weightedly fused in the following manner:

[0097] w=Sigmoid(Conv(P t1 ,P t2 ))

[0098] P fuse =wP t1 +(1-w)P t2

[0099] Among them, is the predicted weight; Sigmoid is the activation function; Conv is the full convolution layer; P t1 is the first result; t2 is the second result; fuse is the third pseudo label information.

[0100] It should be explained here that, on the one hand, due to the drastic feature differences between visible light images and infrared images, the instance masks output by the first group of teacher models contain erroneous predictions; on the other hand, due to the small size of labeled infrared images, it is difficult to meet the learning requirements of the neural network, resulting in omissions in the instance masks output by the second group of teacher models. The pseudo labels predicted in the above situations will have a negative impact on the student model. Therefore, the instance masks output by the two groups of teacher models are weighted and fused, and used as the third pseudo labels of the two groups of teacher-student models for a few cycles of fine-tuning. The third pseudo label is obtained by inputting the second infrared image into the first trained teacher model and the second trained teacher model respectively, and then using the weighted fusion method; weighted fusion can effectively integrate the knowledge of the two teacher models, make up for the shortcomings of a single model, and effectively reduce the negative transfer interference of pseudo labels on the two groups of teacher-student models, thereby generating more reliable third pseudo label information. Using the third pseudo label information, the first trained student model and the second trained student model are fine-tuned, and the network parameters of the corresponding trained teacher model are updated to improve their recognition and classification accuracy for specific instances.

[0101] The network parameters of the corresponding trained teacher model are updated using the following formulas:

[0102] θ t1 ←αθ t1 +(1-α)θ′ s1

[0103] θ s2 ←βθ t2 +(1-β)θ′ s2

[0104] Among them, θ t2 is the network parameter of the second trained teacher model; β is the preset smoothing coefficient, which is 0.999 here; θ′ s2 is the network parameter of the second trained student model; θ′ s1 are the network parameters of the first trained student model.

[0105] It should be explained here that the fine-tuning training process here is similar to the embodiment of generating the first trained student model given in step S3. Fine-tuning training can be performed according to the above embodiment, and the preset number of cycles of fine-tuning training can be 15.

[0106] Step S5: perform instance segmentation on the infrared image of the transmission line using the second converged teacher model.

[0107] In the instance segmentation stage, the prediction of the teacher model is usually more stable than that of the student model, providing more accurate pseudo-labels for unlabeled infrared images. The unlabeled infrared images are predicted by the second teacher model and the corresponding instance masks are output without the participation of the second student model and the first teacher-student model. Therefore, the method does not increase the computational complexity of the model when applied.

[0108] The first group of teacher-student models of the present invention performs cross-domain learning, which uses labeled visible light images and unlabeled infrared images to learn domain-invariant knowledge, such as the structural features of the target, so as to achieve alignment between different domains; while the second group of teacher-student models performs semi-supervised learning, which uses labeled infrared images and unlabeled infrared images to learn domain-specific knowledge, such as the radiation characteristics of the target, so as to promote perception in the same domain and reduce discrimination errors. The above two complementary knowledge can propagate to each other, thereby achieving accurate cross-domain semi-supervised instance segmentation of unlabeled infrared images.

[0109] In a preferred embodiment, in order to reduce the negative impact of erroneous pseudo-labels on the student model, pseudo-labels are screened by mask confidence, where the confidence threshold is set to 0.5. Specifically, the uncertainty of the pseudo-label is calculated by entropy, which is regarded as the confidence of the pseudo-label. Given any pixel in the pseudo-label, if its entropy value is higher than 0.5 after normalization, it will be retained, otherwise it will be removed.

[0110] Another embodiment of the present invention provides a method for instance segmentation of an infrared image of a power transmission line. In addition to steps S1 to S5 of the above embodiment, the method further includes the following steps before step S2:

[0111] Step S1.5, performing a weak data augmentation operation on the visible light image to generate an updated visible light image;

[0112] performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image;

[0113] The weak data augmentation operation includes:

[0114] The current image is cropped to a random size and rotated at a random angle according to a preset probability, and the current image is updated according to the processed image.

[0115] Before step S3, the method further includes the following steps:

[0116] Step S2.5, performing a strong data augmentation operation on the second infrared image to generate an updated second infrared image;

[0117] The strong data augmentation operation includes:

[0118] Randomly adjust the grayscale of the current image and superimpose Gaussian distribution noise, and update the current image according to the processed image.

[0119] It should be explained here that in the present invention, weak data augmentation strategy and strong data augmentation strategy are used to transform images and labels (including pseudo labels). The image after the weak data augmentation strategy will be input into the first teacher model and the second teacher model to maintain the stability of the instance segmentation method, while the image after the strong data augmentation strategy will be input into the first student model and the second student model to improve the generalization of the instance segmentation method. Among them, the weak data augmentation strategy includes random cropping and random flipping with a probability of 0.5, etc., and the strong data augmentation strategy includes changing the grayscale of the image and applying Gaussian distribution noise.

[0120] Based on the above method embodiment, the present invention provides a corresponding device embodiment.

[0121] like Figure 2 As shown, an embodiment of the present invention provides a transmission line infrared image instance segmentation device, including: a training image acquisition module, a teacher model training module, a student model training model, a student model fine-tuning module and an instance segmentation module.

[0122] The training image acquisition module is used to acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model with the same network architecture; the label information is used to characterize the boundaries of each instance in the image;

[0123] The teacher model training module is used to perform supervised training on the first teacher model according to the visible light image and the corresponding label information to generate a first trained teacher model; and perform supervised training on the second teacher model according to the first infrared image and the corresponding label information to generate a second trained teacher model;

[0124] The student model training model is used to train the first student model according to the first pseudo label information of the second infrared image and the second infrared image to generate a first trained student model; train the second student model according to the second infrared image and the second pseudo label information of the second infrared image to generate a second trained student model; wherein the first pseudo label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo label information is obtained by inputting the second infrared image into the second trained teacher model;

[0125] The student model fine-tuning module is used to fine-tune the first trained student model and the second trained student model according to the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model and the second trained teacher model respectively, and then weightedly fusing the first result of the first trained teacher model and the second result of the second trained teacher model;

[0126] The instance segmentation module is used to perform instance segmentation on the infrared image of the transmission line through the second converged teacher model.

[0127] In a preferred embodiment, the above-mentioned transmission line infrared image instance segmentation device further includes: a weak data augmentation module and a strong data augmentation module;

[0128] The weak data augmentation module is used to perform a weak data augmentation operation on the visible light image to generate an updated visible light image; and to perform a weak data augmentation operation on the first infrared image to generate an updated first infrared image; wherein the weak data augmentation operation includes: cropping the current image at a random size and rotating the current image at a random angle according to a preset probability, and updating the current image according to the processed image.

[0129] The strong data augmentation module is used to perform a strong data augmentation operation on the second infrared image to generate an updated second infrared image; wherein the strong data augmentation operation includes: randomly adjusting the grayscale of the current image and superimposing Gaussian distribution noise, and updating the current image according to the processed image.

[0130] It should be noted that the embodiments of the device described above correspond to the above embodiments of the present invention, and can implement any of the methods described above in the present invention. In addition, the embodiments of the above-mentioned device are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0131] Based on the above method embodiment of the present invention, a corresponding electronic device embodiment is provided.

[0132] An embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for instance segmentation of infrared images of transmission lines described in any one of the present invention is implemented, or when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are implemented.

[0133] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0134] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0135] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0136] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0137] Based on the above method embodiment, the present invention provides a storage medium embodiment;

[0138] Another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned power transmission line infrared image instance segmentation methods of the present invention.

[0139] The storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0140] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0141] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for instance segmentation of infrared images of power transmission lines, characterized in that: include: Acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model with the same network architecture; the label information is used to characterize the boundary of each instance in the image; According to the visible light image and the corresponding label information, supervised training is performed on the first teacher model to generate a first trained teacher model; according to the first infrared image and the corresponding label information, supervised training is performed on the second teacher model to generate a second trained teacher model; According to the first pseudo label information of the second infrared image and the second infrared image, the first student model is trained to generate a first trained student model; according to the second pseudo label information of the second infrared image and the second infrared image, the second student model is trained to generate a second trained student model; wherein the first pseudo label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo label information is obtained by inputting the second infrared image into the second trained teacher model; According to the second infrared image and the third pseudo-label information of the second infrared image, fine-tune the first trained student model and the second trained student model, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by respectively inputting the second infrared image into the first trained teacher model and the second trained teacher model, and then weightedly fusing the first result of the first trained teacher model and the second result of the second trained teacher model; The second converged teacher model is used to perform instance segmentation on infrared images of transmission lines.

2. The method for segmenting power transmission line infrared images according to claim 1, characterized in that: Before the first teacher model is supervisedly trained according to the visible light image and the corresponding label information to generate the first trained teacher model, the method further includes: Performing a weak data augmentation operation on the visible light image to generate an updated visible light image; performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image; The weak data augmentation operation includes: The current image is cropped to a random size and rotated at a random angle according to a preset probability, and the current image is updated according to the processed image.

3. The method for segmenting power transmission line infrared images according to claim 2, characterized in that: Before training the first student model according to the second infrared image and the first pseudo label information of the second infrared image to generate the first trained student model, the method further includes: performing a strong data augmentation operation on the second infrared image to generate an updated second infrared image; The strong data augmentation operation includes: Randomly adjust the grayscale of the current image and superimpose Gaussian distribution noise, and update the current image according to the processed image.

4. The method for segmenting power transmission line infrared images according to claim 3, characterized in that: The method of performing supervised training on the first teacher model according to the visible light image and the corresponding label information to generate a first trained teacher model includes: Repeat the cycle process until a preset number of executions is reached, the cycle process comprising: Randomly dividing the visible light image into a plurality of batches of first training samples according to a preset number; Inputting the batches of first training samples into the current first teacher model in sequence, so that each time a batch of first training samples is input, a supervised training operation is performed on the current first teacher model; When the preset number of executions is reached, the current first teacher model is set as the first trained teacher model; The supervised training operation includes: According to the input first training sample, output a first instance mask corresponding to the first training sample through the current first teacher model; Calculating a first loss function value by cross entropy loss according to the first instance mask and the corresponding label information; The optimizer is used to optimize the first teacher model according to the first loss function value, and the optimized model is updated to the current first teacher model.

5. The method for segmenting power transmission line infrared images as claimed in claim 4, characterized in that: The step of training the first student model according to the second infrared image and the first pseudo label information of the second infrared image to generate a first trained student model includes: Repeat the cycle process until a preset number of executions is reached, the cycle process comprising: randomly dividing the second infrared image into a plurality of batches of second training samples according to a preset number; Inputting the batches of second training samples into the current first student model in sequence, so that each time a batch of second training samples is input, a training operation is performed on the current first student model; When the preset number of executions is reached, the current first student model is set as the first trained student model; The training operation includes: According to the input second training sample, output a second instance mask corresponding to the second training sample through the current first student model; Calculating a second loss function value by cross entropy loss according to the second instance mask and the corresponding first pseudo-label information; Utilizing an optimizer to optimize the first student model according to the second loss function value, and updating the optimized model as the current first student model; According to the network parameters of the current first student model, the network parameters of the first trained teacher model are updated by the following formula: i t1 ←th t1 +(1-α)θ s1 Among them, θ t1 is the network parameter of the first trained teacher model; α is the preset smoothing coefficient; θ s1 are the network parameters of the first student model.

6. The method for segmenting power transmission line infrared images according to claim 5, characterized in that: The first result of the first trained teacher model and the second result of the second trained teacher model are weightedly fused in the following manner: w=Sigmoid(Conv(P t1 ,P t2 )) P fuse =wP t1 +(1-in)P t2 Among them, is the predicted weight; Sigmoid is the activation function; Conv is the full convolution layer; P t1 is the first result; t2 is the second result; fuse is the third pseudo label information.

7. A transmission line infrared image instance segmentation device, characterized in that: include: training image acquisition module, teacher model training module, student model training module, student model fine-tuning module and instance segmentation module; The training image acquisition module is used to acquire a visible light image with label information, a first infrared image with label information, a second infrared image without label information, a first teacher-student model, and a second teacher-student model; wherein the teacher-student model includes a teacher model and a student model with the same network architecture; the label information is used to characterize the boundaries of each instance in the image; The teacher model training module is used to perform supervised training on the first teacher model according to the visible light image and the corresponding label information to generate a first trained teacher model; and perform supervised training on the second teacher model according to the first infrared image and the corresponding label information to generate a second trained teacher model; The student model training model is used to train the first student model according to the first pseudo label information of the second infrared image and the second infrared image to generate a first trained student model; train the second student model according to the second infrared image and the second pseudo label information of the second infrared image to generate a second trained student model; wherein the first pseudo label information is obtained by inputting the second infrared image into the first trained teacher model; and the second pseudo label information is obtained by inputting the second infrared image into the second trained teacher model; The student model fine-tuning module is used to fine-tune the first trained student model and the second trained student model according to the second infrared image and the third pseudo-label information of the second infrared image, and update the network parameters of the corresponding trained teacher model to generate a first converged teacher model and a second converged teacher model; wherein the third pseudo-label information is obtained by inputting the second infrared image into the first trained teacher model and the second trained teacher model respectively, and then weightedly fusing the first result of the first trained teacher model and the second result of the second trained teacher model; The instance segmentation module is used to perform instance segmentation on the infrared image of the transmission line through the second converged teacher model.

8. The transmission line infrared image instance segmentation device according to claim 7, characterized in that: Also includes: Weak data augmentation module and strong data augmentation module; The weak data augmentation module is used to perform a weak data augmentation operation on the visible light image to generate an updated visible light image; Performing a weak data augmentation operation on the first infrared image to generate an updated first infrared image; wherein the weak data augmentation operation includes: cropping the current image at a random size and rotating the current image at a random angle according to a preset probability, and updating the current image according to the processed image; The strong data augmentation module is used to perform a strong data augmentation operation on the second infrared image to generate an updated second infrared image; wherein the strong data augmentation operation includes: randomly adjusting the grayscale of the current image and superimposing Gaussian distribution noise, and updating the current image according to the processed image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for segmenting an instance of an infrared image of a power transmission line as claimed in any one of claims 1 to 6 can be implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for segmenting an instance of an infrared image of a power transmission line as described in any one of claims 1 to 6 can be implemented.

Images