Power equipment segmentation method and system based on deep learning and user clicks

Abstract

The present application relates to the technical field of image processing, and provides a power equipment segmentation method and system based on deep learning and user clicks, comprising: acquiring a power equipment infrared image and user clicks; inputting the power equipment infrared image and the user clicks into a trained deep neural network to obtain a segmentation result of the power equipment; wherein in the training process, the deep neural network generates initial clicks by using a random sampling method, and obtains an initial segmentation result by using the deep neural network; if the initial segmentation result needs to be corrected, the clicks are generated by using an iterative sampling method, and in each iteration, the power equipment infrared image, a previous mask, a real mask and the generated clicks are input into the deep neural network to train the deep neural network and update the parameters of the deep neural network. The user is allowed to iteratively control the segmentation result by interactive clicks to correct the segmentation result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a power equipment segmentation method and system based on deep learning and user clicks. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Relevant data shows that more than 70% of power system failures are caused by the failure of power equipment. The failure of power equipment is mostly related to heating caused by loose connection, poor contact and current leakage. For a long time, the heating anomaly of power equipment will cause the creep of metal components and the aging and deterioration of insulating materials, causing serious damage to the equipment and causing power grid failure. There are many power equipment in the transmission and distribution line, and most of the power equipment is subjected to sun and rain for a long time, so the probability of failure is relatively large. Therefore, in order to avoid the occurrence of failure, it is necessary to intelligently monitor the power equipment.

[0004] In recent years, many monitoring technologies have been actively promoted and applied. Among the many monitoring technologies, infrared thermal imaging technology is favored due to its advantages of no power outage, no sampling, no disassembly, etc. In order to improve the intelligent level of electrical equipment state monitoring, some intelligent image processing methods can be used to segment the infrared image and extract the power equipment region of interest, which is convenient for fault diagnosis work. Segmenting the target in the infrared image is an important step in the early processing of the power equipment thermal fault automatic diagnosis process.

[0005] At present, the analysis and labeling of infrared images mainly rely on manual work, and there are few personnel specializing in infrared monitoring of power equipment, which cannot meet the analysis and labeling work of a large amount of infrared images. In addition, the professional knowledge level and labeling experience accumulation of part of the monitoring personnel are insufficient, so it is difficult to quickly, accurately and correctly segment and label the infrared image of the power equipment, which greatly restricts the improvement of the intelligent level of power equipment state monitoring.

[0006] In the field of digital image research, many image segmentation methods have appeared, including region growing, split and merge, morphological watershed, threshold-based methods, edge detection-based methods and deep learning-based methods. However, the current segmentation method for power equipment in the infrared image can only segment the main target, and cannot segment any target of interest; and when the segmentation effect is not ideal, it is difficult to continue to correct. SUMMARY

[0007] To solve the technical problems in the background art, the application provides a power equipment segmentation method and system based on deep learning and user clicks, which trains a deep neural network by an iterative sampling method to generate clicks, so that the trained deep neural network allows users to iteratively control the segmentation result by interactive clicks to modify the segmentation result.

[0008] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0009] The first aspect of the application provides a power equipment segmentation method based on deep learning and user clicks, which includes:

[0010] Obtaining power equipment infrared images and user clicks;

[0011] Inputting the power equipment infrared images and user clicks into the trained deep neural network to obtain the segmentation result of the power equipment;

[0012] In the training process of the deep neural network, an initial click is generated by a random sampling method, and an initial segmentation result is obtained by the deep neural network, if the initial segmentation result needs to be modified, a click is generated by an iterative sampling method, and in each iteration, the power equipment infrared image, the previous mask, the real mask and the generated click are input into the deep neural network to train the deep neural network and update the parameters of the deep neural network.

[0013] Further, the positive click generated by the random sampling method is randomly selected from the set of real mask pixels, and at least several pixels are between any two positive clicks, and any positive click is at least several pixels away from the target boundary.

[0014] Further, the iterative sampling method specifically includes:

[0015] Obtaining difference pixels by comparing the difference between the previous mask and the real mask;

[0016] Using connected components to divide the difference pixels into different clusters, and selecting the largest cluster according to the connected area;

[0017] Generating a point in the largest cluster as a click, and determining whether the click is a positive click or a negative click.

[0018] Further, after the deep neural network calculates the minimum external frame of the previous mask and the generated click, a segmentation network is used to obtain a rough mask.

[0019] Further, the deep neural network compares the difference between the rough mask and the previous mask, obtains a difference mask, calculates the maximum connected region of the generated clicks contained in the difference mask, and obtains a focus region based on the maximum connected region, and the focus region of the rough mask is cropped and expanded to obtain a focus cropped rough mask.

[0020] Further, the deep neural network predicts a detail map and a boundary map based on the focus cropped rough mask, calculates an edge region of the rough mask, and fuses the detail map, the boundary map and the edge region of the rough mask to obtain a refined local mask.

[0021] Further, the deep neural network binarizes the refined local mask, calculates the difference region between the refined local mask and the previous mask, and selects the maximum connected region containing the generated clicks in the difference region as an update region, and updates the refined local mask to the previous mask in the update region.

[0022] The second aspect of the present application provides a power equipment segmentation system based on deep learning and user clicks, which comprises:

[0023] a data acquisition module configured to acquire power equipment infrared images and user clicks;

[0024] an image segmentation module configured to input the power equipment infrared images and the user clicks into the trained deep neural network to obtain a segmentation result of the power equipment;

[0025] In the training process of the deep neural network, an initial click is generated by using a random sampling method, and an initial segmentation result is obtained by using the deep neural network, if the initial segmentation result needs to be corrected, a click is generated by using an iterative sampling method, and in each iteration, the power equipment infrared images, a previous mask, a real mask and the generated click are input into the deep neural network to train the deep neural network and update the parameters of the deep neural network.

[0026] The third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the power equipment segmentation method based on deep learning and user clicks as described above.

[0027] The fourth aspect of the present application provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the steps of the power equipment segmentation method based on deep learning and user clicks as described above.

[0028] Compared with the prior art, the present application has the following advantages:

[0029] The present application trains the deep neural network by the way of generating clicks through iterative sampling method, so that the trained deep neural network allows the user to iteratively control the segmentation result through interactive clicks, and corrects the segmentation result until a satisfactory segmentation effect is achieved.

[0030] The present application can help the labeling personnel to quickly and accurately label the power equipment of interest in the infrared image, thereby greatly saving the labor cost and time cost; compared with the traditional segmentation method, the present application has higher accuracy and faster speed.

[0031] The present application does not need to update the mask of all pixels after each click. Instead, the present application only performs local update in the area expected to be modified by the user. Thus, not only the loss of details that have been labeled is avoided, but also the repeated calculation and redundancy are reduced, thereby greatly saving the calculation amount and improving the running speed of the network. BRIEF DESCRIPTION OF DRAWINGS

[0032] The drawings constituting a part of the specification of the present application are used to provide further understanding of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitation on the present application.

[0033] Figure 1 is a flowchart of the power equipment segmentation method based on deep learning and user clicks of the present application embodiment one;

[0034] Figure 2 is a structural schematic diagram of the deep neural network of the present application embodiment one;

[0035] Fig. 3(a) is a segmentation effect schematic diagram of a current transformer of the present application embodiment one;

[0036] Fig. 3(b) is a segmentation effect schematic diagram of a voltage transformer of the present application embodiment one;

[0037] Fig. 3(c) is a segmentation effect schematic diagram of a circuit breaker of the present application embodiment one;

[0038] Fig. 3(d) is a segmentation effect schematic diagram of a lightning arrester of the present application embodiment one. DETAILED DESCRIPTION

[0039] The present application will be further described below in combination with the drawings and embodiments.

[0040] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0041] Embodiment one

[0042] The embodiment provides a power equipment segmentation method based on deep learning and user clicks.

[0043] The power equipment segmentation method based on deep learning and user clicks provided by the embodiment comprises the following steps as shown in the figure: Figure 1

[0044] (1) A series of power equipment infrared images are collected to form an image semantic segmentation dataset;

[0045] (2) An infrared image power equipment interactive segmentation neural network based on a FocalClick interactive segmentation framework is constructed, mainly including a target cropping module, a rough segmentation module, a focal cropping module, a local refinement module and a gradual merging module;

[0046] (3) A known real mask is used to simulate user input by using an interactive click generation strategy; first, an initial click is generated by using a random sampling method, and if the initial segmentation result (the initial segmentation result is a segmentation result output by the network after the first round of interaction based on the initial click) needs to be further modified (the segmentation result does not reach a preset target IoU value), an iterative sampling method is used to generate subsequent clicks;

[0047] (4) The power equipment infrared image and the generated clicks are input into the deep neural network in batches, the deep neural network is trained by using an Adam optimizer, and the parameters of the deep neural network are updated; that is, in each iteration, the power equipment infrared image, the real mask and the generated clicks are input into the deep neural network, the deep neural network is trained, and the parameters of the deep neural network are updated;

[0048] (5) The power equipment infrared image, the previous mask and the user clicks are input into the trained deep neural network for processing, and through several rounds of user interaction, a user-satisfactory segmentation result is obtained. The user clicks are pixel points selected by the user in the power equipment infrared image (the user adds one pixel point in each round, and thus a satisfactory segmentation result is obtained after n rounds); and the previous mask is the segmentation result obtained after the previous round of interaction, and if it is the first round of interaction, the previous mask is set to 0.

[0049] Specifically, step (2) is: the constructed neural network is based on the FocalClick framework, as shown in the figure Figure 2 ​As shown, it specifically includes a target cropping module, a rough segmentation module, a focus cropping module, a local refinement module, and a gradual merging module. A Gaussian map is used to represent the clicks, and the click map, the image, and the previous mask (the previous mask is the segmentation result obtained after the previous round of interaction, and if it is the first round of interaction, the previous mask is set to 0) are connected and input into the target cropping module to select and crop the region around the target object, adjust its size to a smaller scale, and send it to the rough segmentation module to predict a rough mask. Then, the focus cropping module finds the local region that needs to be refined through morphological analysis, and inputs it into the local refinement module after magnification for further optimization. Finally, the refined local mask is aligned and merged with the full-image mask by gradual merging.

[0050] The target cropping module filters out background information irrelevant to the target object. The module first calculates the minimum bounding box of the previous mask and newly added clicks, and then expands it by a ratio r TC = 1.4. Then, the region within the minimum bounding box in the cropped image, positive and negative click maps, and the previous mask is adjusted to a small scale.

[0051] The rough segmentation module aims to obtain a rough mask of the target through a segmentation network, so as to locate the focus cropping region and perform further local refinement operation. The module selects the classic segmentation network HRNet as a representative. Specifically, the cropped image output by the target cropping module is fused with the previous mask and the positive and negative click maps through convolution layers (Conv) and then through an addition operation, and then processed through a segmentation network to obtain a feature map and a rough mask.

[0052] The focus cropping module aims to locate the region modified by the new clicks. The module first compares the difference between the rough mask and the previous mask to obtain a difference mask M xor . Then, the maximum connected region containing the new clicks in the difference mask M xor is calculated, and a bounding box is generated for the maximum connected region, which is called the focus region. Similar to the target cropping, the focus region in the feature map and the rough mask output by the rough segmentation module is cropped, and expanded by a ratio r FC = 1.4; the focus region in the input original image and click map is also cropped and expanded. Thus, the rough mask, image, click map, and feature map after focus cropping are obtained.

[0053] wherein the local refinement module is configured to restore the details in the rough mask obtained by the focal point cropping. In one aspect, three convolutional layers are used to extract low-level features from the cropped rough mask, the image and the click map. In another aspect, a convolutional layer and an up-sampling layer are used to adjust the length, width and channel number of the cropped feature map to be consistent with those of the low-level features. Then, the two are summed element by element. After that, the summed features are predicted by three convolutional layers to obtain a detail map M d and an edge map M b At the same time, the edge region of the rough mask is calculated M l The local refined mask M r is obtained by fusing the three through the following formula:

[0054] M r = Sigmoid(M b )*M d +(1-Sigmoid(M b ))*M l

[0055] wherein the progressive merging module is configured to align and merge the refined local mask with the full image mask. The module uses morphological analysis to determine the user's intention. After adding a new click, the refined local mask is binarized with a threshold of 0.5, and the difference region between the refined local mask and the previous mask is calculated. Then, in the difference region, the largest connected region containing the new click is selected as the update region. In this update region, the refined local mask is updated to the corresponding position of the previous mask, and in other regions, it remains the same as the previous mask.

[0056] wherein step (3) is specifically: generating an initial positive click (i.e. the first click) and obtaining an initial segmentation result by using a random sampling method; if the initial segmentation result needs to be further corrected, generating subsequent clicks by using an iterative sampling method.

[0057] wherein the random sampling method is specifically:

[0058] According to the actual situation and user habits, the initial click should be a positive click. In the network training process, in order to improve the robustness of the model and adapt to different user inputs, the same image needs to be trained n times. Therefore, for each image, n different initial positive clicks need to be sampled in advance to form an initial click set C for each training of the image. The sampling method is as follows: let G be the set of real mask pixels of the target power equipment in the image, and randomly select n pixels in G as positive clicks, while these clicks need to meet the following filtering conditions: 1) any two clicks are at least d step pixels apart; 2) any click is at least d margina pixel. Wherein, d step and d margin are set values. For each image, only one point is selected from the initial click set C sampled in advance as the initial click for this training.

[0059] Wherein, the iterative sampling method is specifically:

[0060] ①Obtain the difference pixels by comparing the difference between the previous mask and the real mask. In the tth iteration, the previous mask is the click generated based on the (t-1)th iteration, and the mask generated by the deep neural network (i.e., the segmentation result);

[0061] ②Use connected components to divide these difference pixels into different clusters, and select the largest one according to the connected area;

[0062] ③Sample a point in this cluster. If no point has been sampled in this cluster before (because it may have gone through multiple iterations and sampled a point in this cluster in a previous iteration), the center point of this cluster should be selected; if a point has been sampled before, the sampled point should be the farthest from the previously sampled point and the cluster boundary, so as to avoid too close distance between multiple clicks;

[0063] ④Determine the nature of the click. If the pixel position corresponding to the cluster is on the target object, the generated point is a positive click, otherwise it is a negative click. Positive clicks or negative clicks constitute subsequent clicks in a round of interaction. One iteration produces a new pixel point.

[0064] In step (4), random flipping and random scaling are used for data augmentation. The parameter settings of the Adam optimizer are β1=0.9 and β2=0.999. A total of 230 iterations are performed. The initial learning rate is set to 5×10-4, and it is reduced by 10 times after the 190th and 220th iterations. The batch size is set to 32, and parallel training is performed on 4 blocks of GeForce GTX 1080ti graphics cards.

[0065] In step (5), the maximum number of clicks is set to 20, and pictures that cannot achieve the predetermined effect after more than 20 clicks will be recorded.

[0066] As shown in FIG. 3(a), FIG. 3(b), FIG. 3(c), FIG. 3(d), the embodiment is tested on the collected infrared image dataset of power equipment, and there are a total of 4 types of construction objects including current transformers, voltage transformers, switches and arresters. 90% of the data set is randomly selected for training, and the remaining 10% is tested. The standard evaluation criterion NoC (Number of Clicks) of interactive segmentation is used for evaluation. Table 1 shows the NoC value of the embodiment on the test set. NoC@80%, NoC@85%, NoC@90% respectively represent the number of clicks required to achieve 80%, 85%, 90% IoU (Intersection over Union). The smaller the value of NoC, the better the performance.

[0067] Table 1, evaluation index

[0068] Model NoC @ 80% NoC @ 85% NoC @ 90% >＝20@85％ >＝20@90％ The method of the present embodiment 1.42 1.72 2.88 1 9

[0069] As can be seen from Table 1, an average of only 2.88 clicks is required to achieve an IoU of more than 90%. Among the 333 images participating in the test, only 1 image cannot achieve an IoU of 85% after 20 clicks, and only 9 images cannot achieve an IoU of 90% after 20 clicks. The experimental results show that the method can help the labeling personnel to quickly and accurately label the power equipment from the infrared image, and shows the practicability of the application.

[0070] The power equipment segmentation method based on deep learning and user clicks provided by the embodiment adopts an interactive segmentation framework based on FocalClick, and trains network parameters by inputting infrared images containing power equipment and simulated user clicks. During testing, the image with unknown labels is directly passed through a one-time forward propagation of the neural network, and the segmentation result of the power equipment can be obtained.

[0071] The power equipment segmentation method based on deep learning and user clicks provided by the embodiment first collects a batch of infrared images of power equipment, and labels the power equipment in the images; then uses labeled data and simulated user clicks to train an interactive segmentation network to predict the mask of the power equipment in the infrared image. Compared with traditional infrared image power equipment segmentation methods, better performance is achieved while achieving real-time processing speed; it can help the staff to quickly and accurately mask label the power equipment in the infrared image.

[0072] The power equipment segmentation method based on deep learning and user clicks provided in the embodiment allows the user to iteratively control the segmentation result through interactive clicks until a satisfactory segmentation effect is achieved. The method can help the annotator quickly and accurately annotate the power equipment of interest in the infrared image, thereby greatly saving the labor cost and time cost. Compared with the traditional segmentation method, the method has higher accuracy and faster speed.

[0073] The power equipment segmentation method based on deep learning and user clicks provided in the embodiment does not need to update the mask of all pixels after each click. Instead, the application only performs local update in the area expected to be modified by the user. Thus, not only the loss of details that have been annotated is avoided, but also the repeated calculation and redundancy are reduced, thereby greatly saving the calculation amount and improving the running speed of the network.

[0074] Embodiment Two

[0075] The power equipment segmentation system based on deep learning and user clicks provided in the embodiment specifically comprises:

[0076] A data acquisition module configured to acquire a power equipment infrared image and user clicks;

[0077] An image segmentation module configured to input the power equipment infrared image and the user clicks into the trained deep neural network to obtain a segmentation result of the power equipment;

[0078] In the training process of the deep neural network, an initial click is generated by using a random sampling method, and an initial segmentation result is obtained by using the deep neural network. If the initial segmentation result needs to be corrected, a click is generated by using an iterative sampling method, and in each iteration, the power equipment infrared image, the previous mask, the real mask, and the generated click are input into the deep neural network to train the deep neural network and update the parameters of the deep neural network.

[0079] It should be noted that each module in the embodiment corresponds to each step in Embodiment One, and the specific implementation process is the same, which will not be repeated here.

[0080] Embodiment Three

[0081] The embodiment provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the power equipment segmentation method based on deep learning and user clicks as described in Embodiment One.

[0082] Embodiment Four

[0083] The embodiment provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the power equipment segmentation method based on deep learning and user clicks according to the embodiment.

[0084] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operation steps to be performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

Claims

1. A power equipment segmentation method based on deep learning and user clicks, characterized in that, include: Acquire infrared images of power equipment and user clicks; The infrared image of the power equipment and the user's clicks are input into a trained deep neural network to obtain the segmentation result of the power equipment. During the training process, the deep neural network uses a random sampling method to generate initial clicks and obtains initial segmentation results. If the initial segmentation results need to be corrected, an iterative sampling method is used to generate clicks. In each iteration, the infrared image of the power equipment, the previous mask, the real mask, and the generated clicks are input into the deep neural network to train the deep neural network and update its parameters. The iterative sampling method is specifically as follows: Compare the previous mask with the real mask to obtain the difference pixels; The difference pixels are divided into different clusters using connected components, and the largest cluster is selected based on the connected area. Generate a point within the largest cluster as a click, and determine whether the click is a positive or negative click; The previous mask is the segmentation result obtained after the previous round of interaction; the real mask is the known pixel-level label of a series of infrared images of power equipment.

2. The power equipment segmentation method based on deep learning and user clicks as described in claim 1, characterized in that, The positive clicks generated by the random sampling method are randomly selected from the set of real mask pixels, and there must be at least a certain number of pixels between any two positive clicks, and any positive click must be at least a certain number of pixels away from the target boundary.

3. The power equipment segmentation method based on deep learning and user clicks as described in claim 1, characterized in that, After the deep neural network calculates the minimum outer box of the previous mask and the generated click, a segmentation network is used to obtain a coarse mask.

4. The power equipment segmentation method based on deep learning and user clicks as described in claim 3, characterized in that, The deep neural network compares the differences between the coarse mask and the previous mask to obtain a difference mask. It calculates the maximum connected region of the generated click contained in the difference mask and obtains the focal region based on the maximum connected region. The focal region of the coarse mask is then cropped and expanded to obtain a coarse mask with the focal region cropped.

5. The power equipment segmentation method based on deep learning and user clicks as described in claim 4, characterized in that, The deep neural network predicts detail and boundary maps based on a coarse mask after focal clipping, calculates the edge regions of the coarse mask, and fuses the detail, boundary, and edge regions of the coarse mask to obtain a locally refined mask.

6. The power equipment segmentation method based on deep learning and user clicks as described in claim 5, characterized in that, The deep neural network binarizes the refined local mask, calculates the difference region between the refined local mask and the previous mask, and selects the largest connected region containing the generated click as the update region within the difference region. In the update region, the refined local mask is updated onto the previous mask.

7. A power equipment segmentation system based on deep learning and user clicks, characterized in that, include: The data acquisition module is configured to acquire infrared images of power equipment and user clicks. The image segmentation module is configured to input the infrared image of the power equipment and the user's click into a trained deep neural network to obtain the segmentation result of the power equipment. During the training process, the deep neural network uses a random sampling method to generate initial clicks and obtains initial segmentation results. If the initial segmentation results need to be corrected, an iterative sampling method is used to generate clicks. In each iteration, the infrared image of the power equipment, the previous mask, the real mask, and the generated clicks are input into the deep neural network to train the deep neural network and update its parameters. The iterative sampling method is specifically as follows: Compare the previous mask with the real mask to obtain the difference pixels; The difference pixels are divided into different clusters using connected components, and the largest cluster is selected based on the connected area. Generate a point within the largest cluster as a click, and determine whether the click is a positive or negative click; The previous mask is the segmentation result obtained after the previous round of interaction; the real mask is the known pixel-level label of a series of infrared images of power equipment.

8. The power equipment segmentation system based on deep learning and user clicks as described in claim 7, characterized in that, The positive clicks generated by the random sampling method are randomly selected from the set of real mask pixels, and there must be at least a certain number of pixels between any two positive clicks, and any positive click must be at least a certain number of pixels away from the target boundary.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the power equipment segmentation method based on deep learning and user clicks as described in any one of claims 1-6.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the power equipment segmentation method based on deep learning and user clicks as described in any one of claims 1-6.

Images