Overhead transmission line foreign matter defect training sample generation method

Abstract

The present application relates to the field of image processing, and particularly relates to a kind of overhead transmission line foreign matter class defect simulation sample generation method, the method comprises: forming the sample image of defect-free overhead transmission line;Component segmentation network is constructed, and the mask of different key power components is extracted;Line suspension point coordinate set is constructed, and a line suspension point is randomly selected from line suspension point set;Simple three-dimensional modeling is carried out on foreign matter with plane grid, in sample image, foreign matter label rectangular frame is randomly generated, and foreign matter image and foreign matter label rectangular frame are added to sample image, so that line suspension point and foreign matter suspension point coincide with each other.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, more particularly to a method for generating training samples of foreign matter defects of overhead transmission lines. BACKGROUND

[0002] Foreign matter defects of overhead transmission lines are one of the important hidden dangers threatening the safe and stable operation of power transmission networks. The foreign matters that may be suspended on the transmission lines include damaged plastic bags, woven bags, iron wires, rags, branches, bird nests and the like. These foreign matters may be suspended on overhead ground wires and tower power components, and manual identification consumes manpower and resources.

[0003] With the development of neural network technology, it is possible to identify these foreign matters through computer algorithms by using a neural network system. However, the use of a neural network model requires a large amount of data for training.

[0004] The image data captured from daily inspection tasks mainly includes defect-free samples. The collected foreign matter defect samples are single in form, small in quantity and unbalanced. Therefore, the development of an unmanned aerial vehicle (UAV) inspection intelligent defect detection and identification algorithm lacks necessary training data. The existing overhead transmission line defect detection and identification method based on deep learning is prone to false detection and missed detection for foreign matter defects.

[0005] Therefore, a sample expansion technique for foreign matter defects is needed to expand the number of foreign matter defect samples, so that the foreign matter defect samples in the data set for training the defect detection and identification algorithm are more diverse. SUMMARY

[0006] OBJECTIVE

[0007] The existing unmanned aerial vehicle (UAV) inspection image defect intelligent detection and identification algorithm requires a large amount of sample data for training. However, the data collected in daily inspection tasks mainly includes defect-free data, which makes it difficult to train a defect detection algorithm with high generalization ability.

[0008] The present application provides an effective solution to the problem of a small number of foreign matter defect samples and a single form in the current unmanned aerial vehicle (UAV) inspection process. The generated foreign matter model takes into account the suspension of foreign matters and introduces the deformation of foreign matters under the action of random wind. The generated foreign matter sample is more diverse and closer to the real situation.

[0009] Based on a large number of defect-free samples collected during the unmanned aerial vehicle (UAV) inspection process, the present application uses generative artificial intelligence to generate partial images of damaged woven bags, plastic bags, rags and other foreign matters with different forms from the defect-free samples. The internal repair algorithm is used to fuse the original defect-free samples, realize the expansion of foreign matter defect samples, and generate a label frame for the generated foreign matter defect part for the training of an intelligent identification algorithm.

[0010] To achieve the above object, the present application adopts the following technical solutions:

[0011] An overhead power transmission line foreign object defect simulation sample generation method, the method comprising:

[0012] Step (1), flying along the overhead power transmission line with a drone to take pictures of the overhead power transmission line at a horizontal angle, forming a sample image of the overhead power transmission line without defects;

[0013] Step (2), constructing a component segmentation network to extract the masks of different key power components in the sample image, each mask corresponding to a power component, and the coverage area of each power component forming a coordinate point set;

[0014] Step (3), traversing all pixels in the key power component masks extracted by the component segmentation network, for each pixel, determining the contact points of two power components by judging whether the 8-neighborhood of the pixel contains pixels of other power component masks, judging whether each contact point belongs to a line suspension point according to the contact point category, constructing a line suspension point coordinate set, and randomly selecting a line suspension point from the line suspension point set for step (4);

[0015] Step (4), performing simple three-dimensional modeling of the foreign object with a planar grid, randomly selecting a vertex in the planar grid as a foreign object suspension point, and using the PBD algorithm to iteratively calculate the positions and velocities of all vertices in the grid under the combined action of gravity and random wind force, fitting the region surrounded by all vertex coordinates on the planar grid into a mask after a predetermined number of iterations, so that the foreign object suspension point in the mask coincides with the line suspension point;

[0016] Step (5) generates a foreign object image based on the mask and foreign object information, fuses the foreign object image into the position of the sample image mask, calculates the coordinates of the circumscribed rectangle of the mask generated in step (3), and generates a foreign object annotation rectangle frame in the sample image, and uses the foreign object annotation rectangle frame to label the foreign object in the sample image.

[0017] Further, the neural network constructed in step (2) is a Mask R-CNN network.

[0018] Further, it further comprises the step of constructing a foreign object stable diffusion model, which is constructed based on the WIT, RedCaps, MMDialog, and CxC image-text public data sets.

[0019] Further, the method further comprises: randomly selecting a plurality of line suspension points from the plurality of line suspension points; and for each selected line suspension point, adding a foreign object image at the suspension position corresponding to the line suspension point, so that the foreign object suspension point coincides with the line suspension point.

[0020] Further, the power component comprises: a clamp, a ground wire, an insulator, and a coupling fitting.

[0021] Further, the method further comprises: for different sample images, randomly adding different types of foreign object images in the sample images.

[0022] Further, the process of generating the foreign object image in step (5) comprises: inputting the generated mask into a foreign object stable diffusion model, and randomly selecting a foreign object type and a pattern color description prompt word from a prompt word library, and generating a foreign object image of a region corresponding to the mask according to the foreign object type and the pattern color description prompt word.

[0023] Further, the method further comprises: repeating the processes of steps (3)-(5) for different suspension points, thereby generating different defect sample images.

[0024] Advantages

[0025] The present application can quickly generate a large number of realistic and defect-morphology-varied foreign object defect sample images from a large number of defect-free unmanned aerial vehicle power inspection sample images, and provide defect simulation training samples for development of a defect detection algorithm, so as to improve the detection accuracy of the defect detection algorithm for foreign object defects. Moreover, the present application solves the problem of logically unreasonable generated defect images, and simulates defect sample images close to real scenes based on the possible suspension points. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The present application is an overall schematic flowchart of an overhead transmission line foreign object defect simulation sample generation method;

[0027] Figure 2 The present application is a defect-free unmanned aerial vehicle power inspection sample image;

[0028] Figure 3 The present application is a sample image for Figure 2 The present application is an example image after segmentation of a power component in the sample image, and the segmented reference numerals in the image represent: 1. background; 2. tower; 3. insulator; 4. grading ring; 5. coupling fitting; 6-12. conductor; 13-16. clamp; 17-20. anti-vibration hammer;

[0029] Figure 4 The present application is a suspension point set (marked in white in the image) selected from the power component mask;

[0030] Figure 5 An initial planar grid example for simulating foreign matter, where P is a fixed foreign matter hanging point;

[0031] Figure 6 A generated foreign matter mask map as an example for controlling the diffusion model to generate foreign matter (the diffusion model only superimposes random noise on the mask area, and then generates a high-quality local image by gradually reducing noise in the mask area);

[0032] Figure 7 A foreign matter defect simulation sample example, as can be seen from the figure, the simulated foreign matter is very close to the real hanging situation. DETAILED DESCRIPTION

[0033] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0034] The flow of generating the overhead transmission line foreign matter defect training sample is as shown in Figure 1

[0035] As shown in the figure, the method for automatically generating the foreign matter defect training sample of the embodiment includes the following processes:

[0036] (1) An unmanned aerial vehicle flies along the overhead transmission line to take pictures of the overhead transmission line at eye level or any other required angle, forming sample images of the defect-free overhead transmission line. The sample images can be taken from both sides respectively, and the target image area between the two taking processes can have a certain overlap.

[0037] (2) An instance segmentation network of the main components contained in the overhead transmission line is constructed. The key parts in the power tower, including the wire clamp, ground wire, insulator, and connecting hardware, are labeled in the real inspection image without defects, and a power component instance segmentation training set is established. The instance segmentation network model of the power tower component is trained by using the constructed power component instance segmentation training set.

[0038] For the overhead transmission line, a list database of key parts is first established, and each key component is labeled one by one according to the list in the database.

[0039] The instance segmentation network adopts Mask R-CNN to realize instance segmentation of the input inspection image, and outputs a power component mask map, as shown in Figure 3 ​All the defect-free inspection samples collected are input into the instance segmentation network for instance segmentation to extract the masks of key power components. The Mask R-CNN network is improved based on the Faster R-CNN architecture, and a branch is added to predict the segmentation mask of each instance based on the Faster R-CNN.

[0040] In an implementation manner, the Mask R-CNN includes:

[0041] Backbone (backbone network), using a pre-trained convolutional neural network (such as ResNet or ResNeXt) as a feature extractor to extract high-level feature maps from input images.

[0042] Region Proposal Network (RPN, region proposal network), this component is used to generate candidate regions (Region of Interest, RoI), that is, image regions that may contain objects. These RoIs will be used for more accurate target detection and segmentation later.

[0043] RoI Align, used to more accurately crop the corresponding RoI features from the feature map through bilinear interpolation, avoiding information loss due to quantization, thereby improving the quality of mask prediction;

[0044] Classification and Bounding Box Regression Head: used to classify each RoI (determine which category it belongs to) and adjust the position of the bounding box to better enclose the object;

[0045] Mask Prediction Head: In addition to classification and bounding box regression, there is a special branch for each RoI to predict a binary mask corresponding to the RoI. This allows the model to provide pixel-level segmentation results for each detected object.

[0046] (3) Traverse all pixels in the key power component mask extracted by the instance segmentation network, for each pixel, determine the contact point of two power components by judging whether the 8-neighborhood of the pixel contains pixels of other power component masks, determine whether each contact point belongs to a line suspension point according to the contact point category, construct a line suspension point coordinate set, and randomly select a line suspension point from the line suspension point set. Whether the contact point belongs to the suspension point is determined according to the category of the contact point, that is, according to the easy-to-suspend position of the suspended object in the line, such as the contact points between fittings and fittings, wires and clamps, wires and anti-vibration hammers, which are easy-to-suspend foreign object suspension points and should be screened out.

[0047] Through the extracted mask, the coordinate positions of the parts prone to hanging foreign objects, such as the connection of each fitting, the contact points of the conductor and the clamp, and the contact points of the conductor and the damping hammer, in the image are calculated.

[0048] For each sample generation, a random line hanging point coordinate is selected for subsequent matching with the foreign object hanging point coordinate.

[0049] (4) A hanging point is selected from the hanging points of the possible foreign objects, a simple three-dimensional modeling of the foreign objects is performed using a planar grid, a vertex in the planar grid is randomly selected as a foreign object hanging point, the positions and velocities of all vertices in the grid under the combined action of gravity and random wind force are iteratively calculated using the PBD algorithm, and after a predetermined number of iterations, the figure area surrounded by all vertex coordinates on the planar grid is fitted into a mask.

[0050] For example, a simple three-dimensional modeling of damaged woven bags, plastic bags, rags and other foreign objects is performed using a planar grid, a vertex in the planar grid is randomly selected as a hanging point (as shown in Figure 5 The positions and velocities of all vertices in the grid under the combined action of gravity and random wind force are iteratively calculated using the PBD algorithm (the specific process of the algorithm is described in patent US7616204), and after a sufficient number of iterations, the figure area surrounded by all vertex coordinates on the grid is fitted into a mask. The mask is smoothed using morphological closing operation to generate the final mask. The PBD algorithm can be executed using NVIDIA PhysX, Houdini or Blender software. NVIDIA PhysX is a widely used physics engine that supports rigid body, soft body, cloth and fluid simulation.

[0051] Specifically, the PBD algorithm updates the positions and velocities of all vertices of the planar grid each time. As shown in Figure 5In this embodiment, the grid size is 10x10, the initial speed of all vertices is 0, and after more than 50 iterations, the coordinates of all vertices no longer have large changes. The point set composed of all vertex coordinates updated to the new coordinate position is re-calculated using the Alpha Shapes algorithm to obtain the mask. (Alpha Shapes algorithm implementation reference H. Edelsbrunner, D. Kirkpatrick and R. Seidel, "On the shape of a set of points in the plane," in IEEE Transactions on Information Theory, vol. 29, no. 4, pp. 551-559, July 1983, doi: 10.1109 / TIT.1983.1056714.)

[0052] The mask generated in step (3) is input into the foreign object stable diffusion model, and the foreign object type (plastic bag, woven bag, rags, etc.) and pattern color description prompt words are randomly selected from the prompt library to generate a foreign object graph for the specified area of the mask, and superimposed and fused into the original graph mask area (in the figure, the mask area is superimposed with noise to generate a target foreign object graph, and fused with the background), thereby synthesizing a foreign object defect sample (i.e., fusing the generated foreign object image into the original sample image). The foreign object stable diffusion model is trained based on the WIT, RedCaps, MMDialog, CxC, etc. image-text public data sets and the image-text pair data set constructed from the existing power grid inspection images to realize the stable diffusion model for generating a local foreign object graph under the restriction of the input mask. That is, various image-text public data sets and foreign object-text pair data sets in power grid inspection images are obtained, the stable diffusion model is trained, and then based on the foreign object type (plastic bag, woven bag, rags, etc.) and pattern color description prompt words, the stable diffusion model is used to generate a local foreign object graph. The construction of the stable diffusion model is described in the article "Blended Diffusion for Text-driven Editing of Natural Images" by Omri Avrahami et al.

[0053] (5) Based on this mask, a labeled rectangular frame circumscribed around the mask is generated, saved as a labeled file, and specifically, the coordinates of the circumscribed rectangle of the mask generated in step (3) are calculated, and in the sample image, a foreign object labeled rectangular frame is randomly generated, and the foreign object labeled rectangular frame is added to the sample image, so that the line suspension point and the foreign object suspension point coincide with each other.

[0054] As Figure 7 shown is a training sample obtained after adding a foreign object to a defect-free sample image.

[0055] Although the preferred embodiments of the present application have been described above with reference to the accompanying drawings, the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative and are not restrictive. A person of ordinary skill in the art can make many specific modifications under the inspiration of the present application without departing from the spirit of the present application and the scope of protection of the claims, and these all belong to the scope of protection of the present application.

Claims

1. A method for generating simulated samples of foreign object defects in overhead transmission lines, characterized in that, The method includes: Step (1): Use a drone to fly along the overhead power transmission line and take pictures of the overhead power transmission line at a level angle to form a sample image of the defect-free overhead power transmission line. Step (2): Construct a component segmentation network to extract masks of different key power components in the sample images. Each mask corresponds to a power component, and the coverage area of ​​each power component forms a set of coordinate points. Step (3): Traverse all pixels in the key power component mask extracted by the component segmentation network. For each pixel, determine the contact point between two power components by judging whether the 8-neighborhood of the pixel contains pixels of other power component masks. Determine whether each contact point belongs to the line suspension point according to the contact point category. Construct a set of line suspension point coordinates. Randomly select a line suspension point from the set of line suspension points for step (4). Step (4): Use a planar mesh to perform a simple three-dimensional model of the foreign object. Randomly select a vertex of the planar mesh as the suspension point of the foreign object. Use the PBD algorithm to iteratively calculate the position and velocity of all vertices in the mesh under the combined action of gravity and random wind. After a predetermined number of iterations, fit the graphic area enclosed by the coordinates of all vertices on the planar mesh into a mask so that the suspension point of the foreign object in the mask coincides with the suspension point of the line. Step (5) Generate a foreign object image based on the mask and foreign object information, fuse the foreign object image into the mask location of the sample image, and calculate the coordinate values ​​of the bounding rectangle of the mask generated in step (3). Generate a foreign object annotation rectangle in the sample image and use the foreign object annotation rectangle to annotate the foreign objects in the sample image.

2. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The neural network constructed in step (2) is the Mask R-CNN network.

3. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, It also includes the step of constructing a foreign matter stable diffusion model, which is constructed and trained based on the WIT, RedCaps, MMDialog, and CxC image-text public datasets.

4. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The method further includes: randomly selecting several line suspension points from each line suspension point, and for each selected line suspension point, adding a foreign object diagram at the suspension position corresponding to the suspension point, so that the foreign object suspension point coincides with the line suspension point.

5. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The electrical components include: wire clamps, ground wires, insulators, and connecting hardware.

6. The method for generating simulated foreign object defects in overhead transmission lines according to claim 5, characterized in that, The method also includes randomly adding images of different types of foreign objects to different sample images.

7. The method for generating simulated foreign object defects in overhead transmission lines according to claim 5, characterized in that, The process of generating the foreign object image in step (5) includes: inputting the generated mask into the foreign object stable diffusion model, randomly selecting foreign object type and pattern color description prompts from the prompt word library, and generating a foreign object image of the area corresponding to the mask based on the foreign object type and pattern color description prompts.

8. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The method also includes selecting different suspension points and repeating steps (3)-(5) to generate different defect sample images.

Images