Method for generating foreign matter defect training sample of overhead transmission line

Abstract

The invention relates to the field of image processing, in particular to an overhead transmission line foreign matter defect simulation sample generation method, which comprises the following steps: forming a sample image of a defect-free overhead transmission line; constructing a component segmentation network, and extracting masks of different key power components; constructing a line suspension point coordinate set, and randomly selecting a line suspension point from the line suspension point set; and performing simple three-dimensional modeling on the foreign matter by using a plane grid, randomly generating a foreign matter labeling rectangular frame in a sample image, and adding the foreign matter image and the foreign matter labeling rectangular frame into the sample image, so that a line suspension point and a foreign matter suspension point coincide with each other.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically to a method for generating training samples of foreign body defects in overhead transmission lines. Background Art

[0002] Foreign objects on overhead transmission lines are one of the major hidden dangers that threaten the safe and stable operation of transmission networks. There are many types of foreign objects that may be hanging on transmission lines, including damaged plastic bags, woven bags, wires, rags, branches, bird nests, etc. These foreign objects may be hanging on overhead ground wires and pole tower power components, and manual identification consumes manpower and material resources.

[0003] With the development of neural network technology, it is possible to identify these foreign objects through computer algorithms through neural network systems, but the use of neural network models requires a large amount of data for training.

[0004] The image data captured during daily inspection tasks are mainly defect-free samples, and the collected foreign body defect samples are of a single type, small in number and unbalanced. Therefore, there is a lack of necessary training data for the development of intelligent defect detection and recognition algorithms for drone inspections. The existing overhead transmission line defect detection and recognition methods based on deep learning are prone to false detection and missed detection for foreign body defects.

[0005] Therefore, a sample expansion technology for foreign body defects is needed to expand the number of samples of foreign body defects, so that the foreign body defect samples in the data set for training defect detection and recognition algorithms are richer and more diverse. Summary of the invention

[0006] Purpose of the Invention

[0007] The training of existing intelligent detection and recognition algorithms for defects in drone inspection images requires a large amount of sample data, but the data collected in daily inspection tasks are mainly defect-free data, making it difficult to train a defect detection algorithm with high generalization ability.

[0008] The present invention proposes an effective solution, which can solve the problem of few foreign body defect samples and single morphology in the current drone inspection process. The generated foreign body model not only takes into account the problem of foreign body suspension, but also introduces the problem of foreign body deformation under the action of random wind. The generated foreign body samples are richer and closer to the actual situation.

[0009] The present invention is based on a large number of defect-free samples collected during the drone inspection process. It uses generative artificial intelligence to generate local images of foreign objects such as damaged woven bags, plastic bags, rags, etc. in various shapes for the defect-free samples, and integrates them into the original defect-free samples with an internal repair algorithm to achieve the expansion of foreign object defect samples. At the same time, a labeling box is generated for the generated foreign object defect part for the training of the intelligent recognition algorithm.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A method for generating a simulated sample of a foreign body defect in an overhead transmission line, the method comprising:

[0012] Step (1), using a drone to patrol along the overhead power transmission line, photographing the overhead power transmission line at a horizontal angle, and forming a sample image of a defect-free overhead power transmission line;

[0013] Step (2), constructing a component segmentation network, extracting masks of different key power components in the sample image, each mask corresponds to a power component, and the coverage area of ​​each power component constitutes a coordinate point set;

[0014] Step (3), traverse all pixels in the key power component mask extracted by the component segmentation network, for each pixel, determine the contact point of two power components by judging whether the neighborhood of the pixel 8 contains pixels of other power component masks, judge whether each contact point belongs to a line suspension point according to the contact point category, construct a line suspension point coordinate set, and randomly select a line suspension point from the line suspension point set to perform step (4);

[0015] Step (4), a simple three-dimensional modeling of the foreign object is performed using a plane grid, a vertex in the plane grid is randomly selected as the foreign object hanging point, and the PBD algorithm is used to iteratively calculate the position and speed of all vertices in the grid under the combined action of gravity and random wind force. After a predetermined number of iterations, the graphic area surrounded by the coordinates of all vertices on the plane grid is fitted into a mask, so that the foreign object hanging point in the mask coincides with the line hanging point;

[0016] Step (5) generates a foreign body image based on the mask and foreign body information, fuses the foreign body image to the location of the sample image mask, and calculates the coordinate values ​​of the circumscribed rectangle of the mask generated in step (3). In the sample image, a foreign body annotation rectangular frame is generated, and the foreign body annotation rectangular frame is used to annotate the foreign body in the sample image.

[0017] Furthermore, the neural network constructed in step (2) is a Mask R-CNN network.

[0018] Furthermore, the method also includes the step of constructing a foreign body stable diffusion model, wherein the foreign body stable diffusion model is constructed based on WIT, RedCaps, MMDialog, and CxC image-text public data sets.

[0019] Furthermore, the method also includes: randomly selecting a number of line suspension points from each line suspension point, and for each selected line suspension point, adding a foreign object map at the suspension position corresponding to the suspension point so that the foreign object suspension point coincides with the line suspension point.

[0020] Furthermore, the electrical components include: a wire clamp, a ground wire, an insulator, and connecting hardware.

[0021] Furthermore, the method also includes randomly adding different types of foreign body images to the sample images for different sample images.

[0022] Furthermore, the process of generating a foreign body image in step (5) includes: inputting the generated mask into a foreign body stable diffusion model, and randomly selecting foreign body types and pattern color description prompt words from a prompt word library, and generating a foreign body image of the mask corresponding area according to the foreign body types and pattern color description prompt words.

[0023] Furthermore, the method also includes selecting different suspension points to repeat the process of steps (3)-(5), thereby generating different defect sample images.

[0024] Beneficial Effects

[0025] The present invention can use a large number of defect-free UAV power inspection samples to quickly generate a large number of realistic foreign body defect samples with various defect shapes, and provide defect simulation training samples for the development of defect detection algorithms, so as to improve the detection accuracy of defect detection algorithms for foreign body defects. In addition, the present invention solves the problem of unreasonable logic of generated defect images, and uses the constructed possible hanging points as a reference to simulate defect samples close to real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an overall schematic flow chart of the method for generating simulated samples of foreign body defects in overhead power transmission lines of the present invention;

[0027] Figure 2 This is a sample image of a defect-free drone power inspection example;

[0028] Figure 3 For Figure 2 An example diagram of the power components in the sample image after segmentation, where the segmented reference numerals in the figure represent: 1. background; 2. pole tower; 3. insulator; 4. grading ring; 5. connection hardware; 6-12. conductor; 13-16. wire clamp; 17-20. anti-vibration hammer;

[0029] Figure 4 is the set of suspension points selected from the power component mask (marked in white in the figure);

[0030] Figure 5 is an example of the initial plane mesh used to simulate foreign objects, where P is a fixed foreign object hanging point;

[0031] Figure 6 The generated foreign body mask image is used as an example to control the diffusion model to generate foreign bodies (the diffusion model only superimposes random noise on the mask area, and then generates a high-quality local image by gradually reducing the noise in the mask area);

[0032] Figure 7 This is an example of a simulated sample of a foreign body defect. It can be seen from the figure that the simulated foreign body is very close to the actual suspension situation. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0034] The process of generating training samples of foreign body defects in overhead transmission lines is as follows: Figure 1 shown.

[0035] As shown in the figure, the method for automatically generating foreign body defect training samples in this embodiment includes the following process:

[0036] (1) Use a drone to fly along the overhead transmission line and photograph the overhead transmission line at eye level or any other desired angle to form a sample image of a defect-free overhead transmission line. Sample images can be taken from both sides of the line, and the image is taken once every flight period. There can be a certain overlap between the target image areas in the two shooting processes.

[0037] (2) Construct an instance segmentation network for the main components of overhead transmission lines. Use real inspection images without defects to mark the key parts of power towers, including wire clamps, ground wires, insulators, and connection hardware, and establish a training set for instance segmentation of power components. Use the constructed training set for instance segmentation of power components to train the instance segmentation network model for power tower components.

[0038] For overhead transmission lines, we first establish a database of key parts and then mark each key component one by one according to the list in the database.

[0039] The instance segmentation network uses Mask R-CNN to implement instance segmentation of the input inspection image and output the mask image of the power component, such as Figure 3As shown in the figure. All the collected defect-free inspection samples are input into the instance segmentation network for instance segmentation, and the masks of key power components are extracted. The Mask R-CNN network is improved based on the Faster R-CNN architecture. On the basis of Faster R-CNN, branches are added to predict the segmentation mask of each instance.

[0040] In one implementation, Mask R-CNN includes:

[0041] Backbone uses a pre-trained convolutional neural network (such as ResNet or ResNeXt) as a feature extractor to extract high-level feature maps from the input image.

[0042] Region Proposal Network (RPN), which is used to generate candidate regions (Region of Interest, RoI), that is, image regions that may contain objects. These RoIs will then be used for more accurate object detection and segmentation.

[0043] RoI Align, which is used to more accurately crop the corresponding RoI features from the feature map through bilinear interpolation, avoiding information loss caused by quantization, thereby improving the quality of mask prediction;

[0044] Classification and Bounding Box Regression Head: used to classify each RoI (determine which category it belongs to) and adjust the position of the bounding box to better surround the object;

[0045] Mask Prediction Head: For each RoI, in addition to classification and bounding box regression, there is a dedicated branch for predicting a binary mask corresponding to that RoI. This enables the model to provide pixel-level segmentation results for each detected object.

[0046] (3) Traverse all pixels in the key power component mask extracted by the instance segmentation network. For each pixel, determine the contact point of the two power components by judging whether the pixel's 8 neighborhood contains pixels of other power component masks. Judge whether each contact point belongs to a line suspension point according to the contact point category, construct a line suspension point coordinate set, and randomly select a line suspension point from the line suspension point set. Whether a contact point belongs to a suspension point is judged according to the contact point category, that is, according to the easy hanging position of the hanging object in the line. For example, the contact points between hardware and hardware, wire and wire clamp, wire and anti-vibration hammer are hanging points that are easy to hang foreign objects and should be screened out.

[0047] Through the extracted masks, the coordinate positions of the parts in the image where foreign objects are easily hung, such as the connection points of each hardware, the contact points between the wire and the wire clamp, and the contact points between the wire and the anti-vibration hammer, are calculated.

[0048] For each sample generation, a line suspension point coordinate is randomly selected for subsequent matching with the foreign object suspension point coordinate.

[0049] (4) A hanging point is selected from the possible hanging points of the foreign object, and a simple three-dimensional model is performed on the foreign object using a plane mesh. A vertex in the plane mesh is randomly selected as the hanging point of the foreign object, and the PBD algorithm is used to iteratively calculate the position and velocity of all vertices in the mesh under the combined action of gravity and random wind force. After a predetermined number of iterations, the graphic area surrounded by the coordinates of all vertices on the plane mesh is fitted into a mask.

[0050] For example, a plane mesh is used to perform simple 3D modeling of damaged woven bags, plastic bags, rags and other foreign objects, and a vertex in the plane mesh is randomly selected as a hanging point (such as Figure 5 As shown in Figure 1, the PBD algorithm (the specific process of the algorithm refers to patent US7616204) is used to iteratively calculate the positions and velocities of all vertices in the mesh under the combined action of gravity and random wind force. After a sufficient number of iterations, the graphic area surrounded by the coordinates of all vertices on the mesh is fitted into a mask. The mask is smoothed using morphological closing operations to generate the final mask. The PBD algorithm can be executed using NVIDIA PhysX, Houdini, or Blender software. NVIDIA PhysX is a widely used physics engine that supports rigid body, soft body, cloth, and fluid simulation.

[0051] Specifically, the PBD algorithm updates the positions and movement speeds of all the vertices of the plane mesh each time. Figure 5In this embodiment, the grid size is 10×10, the initial velocity of all vertices is 0, and after more than 50 iterations, the coordinates of all vertices no longer change significantly. The Alpha Shapes algorithm is used to recalculate the boundary of the point set composed of all vertex coordinates updated to the new coordinate position, thereby obtaining a mask. (For the implementation of the Alpha Shapes algorithm, refer to H.Edelsbrunner, D.Kirkpatrick and R.Seidel,"On the shape of a set of points in the plane," in IEEE Transactions on Information Theory, vol.29, no.4, pp.551-559, July 1983, doi:10.1109 / TIT.1983.1056714.)

[0052] The mask generated in step (3) is input into the foreign body stable diffusion model, and the foreign body type (plastic bag, woven bag, rags, etc.) and pattern color description prompt words are randomly selected from the prompt word library to generate a foreign body map of the mask specified area, and superimposed and fused into the mask area of ​​the original image (noise is superimposed in the mask area of ​​the image to generate a target foreign body map, which is fused with the background), thereby synthesizing a foreign body defect sample (i.e., the generated foreign body image is fused into the original sample image). The foreign body stable diffusion model is implemented by training a stable diffusion model based on image-text public datasets such as WIT, RedCaps, MMDialog, CxC, and image-text pair datasets constructed from existing power grid inspection images, and is used to generate a local foreign body map under the restriction of the mask area based on the input mask. That is, various image-text public datasets and foreign body-text pair datasets in power grid inspection images are obtained to train the stable diffusion model, and then based on the foreign body type (plastic bag, woven bag, rags, etc.) and pattern color description prompt words, a local foreign body map is generated using the stable diffusion model. For the construction of the stable diffusion model, see the article “Blended Diffusion for Text-driven Editing of Natural Images” by Omri Avrahami et al.

[0053] (5) Based on the mask, a labeled rectangular frame circumscribed to the mask is generated and saved as a labeling file. Specifically, the coordinate values ​​of the circumscribed rectangle of the mask generated in step (3) are calculated, and in the sample image, a foreign object labeled rectangular frame is randomly generated, and the foreign object labeled rectangular frame is added to the sample image so that the line hanging point and the foreign object hanging point coincide with each other.

[0054] like Figure 7 Shown are training samples obtained after adding foreign matter to defect-free sample images.

[0055] Although the preferred embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms of specific changes without departing from the scope of protection of the invention and the claims, all of which fall within the scope of protection of the present invention.

Claims

1. A method for generating simulated samples of foreign body defects in overhead transmission lines, characterized in that: The method comprises: Step (1), using a drone to patrol along the overhead power transmission line, photographing the overhead power transmission line at a horizontal angle, and forming a sample image of a defect-free overhead power transmission line; Step (2), constructing a component segmentation network, extracting masks of different key power components in the sample image, each mask corresponds to a power component, and the coverage area of ​​each power component constitutes a coordinate point set; Step (3), traverse all pixels in the key power component mask extracted by the component segmentation network, for each pixel, determine the contact point of two power components by judging whether the neighborhood of the pixel 8 contains pixels of other power component masks, judge whether each contact point belongs to a line suspension point according to the contact point category, construct a line suspension point coordinate set, and randomly select a line suspension point from the line suspension point set to perform step (4); Step (4), a simple three-dimensional modeling of the foreign object is performed using a plane grid, a vertex in the plane grid is randomly selected as the foreign object hanging point, and the PBD algorithm is used to iteratively calculate the position and speed of all vertices in the grid under the combined action of gravity and random wind force. After a predetermined number of iterations, the graphic area surrounded by the coordinates of all vertices on the plane grid is fitted into a mask, so that the foreign object hanging point in the mask coincides with the line hanging point; Step (5) generates a foreign body image based on the mask and foreign body information, fuses the foreign body image to the location of the sample image mask, and calculates the coordinate values ​​of the circumscribed rectangle of the mask generated in step (3). In the sample image, a foreign body annotation rectangular frame is generated, and the foreign body annotation rectangular frame is used to annotate the foreign body in the sample image.

2. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 1, characterized in that: The neural network constructed in step (2) is a Mask R-CNN network.

3. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 1, characterized in that: The method also includes the step of constructing a foreign body stable diffusion model, wherein the foreign body stable diffusion model is constructed and trained based on WIT, RedCaps, MMDialog, and CxC image-text public data sets.

4. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 1, characterized in that: The method further includes: randomly selecting a number of line suspension points from each line suspension point, and for each selected line suspension point, adding a foreign object map at a suspension position corresponding to the suspension point, so that the foreign object suspension point coincides with the line suspension point.

5. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 1, characterized in that: The electric power components include: a wire clamp, a ground wire, an insulator, and connecting hardware.

6. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 5, characterized in that: The method further includes randomly adding different types of foreign body images to the sample images for different sample images.

7. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 5, characterized in that: The process of generating a foreign body image in step (5) includes: inputting the generated mask into a foreign body stable diffusion model, and randomly selecting foreign body types and pattern color description prompt words from a prompt word library, and generating a foreign body image of the mask corresponding area according to the foreign body types and pattern color description prompt words.

8. The method for generating simulated samples of foreign body defects in overhead transmission lines according to claim 1, characterized in that: The method further includes selecting different suspension points to repeat the process of steps (3)-(5), thereby generating different defect sample images.

Images