Power transmission line foreign matter detection method and device, computer device and storage medium

Abstract

The application relates to a power transmission line foreign matter detection method and device, computer equipment and a storage medium. The method comprises the following steps: constructing a target detection model; acquiring an image to be processed, inputting the image to be processed into the target detection model, determining a prediction box in the image to be processed through the target detection model; segmenting an image in the prediction box into a plurality of superpixels, and determining an importance score of each superpixel; and assigning values to each pixel in the prediction box based on the importance score, to obtain a visualized image of a detection target. The foreign matter can be visually presented by using the method, and a patrol personnel can more intuitively determine the foreign matter.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a power transmission line foreign matter detection method and device, computer equipment and storage medium. BACKGROUND

[0002] The power transmission line is a power transmission system for transmitting electric energy from a power plant or a substation to users. It is composed of power transmission lines, power transmission towers and other related equipment. In order to ensure the safety of the power transmission line, power inspection needs to be carried out on the power transmission line.

[0003] In the traditional technology, after the photos and images of the power transmission line are taken in the process of power inspection, it is determined whether there is foreign matter in the photos and images, and the position of the foreign matter is further determined.

[0004] However, after a large number of images are obtained in the inspection process, if the confirmation of the foreign matter is carried out manually one by one, it will result in a great workload and poor efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a power transmission line foreign matter detection method and device, computer equipment and storage medium which can visually present the foreign matter in the inspection image in view of the above technical problems.

[0006] In a first aspect, the present application provides a power transmission line foreign matter detection method. The method comprises:

[0007] constructing a target detection model;

[0008] obtaining a to-be-processed image, inputting the to-be-processed image into the target detection model, and determining a prediction box in the to-be-processed image through the target detection model;

[0009] segmenting an image in the prediction box into a plurality of superpixels, and determining an importance score of each superpixel;

[0010] assigning values to each pixel in the prediction box based on the importance score, and obtaining a visualized image of a detection target.

[0011] In one of the embodiments, the target detection model comprises a feature extraction network and an intermediate network; the prediction box in the to-be-processed image is determined through the target detection model, which comprises:

[0012] extracting a feature map of the to-be-processed image having a detection target through the feature extraction network;

[0013] inputting the feature map into the intermediate network, and obtaining a plurality of candidate boxes through the intermediate network; the candidate boxes at least comprise a confidence degree of the detection target;

[0014] removing the candidate boxes with a confidence degree value less than a preset confidence threshold, and removing overlapping candidate boxes, to obtain the prediction box.

[0015] In one embodiment, the target detection model is constructed based on a convolutional neural network, and the target detection model comprises at least a convolutional layer and a pooling layer, and the convolutional layer and the pooling layer are configured to skip pixels without valid values in a pixel calculation process.

[0016] In one embodiment, the image in the prediction box is segmented into a plurality of superpixels, including:

[0017] Obtaining parameter information of each pixel; the parameter information includes color information and position information;

[0018] Based on the position information, adjacent pixels with color information difference less than a preset color threshold are regarded as the same region, and a plurality of superpixels are obtained based on a plurality of different regions.

[0019] In one embodiment, the importance score of each superpixel is determined, including:

[0020] Obtaining a plurality of groups of sampling images with superpixels; wherein, each group of sampling images randomly removes part of the superpixels;

[0021] Obtaining a reserved area of the reserved superpixels in each group of sampling images, and determining a sampling score of each group of sampling images;

[0022] Determining the weight of each superpixel according to the sampling score and the reserved area in each group of sampling images;

[0023] Weighted average of each sampling score based on the weight, to obtain the importance score of each superpixel.

[0024] In one embodiment, the sampling score in each group of sampling images is determined, including:

[0025] Inputting the sampling image with the reserved superpixels into the target detection model to determine the sampling bounding box through the target detection model;

[0026] Determining the sampling score according to the difference between the sampling bounding box and the candidate box.

[0027] In one embodiment, the importance score is assigned to each pixel in the prediction box to obtain a visualization image of the detection target, including:

[0028] Assigning the importance score to each pixel in the prediction box to obtain a heat map with the importance score of each pixel;

[0029] Filtering the heat map, and loading the heat map on the image to be processed through a visualization interface to obtain a visualization image of the detection target.

[0030] In a second aspect, the application also provides a power transmission line foreign matter detection device. The device comprises:

[0031] a model construction module, configured to construct a target detection model;

[0032] a prediction box determination module, configured to acquire a to-be-processed image, input the to-be-processed image into the target detection model, and determine a prediction box in the to-be-processed image by using the target detection model;

[0033] an importance score determination module, configured to segment an image in the prediction box into a plurality of superpixels, and determine an importance score of each superpixel;

[0034] a visualized image determination module, configured to assign a value to each pixel in the prediction box based on the importance score, and obtain a visualized image of a detection target.

[0035] In a third aspect, the present application further provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0036] constructing a target detection model;

[0037] acquiring a to-be-processed image, inputting the to-be-processed image into the target detection model, and determining a prediction box in the to-be-processed image by using the target detection model;

[0038] segmenting an image in the prediction box into a plurality of superpixels, and determining an importance score of each superpixel;

[0039] assigning a value to each pixel in the prediction box based on the importance score, and obtaining a visualized image of a detection target.

[0040] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program implements the following steps when executed by a processor:

[0041] constructing a target detection model;

[0042] acquiring a to-be-processed image, inputting the to-be-processed image into the target detection model, and determining a prediction box in the to-be-processed image by using the target detection model;

[0043] segmenting an image in the prediction box into a plurality of superpixels, and determining an importance score of each superpixel;

[0044] assigning a value to each pixel in the prediction box based on the importance score, and obtaining a visualized image of a detection target.

[0045] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program implements the following steps when executed by a processor:

[0046] construct a target detection model;

[0047] obtain a to-be-processed image, input the to-be-processed image into the target detection model, and determine a prediction box in the to-be-processed image through the target detection model;

[0048] segment an image in the prediction box into a plurality of superpixels, and determine an importance score of each superpixel;

[0049] value each pixel in the prediction box based on the importance score, and obtain a visualized image of a detection target.

[0050] The power transmission line foreign matter detection method, device, computer device, and storage medium described above first construct a target detection model, input a to-be-processed image into the target detection model to determine a prediction box, so as to determine the position of foreign matter. Then, the image in the prediction box is segmented into a plurality of superpixels according to the same or similar features, and an importance score of each superpixel is determined. The importance score represents the importance of each superpixel. After each pixel is valued based on the importance score, the pixel region with a higher importance in the prediction box can be highlighted, so that the foreign matter is visualized, and a visualized image of the foreign matter is obtained. Compared with the method of determining the foreign matter in the image through manual recognition in the prior art, the foreign matter detection method provided in this application can present the foreign matter visually, and help the inspection personnel to determine the foreign matter more intuitively. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 An application environment diagram of the power transmission line foreign matter detection method in one embodiment;

[0052] Figure 2 A flowchart of the power transmission line foreign matter detection method in one embodiment;

[0053] Figure 3 A flowchart of the step of determining the prediction box in the to-be-processed image in one embodiment;

[0054] Figure 4 A superpixel segmentation result obtained in one embodiment;

[0055] Figure 5 A flowchart of the step of determining the importance score of each superpixel in one embodiment;

[0056] Figure 6 A flowchart of the step of calculating the sampling score in one embodiment;

[0057] Figure 7 A method diagram of calculating the weight in one embodiment;

[0058] Figure 8 A comparison diagram of an image obtained in one embodiment and an image obtained by filling in the missing part;

[0059] Figure 9 Flowchart of the method for detecting foreign matter on a power transmission line in another embodiment;

[0060] Figure 10 Block diagram of the structure of the device for detecting foreign matter on a power transmission line in an embodiment;

[0061] Figure 11 Internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0062] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0063] The method for detecting foreign matter on a power transmission line provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 . The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The image to be processed is obtained through the terminal 102, and transmitted to the server 104 for processing. The server 104 presents the foreign matter visually on the image to be processed, which is convenient for manual observation and review. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0064] In an embodiment, as shown in Figure 2 , a method for detecting foreign matter on a power transmission line is provided. Taking the server 104 in Figure 1 as an example, the method includes the following steps S202 to S208:

[0065] S202, constructing a target detection model.

[0066] The target detection model is used to determine the candidate box that may contain the target of interest in the image. The candidate box can also be referred to as a bounding box or a region of interest. The target detection model can be based on a deep learning method, illustratively, based on a CNN (Convolutional Neural Networks).

[0067] In an embodiment, the target detection model is constructed based on a convolutional neural network. The target detection model at least includes a convolutional layer and a pooling layer, which are set to skip the pixels without effective values in the pixel calculation process.

[0068] That is, the convolutional layer and the pooling layer in the embodiment are missing adaptive convolutional layer and pooling layer, which are different from general convolutional layer and pooling layer. The missing adaptive convolutional layer and the pooling layer directly skip the pixels without any effective value in the process of inference calculation, instead of filling zeros. Illustratively, for a pre-trained target detection model, the general convolutional layer and the pooling layer in the network can be replaced by the missing adaptive convolutional layer and the pooling layer, which can also be called missing adaptive convolutional layer and the pooling layer. At this time, the target detection model is defined as a missing adaptive target detection model.

[0069] The missing adaptive needs to consider two types of missing values in the calculation process of the pixels. The first type is the effective feature that needs to be considered. The second type is the external filling feature outside the spatial range of the effective feature mapping, which is necessary for the convolution and the pooling operation. Because it is necessary to keep the spatial size of the feature unchanged.

[0070] Specifically, the missing adaptive convolution is defined as formula (1): represents the feature map of the lth layer, is the corresponding binary mask, where, is a real matrix with width W and height H, M l The 1 in the binary mask represents that the feature value is valid, and the 0 represents that the feature value is missing. Here, the channel dimension is provided for symbol simplification. Let represents the set of valid indexes in the convolution or pooling window centered at position (i, j), that is,

[0071] Therefore, the proposed missing adaptive convolution can be defined as formula (1):

[0072]

[0073] where K l represents the convolution kernel, represents the empty set; ||·|| represents the number of elements in the set; represents a scale factor to compensate for the missing feature values on the output distribution, that is, if there is no feature missing, the missing adaptive convolution degenerates into a regular convolution. Similarly, the missing adaptive average pooling is defined as formula (2):

[0074]

[0075] ​In the proposed missing-adaptive convolution and pooling operations, the missing values are directly skipped instead of being filled with other alternative values in inference, significantly avoiding the missing bias problem introduced by these filled values. The symbol NULL in the above formula represents that the value is invalid due to the missing value in the input. Therefore, the missing adaptation updates the mask M according to the zero values in the output. Taking the convolution operation as an example, as shown in formula (3):

[0076]

[0077] In addition, the target detection network modified by the missing adaptation can input a mask with the same size as the to-be-processed image when performing inference, to indicate whether the pixels at the corresponding positions in the to-be-processed image are retained or deleted. 0 indicates that the pixels at the corresponding positions need to be discarded, and 1 indicates that the pixels at the corresponding positions need to be retained.

[0078] In this embodiment, the target detection network is a convolutional neural network, wherein the convolutional layer and the pooling layer skip the pixels without valid values in the pixel calculation process, that is, missing adaptation is achieved. In this way, the missing values are directly skipped in the calculation process instead of being filled with alternative values, significantly avoiding the missing bias problem introduced by the alternative values.

[0079] S204, obtaining a to-be-processed image, inputting the to-be-processed image into a target detection model, and determining a prediction box in the to-be-processed image by the target detection model.

[0080] The to-be-processed image is an image for which it is necessary to determine whether there is a foreign object and to determine the position of the foreign object. The to-be-processed image can be obtained by the terminal 102 photographing a power equipment during power inspection, and the terminal 102 transmits the to-be-processed image to the server 104. After obtaining the to-be-processed image, the to-be-processed image is input into the target detection model.

[0081] The prediction box is a rectangular box generated by the target detection model, used to identify the position of a target object in the image, for example, the region where the foreign object is in the to-be-processed image in this embodiment. Generally, the prediction box is often accompanied by a prediction class label, that is, it tells us the possible target class in the region.

[0082] S206, segmenting the image in the prediction box into a plurality of superpixels, and determining the importance scores of the superpixels.

[0083] After determining the prediction box, the image in the prediction box is segmented into a plurality of superpixels. Superpixels are a group of tightly connected pixel regions formed by aggregating adjacent pixels together. The image in the prediction box is divided into continuous regions with consistent semantics, thereby reducing the redundant information in the image. Generally, the pixels in the same superpixel have similar texture, color, brightness, and other characteristics.

[0084] The importance score is an index for measuring the relative importance of the superpixel. Generally, the foreign object image in the prediction box obtains a relatively high importance score.

[0085] In S208, each pixel in the prediction box is assigned based on the importance score, and a visualization image of the detection target is obtained.

[0086] After determining the importance score of each superpixel, each pixel is assigned. The higher the importance score, the higher the importance of the corresponding superpixel region, indicating that the foreign object is more likely. Therefore, by assigning the pixel based on the importance score, the pixel of the foreign object can be highlighted, making the foreign object more prominent relative to the background, helping the inspection personnel and model developers to more intuitively understand and explain the results.

[0087] In the above power line foreign object detection method, a target detection model is first constructed, and the image to be processed is input into the target detection model to determine a prediction box to determine the position of the foreign object. Then, the image in the prediction box is segmented into a plurality of superpixels according to the same or similar features, and the importance score of each superpixel is determined. The importance score represents the importance of each superpixel. After assigning each pixel based on the importance score, the pixel region with higher importance in the prediction box can be highlighted, thereby visualizing the foreign object and obtaining a foreign object visualization image. Compared with the method of determining the foreign object in the image by manual recognition in the traditional technology, the foreign object detection method provided in the present application can visualize the foreign object and help the inspection personnel to more intuitively determine the foreign object.

[0088] In one embodiment, as shown in Figure 3 The target detection model includes a feature extraction network and an intermediate network, and step S204 of determining the prediction box in the image to be processed by the target detection model includes:

[0089] In S302, the feature extraction network extracts a feature map of the image to be processed having a detection target.

[0090] The feature extraction network is used to extract a feature map from the image to be processed. At this time, the extracted feature map can have multiple feature maps as a feature map set.

[0091] In the case of inputting a mask to the target detection model, no feature is discarded at this time, that is, the input mask value is all 1, which means that all pixels need to be retained.

[0092] In S304, the feature map is input into the intermediate network to obtain a plurality of candidate boxes through the intermediate network; the candidate box at least includes the confidence of the detection target.

[0093] The intermediate network is used to determine the bounding boxes of the feature map. The feature map extracted by the feature extraction network is input into the intermediate network, and a plurality of bounding boxes are obtained by the intermediate network as candidate prediction results. Generally, these bounding boxes are various and densely distributed on the image, covering the positions where the target may exist.

[0094] Illustratively, the intermediate network can be a Header model. After the feature map is input into the Header model, a plurality of candidate prediction results are obtained, wherein N represents the number of candidate prediction results, represents the length of each candidate prediction result. Therefore, each prediction result can be represented as a vector x, y, w, h, c respectively represent the horizontal coordinate, the vertical coordinate, the width, the height and the confidence that the target exists in the detection box, represents the confidence that the detected target is of different types.

[0095] S306, remove the bounding boxes with confidence values less than the preset confidence threshold, and remove the overlapping bounding boxes to obtain the prediction boxes.

[0096] After obtaining a plurality of bounding boxes, the bounding boxes with confidence values less than the preset confidence threshold are removed. The confidence represents the accuracy of the detection target. By removing the bounding boxes with smaller confidence, the accuracy of the foreign object judgment is ensured. Illustratively, the preset confidence threshold can be set to 1, and the bounding boxes with confidence values less than 1 are filtered out.

[0097] Then, the remaining bounding boxes are screened to remove the overlapping bounding boxes to obtain unique bounding boxes as prediction boxes. Illustratively, the NMS (Non-Maximum Suppression) algorithm can be used to remove the overlapping prediction boxes. The NMS selects the box with the highest confidence according to the score and the overlapping degree, and eliminates the similar boxes. In this embodiment, the detection target in the prediction box found by the target detection model is taken as the subsequent explanation target, i.e., the target evaluated by the importance score.

[0098] In this embodiment, through the cooperation of the feature extraction network and the intermediate network, a plurality of bounding boxes are first determined, then the bounding boxes are screened and removed, and finally the prediction boxes containing the detection target are obtained, so as to facilitate the subsequent extraction of foreign objects based on the prediction boxes.

[0099] In one embodiment, the step S206 of segmenting the image in the prediction box into a plurality of superpixels includes: obtaining parameter information of each pixel; the parameter information includes color information and position information; based on the position information, adjacent pixels with color information difference less than a preset color threshold are taken as the same region, and a plurality of superpixels are obtained based on a plurality of different regions.

[0100] Pixels within the same superpixel have similar texture, color, brightness and other characteristics, and thus the image can be segmented based on the color information of each pixel. As an example, the image in the prediction box can be segmented into multiple superpixels by a QuickShift algorithm. The core idea of the QuickShift algorithm is to merge superpixels by calculating the similarity between pixels. For example, the RGB (Red Green Blue) values of adjacent pixels are subtracted to obtain a difference value, and a preset color threshold is set. Adjacent pixels with a difference value less than the preset color threshold are considered to have a higher similarity. The similarity measure is based on color information and position information, so that pixels with similar colors have a higher similarity score.

[0101] Specifically, the QuickShift algorithm creates a set of binary superpixel masks where each binary superpixel mask belongs to the position of a pixel in the kth superpixel, and w and h represent the width and height of the image, respectively. These superpixels represent a plurality of pixels that are continuously similar, and their existence or nonexistence is controlled by the binary mask. Then, the superpixels of the target are obtained according to the prediction box range of each explanation target represent the number of superpixels related to the nth target detection result, where each superpixel has at least one pixel within the detection box range of the explanation target. The resulting superpixel segmentation result is shown in Figure 4 where some superpixels around the candidate box are only partially within the candidate box.

[0102] In this embodiment, based on the color information and position information of each pixel in the prediction box, pixels with a higher similarity are considered to be in the same region, thereby segmenting the image in the prediction box into multiple superpixels. Superpixels can better capture the local structure and texture information of the image, and reduce the data dimension while preserving the boundary information, thereby improving the calculation speed.

[0103] As shown in Figure 5 In one embodiment, the importance score of each superpixel is determined in step S206, including:

[0104] S502, obtaining a plurality of groups of sampling images with superpixels; wherein each group of sampling images randomly removes part of the superpixels.

[0105] After segmenting the image in the prediction box into multiple superpixels, the image to be processed is randomly sampled into multiple groups to obtain multiple groups of sampling images. During sampling, part of the superpixels can be randomly deleted. When the number of sampling groups is sufficient, even if a superpixel in a certain group is deleted, it will appear in other groups, thereby ensuring that the importance score of each superpixel can be obtained.

[0106] The L sets of random samples for the nth explanatory target can be represented as: Where each sampling mask vector m l For a length of The vector is a binary vector, where 0 indicates the deletion of the corresponding superpixel and 1 indicates the retention of the corresponding superpixel. Indicatively, the total number of groups in the sampled image can be 200.

[0107] S504, obtain the retained area of ​​superpixels in each group of sampled images, and determine the sampling score of each group of sampled images.

[0108] In each group, some superpixels are removed, and the remaining superpixels are retained. The retained area refers to the total area of ​​the retained superpixels.

[0109] In one embodiment, determining the sampling score of each group of sampled images includes: inputting the sampled images with retained superpixels into an object detection model, determining the sampling bounding boxes through the object detection model, and determining the sampling score based on the difference between the sampling bounding boxes and the candidate bounding boxes.

[0110] like Figure 6 As shown, in this embodiment, the similarity between the sampled image obtained from sampling and the original image to be processed is used as the importance score of the superpixel. Specifically, the sampled image with retained superpixels is input into the object detection model to obtain a sampled bounding box similar to the prediction box. The removal and retention of superpixels can be represented by a sampling mask, and the image to be processed with the sampling mask can represent the retained superpixels.

[0111] Each sampling mask and the image to be processed are input into the missing adaptive target detection model to obtain a new detection result vector. Then, the detection results of o′ and the original candidate box are calculated. The similarity score S(o, o′) is taken as the importance score of each retained superpixel. The similarity score is mainly the product of three parts, which can be expressed as the following formula (4):

[0112] S(o′,o)=S1(o′,o)·S2(o′,o)·S3(o′,o) (4)

[0113] Each part is shown in formulas (5) to (7):

[0114] S1(o′,o)=IoU(x′,y′,w′,h′,x,y,w,h) (5)

[0115] S2(o′,o)=c′ (6)

[0116]

[0117] Wherein, the IoU represents the intersection over union of the original prediction box and the sampled prediction box. Since only the detection result with higher confidence is kept in the prediction box, s2 is simplified as the confidence c' of the modified image. The larger S1(o', o) is, the more similar the modified image is to the original image, i.e. the more important the kept superpixels are.

[0118] In the embodiment, a calculation manner of the importance score is specifically disclosed. The sampled image is input into the target detection model to obtain a sampling bounding box. The sampling score of each sampling image is determined according to the difference between the sampling bounding box and the prediction box.

[0119] S506, the weight of each superpixel is determined according to the sampling score and the kept area in each group of sampling images.

[0120] As shown in Figure 7 , after obtaining the importance score S(o', o) of each random sampling image, the same score is assigned to the kept superpixel under each sampling image, and the proportion of the area of the superpixel to the area of all kept superpixels is taken as the weight.

[0121] Each superpixel finally obtains a series of importance scores and the ratio of the area of the superpixel to the area of all kept superpixels in the sampling image All elements in are normalized, i.e. each r l is normalized, so that the weight

[0122] S508, the importance score of each superpixel is obtained by weighted average of the sampling scores based on the weight.

[0123] The importance score of each superpixel is obtained by weighted average of the sampling scores based on the weight. If the superpixel is not sampled, the corresponding value is set to 0, then s k , r k and w k can be expressed as a vector with a length of L, i.e. The weighted average can be expressed as diag(·) is the diagonal element of the matrix.

[0124] In the embodiment, a plurality of groups of sampling images are randomly sampled in the manner of removing part of the superpixels. After determining the sampling scores of the sampling images, the weight of each superpixel is determined based on the total area of the kept superpixels and the sampling scores. Then, the importance score of each superpixel is obtained by weighted average of the sampling scores based on the weight. In this way, the importance score of each superpixel is determined, so as to represent the importance degree of the detection target pixel in the prediction box through the importance score.​

[0125] In one embodiment, the pixels in the prediction box are assigned based on the importance score to obtain a visual image of the detection target, including: assigning the pixels in the prediction box based on the importance score to obtain a heat map with each pixel importance score; filtering the heat map, and loading the heat map on the image to be processed through a visualization interface to obtain a visual image of the detection target.

[0126] Each superpixel is assigned a corresponding importance score value to obtain a heat map of the importance of each pixel. In order to make the heat map smoother, the heat map can be further processed using Gaussian filtering. Finally, the smoothed heat map is visualized on the image to be processed through the visualization interface of the Captum framework to obtain an image in which the detection target is prominently displayed.

[0127] The visualization result is shown in the left of Figure 8 , Figure 8 The left image represents that the convolutional layer and the pooling layer of the target detection model are not the result of missing adaptation, and the right image represents that the convolutional layer and the pooling layer are the result of missing adaptation. From Figure 8 It can be obtained that when the convolutional layer and the pooling layer of the target detection model are set to missing adaptation, the image of the detection target can be more prominent and clearer.

[0128] In this embodiment, after the importance score is assigned to the superpixel, a heat map with the importance score is obtained, and the heat map is filtered and then loaded in the original image to be processed, so that the detection target is prominently displayed, thereby helping the inspection personnel and the model developer to observe more intuitively.

[0129] In one embodiment, please refer to Figure 9 A foreign matter detection method for a power transmission line includes the following steps:

[0130] S902, a target detection model is constructed; the target detection model includes a feature extraction network and an intermediate network; the target detection model is constructed based on a convolutional neural network, and the target detection model at least includes a convolutional layer and a pooling layer, and the convolutional layer and the pooling layer are set to skip the pixels without effective values in the pixel calculation process.

[0131] S904, an image to be processed is obtained, and the image to be processed is input into the target detection model to extract a feature map with a detection target of the image to be processed through the feature extraction network.

[0132] S906, the feature map is input into the intermediate network to obtain a plurality of candidate boxes through the intermediate network; the candidate boxes at least include a confidence of the detection target.

[0133] S908, remove the candidate box whose confidence value is less than the preset confidence threshold, and remove the overlapping candidate boxes to obtain a prediction box.

[0134] S910, obtain parameter information of each pixel; the parameter information includes color information and position information; based on the position information, adjacent pixels with color information difference less than a preset color threshold are regarded as a same region, and a plurality of superpixels are obtained based on a plurality of different regions.

[0135] S912, obtain a plurality of groups of sampling images with superpixels; wherein, each group of sampling images randomly removes part of the superpixels.

[0136] S914, obtain a reserved area of the superpixels reserved in each group of sampling images, input the sampling images with the reserved superpixels into the target detection model, and determine a sampling bounding box through the target detection model.

[0137] S916, determine a sampling score according to the difference between the sampling bounding box and the candidate box; and determine a weight of each superpixel according to the sampling score and the reserved area in each group of sampling images.

[0138] S918, weight average each sampling score based on the weight to obtain an importance score of each superpixel.

[0139] S920, assign a value to each pixel in the prediction box based on the importance score to obtain a heat map with the importance score of each pixel.

[0140] S922, filter the heat map, and load the heat map on the image to be processed through a visualization interface to obtain a visualized image of the detected target.

[0141] In this embodiment, a target detection model based on a convolutional neural network is constructed, the target detection model includes a feature extraction network and an intermediate network, and the convolutional layer and the pooling layer in the target detection model are set to skip the pixels without effective values in the pixel calculation process, that is, to be self-defect adaptive. After the image to be processed is input into the target detection model, the feature map is extracted through the feature extraction network, a plurality of candidate boxes of the feature map are obtained through the intermediate network, and the redundant candidate boxes are filtered and removed until a prediction box is obtained. The image in the prediction box is divided into a plurality of superpixels according to the color information and the position information in the prediction box, the importance score of each superpixel is determined, each pixel is assigned a value based on the importance score, a heat map with the importance score is obtained, and the heat map is loaded on the original image to be processed after being filtered, so that a visualized image of the detected target is obtained, thereby facilitating the identification of the inspection personnel.

[0142] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiments of the present application also provide a power line foreign matter detection device for implementing the power line foreign matter detection method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more power line foreign matter detection device embodiments provided below can refer to the limitations of the power line foreign matter detection method described above, and will not be repeated here.

[0144] In one embodiment, as shown in Figure 10 A power line foreign matter detection device 1000 is provided, including a model construction module 1002, a prediction box determination module 1004, an importance score determination module 1006, and a visual image determination module 1008, wherein:

[0145] The model construction module 1002 is configured to construct a target detection model.

[0146] The prediction box determination module 1004 is configured to obtain a to-be-processed image, input the to-be-processed image into the target detection model, and determine a prediction box in the to-be-processed image by using the target detection model.

[0147] The importance score determination module 1006 is configured to divide an image in the prediction box into a plurality of superpixels, and determine an importance score of each superpixel.

[0148] The visual image determination module 1008 is configured to assign a value to each pixel in the prediction box based on the importance score, and obtain a visual image of a detection target.

[0149] In an embodiment, the target detection model comprises a feature extraction network and an intermediate network; the prediction box determination module 1004 is specifically configured to: extract, by the feature extraction network, a feature map of the detection target of the to-be-processed image; input the feature map into the intermediate network, and obtain a plurality of candidate boxes by the intermediate network; the candidate boxes at least comprise confidence of the detection target; remove the candidate boxes with confidence values less than a preset confidence threshold, and remove overlapping candidate boxes to obtain the prediction box.

[0150] In an embodiment, the target detection model is constructed based on a convolutional neural network, and the target detection model at least comprises a convolutional layer and a pooling layer, which are configured to skip the pixels without effective values in the pixel calculation process.

[0151] In an embodiment, the importance score determination module 1006 is specifically configured to: obtain parameter information of each pixel; the parameter information comprises color information and position information; based on the position information, adjacent pixels with color information difference less than a preset color threshold are regarded as the same region, and a plurality of superpixels are obtained based on a plurality of different regions.

[0152] In an embodiment, the importance score determination module 1006 is specifically configured to: obtain a plurality of groups of sampling images with superpixels; wherein, each group of sampling images removes part of the superpixels randomly; obtain a reserved area of the reserved superpixels in each group of sampling images, and determine a sampling score of each group of sampling images; determine a weight of each superpixel according to the sampling score and the reserved area in each group of sampling images; and perform weighted average on each sampling score based on the weight to obtain the importance score of each superpixel.

[0153] In an embodiment, the importance score determination module 1006 is specifically configured to: determine each sampling score in each group of sampling images, comprising: inputting the sampling image with the reserved superpixels into the target detection model, and determining a sampling bounding box by the target detection model; and determining the sampling score according to the difference between the sampling bounding box and the candidate box.

[0154] In an embodiment, the visualized image determination module 1008 is specifically configured to: assign values to each pixel in the prediction box based on the importance score to obtain a heat map with the importance score of each pixel; filter the heat map, and load the heat map on the to-be-processed image through a visualization interface to obtain a visualized image of the detection target.

[0155] The above-mentioned various modules in the power transmission line foreign matter detection device can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0156] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 11 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store acquired image data to be processed. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through network connection. The computer program is executed by the processor to implement a foreign matter detection method for a power transmission line.

[0157] Those skilled in the art can understand that Figure 11 The structure shown in the above embodiment is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0158] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0159] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0160] In one embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0161] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0162] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0163] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method of detecting a foreign object on a power transmission line, characterized by, The method comprises: constructing a target detection model; obtaining a to-be-processed image, inputting the to-be-processed image into the target detection model, and determining a prediction box in the to-be-processed image by the target detection model; segmenting an image in the prediction box into a plurality of superpixels; obtaining a plurality of groups of sampling images with the superpixels; wherein each group of sampling images randomly removes part of the superpixels; obtaining a reserved area of reserved superpixels in each group of sampling images, and determining a sampling score of each group of sampling images; determining a weight of each superpixel according to the sampling score and the reserved area in each group of sampling images; weighting and averaging each sampling score based on the weight to obtain an importance score of each superpixel; assigning values to each pixel in the prediction box based on the importance score to obtain a visual image of a detection target.

2. The method of claim 1, wherein, The target detection model comprises a feature extraction network and an intermediate network; the target detection model determines the prediction box in the to-be-processed image by: extracting a feature map of the to-be-processed image with a detection target by the feature extraction network; inputting the feature map into the intermediate network to obtain a plurality of candidate boxes by the intermediate network; the candidate boxes at least include a confidence of the detection target; removing candidate boxes with a confidence value less than a preset confidence threshold and overlapping candidate boxes to obtain the prediction box.

3. The method of claim 1, wherein, The target detection model is constructed based on a convolutional neural network, and at least comprises a convolutional layer and a pooling layer, which are set to skip pixels without effective values in the pixel calculation process.

4. The method of claim 1, wherein, The segmentation of the image in the prediction box into a plurality of superpixels comprises: obtaining parameter information of each pixel; the parameter information includes color information and position information; based on the position information, adjacent pixels with color information difference less than a preset color threshold are regarded as the same region, and a plurality of superpixels are obtained based on a plurality of different regions.

5. The method of claim 1, wherein, The determination of the sampling score of each group of sampling images comprises: inputting the sampling image with the reserved superpixels into the target detection model, determining a sampling selection box by the target detection model; determining the sampling score according to the difference between the sampling selection box and the candidate box.

6. The method of claim 1, wherein, The assignment of values to each pixel in the prediction box based on the importance score to obtain a visual image of a detection target comprises: assigning values to each pixel in the prediction box based on the importance score to obtain a heat map with an importance score of each pixel; filtering the heat map, and loading the heat map on the to-be-processed image through a visualization interface to obtain a visual image of a detection target.

7. A foreign object detection device for a power transmission line, characterized by The device comprises: a model construction module for constructing a target detection model; a prediction box determination module for obtaining a to-be-processed image, inputting the to-be-processed image into the target detection model, and determining a prediction box in the to-be-processed image by the target detection model; The importance score determination module is configured to divide an image in the prediction box into a plurality of superpixels, obtain a plurality of groups of sampling images with the superpixels, obtain a reserved area of a reserved superpixel in each group of sampling images, determine a sampling score of each group of sampling images, determine a weight of each superpixel according to the sampling score and the reserved area in each group of sampling images, and obtain an importance score of each superpixel by weighted average of the sampling scores based on the weights, wherein a part of the superpixels are randomly removed in each group of sampling images. The visualized image determination module is configured to assign a value to each pixel in the prediction box based on the importance score, and obtain a visualized image of the detection target.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images