Power transmission line nest hazard grade identification method and device, and computer equipment

Abstract

The invention relates to a power transmission line nest hazard grade identification method and device and computer equipment. The method comprises the following steps: acquiring a target power transmission inspection image, wherein the target power transmission inspection image comprises a bird nest and an insulator; detecting a bird nest and an insulator in the target power transmission inspection image to obtain a target detection result; performing instance segmentation on the target power transmission inspection image to obtain a first segmented region image including a bird nest and a second segmented region image including an insulator; based on the target detection result, the first segmented region image and the second segmented region image, determining a spatial position relationship between the bird nest and the insulator; and determining a bird nest hazard level based on the spatial position relationship. By adopting the method, the accuracy of bird nest hazard grade identification can be improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for identifying the hazard level of bird nests in power transmission lines. Background Art

[0002] Birds often build nests on the towers of power transmission lines. The nesting materials fall on the power equipment, which can easily cause line discharge tripping accidents, seriously affecting the safe and stable operation of the power grid. At present, the research on bird nest detection generally uses traditional image processing algorithms, such as edge detection and feature matching, to capture key features in the image to achieve bird nest recognition. However, due to the complex background of drone inspection images, the size of bird nests is not fixed, and the shapes are diverse, which makes it difficult to identify bird nests and there are many false detections. As a result, the accuracy of bird nest hazard level recognition is not high. The traditional bird nest recognition method cannot adapt to complex and changing environments.

[0003] This shows that the current bird nest detection method is not very accurate in determining the hazard level of bird nests. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for identifying the bird nest hazard level of a transmission line, which can identify the bird nest hazard level, in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for identifying the hazard level of bird nests on power transmission lines, comprising:

[0006] Acquire a target power transmission inspection image, where the target power transmission inspection image includes a bird's nest and an insulator;

[0007] Detect bird nests and insulators in the target power transmission inspection image and obtain target detection results;

[0008] Performing instance segmentation on the target power transmission inspection image to obtain a first segmented region image including the bird's nest and a second segmented region image including the insulator;

[0009] Determine the spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented region image and the second segmented region image;

[0010] Determine the hazard level of bird nests based on spatial location relationships.

[0011] In one embodiment, the target detection result includes a first detection frame of the bird's nest and a second detection frame of the insulator; and determining the spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented region image, and the second segmented region image includes:

[0012] Expanding the sizes of the first detection frame and the second detection frame;

[0013] Determine an intersection-over-union ratio of the first detection frame after the size expansion and the second detection frame after the size expansion;

[0014] The I / O ratio is compared with a preset I / O ratio threshold to determine the spatial position relationship between the bird's nest and the insulator.

[0015] In one embodiment, comparing the intersection-and-parallel ratio with a preset intersection-and-parallel ratio threshold to determine the spatial position relationship between the bird's nest and the insulator includes:

[0016] If the intersection-and-parallel ratio is less than the preset intersection-and-parallel ratio threshold, it is determined that the bird's nest is not located above the insulator;

[0017] If the I / O ratio is greater than or equal to the I / O ratio threshold, it is determined that the bird's nest is located above the insulator.

[0018] In one embodiment, the method for identifying bird nest hazards in power transmission lines further includes:

[0019] When the IoU is greater than or equal to the IoU threshold, determining an image depth difference value between the first segmented region image and the second segmented region image;

[0020] If the image depth difference value is less than the preset depth difference threshold, it is determined that the bird's nest is located directly above the insulator;

[0021] If the image depth difference value is greater than a preset depth difference threshold, it is determined that the bird's nest is located above the misalignment of the insulator.

[0022] In one embodiment, determining the hazard level of the bird nest based on the spatial position relationship includes:

[0023] In the case where the bird nest is located directly above the insulator, the bird nest hazard level is determined to be a significant hazard;

[0024] When the bird's nest is not located above the insulator, or the bird's nest is located above the misalignment of the insulator, the bird's nest hazard level is determined to be a minor hazard.

[0025] In one embodiment, determining the image depth difference value between the first segmented region image and the second segmented region image includes:

[0026] Determine a first minimum bounding box of the first detection box and a second minimum bounding box of the second detection box, and expand the sizes of the first minimum bounding box and the second minimum bounding box;

[0027] Determine first depth information of a first minimum bounding box after the size is expanded, and determine second depth information of a second minimum bounding box after the size is expanded;

[0028] The first image depth information of the first segmented region image is determined from the first depth information, the second image depth information of the second segmented region image is determined from the second depth information, and an image depth difference value between the first image depth information and the second image depth information is determined.

[0029] In one embodiment, detecting bird nests and insulators in a target power transmission inspection image to obtain a target detection result includes:

[0030] Taking the target power transmission inspection image as input, the trained target detection model is called to obtain the target detection result;

[0031] Among them, the target detection model is trained based on the following method:

[0032] Construct an initial target detection model, which includes a multi-scale attention mechanism and a three-scale feature fusion module;

[0033] Obtain historical power transmission inspection images, which include bird nests and insulators;

[0034] Taking the historical power transmission inspection images as input, the initial target detection model is called to determine the category and location information of the prediction box. The prediction box is used to identify the bird's nest and insulator in the historical power transmission inspection images. The historical power transmission inspection images carry the category labels and location information of the real boxes of the bird's nest and insulator.

[0035] Based on the category label and location information of the real box, as well as the category and location information of the predicted box, determine the difference between the predicted box and the real box;

[0036] Based on the difference, the parameters of the initial target detection model are iteratively updated until the preset training end condition is reached to obtain the trained target detection model.

[0037] In one embodiment, the target detection model includes a backbone network, a neck network, and a head network, and the initial target detection model determines the category of the prediction box based on the following method:

[0038] Based on the multi-scale attention mechanism, convolution layer and pooling layer in the backbone network of the initial target detection model, feature extraction is performed on the historical power transmission inspection images, and the feature importance of the historical power transmission inspection images is adjusted to obtain multiple first feature images of different sizes;

[0039] Inputting a plurality of first feature images of different sizes into the neck network in the initial target detection model, and performing multi-scale feature fusion on the plurality of first feature images of different sizes based on the three-scale feature fusion module in the neck network to obtain a plurality of second feature images of different sizes;

[0040] Inputting a plurality of second feature images of different sizes into a head network in the target detection model, and generating a plurality of prediction boxes for identifying bird nests and insulators based on the head network and the plurality of second feature images of different sizes;

[0041] Based on the preset label assignment strategy, the category probability distribution of each prediction box is predicted, and based on the category probability distribution, the category of the prediction box is determined.

[0042] In one embodiment, based on a preset label assignment strategy, the category probability distribution of each prediction box is predicted, including:

[0043] For each predicted box, determine the position similarity and size similarity between the predicted box and the real box;

[0044] The similarity between the predicted box and the true box is determined based on the position similarity and size similarity, and based on the similarity, the category probability distribution of the predicted box is determined.

[0045] In a second aspect, the present application also provides a device for identifying the hazard level of bird nests on power transmission lines, comprising:

[0046] An image acquisition module is used to acquire a target power transmission inspection image, where the target power transmission inspection image includes a bird's nest and an insulator;

[0047] A target object detection module is used to detect bird nests and insulators in the target power transmission inspection image and obtain target detection results;

[0048] An instance segmentation module is used to perform instance segmentation on the target power transmission inspection image to obtain a first segmented area image including a bird's nest and a second segmented area image including an insulator;

[0049] A spatial position relationship determination module, used to determine the spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented area image and the second segmented area image;

[0050] The hazard level determination module is used to determine the hazard level of the bird's nest based on the spatial position relationship.

[0051] In a third aspect, the present application further provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the above-mentioned embodiments of the method for identifying the hazard level of bird nests on transmission lines are implemented.

[0052] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any one of the above-mentioned embodiments of the method for identifying the hazard level of bird nests on transmission lines.

[0053] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in any one of the above-mentioned embodiments of the method for identifying the hazard level of bird nests on transmission lines.

[0054] The above-mentioned method, device, computer equipment, computer-readable storage medium and computer program product for identifying the hazard level of bird nests on transmission lines first obtain a target transmission inspection image containing bird nests and insulators, and obtain a target detection result by performing target detection on the bird nests and insulators in the transmission inspection image. Secondly, instance segmentation is performed on the target transmission inspection image to obtain segmented area images containing the bird nests and insulators respectively. Thirdly, the spatial position relationship between the bird nest and the insulator is determined based on the target detection result, the first segmented area image and the second segmented area image, thereby reducing the possibility of misjudging the position relationship between the bird nest and the insulator due to the shooting angle problem of the transmission line inspection image, and improving the accuracy of determining the position relationship between the bird nest and the insulator. Finally, considering that the bird nest at different positions of the insulator has different degrees of hazard to the safety of the power grid, the hazard level of the bird nest in the target transmission inspection image is determined based on the spatial position relationship, thereby improving the accuracy of identifying the hazard level of the bird nest. Furthermore, by identifying the hazard level of bird nests, more effective measures can be taken to prevent short circuits, tripping and other power failures caused by bird nests according to different hazard levels, which is beneficial to improving the stability and safety of the power grid. At the same time, it helps power companies to rationally arrange repair and maintenance resources, ensuring that limited manpower and material resources can be concentrated where they are most needed, thereby improving the maintenance efficiency of transmission lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A diagram showing an application environment of a method for identifying bird nest hazard levels of power transmission lines in one embodiment;

[0057] Figure 2 A schematic diagram of a flow chart of a method for identifying bird nest hazard levels of power transmission lines in one embodiment;

[0058] Figure 3 A schematic diagram of a flow chart of a method for identifying bird nest hazard levels of power transmission lines in another embodiment;

[0059] Figure 4 A schematic diagram of a flow chart of a method for identifying bird nest hazard levels of power transmission lines in another embodiment;

[0060] Figure 5 is a schematic diagram of a target power transmission inspection image in one embodiment;

[0061] Figure 6 is a depth map of a target power transmission inspection image in one embodiment;

[0062] Figure 7 is a structural diagram of an initial target detection model in one embodiment;

[0063] Figure 8 is a structural diagram of a multi-scale attention mechanism in one embodiment;

[0064] Fig. 9 is a structural diagram of a three-scale feature fusion module in an embodiment;

[0065] Fig.10 It is a structural block diagram of a device for identifying bird nest hazard level of a power transmission line in one embodiment;

[0066] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] The method for identifying the hazard level of bird nests in power transmission lines provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0069] Specifically, the operator may upload the collected target power transmission inspection image to the server 104 through the terminal 102, and then send a bird's nest hazard level identification message to the server 104 through the terminal 102. The server 104 obtains the target power transmission inspection image, which includes the bird's nest and the insulator. Secondly, the bird's nest and the insulator in the target power transmission inspection image are detected to obtain the target detection result, and the target power transmission inspection image is instance segmented to obtain a first segmented area image including the bird's nest and a second segmented area image including the insulator. Finally, based on the target detection result, the first segmented area image and the second segmented area image, the spatial position relationship between the bird's nest and the insulator is determined, and based on the spatial position relationship, the bird's nest hazard level is determined.

[0070] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0071] In an exemplary embodiment, Figure 2 As shown, a method for identifying bird nest hazards in power transmission lines is provided. Figure 1 The server 104 in the example is used for explanation, and includes the following S100 to S500. Among them:

[0072] S100, obtaining a target power transmission inspection image, where the target power transmission inspection image includes a bird's nest and an insulator.

[0073] Among them, the transmission inspection image is the image of the transmission line inspection, which can be collected by professionals carrying cameras, or by transmission line image monitoring devices, or by drones carrying high-definition cameras, or by robots equipped with various sensors (such as infrared thermal imagers, visible light cameras). The transmission line includes wires (conductors), towers, insulators, hardware, lightning conductors and grounding devices, among which insulators are installed in different of or between the conductor and the grounding member, can withstand Devices that are subject to electrical and mechanical stress, mainly used to achieve electrical insulation and mechanical fixation.

[0074] In practical applications, drones can be used to inspect and photograph transmission lines to obtain target transmission inspection images that include bird nests and insulators.

[0075] S200, detecting bird nests and insulators in a target power transmission inspection image to obtain a target detection result.

[0076] The target detection result may include a detection frame and a category label of a bird's nest and an insulator. The detection frame may include positioning information, and the category label may include "bird's nest" and "insulator".

[0077] In practical applications, the features of bird nests and insulators in historical power transmission inspection images can be extracted through edge detection algorithms (such as Canny edge detection), and classifiers (such as support vector machines SVM or random forests, etc.) can be trained using feature vectors. The trained classifiers can be used to detect bird nests and insulators in target power transmission images to obtain target detection results. It is also possible to train deep learning models to detect bird nests and insulators in target power transmission inspection images to obtain target detection results.

[0078] S300, performing instance segmentation on the target power transmission inspection image to obtain a first segmented area image including a bird's nest and a second segmented area image including an insulator.

[0079] In practical applications, the target power transmission inspection image can be segmented and labeled at the pixel level according to the target detection result. The labeled target inspection image can be used to pre-train the constructed deep learning model, such as Mask R-CNN (Mask Region-based Convolutional Neural Network, a region-based convolutional neural network used to generate masks) or YOLO (You Only Look Once) model, etc. The trained deep learning model is used to perform instance segmentation on the bird's nest and insulator in the target power transmission inspection image to obtain the first segmented area image containing the bird's nest and the second segmented area image containing the insulator.

[0080] S400, determining a spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented region image and the second segmented region image.

[0081] The spatial position relationship between the bird's nest and the insulator may include the position of the bird's nest relative to the insulator, such as above, below, left and right.

[0082] In practical applications, the detection box of the target detection is combined with the segmentation mask of the instance segmentation through the target detection result and the segmentation area image to determine the specific position and shape of the bird's nest and the insulator, and then the spatial position relationship between the bird's nest and the insulator is determined through center point coordinate analysis, intersection-over-union calculation or direction vector calculation. Specifically, the center point coordinate analysis can be to determine the center point coordinates of the bird's nest and the insulator, and determine the spatial position relationship between the bird's nest and the insulator by comparing the center point coordinates. The intersection-over-union calculation can be to determine the intersection-over-union ratio between the bird's nest and the insulator, determine their overlap degree, and determine the spatial position relationship according to the intersection-over-union ratio and the intersection-over-union ratio threshold. Among them, the intersection-over-union ratio is the ratio of the intersection area of ​​the target to the union area. The higher the intersection-over-union ratio, the higher the overlap degree of the target object.

[0083] S500: Determine the hazard level of the bird nest based on the spatial position relationship.

[0084] After obtaining the spatial position relationship between the bird's nest and the insulator, the bird's nest hazard level can be determined according to the preset evaluation indicators. Specifically, the evaluation indicators may include the spatial position relationship between the bird's nest and the insulator, the size of the bird's nest, and the material type of the bird's nest. A scoring standard is set for each indicator. For example, the scoring standard for the spatial position relationship indicator may be: the bird's nest is located above the insulator, the score is higher, and the bird's nest is located below the insulator, the score is lower; the scoring standard for the bird's nest size indicator may be: the larger the size of the bird's nest, the higher the score, and the smaller the size of the bird's nest, the lower the score; the scoring standard for the bird's nest material type indicator may be: the more conductive materials, the higher the score, and vice versa, the lower the score.

[0085] Assign weights to the scores of each evaluation indicator, determine the bird nest hazard score by weighted method, and then set the bird nest hazard level according to multiple bird nest hazard score ranges. For example, if the bird nest hazard score is greater than 80 points, the bird nest hazard level is determined to be level one hazard; if the bird nest hazard score is between 80 and 60 points, the bird nest hazard level is determined to be level two hazard; if the bird nest hazard score is less than 60 points, the bird nest hazard level is determined to be level three hazard.

[0086] In practical applications, after obtaining the spatial positions of bird nests and insulators, the size of the bird nest can be determined based on the size of the bird nest in the target detection results and the actual reference objects (such as poles, insulators, etc.). The type of bird nest material is identified through the trained deep learning model. According to the scoring criteria corresponding to each evaluation indicator, the spatial position relationship score, the size score of the bird nest, and the material type score of the bird nest are obtained. The bird nest hazard score is determined by the weight of each evaluation indicator, and then the bird nest hazard level is determined based on multiple bird nest hazard score ranges.

[0087] In the above-mentioned method for identifying the hazard level of bird nests in transmission lines, first, a target transmission inspection image containing bird nests and insulators is obtained, and a target detection result is obtained by performing target detection on the bird nests and insulators in the transmission inspection image. Secondly, the target transmission inspection image is instance segmented to obtain segmented area images containing the bird nests and insulators, respectively. Thirdly, the spatial position relationship between the bird nest and the insulator is determined based on the target detection result, the first segmented area image and the second segmented area image, thereby reducing the possibility of misjudging the position relationship between the bird nest and the insulator due to the shooting angle problem of the transmission line inspection image, and improving the accuracy of determining the position relationship between the bird nest and the insulator. Finally, considering that the bird nest at different positions of the insulator has different degrees of hazard to the safety of the power grid, the hazard level of the bird nest in the target transmission inspection image is determined based on the spatial position relationship, thereby improving the accuracy of identifying the hazard level of the bird nest. Furthermore, by identifying the hazard level of bird nests, more effective measures can be taken to prevent short circuits, tripping and other power failures caused by bird nests according to different hazard levels, which is beneficial to improving the stability and safety of the power grid. At the same time, it helps power companies to rationally arrange repair and maintenance resources, ensuring that limited manpower and material resources can be concentrated where they are most needed, thereby improving the maintenance efficiency of transmission lines.

[0088] In an exemplary embodiment, the target detection result includes a first detection frame of a bird's nest and a second detection frame of an insulator, such as Figure 3 As shown, S400 includes S420 to S460. Among them:

[0089] S420: Expand the sizes of the first detection frame and the second detection frame.

[0090] In practical applications, the first detection frame and the second detection frame include the center coordinates of the detection frame and the width and height of the detection frame, respectively. For the first detection frame, the height and width of the first detection frame are expanded, and the expanded size can be 1 / 3 of the width of the first detection frame. It can be understood that the expansion size can be set according to the target detection result, and by expanding the size of the detection frame, the possibility of incomplete target object area contained in the detection frame is reduced. For the second detection frame, the width and height of the second detection frame are expanded, and the expansion size is 1 / 2 of the width of the second detection frame.

[0091] S440: Determine an intersection-over-union ratio between the first detection frame after the size expansion and the second detection frame after the size expansion.

[0092] Among them, the intersection-over-union ratio represents the overlap degree of two detection frames, and the intersection-over-union ratio is the ratio of the intersection area to the union area of ​​the two detection frames.

[0093] In practical applications, the coordinates of the upper left corner and the upper right corner of the intersection area of ​​the first detection frame after size expansion and the second detection frame after size expansion can be determined first, and the intersection area can be determined based on the coordinates of the intersection area. The union area of ​​the first detection frame after size expansion and the second detection frame after size expansion is determined, and the intersection-union ratio of the first detection frame after size expansion and the second detection frame after size expansion is intersection area / union area.

[0094] S460: Compare the I / O ratio with a preset I / O ratio threshold to determine the spatial position relationship between the bird's nest and the insulator.

[0095] In practical applications, there may be multiple preset I / O ratio thresholds, for example, the I / O ratio threshold may include a first I / O ratio threshold (such as 0.5) and a second I / O ratio threshold (such as 0.1). After obtaining the I / O ratio, the I / O ratio is compared with the preset I / O ratio threshold, and the spatial position relationship between the bird's nest and the insulator is determined according to the comparison result. Exemplarily, if the I / O ratio is greater than the first I / O ratio threshold, it is determined that the bird's nest is located above the insulator; if the I / O ratio is between the first I / O ratio threshold and the second I / O ratio threshold, it is determined that the bird's nest is located on the left or right side of the insulator; if the I / O ratio is less than the second I / O ratio threshold, it is determined that the bird's nest is located below the insulator.

[0096] In this embodiment, by expanding the detection frame of the bird's nest and the insulator, the surrounding information of the detection frame of the bird's nest and the insulator is increased, and the possibility of incomplete bird's nest area and insulator area in the detection frame is reduced, so that the intersection and union ratio is determined based on the expanded detection frame, and the intersection and union ratio and the intersection and union ratio threshold are compared to determine the spatial position relationship between the bird's nest and the transmission tower, thereby improving the accuracy of determining the spatial position relationship.

[0097] Based on the I / O ratio and the I / O ratio threshold, the spatial position relationship between the bird's nest and the insulator is determined. In an exemplary embodiment, S460 further includes S462 to S464. Among them:

[0098] S462, if the I / O ratio is less than a preset I / O ratio threshold, it is determined that the bird's nest is not located above the insulator;

[0099] In practical applications, if the I / O ratio is less than a preset I / O ratio threshold (such as 0.5), it is determined that the bird's nest is not located above the insulator.

[0100] S464: If the intersection-and-parallel ratio is greater than or equal to the intersection-and-parallel ratio threshold, it is determined that the bird's nest is located above the insulator.

[0101] In practical applications, if the I / O ratio is greater than or equal to a preset I / O ratio threshold (eg, 0.5), it is determined that the bird's nest is located above the insulator.

[0102] In this embodiment, the vertical positional relationship between the bird's nest and the insulator is determined by comparing the intersection-parallel ratio with a preset intersection-parallel ratio threshold, thereby improving the accuracy of determining the positional relationship between the bird's nest and the insulator.

[0103] The spatial position relationship between the bird's nest and the insulator is determined by the image depth difference between the bird's nest and the insulator. In an exemplary embodiment, Figure 4 As shown, the method for identifying bird nest hazards in power transmission lines also includes S482 to S486. Among them:

[0104] S482: When the IoU is greater than or equal to the IoU threshold, determine the image depth difference between the first segmented region image and the second segmented region image.

[0105] When the intersection-and-parallel ratio is greater than or equal to the intersection-and-parallel ratio threshold, the bird's nest is located above the insulator. However, due to the shooting angle of the target transmission inspection image, there may be two situations in the spatial position relationship between the bird's nest and the insulator, such as Figure 5 As shown, Figure 5 The left picture shows that the bird's nest is located on the cross arm and above the misalignment of the insulator, while the right picture shows that the bird's nest is located above the hanging point of the insulator, that is, directly above the insulator. The different spatial position relationships between the bird's nest and the insulator correspond to different degrees of bird's nest damage.

[0106] Therefore, in practical applications, when the intersection-and-union ratio is greater than or equal to the intersection-and-union ratio threshold, a pre-trained depth estimation model, such as DenseDepth or MonoDepth, can be used to generate a depth map of the first segmented region image and the second segmented region image, and based on the depth map, the depth difference value between the first segmented region image and the second segmented region image can be determined.

[0107] S484: If the image depth difference value is less than a preset depth difference threshold, it is determined that the bird's nest is located directly above the insulator.

[0108] like Figure 6 As shown, Figure 6 The middle right picture is Figure 5 In the depth map on the right side of the figure, the bird's nest is located directly above the insulator, and the depth difference between the bird's nest area and the insulator area is small. In actual applications, if the image depth difference value is less than a preset depth difference threshold (such as 50 pixels), it is determined that the bird's nest is located directly above the insulator.

[0109] S486: If the image depth difference value is greater than a preset depth difference threshold, it is determined that the bird's nest is located above the misalignment of the insulator.

[0110] like Figure 6 As shown, Figure 6 The left picture is Figure 5In the depth map on the left side of the figure, the bird's nest is located on the cross arm and above the insulator offset. The depth of the bird's nest area is significantly different from the depth of the insulator area. In actual applications, if the image depth difference value is greater than a preset depth difference threshold (such as 50 pixels), it is determined that the bird's nest is located above the insulator offset.

[0111] In this embodiment, when the intersection-and-joint ratio is greater than or equal to the intersection-and-joint ratio threshold, the bird's nest is located above the insulator. Through the image depth difference value between the first segmented area image and the second segmented area image, it is further judged that the bird's nest is located directly above the insulator, or above the insulator, thereby reducing the possibility of misjudging the positional relationship between the bird's nest and the insulator due to the shooting angle problem of the transmission line inspection image, and improving the accuracy of determining the spatial positional relationship between the bird's nest and the insulator.

[0112] Determine the bird nest hazard level based on the spatial position relationship. In an exemplary embodiment, S500 includes S520 to S540. Among them:

[0113] S520, when the bird nest is located directly above the insulator, the bird nest hazard level is determined to be a major hazard.

[0114] In actual applications, the material of the bird's nest may contain conductive materials, such as branches or wires. These conductive materials fall and float on the insulator, which can easily short-circuit the insulator and cause line discharge tripping faults, which may further pose a fire risk. If the bird's nest is located directly above the insulator, the bird's nest hazard level is determined to be a major hazard.

[0115] S540, when the bird nest is not located above the insulator, or the bird nest is located above the misalignment of the insulator, the bird nest hazard level is determined to be a minor hazard.

[0116] In actual application, if the bird's nest is not located above the insulator, or the bird's nest is located above the misalignment of the insulator, the bird's nest hazard level is determined to be a secondary hazard.

[0117] In this embodiment, the bird nest hazard level is graded according to the spatial position relationship between the bird nest and the insulator, which is conducive to taking more effective measures to prevent short circuits, tripping and other power failures caused by bird nests according to different hazard levels, and is conducive to improving the stability and safety of the power grid. At the same time, it helps the power company to reasonably arrange repair and maintenance resources to ensure that limited manpower and material resources can be concentrated where they are most needed, thereby improving the maintenance efficiency of the transmission line.

[0118] Determine the image depth difference between the bird's nest and the insulator. In an exemplary embodiment, S462 includes S4622 to S4626.

[0119] S4622: Determine a first minimum bounding box of the first detection box and a second minimum bounding box of the second detection box, and expand the sizes of the first minimum bounding box and the second minimum bounding box.

[0120] The first minimum external bounding frame represents the minimum external bounding rectangle of the bird's nest, and the second minimum external bounding frame represents the minimum external bounding rectangle of the insulator.

[0121] In practical applications, the first minimum bounding box of the first detection box and the second minimum bounding box of the second detection box can be determined by an edge detection algorithm. The width and height of the first minimum bounding box are expanded, and the expanded size can be 1 / 5 of the first minimum bounding box, and the width and height of the second minimum bounding box are expanded, and the expanded size can be 1 / 5 of the second minimum bounding box.

[0122] S4624, determining first depth information of the first minimum circumscribed frame after the size is expanded, and determining second depth information of the second minimum circumscribed frame after the size is expanded.

[0123] The first depth information includes the distance from each pixel of the first minimum bounding frame after the size expansion to the camera, and the second depth information includes the distance from each pixel of the second minimum bounding frame after the size expansion to the camera.

[0124] In practical applications, the first depth information of the first minimum bounding box after size expansion and the second depth information of the second minimum bounding box after size expansion can be determined by the depth estimation model. Specifically, a pre-trained depth estimation model, such as the Depth Anything V2 depth estimation model, can be called, and the first minimum bounding box after size expansion and the second minimum bounding box after size expansion are used as inputs of the model to predict the depth map of the first minimum bounding box after size expansion and the depth map of the second minimum bounding box after size expansion, respectively. The depth map includes the distance from each pixel to the camera, and the first depth information and the second depth information are obtained through the depth map.

[0125] S4626, determining first image depth information of the first segmented region image from the first depth information, determining second image depth information of the second segmented region image from the second depth information, and determining an image depth difference value between the first image depth information and the second image depth information.

[0126] In practical applications, after obtaining the first depth information and the second depth information, the image depth information of the first segmented region image can be extracted from the first depth information through the instance segmentation mask of the first segmented image region, and the depth mean or median in the image depth information is taken as the first image depth information of the first segmented region image. The image depth information of the second segmented region image can be extracted from the second depth information through the instance segmentation mask of the second segmented image region, and the depth mean or median in the image depth information is taken as the second image depth information of the second segmented region image. After obtaining the first image depth information and the second image depth information, the image depth difference value is obtained by the difference between the first image depth information and the second image depth information.

[0127] In this embodiment, by detecting the minimum bounding box of the bird's nest and the insulator in the target inspection image and expanding the minimum bounding box, the image depth information of the bird's nest and the insulator determined based on the minimum bounding box can be improved, thereby facilitating improving the accuracy of determining the spatial position relationship between the bird's nest and the insulator based on the image depth information.

[0128] By detecting bird nests and insulators in the target power transmission inspection, a target detection result is obtained. In an exemplary embodiment, S200 includes S210 to S260. Among them:

[0129] S210, taking the target power transmission inspection image as input, calling the trained target detection model, and obtaining the target detection result.

[0130] The target detection result includes a first detection frame of the bird's nest and a second detection frame of the insulator.

[0131] In practical applications, after acquiring the target power transmission inspection image, the trained target detection model is called to detect the bird's nest and insulator in the target power transmission inspection image, and mark the first detection frame of the bird's nest and the second detection frame of the insulator.

[0132] The target detection model is trained based on the following method:

[0133] S220, constructing an initial target detection model, the initial target detection model includes a multi-scale attention mechanism and a three-scale feature fusion module.

[0134] In practical applications, the initial target detection model can be constructed by introducing a multi-scale attention mechanism and a three-scale feature fusion module in a deep learning model, such as the R-CNN model (Regions with CNN features), the YOLO model, or the SSD (Single Shot MultiBox Detector) model.

[0135] S230, obtaining a historical power transmission inspection image, where the historical power transmission inspection image includes a bird's nest and an insulator.

[0136] In practical applications, historical transmission inspection images including bird nests and insulators taken by drones during the historical time period are obtained. For example, historical transmission inspection images taken by drones during the past year are obtained.

[0137] S240, taking the historical power transmission inspection image as input, calling the initial target detection model, and determining the category and location information of the prediction box, the prediction box is used to identify the bird's nest and insulator in the historical power transmission inspection image, and the historical power transmission inspection image carries the category label of the real box of the bird's nest and insulator and the location information of the real box.

[0138] In practical applications, after obtaining historical power transmission inspection images, the annotation information is used to annotate the historical image inspection images. The annotation information includes the category labels of the real frames of the bird's nest and insulator and the location information of the real frames. The location information of the real frames may include: the coordinates of the center point of the real frame and the width and height of the real frame.

[0139] The historical power transmission inspection images containing labeled information are used as model input, and the initial target detection box model is called. The initial target detection model generates prediction boxes for identifying bird nests and insulators in the historical power transmission inspection images by performing feature extraction and feature fusion on the historical power transmission inspection images. The category of each prediction box is predicted to obtain the category and location information of the prediction box.

[0140] S250, based on the category label and position information of the real box, and the category and position information of the predicted box, determine the difference between the predicted box and the real box.

[0141] In practical applications, the difference between the predicted box and the real box can be determined by calculating the difference between the position information of the predicted box and the position information of the real box, the difference between the category of the predicted box and the real category, and the intersection and union ratio of the predicted box and the real box. The difference can be calculated by a loss function, and the loss function includes but is not limited to a mean square error loss function, a cross entropy loss function, and a smoothed L1 loss function.

[0142] S260, based on the difference, iteratively update the parameters of the initial target detection model until a preset training end condition is reached to obtain a trained target detection model.

[0143] In practical applications, after the prediction box category and location information are obtained, the loss function value (difference) is calculated through the loss function, the category label and location information of the real box, and the back propagation is started from the output layer of the target detection model according to the loss function value, the parameter gradient value of each layer is calculated, and the parameters of the target detection model are updated to minimize the loss function value. The preset training end condition can be that the loss function value is continuously less than the preset loss threshold within the preset number of iterations, and the training of the target detection model is stopped to obtain the trained target detection model.

[0144] In other embodiments, insulators are divided into horizontal insulators and vertical insulators. Considering that there is a small possibility of a bird's nest above the horizontal insulator, in order to improve the efficiency of target detection, the category label and position information of the real box of the vertical insulator in the historical power transmission inspection image can be data labeled, and the model is trained according to the steps of training the initial target detection model above to obtain a trained target detection model for detecting bird's nests and vertical insulators in the target power transmission inspection image.

[0145] In this embodiment, by constructing and training a target detection model including a multi-scale attention mechanism and a three-scale feature fusion module, the features of the target power transmission inspection image can be captured more effectively, so that the target power transmission inspection image is detected by the trained target detection model to obtain the target detection result, thereby improving the accuracy of the target detection result.

[0146] In other embodiments, after obtaining the trained target detection model, the method further includes: constructing an initial instance segmentation model based on a deep learning model (such as a YOLOv5 model), obtaining historical power transmission inspection images with category labels of real boxes of bird's nests and insulators and location information of real boxes, performing pixel-level mask annotation on the historical power transmission inspection images, and training the initial instance segmentation model using the annotated historical power transmission inspection images to obtain a trained instance segmentation model.

[0147] In other embodiments, the target power transmission inspection image is subjected to instance segmentation to obtain a first segmented region image including a bird's nest and a second segmented region image including an insulator, and further includes: for a first detection frame of a bird's nest in the target detection result, a region image of the first detection frame is cropped from the target power transmission inspection image, for a second detection frame of an insulator in the target detection result, a region image of the second detection frame is cropped from the target power transmission inspection image, and based on the region image of the first detection frame, the region image of the second detection frame and the trained instance segmentation model, a first segmented image including a bird's nest and a second segmented region image including an insulator are obtained. By cropping the target power transmission inspection image according to the detection frame of the target detection result, on the one hand, in the instance segmentation process, the number of pixels to be processed is reduced, and the computational efficiency is improved. On the other hand, instance segmentation of the cropped image can reduce the influence of background noise, thereby improving segmentation accuracy.

[0148] In an exemplary embodiment, the target detection model includes a backbone network, a neck network and a head network, and S240 includes S242 to S248. Among them:

[0149] The initial object detection model determines the category of the predicted box based on the following method:

[0150] S242, based on the multi-scale attention mechanism, convolution layer and pooling layer in the backbone network of the initial target detection model, feature extraction is performed on the historical power transmission inspection images, and the feature importance of the historical power transmission inspection images is adjusted to obtain multiple first feature images of different sizes.

[0151] In this embodiment, the initial target detection model is a YOLOv5 model that integrates a multi-scale attention mechanism and a three-scale feature fusion module. Specifically, a multi-scale attention mechanism is inserted into the backbone network, and a three-scale feature fusion module is added to the neck network.

[0152] Among them, the backbone network is used for feature extraction. The multi-scale attention mechanism (MSAM) is an extension of the self-attention mechanism. Its main purpose is to capture the multi-scale features of the image by adaptively paying attention to the different scales of information in the image and calculating the attention weights at different levels.

[0153] The multi-system attention mechanism includes two parts: channel attention mechanism and spatial attention mechanism. The channel attention mechanism can deeply analyze and model the intrinsic correlation and importance between different input feature channels. By dynamically adjusting the attention weights of each channel, it strengthens those feature channels that contribute significantly to target detection, while weakening or ignoring those relatively unimportant channels, thereby optimizing feature expression and improving the overall performance and efficiency of the initial target detection model. The spatial attention mechanism focuses on the spatial dimensions of the input feature image, guiding the backbone network to pay more attention to those information-rich and important areas during the training process. This mechanism enables the model to use spatial information more effectively, ignore background noise or irrelevant details, and focus on the important parts of the historical transmission inspection images, thereby enhancing the model's understanding ability and prediction accuracy.

[0154] Due to the complex background of historical power transmission inspection images, there may be many objects similar to the detection targets, such as hay and dead vines, which are easily mistaken for bird nests. Therefore, the multi-scale attention mechanism is introduced to enable the initial target detection model to focus on the key feature information in the historical power transmission inspection images, suppress the interference of background information, and improve the efficiency of extracting the feature information of the detection target.

[0155] In practical applications, such as Figure 7 As shown in the figure, after the historical power transmission inspection images are input into the initial target detection model, the high-level, middle-level and low-level convolutional layers (Conv, C3), pooling layers (Spatial Pyramid Pooling Fusion, SPPF) and multi-scale attention mechanisms in the backbone network of the initial target detection model are used to extract features at different scales. The channel attention mechanism of the multi-scale attention mechanism dynamically adjusts the attention weights of each channel to highlight important channels, and the spatial attention mechanism highlights important spatial positions, thereby extracting features from the input historical power transmission inspection images, gradually reducing the size of the feature image while increasing the number of channels, and extracting multiple first feature images of different scales through the backbone network. Among them, as Figure 8As shown in the figure, the channel attention mechanism includes a one-dimensional convolution and an adaptive average pooling layer (adaptiveavgpool). Through the adaptive average pooling layer and the convolution layer, the attention weight of each channel is learned. The spatial attention mechanism reduces the dimension of the feature image through a 1×1 convolution operation. It is evenly divided into two branches through channel splitting (channel split), and each branch performs two dilated convolutions. In order to enable the multi-scale attention mechanism to obtain multi-scale feature information, the dilation rates of the two branches are set to 1 and 4 respectively. The feature fusion operation is completed by connecting the two branches. After multiplying the results of the channel attention module and the spatial attention module, the sigmoid function is used to compress the product result. Among them, the main function of the dilated convolution is to significantly increase the effective receptive field of each convolution kernel while maintaining the resolution. This feature enables the network to capture a wider range of contextual information. Different dilation rates capture rich feature information from multiple scales. This ability to fuse multi-scale information improves the accuracy of the model when processing complex visual tasks.

[0156] S243, inputting the multiple first feature images of different sizes into the neck network in the initial target detection model, and performing multi-scale feature fusion on the multiple first feature images of different sizes based on the three-scale feature fusion module in the neck network to obtain multiple second feature images of different sizes.

[0157] Among them, the neck network is used to further perform feature fusion based on the features extracted by the backbone network. The triple feature encoding module (TFE) is a feature fusion mechanism, which is mainly used to fuse the features of feature images of large, medium and small sizes.

[0158] In practical applications, multiple first feature images of different sizes are input into the neck network, and feature fusion is performed through the three-scale feature module. Specifically, Fig. 9 As shown in the figure, the large-scale feature map passes through a convolution module and downsampling operations, which is conducive to retaining the high-resolution features and semantic information of different targets in the historical power transmission inspection images; the medium-scale feature map passes through a convolution module; the small-scale feature map passes through a convolution module and upsampling operations, which can maintain the richness of local features. Finally, the three feature maps of the same size, large, medium, and small, are convolved once, and then spliced ​​by channel to obtain the fused first feature image. The first feature images of different scales are fused through the three-scale feature module to capture richer feature information, especially to improve the ability to extract small target information.

[0159] The convolution module consists of convolution operation, batch normalization (BN) and SiLU activation function. Downsampling operations include maximum pooling and average pooling, and upsampling methods include bilinear interpolation and transposed convolution.

[0160] The convolution layer in the neck network further performs feature fusion and upsampling operations on the first feature image after the three-scale feature modules are fused to obtain multiple second feature images of different scales, so as to provide more advanced semantic information and the ability to adapt to images of different scales.

[0161] S246, inputting the multiple second feature images of different sizes into the head network in the target detection model, and generating multiple prediction boxes for identifying the bird's nest and the insulator based on the head network and the multiple second feature images of different sizes.

[0162] In actual applications, after the head network receives multiple second feature images of different sizes, these second feature images after feature fusion will be sent to the corresponding prediction heads respectively. Each prediction head is responsible for performing bounding box regression at a specific scale and generating multiple prediction boxes for identifying bird nests and insulators.

[0163] S248, based on a preset label assignment strategy, predict the category probability distribution of each prediction box, and determine the category of the prediction box based on the category probability distribution.

[0164] In practical applications, the preset label assignment strategy can be: calculate the intersection-over-intersection ratio of the predicted box and the true box, distinguish positive and negative samples based on the intersection-over-intersection ratio threshold, specifically, determine the predicted box with an intersection-over-intersection ratio higher than the intersection-over-intersection ratio threshold as a positive sample, and determine the predicted box with an intersection-over-intersection ratio lower than the intersection-over-intersection ratio threshold as a negative sample. For positive samples, the original classification score of each predicted box is output by the classification branch of the initial target detection model, and then converted into a probability distribution through an activation function (such as Softmax) to obtain the category probability distribution of the predicted box. After obtaining the category probability distribution of each predicted box, the category with the largest probability can be selected as the category of the predicted box.

[0165] In this embodiment, a multi-scale attention mechanism is introduced into the backbone network of the initial target detection model, so that the backbone network can focus on the key feature information of the input historical power transmission inspection images and suppress the interference of background information. A three-scale feature fusion module is introduced into the neck network to fuse feature map information of different scales to capture richer feature information, thereby improving the accuracy of determining the prediction box category.

[0166] The category probability distribution of each prediction box is predicted through a preset label assignment strategy. In an exemplary embodiment, S248 includes S2482 to S2484.

[0167] S2482, for each predicted box, determine the position similarity and size similarity between the predicted box and the real box.

[0168] S2484, determining the similarity between the predicted box and the true box based on the position similarity and the size similarity, and determining the category probability distribution of the predicted box based on the similarity.

[0169] In practical applications, in order to improve the detection effect of small targets (such as bird nests), the label assignment of the prediction box can be based on the similarity distance. Specifically, for each prediction box, the position similarity and size similarity between the prediction box and the real box are determined, which can be obtained by the following formula:

[0170]

[0171]

[0172] In the formula, sim L is the position similarity between the predicted box and the real box; sim S is the size similarity between the predicted box and the real box; (x g ,y g ), w g 、h g They are the center point coordinates, width and height of the real frame respectively; (x a ,y a ), w a 、h a The center coordinates and width and height of the predicted box are respectively. The differences between predicted boxes and true boxes of different sizes are eliminated by parameters m and n respectively.

[0173] Where m and n are defined as follows:

[0174]

[0175] Where m is the average ratio of the sum of the widths of all the annotation boxes and the prediction boxes in each historical power transmission inspection image; M is the number of historical power transmission inspection images; N i and Q i are the number of annotation boxes and the number of prediction boxes of the i-th image; x ij ,y ij 、w ij 、h ij are the x-axis coordinate, y-axis coordinate, width and height of the center point of the j-th annotation box in the i-th image; ik ,y ik、w ik 、h ik are the x-axis coordinate, y-axis coordinate, width and height of the center point of the k-th prediction box in the i-th image. Among them, the normalization parameters m and n are calculated based on the historical transmission inspection images and can automatically adapt to different data sets.

[0176] After obtaining the position similarity and size similarity between the predicted box and the true box, the similarity between the predicted box and the true box is determined based on the position similarity and size similarity, which can be obtained by the following formula:

[0177]

[0178] Where SimD is the similarity between the predicted box and the true box.

[0179] After obtaining the similarity between the predicted box and the true box, the category probability distribution of the predicted box is determined based on the similarity. This can be done by normalizing the similarity between the predicted box and the true box using an exponential function (such as the Softmax function or the Sigmoid function, etc.), converting it into a probability value, and obtaining the category probability distribution of the predicted box by weighted averaging.

[0180] In this embodiment, the similarity of the position and shape of the predicted box and the real box is taken into account. The similarity between the predicted box and the real box is determined by the position similarity and size similarity between the predicted box and the real box, which can well reflect the similarity relationship between the predicted box and the real box, improve the detection performance of the target detection model for small targets, and improve the accuracy of the prediction box category prediction.

[0181] In order to make a clearer description of the method for identifying the hazard level of bird nests on power transmission lines provided in the present application, a specific embodiment is described below. The specific embodiment includes the following steps:

[0182] S1, obtain the target power transmission inspection image, the target power transmission inspection image contains the bird's nest and the insulator, take the target power transmission inspection image as input, call the trained target detection model, and obtain the target detection result.

[0183] S2, performing instance segmentation on the target power transmission inspection image to obtain a first segmented area image including the bird's nest and a second segmented area image including the insulator.

[0184] S3, expand the sizes of the first detection frame and the second detection frame; determine the intersection and union ratio of the first detection frame after the size expansion and the second detection frame after the size expansion, if the intersection and union ratio is less than a preset intersection and union ratio threshold, it is determined that the bird's nest is not above the insulator; if the intersection and union ratio is greater than or equal to the intersection and union ratio threshold, it is determined that the bird's nest is above the insulator.

[0185] S4, when the IoR is greater than or equal to the IoR threshold, determine a first minimum bounding box of the first detection frame and a second minimum bounding box of the second detection frame, and expand the sizes of the first minimum bounding box and the second minimum bounding box; determine first depth information of the first minimum bounding box after the size expansion, and determine second depth information of the second minimum bounding box after the size expansion; determine first image depth information of the first segmented area image from the first depth information, determine second image depth information of the second segmented area image from the second depth information, and determine an image depth difference value between the first image depth information and the second image depth information.

[0186] S5, if the image depth difference value is less than the preset depth difference threshold, it is determined that the bird's nest is located directly above the insulator; if the image depth difference value is greater than the preset depth difference threshold, it is determined that the bird's nest is located above the insulator.

[0187] S6. When the bird's nest is located directly above the insulator, the bird's nest hazard level is determined to be a major hazard; when the bird's nest is not located above the insulator, or the bird's nest is located above the dislocation of the insulator, the bird's nest hazard level is determined to be a minor hazard.

[0188] In this embodiment, before acquiring the target power transmission inspection image, a target detection model is constructed and trained. The construction and training process of the target detection frame model is as follows:

[0189] S101, acquiring a target power transmission inspection image to construct an initial target detection model, wherein the initial target detection model includes a multi-scale attention mechanism and a three-scale feature fusion module.

[0190] S102, obtaining a historical power transmission inspection image, the historical power transmission inspection image contains a bird's nest and an insulator; using the historical power transmission inspection image as input, calling an initial target detection model, the historical power transmission inspection image carries the category label of the real frame of the bird's nest and the insulator and the position information of the real frame.

[0191] S103, based on the multi-scale attention mechanism, convolution layer and pooling layer in the backbone network of the initial target detection model, feature extraction is performed on the historical power transmission inspection image, and the feature importance of the historical power transmission inspection image is adjusted to obtain a plurality of first feature images of different sizes;

[0192] S104, inputting the multiple first feature images of different sizes into the neck network in the initial target detection model, and performing multi-scale feature fusion on the multiple first feature images of different sizes based on the three-scale feature fusion module in the neck network to obtain multiple second feature images of different sizes;

[0193] S105, inputting a plurality of second feature images of different sizes into a head network in a target detection model, and generating a plurality of prediction boxes for identifying bird nests and insulators based on the head network and the plurality of second feature images of different sizes, wherein the prediction boxes are used to identify bird nests and insulators in historical power transmission inspection images.

[0194] S106, for each predicted box, determine the position similarity and size similarity between the predicted box and the real box; determine the similarity between the predicted box and the real box based on the position similarity and size similarity, determine the category probability distribution of the predicted box based on the similarity, and determine the category of the predicted box based on the category probability distribution.

[0195] S107, based on the category label and position information of the real box, as well as the category and position information of the predicted box, determine the difference between the predicted box and the real box; based on the difference, iteratively update the parameters of the initial target detection model until the preset training end condition is reached, thereby obtaining the trained target detection model.

[0196] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0197] In an exemplary embodiment, Fig.10 As shown, a device 600 for identifying the hazard level of bird nests in power transmission lines is provided, comprising: an image acquisition module 610, a target object detection module 620, an instance segmentation module 630, a spatial position relationship determination module 640 and a hazard level determination module 650, wherein:

[0198] An image acquisition module 610 is used to acquire a target power transmission inspection image, where the target power transmission inspection image includes a bird's nest and an insulator;

[0199] A target object detection module 620 is used to detect bird nests and insulators in the target power transmission inspection image and obtain a target detection result;

[0200] An instance segmentation module 630 is used to perform instance segmentation on the target power transmission inspection image to obtain a first segmented region image including a bird's nest and a second segmented region image including an insulator;

[0201] A spatial position relationship determination module 640, configured to determine a spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented region image and the second segmented region image;

[0202] The hazard level determination module 650 is used to determine the hazard level of the bird's nest based on the spatial position relationship.

[0203] In an exemplary embodiment, the spatial position relationship determination module 640 is also used to expand the size of the first detection frame and the second detection frame; determine the intersection and union ratio of the first detection frame after the size expansion and the second detection frame after the size expansion; compare the intersection and union ratio with a preset intersection and union ratio threshold to determine the spatial position relationship between the bird's nest and the insulator.

[0204] In an exemplary embodiment, the spatial position relationship determination module 640 is also used to determine that the bird's nest is not located below the insulator if the intersection-and-parallel ratio is less than a preset intersection-and-parallel ratio threshold; if the intersection-and-parallel ratio is greater than or equal to the intersection-and-parallel ratio threshold, determine that the bird's nest is located above the insulator.

[0205] In an exemplary embodiment, the transmission line bird nest hazard level identification device 600 further includes an image depth information determination module 660, which is used to determine the image depth difference value between the first segmented region image and the second segmented region image when the intersection-and-union ratio is greater than or equal to the intersection-and-union ratio threshold;

[0206] The spatial position relationship determination module 640 is also used to determine that the bird's nest is located directly above the insulator if the image depth difference value is less than a preset depth difference threshold; if the image depth difference value is greater than the preset depth difference threshold, then determine that the bird's nest is located above the insulator.

[0207] In an exemplary embodiment, the hazard level determination module 650 is also used to determine the bird nest hazard level as a major hazard when the bird nest is located directly above the insulator; and to determine the bird nest hazard level as a minor hazard when the bird nest is not located above the insulator, or the bird nest is located above the insulator.

[0208] In an exemplary embodiment, the image depth information determination module 660 is also used to determine a first minimum external bounding box of a first detection frame, and a second minimum external bounding box of a second detection frame, and to expand the size of the first minimum external bounding box and the second minimum external bounding box; determine the first depth information of the first minimum external bounding box after the size expansion, and determine the second depth information of the second minimum external bounding box after the size expansion; determine the first image depth information of the first segmented area image from the first depth information, determine the second image depth information of the second segmented area image from the second depth information, and determine the image depth difference value between the first image depth information and the second image depth information.

[0209] In an exemplary embodiment, the target object detection module 620 is further used to take the target power transmission inspection image as input, call the trained target detection model, and obtain the target detection result.

[0210] The image acquisition module 610 is also used to acquire historical power transmission inspection images, which include bird nests and insulators.

[0211] The transmission line bird nest hazard level identification device 600 also includes a target detection model construction module 670 for constructing an initial target detection model, the initial target detection model including a multi-scale attention mechanism and a three-scale feature fusion module;

[0212] The transmission line bird nest hazard level identification device 600 also includes a target detection model training module 680, which is used to take historical transmission inspection images as input, call the initial target detection model, determine the category and position information of the prediction box, the prediction box is used to identify the bird nests and insulators in the historical transmission inspection images, and the historical transmission inspection images carry the category labels of the real boxes of the bird nests and insulators and the position information of the real boxes; based on the category labels and position information of the real boxes, as well as the category and position information of the prediction boxes, determine the difference between the prediction boxes and the real boxes; based on the difference, iteratively update the parameters of the initial target detection model until the preset training end conditions are reached, and obtain the trained target detection model.

[0213] In an exemplary embodiment, the target detection model training module 680 is also used to extract features from historical power transmission inspection images based on the multi-scale attention mechanism, convolution layer and pooling layer in the backbone network of the initial target detection model, adjust the feature importance of the historical power transmission inspection images, and obtain multiple first feature images of different sizes; input the multiple first feature images of different sizes into the neck network in the initial target detection model, and based on the three-scale feature fusion module in the neck network, perform multi-scale feature fusion on the multiple first feature images of different sizes to obtain multiple second feature images of different sizes; input the multiple second feature images of different sizes into the head network in the target detection model, and generate multiple prediction frames for identifying bird nests and insulators based on the head network and the multiple second feature images of different sizes; based on a preset label assignment strategy, predict the category probability distribution of each prediction frame, and determine the category of the prediction frame based on the category probability distribution.

[0214] In an exemplary embodiment, the target detection model training module 680 is also used to determine, for each predicted box, the position similarity and size similarity between the predicted box and the real box; determine the similarity between the predicted box and the real box based on the position similarity and size similarity, and determine the category probability distribution of the predicted box based on the similarity.

[0215] Each module in the above-mentioned power transmission line bird nest hazard level identification device 600 can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0216] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as target power transmission inspection images. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying the hazard level of bird nests in power transmission lines is implemented.

[0217] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0218] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in any one of the above-mentioned embodiments of the method for identifying the hazard level of bird nests on power transmission lines are implemented.

[0219] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any one of the above-mentioned methods for identifying the hazard level of bird nests in power transmission lines are implemented.

[0220] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in any one of the above-mentioned methods for identifying the hazard level of bird nests in power transmission lines.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0222] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0223] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0224] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying the hazard level of bird nests in power transmission lines, characterized in that: The method comprises: Acquire a target power transmission inspection image, wherein the target power transmission inspection image includes a bird's nest and an insulator; Detecting bird nests and insulators in the target power transmission inspection image to obtain a target detection result; Performing instance segmentation on the target power transmission inspection image to obtain a first segmented region image including the bird's nest and a second segmented region image including the insulator; Determine a spatial positional relationship between the bird's nest and the insulator based on the target detection result, the first segmented area image, and the second segmented area image; Based on the spatial position relationship, the hazard level of the bird's nest is determined.

2. The method according to claim 1, characterized in that The target detection result includes a first detection frame of the bird's nest and a second detection frame of the insulator; and determining the spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented region image, and the second segmented region image includes: Expanding the sizes of the first detection frame and the second detection frame; Determine an intersection-over-union ratio of the first detection frame after the size expansion and the second detection frame after the size expansion; The I / O ratio is compared with a preset I / O ratio threshold to determine the spatial position relationship between the bird's nest and the insulator.

3. The method according to claim 2, characterized in that The step of comparing the I / O ratio with a preset I / O ratio threshold to determine the spatial position relationship between the bird's nest and the insulator includes: If the intersection-and-parallel ratio is less than a preset intersection-and-parallel ratio threshold, it is determined that the bird's nest is not located above the insulator; If the I / O ratio is greater than or equal to the I / O ratio threshold, it is determined that the bird's nest is located above the insulator.

4. The method according to claim 3, characterized in that The method further comprises: When the IoU is greater than or equal to the IoU threshold, determining an image depth difference value between the first segmented region image and the second segmented region image; If the image depth difference value is less than a preset depth difference threshold, it is determined that the bird's nest is located directly above the insulator; If the image depth difference value is greater than a preset depth difference threshold, it is determined that the bird's nest is located above the misalignment of the insulator.

5. The method according to claim 4, characterized in that Determining the bird nest hazard level based on the spatial position relationship includes: In the case where the bird's nest is located directly above the insulator, the bird's nest hazard level is determined to be a major hazard; When the bird's nest is not located above the insulator, or the bird's nest is located above the misalignment of the insulator, the bird's nest hazard level is determined to be a secondary hazard.

6. The method according to claim 4, characterized in that The determining of the image depth difference value between the first segmented region image and the second segmented region image includes: Determine a first minimum outer bounding box of the first detection box and a second minimum outer bounding box of the second detection box, and expand the sizes of the first minimum outer bounding box and the second minimum outer bounding box; Determine first depth information of a first minimum bounding box after the size is expanded, and determine second depth information of a second minimum bounding box after the size is expanded; The first image depth information of the first segmented region image is determined from the first depth information, the second image depth information of the second segmented region image is determined from the second depth information, and an image depth difference value between the first image depth information and the second image depth information is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The detecting of the bird's nest and the insulator in the target power transmission inspection image to obtain the target detection result comprises: Taking the target power transmission inspection image as input, calling the trained target detection model to obtain a target detection result; The target detection model is trained based on the following method: Constructing an initial target detection model, wherein the initial target detection model includes a multi-scale attention mechanism and a three-scale feature fusion module; Acquire a historical power transmission inspection image, wherein the historical power transmission inspection image includes a bird's nest and an insulator; Taking the historical power transmission inspection image as input, calling the initial target detection model, and determining the category and position information of the prediction box, wherein the prediction box is used to identify the bird's nest and the insulator in the historical power transmission inspection image, and the historical power transmission inspection image carries the category label of the real box of the bird's nest and the insulator and the position information of the real box; Determine the difference between the predicted frame and the true frame based on the category label and position information of the true frame and the category and position information of the predicted frame; Based on the difference, the parameters of the initial target detection model are iteratively updated until a preset training end condition is reached to obtain a trained target detection model.

8. The method according to claim 7, characterized in that The target detection model includes a backbone network, a neck network and a head network. The initial target detection model determines the category of the prediction box based on the following method: Based on the multi-scale attention mechanism, convolution layer and pooling layer in the backbone network of the initial target detection model, feature extraction is performed on the historical power transmission inspection image, and the feature importance of the historical power transmission inspection image is adjusted to obtain a plurality of first feature images of different sizes; Inputting a plurality of first feature images of different sizes into the neck network in the initial target detection model, and performing multi-scale feature fusion on the plurality of first feature images of different sizes based on the three-scale feature fusion module in the neck network to obtain a plurality of second feature images of different sizes; Inputting the plurality of second feature images of different sizes into a head network in the target detection model, and generating a plurality of prediction frames for identifying the bird's nest and the insulator based on the head network and the plurality of second feature images of different sizes; Based on a preset label assignment strategy, a category probability distribution of each prediction box is predicted, and based on the category probability distribution, a category of the prediction box is determined.

9. The method according to claim 8, characterized in that The predicting of the category probability distribution of each prediction box based on the preset label assignment strategy includes: For each of the predicted frames, determining a position similarity and a size similarity between the predicted frame and the real frame; The similarity between the predicted box and the real box is determined based on the position similarity and the size similarity, and the category probability distribution of the predicted box is determined based on the similarity.

10. A device for identifying the hazard level of bird nests on power transmission lines, characterized in that: The device comprises: An image acquisition module is used to acquire a target power transmission inspection image, wherein the target power transmission inspection image includes a bird's nest and an insulator; A target object detection module, used to detect bird nests and insulators in the target power transmission inspection image to obtain a target detection result; An instance segmentation module is used to perform instance segmentation on the target power transmission inspection image to obtain a first segmented area image including the bird's nest and a second segmented area image including the insulator; A spatial position relationship determination module, configured to determine a spatial position relationship between the bird's nest and the insulator based on the target detection result, the first segmented region image, and the second segmented region image; The hazard level determination module is used to determine the hazard level of the bird's nest based on the spatial position relationship.

Images