Power transmission line safety monitoring and early-warning method and system based on computer vision

Abstract

Disclosed in the present invention are a power transmission line safety monitoring and early-warning method and system based on computer vision. The method comprises: collecting an on-site picture of a power transmission line; identifying a power transmission channel area and a dynamic target in the on-site picture; on the basis of an identification result for the power transmission channel area and the dynamic target, determining an interference result for the dynamic target and the power transmission channel area, specifically involving: performing overlapping contrast on trajectory information of the dynamic target and the power transmission channel area, so as to give an overlapping result, and on the basis of the overlapping result, screening out the dynamic target, the trajectory information of which overlaps with the power transmission channel area; and determining alarm information on the basis of the interference result. The present invention can automatically define a power transmission channel area near a power transmission tower, thereby greatly reducing the workload of a line monitoring person manually defining a power transmission channel, and improving the applicability of large-range application of an algorithm. The accuracy of power transmission line safety early-warning can be improved.

Description

A computer vision-based transmission line safety monitoring and early warning method and system Technical Field

[0001] The present invention relates to the field of computer vision tracking technology, and in particular to a power transmission line safety monitoring and early warning method and system based on computer vision. Background Art

[0002] With the acceleration of urbanization, numerous construction activities are taking place near power transmission lines. The operation of large construction equipment such as tower cranes, cement pouring trucks, excavators, and dump trucks threatens the safe operation of power facilities. Therefore, the power industry urgently needs tools that can identify and respond to dynamic abnormal targets online. Dynamic abnormal targets are highly mobile and their movements are difficult to predict, causing unexpected damage to transmission lines and towers. Therefore, anomaly detection and identification must be fast and accurate to enable timely action when necessary.

[0003] Currently, transmission channel anomaly detection typically focuses on static object detection, and is broadly categorized by detection method into manual inspections and camera inspections. Manual inspections typically involve professionals traveling along transmission lines in power operation and maintenance vehicles or helicopters, visually monitoring the transmission lines. Based on their experience, they determine whether vegetation or buildings are encroaching on the transmission lines. Camera inspections are categorized into fixed camera inspections and mobile camera inspections. Fixed camera inspections involve surveillance cameras mounted on specific towers capturing images of the transmission lines at regular intervals. These images are then transmitted to a data storage center, where line operation and maintenance personnel analyze the images and determine the safety of the lines. Mobile camera inspections typically involve unmanned aerial vehicles equipped with surveillance cameras that patrol along transmission lines and transmit the video feeds back to operators, who then assess the safety of the lines. For example, CN106932688A discloses a transmission line detector and a drone-based transmission line detection system. The transmission line detector includes a housing, a camera mounted on the outer wall of the housing, and an image preprocessor, a line detector, and a transmission line identifier mounted inside the housing. The camera is configured to capture image information of a target area; the image preprocessor is configured to process the image information; the line detector is configured to mark detected lines on the processed image information; the transmission line identifier is configured to mark the identified transmission line on the image information after marking the lines and output the image information after marking the lines. The transmission line detector also includes a communication module for wirelessly transmitting the received image information after marking the lines to an associated terminal. This system can accurately detect transmission lines in complex background environments, providing convenience for power workers and improving the effectiveness and reliability of power inspections. However, neither manual nor camera inspections can provide real-time, uninterrupted online monitoring and safety assessment of transmission channels, rendering existing technologies incapable of monitoring and early warning of dynamic targets on site.

[0004] The main challenges of dynamic anomaly detection are as follows. Large-scale autonomous monitoring of power transmission channels requires algorithms that can adapt to the diversity of transmission scenarios and the complexity of on-site weather conditions, and automatically define transmission channels. Algorithms are also needed to track dynamic targets and identify their dynamic trajectories and operating states to avoid misjudgments and missed detections.

[0005] Therefore, how to provide a transmission line monitoring method that is applicable to different transmission scenarios and has dynamic target tracking and high recognition and warning accuracy is an urgent problem to be solved in this field.

[0006] Summary of the Invention

[0007] In response to the above-mentioned deficiencies in the prior art, the present invention provides a computer vision-based power transmission line safety monitoring and early warning method and system, which can improve the accuracy of power transmission line safety early warning.

[0008] In a first aspect, the present invention provides a computer vision-based transmission line safety monitoring and early warning method, comprising:

[0009] Collect on-site pictures of transmission lines;

[0010] Identify transmission channel areas and dynamic targets in on-site images;

[0011] Based on the identification results of the transmission channel area and the dynamic target, determining the interference result between the dynamic target and the transmission channel area;

[0012] An alarm message is determined based on the interference result.

[0013] Furthermore, the transmission channel area is identified from the on-site pictures, including:

[0014] Perform edge processing on the scene image to obtain edge information belonging to the straight line segment;

[0015] The edge information of the straight line segment is filtered and processed to give the transmission line;

[0016] The given transmission line is mapped to the ground to form the transmission channel area.

[0017] Furthermore, edge processing is performed on the scene image to obtain edge information belonging to the straight line segment, including:

[0018] Convert the scene image into a grayscale image and perform edge detection to obtain edge information;

[0019] Based on the grayscale threshold, the edge information is divided to obtain the divided edge information;

[0020] Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale image;

[0021] Perform Hough transform on the binary grayscale image to give the polar coordinates of the edge information;

[0022] Screening the polar coordinates of edge information to provide multiple sets of edge information that meet predetermined conditions;

[0023] Treat the edge information of the same group as the edge information of a straight line segment;

[0024] The predetermined condition is that the number of edge information in the same group is not less than a predetermined number, and the polar coordinates of all edge information in the same group form a straight line segment in the polar coordinate system.

[0025] Furthermore, the binary grayscale image is processed by Hough transform to give the polar coordinates of the edge information, including: y = ax + b → r = xcosθ + ysinθ

[0026] Where (x, y) is the coordinate of the straight line segment in the binary grayscale image in the rectangular coordinate system, a is the slope of the straight line segment, b is the intercept of the straight line segment, (r, θ) is the polar coordinate of the straight line segment in the polar coordinate system after Hough transform processing, r is the distance from the origin to the straight line segment, and θ is the angle between r and the horizontal axis in the rectangular coordinate system.

[0027] The present invention performs Hough transform on a binary grayscale image to form polar coordinates, thereby screening out straight line segments in the binary grayscale image, and splicing the screened straight line segments in a rectangular coordinate system.

[0028] Furthermore, the edge information of the straight line segment is filtered and processed to give the transmission line, including:

[0029] Classify straight line segments with the same slope into the same category;

[0030] Connect the ends of the two straight line segments with overlapping points in each category to form a straight line segment;

[0031] Based on each type of connected straight line segments, the lengths of all straight line segments are obtained;

[0032] Compare the length of the straight line segments in each category with the length threshold, and give the straight line segments that are greater than the length threshold;

[0033] The straight line segments in the same class that are greater than the length threshold are regarded as transmission lines in the same location area;

[0034] Integrate the transmission lines of all location areas to give the transmission lines.

[0035] Furthermore, the straight line segments with the same slope are divided into the same category, including:

[0036] Subtract the slopes of all straight line segments pairwise to give the slope difference result;

[0037] Based on the slope difference threshold, all slope difference results are screened and the slope difference results within the slope difference threshold are given, specifically: |k1-k2|<α

[0038] Where k1 and k2 are the slopes of the two straight line segments, α is the slope difference threshold, and α is 0.2;

[0039] According to the filtered slope difference results, the two straight lines corresponding to each slope difference result are divided into the same category.

[0040] Furthermore, the ends of the two straight line segments with overlapping points in each category are connected to form a straight line segment, including:

[0041] Get the endpoint coordinates of all line segments in each category;

[0042] The endpoint distance of any two straight line segments in each category is solved;

[0043] Based on the distance threshold, all the endpoint distances solved are screened and the endpoint distances within the distance threshold are given, specifically:

[0044] Where x1 and y1 are the coordinates of the endpoints of one straight line segment, x2 and y2 are the coordinates of the endpoints of the other straight line segment, and β is the distance threshold, which is 20 pixels.

[0045] Divide all two straight line segments corresponding to the given endpoint distances into two straight line segments with overlapping points, and connect the corresponding two endpoints.

[0046] Furthermore, the given transmission line is mapped to the ground to form a transmission channel area, including:

[0047] Based on Hough transform, the given transmission line is identified and the endpoints of the two transmission lines located outside in each location area are obtained;

[0048] Connect the endpoints of the two transmission lines located outside of all location areas to obtain an outer envelope sub-area corresponding to the location area;

[0049] Integrating all outer envelope sub-areas to form the outer envelope area of ​​the transmission line;

[0050] Based on the connected endpoint coordinates, the boundary coordinates of the outer envelope area are obtained;

[0051] The boundary coordinates of the outer envelope area are mapped symmetrically along the horizontal axis of the on-site picture to form the transmission channel area.

[0052] Furthermore, after obtaining the endpoints of the two transmission lines located outside each location area, the method includes:

[0053] Obtain the endpoint coordinates of the two transmission lines located outside of all location areas;

[0054] For the two outer transmission lines in each location area, determine the upper end point and the lower end point of each transmission line;

[0055] Perform coordinate difference processing on the two lower endpoint ordinates and the two upper endpoint ordinates in each position area;

[0056] Compare all coordinate difference processing results with the endpoint difference threshold, and give the coordinate difference processing results that exceed the endpoint difference threshold;

[0057] Based on the given coordinate difference processing results, the endpoints of the two transmission lines corresponding to them are obtained;

[0058] Based on the endpoints of the two corresponding transmission lines, the lengths of the two transmission lines corresponding to the endpoints are calculated, and a line segment with the same slope and a certain length as the smaller transmission line is added along its corresponding endpoint; wherein the length of the added line segment is equal to the difference in length between the two transmission lines.

[0059] Furthermore, the coordinate difference is processed as the absolute value of the difference between the vertical coordinates of the two endpoints, and the endpoint difference threshold is 200 pixels.

[0060] Furthermore, dynamic target recognition is performed on the scene image, including:

[0061] Perform target detection on the scene image based on the pre-built target detection model, and provide the detection frame, target type and location information of all targets in the scene image;

[0062] Obtain the target detection result of the scene image collected before the current scene image;

[0063] Combine the target detection results of the current scene image and the previously collected scene image to provide the current trajectory information of the dynamic target;

[0064] The historical trajectory information of the dynamic target is obtained, and combined with the current trajectory information of the dynamic target, the trajectory information of the dynamic target is obtained.

[0065] Furthermore, the target detection results of the current scene image and the previously collected scene image are combined to provide the current trajectory information of the dynamic target, including:

[0066] Overlap comparison is performed on all detection frames in the current scene image and all detection frames in the scene image collected previously, and the overlap comparison result is given;

[0067] Based on the overlapping comparison results, each detection frame in the current scene image and the detection frames in the previously collected scene image that overlap are taken as a group of detection frames;

[0068] Each group of detection frames is screened based on the overlap threshold, and the detection frames in the previously collected scene image whose overlap comparison results with the detection frames of the current scene image are less than the overlap threshold are eliminated;

[0069] Compare the geometric centers of the remaining detection frames in the previous scene image in each group of detection frames with the geometric centers of the detection frames in the current scene image, and give a detection frame in the previous scene image that is closest to the geometric center of the detection frame in the current scene image;

[0070] The geometric center of the detection frame in the current scene image within each group of detection frames is connected with the geometric center of a detection frame in the given previous scene image to obtain the trajectory information of the target corresponding to the detection frame in the current scene image.

[0071] Furthermore, all detection frames in the current scene image are overlapped and compared with all detection frames in the scene image collected previously, and the overlap comparison results are given, specifically:

[0072] Where IOU(box1,box2) is the overlapping comparison result, Area(box1) is the detection box in the previous scene picture, and Area(box2) is the detection box in the current scene picture.

[0073] Furthermore, before connecting the geometric center of the detection frame in the current scene image within each group of detection frames with the geometric center of a detection frame in the given previously acquired scene image, the method further includes:

[0074] Obtain the detection frames in the current scene image where the overlap comparison result shows no overlap;

[0075] Calculate the distance between the geometric center of the non-overlapping detection frame and the geometric center of all detection frames in the previously collected scene image, and give the detection frame in the previously collected scene image with the smallest distance to the non-overlapping detection frame;

[0076] The detection frames without overlap and the detection frames with the smallest distance are divided into one group.

[0077] Furthermore, the target detection results of the current scene image and the previously collected scene image are combined to provide the current trajectory information of the dynamic target, which also includes:

[0078] Based on the trajectory information of all targets in the current scene image, the corresponding overlapping comparison results of all target trajectory information within the preset alarm time are given, as well as the distance between the geometric center of the detection frame in the current scene image and the geometric center of a corresponding detection frame in the previously collected scene image within the preset alarm time;

[0079] Based on the dynamic overlap threshold and dynamic distance threshold, the overlap comparison results and the distance to the geometric center corresponding to all target trajectory information are screened to provide the trajectory information of the dynamic target, including:

[0080] Eliminate the target trajectory information whose overlap comparison result is greater than the dynamic overlap threshold and whose geometric center distance is less than the dynamic distance threshold within the preset alarm time, and use the trajectory information of the remaining targets as the trajectory information of the dynamic targets.

[0081] Furthermore, based on the dynamic overlap threshold and dynamic distance threshold, the overlap comparison results and the distances to the geometric centers corresponding to all target trajectory information are screened to satisfy the following relationship:

[0082] Where a is the time interval between the current scene image and the previous moment and the preset alarm time, N is the number of frames of the scene image within the preset alarm time, is the detection frame in the nth frame of the scene picture in time period a, is the horizontal coordinate of the geometric center of the detection frame in the nth frame of the scene image within time period a, is the vertical coordinate of the geometric center of the detection box in the n-th frame scene image in time period a, μ is the dynamic overlap threshold, and τ is the dynamic distance threshold.

[0083] Furthermore, the dynamic overlap threshold is 95% and the dynamic distance threshold is 10 pixels.

[0084] Furthermore, based on the identification results of the transmission channel area and the dynamic target, the interference result between the dynamic target and the transmission channel area is determined, including:

[0085] Overlap and compare the trajectory information of the dynamic target with the transmission channel area and give the overlapping results;

[0086] Based on the overlapping results, dynamic targets whose trajectory information overlaps with the transmission channel area are screened out.

[0087] Furthermore, the alarm information is determined based on the interference result, including:

[0088] According to the location information and transmission channel area of ​​all targets in the current scene picture, and combined with the trajectory information of dynamic targets overlapping with the transmission channel area, the dynamic targets located in the transmission channel area are given;

[0089] Based on the historical alarm information within the predetermined time, the historical dynamic target of the alarm is given;

[0090] Match the dynamic targets in the transmission channel area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched;

[0091] Generates alarm information based on unsuccessful matching of dynamic targets.

[0092] Furthermore, an alarm message is generated based on the unsuccessful matching of the dynamic target, including:

[0093] According to the trajectory information of the dynamic target that was not successfully matched, the geometric center of the detection box at different times is obtained;

[0094] Obtain pixel change values ​​of unmatched dynamic targets based on the geometric center of the detection frame at different times;

[0095] Compare the obtained pixel change value with the pixel change threshold to obtain the dynamic target within the pixel change threshold range;

[0096] According to the dynamic target within the pixel change threshold range, a historical scene picture containing the dynamic target is obtained;

[0097] Based on the historical and current on-site pictures of the dynamic target within the pixel change threshold range, the historical or current on-site picture with the minimum distance between the dynamic target and the boundary of the transmission channel area is obtained;

[0098] Based on the historical scene picture with the smallest distance or the current scene picture, the corresponding target collection start time is given;

[0099] According to the target acquisition start time and the acquisition time corresponding to the current scene picture, the interference time of the dynamic target entering the transmission channel area is given;

[0100] Based on the intervention time and the preset alarm time, a dynamic target with an intervention time not less than the preset alarm time is given;

[0101] Generate corresponding alarm information based on dynamic targets whose interference time is not less than the preset alarm time.

[0102] In a second aspect, the present invention further provides a power transmission line safety monitoring and early warning system based on computer vision, which adopts the above-mentioned power transmission line safety monitoring and early warning method based on computer vision, and the system includes:

[0103] A data acquisition module, which is used to collect on-site pictures of the transmission line;

[0104] Image recognition module, which is used to identify transmission channel areas and dynamic targets in on-site images;

[0105] An interference judgment module, which is used to determine the interference result between the dynamic target and the transmission channel area based on the identification results of the transmission channel area and the dynamic target;

[0106] The early warning determination module is used to determine the alarm information based on the interference result.

[0107] Furthermore, the image recognition module is also used to:

[0108] Perform edge processing on the scene image to obtain edge information belonging to the straight line segment;

[0109] The edge information of the straight line segment is filtered and processed to give the transmission line;

[0110] The given transmission line is mapped to the ground to form the transmission channel area.

[0111] Furthermore, the image recognition module is also used to:

[0112] Convert the scene image into a grayscale image and perform edge detection to obtain edge information;

[0113] Based on the grayscale threshold, the edge information is divided to obtain the divided edge information;

[0114] Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale image;

[0115] Perform Hough transform on the binary grayscale image to give the polar coordinates of the edge information;

[0116] Screening the polar coordinates of edge information to provide multiple sets of edge information that meet predetermined conditions;

[0117] The edge information of the same group is regarded as the edge information of a straight line segment.

[0118] Furthermore, the image recognition module is also used to:

[0119] Classify straight line segments with the same slope into the same category;

[0120] Connect the ends of the two straight line segments with overlapping points in each category to form a straight line segment;

[0121] Based on each type of connected straight line segments, the lengths of all straight line segments are obtained;

[0122] Compare the length of the straight line segments in each category with the length threshold, and give the straight line segments that are greater than the length threshold;

[0123] The straight line segments in the same class that are greater than the length threshold are regarded as transmission lines in the same location area;

[0124] Integrate the transmission lines of all location areas to give the transmission lines.

[0125] Furthermore, the image recognition module is also used to:

[0126] Perform target detection on the scene image based on the pre-built target detection model, and provide the detection frame, target type and location information of all targets in the scene image;

[0127] Obtain the target detection result of the scene image collected before the current scene image;

[0128] Combine the target detection results of the current scene image and the previously collected scene image to provide the current trajectory information of the dynamic target;

[0129] The historical trajectory information of the dynamic target is obtained, and combined with the current trajectory information of the dynamic target, the trajectory information of the dynamic target is obtained.

[0130] Furthermore, the interference judgment module is also used to:

[0131] Overlap and compare the trajectory information of the dynamic target with the transmission channel area and give the overlapping results;

[0132] Based on the overlapping results, dynamic targets whose trajectory information overlaps with the transmission channel area are screened out.

[0133] Furthermore, the early warning determination module is also used to:

[0134] According to the location information and transmission channel area of ​​all targets in the current scene picture, and combined with the trajectory information of dynamic targets overlapping with the transmission channel area, the dynamic targets located in the transmission channel area are given;

[0135] Based on the historical alarm information within the predetermined time, the historical dynamic target of the alarm is given;

[0136] Match the dynamic targets in the transmission channel area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched;

[0137] Generates alarm information based on unsuccessful matching of dynamic targets.

[0138] Furthermore, the early warning determination module is also used to:

[0139] According to the location information and transmission channel area of ​​all targets in the current scene picture, and combined with the trajectory information of dynamic targets overlapping with the transmission channel area, the dynamic targets located in the transmission channel area are given;

[0140] Based on the historical alarm information within the predetermined time, the historical dynamic target of the alarm is given;

[0141] Match the dynamic targets in the transmission channel area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched;

[0142] Generates alarm information based on unsuccessful matching of dynamic targets.

[0143] Furthermore, the early warning determination module is also used to:

[0144] According to the trajectory information of the dynamic target that was not successfully matched, the geometric center of the detection box at different times is obtained;

[0145] Obtain pixel change values ​​of unmatched dynamic targets based on the geometric center of the detection frame at different times;

[0146] Compare the obtained pixel change value with the pixel change threshold to obtain the dynamic target within the pixel change threshold range;

[0147] According to the dynamic target within the pixel change threshold range, a historical scene picture containing the dynamic target is obtained;

[0148] Based on the historical and current on-site pictures of the dynamic target within the pixel change threshold range, the historical or current on-site picture with the minimum distance between the dynamic target and the boundary of the transmission channel area is obtained;

[0149] Based on the historical scene picture with the smallest distance or the current scene picture, the corresponding target collection start time is given;

[0150] According to the target acquisition start time and the acquisition time corresponding to the current scene picture, the interference time of the dynamic target entering the transmission channel area is given;

[0151] Based on the intervention time and the preset alarm time, a dynamic target with an intervention time not less than the preset alarm time is given;

[0152] Generate corresponding alarm information based on dynamic targets whose interference time is not less than the preset alarm time.

[0153] In a third aspect, the present invention further provides a power transmission line safety monitoring and early warning device based on computer vision, comprising a power transmission channel identification unit, a dynamic target identification unit, and an early warning information generation unit;

[0154] The transmission channel identification unit is used to collect on-site pictures of the transmission lines;

[0155] The dynamic target recognition unit is used to identify the transmission channel area and dynamic targets in the on-site pictures;

[0156] The early warning information generating unit is used to determine the interference result between the dynamic target and the power transmission channel area based on the identification result of the power transmission channel area and the dynamic target; and is also used to determine the alarm information based on the interference result.

[0157] The present invention provides a computer vision-based power transmission line safety monitoring and early warning method and system, which have at least the following beneficial effects:

[0158] (1) It can provide a method for identifying dangerous transmission channel areas. It can also identify dynamic targets, thereby generating alarm information for targets in motion and improving the accuracy of transmission line safety warnings.

[0159] (2) The transmission channel area near the transmission tower can be automatically defined, which greatly reduces the workload of line monitoring personnel in manually defining the transmission channel and improves the applicability of the algorithm for large-scale application.

[0160] (3) By performing anti-shake processing, trajectory analysis processing, and speed analysis processing on moving targets, static targets and non-threatening dynamic targets are effectively screened out, greatly reducing the false alarm rate and improving the work efficiency of background monitoring personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0161] FIG1 is a flow chart of a computer vision-based power transmission line safety monitoring and early warning method provided by the present invention;

[0162] FIG2 is a flow chart of obtaining edge information of a straight line segment according to an embodiment of the present invention;

[0163] FIG3 is a flow chart showing a power transmission line according to an embodiment of the present invention;

[0164] FIG4 is a flow chart of forming a power transmission channel area according to an embodiment of the present invention;

[0165] FIG5 is a flow chart of providing dynamic target trajectory information according to an embodiment of the present invention;

[0166] FIG6 is a flowchart of generating alarm information according to an embodiment of the present invention;

[0167] FIG7 is a schematic diagram of a computer vision-based power transmission line safety monitoring and early warning system provided by the present invention;

[0168] FIG8 is a schematic diagram of a computer vision-based power transmission line safety monitoring and early warning device provided by the present invention.

[0169] Explanation of the accompanying symbols: 1-transmission channel identification unit, 2-dynamic target identification unit, 3-early warning information generation unit, 4-edge detection subunit, 5-edge screening subunit, 6-line segment identification subunit, 7-transmission line fitting subunit, 8-transmission line boundary identification subunit, 9-transmission channel generation subunit, 10-video stack subunit, 11-target recognition model subunit, 12-target tracking subunit, 13-dynamic trajectory analysis unit, 14-early warning information comparison subunit, 15-historical warning cache subunit, 16-early warning information return subunit. DETAILED DESCRIPTION

[0170] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0171] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "an," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, and unless the context clearly indicates otherwise, "a plurality" generally includes at least two.

[0172] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or device comprising the element.

[0173] As shown in FIG1 , the present invention provides a computer vision-based transmission line safety monitoring and early warning method, comprising:

[0174] Collect on-site pictures of transmission lines;

[0175] Identify transmission channel areas and dynamic targets in on-site images;

[0176] Based on the identification results of the transmission channel area and the dynamic target, determining the interference result between the dynamic target and the transmission channel area;

[0177] An alarm message is determined based on the interference result.

[0178] After collecting on-site pictures of the power transmission lines, identifying the power transmission channel area in the on-site pictures may include:

[0179] Perform edge processing on the scene image to obtain edge information belonging to the straight line segment;

[0180] The edge information of the straight line segment is filtered and processed to give the transmission line;

[0181] The given transmission line is mapped to the ground to form the transmission channel area.

[0182] By defining the transmission channel area near the transmission tower, the workload of line monitoring personnel in manually defining the transmission channel is reduced, and the applicability of a wide range of applications is improved.

[0183] As shown in FIG2 , in the process of identifying the transmission channel area in the on-site image, edge processing is performed on the on-site image to obtain edge information belonging to the straight line segment, which may include:

[0184] Convert the scene image into a grayscale image and perform edge detection to obtain edge information;

[0185] Based on the grayscale threshold, the edge information is divided to obtain the divided edge information;

[0186] Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale image;

[0187] Perform Hough transform on the binary grayscale image to give the polar coordinates of the edge information;

[0188] Screening the polar coordinates of edge information to provide multiple sets of edge information that meet predetermined conditions;

[0189] Treat the edge information of the same group as the edge information of a straight line segment;

[0190] The predetermined conditions are that the number of edge information in the same group is not less than a predetermined number, and the polar coordinates of all edge information in the same group form a straight line segment in the polar coordinate system; the grayscale threshold is 50. Edge information refers to edge points with corresponding coordinates in the grayscale image.

[0191] After converting the scene image into a grayscale image, the grayscale image can be processed with the Sobel operator to complete edge detection and obtain edge information.

[0192] When performing binary grayscale conversion on the divided edge information, the edge information exceeding the conversion threshold is set to 1, and the edge information below the conversion threshold is set to 0.

[0193] After performing a Hough transform on a binary grayscale image, the edge points corresponding to the edge information are represented by polar coordinates in a polar coordinate system. When screening and giving multiple sets of edge information that meet the predetermined conditions, if more than 100 edge information can be described by a polar coordinate straight line formula, then the edge information of this straight line segment exists in the image and is retained in the edge detection results. The remaining edge information that does not meet this requirement is deleted.

[0194] Among them, the binary grayscale image is processed by Hough transform to give the polar coordinates of the edge information, including: y=ax+b→r=xcosθ+ysinθ

[0195] Where (x, y) is the Cartesian coordinate system of the line segment in the binary grayscale image, a is the slope of the line segment, b is the intercept of the line segment, (r, θ) is the polar coordinate system of the line segment after the Hough transform, r is the distance from the origin to the line segment, and θ is the angle between r and the horizontal axis of the Cartesian coordinate system. The polar coordinate transform is used to screen out the line segments in the scene image, and all the screened line segments are subsequently processed in the Cartesian coordinate system.

[0196] As shown in FIG3 , after obtaining the edge information belonging to the straight line segment, the edge information of the straight line segment is filtered and processed to obtain the transmission line, which may include:

[0197] Classify straight line segments with the same slope into the same category;

[0198] Connect the ends of the two straight line segments with overlapping points in each category to form a straight line segment;

[0199] Based on each type of connected straight line segments, the lengths of all straight line segments are obtained;

[0200] Compare the length of the straight line segments in each category with the length threshold, and give the straight line segments that are greater than the length threshold;

[0201] The straight line segments in the same class that are greater than the length threshold are regarded as transmission lines in the same location area;

[0202] Integrate the transmission lines of all location areas to give the transmission lines;

[0203] Among them, straight line segments with the same slope are divided into the same category, including:

[0204] Subtract the slopes of all straight line segments pairwise to give the slope difference result;

[0205] Based on the slope difference threshold, all slope difference results are screened and the slope difference results within the slope difference threshold are given, specifically: |k1-k2|<α

[0206] Where k1 and k2 are the slopes of the two straight line segments, α is the slope difference threshold, and α is 0.2;

[0207] According to the filtered slope difference results, the two straight lines corresponding to each slope difference result are divided into the same category.

[0208] Connect the ends of two straight line segments with overlapping points in each category to form a straight line segment, including:

[0209] Get the endpoint coordinates of all line segments in each category;

[0210] The endpoint distance of any two straight line segments in each category is solved;

[0211] Based on the distance threshold, all the endpoint distances solved are screened and the endpoint distances within the distance threshold are given, specifically:

[0212] Where x1 and y1 are the coordinates of the endpoints of one straight line segment, x2 and y2 are the coordinates of the endpoints of the other straight line segment, and β is the distance threshold, which is 20 pixels.

[0213] Divide all two straight line segments corresponding to the given endpoint distances into two straight line segments with overlapping points, and connect the corresponding two endpoints.

[0214] In this embodiment, the polar coordinates of edge information of all groups of the same class form straight line segments with the same slope in the polar coordinate system; the length threshold is 1 / 20 of the Y-axis height of the binary grayscale image, that is, when the length of the straight line segment is greater than 1 / 20 of the Y-axis height of the binary grayscale image, the straight line segment is determined to be a transmission line.

[0215] As shown in FIG4 , after the transmission line is obtained, the given transmission line can be mapped to the ground to form a transmission channel area, specifically including:

[0216] Based on Hough transform, the given transmission lines are identified and the starting ends of the two transmission lines located outside each location area are obtained;

[0217] Connect the endpoints of the two transmission lines located outside of all location areas to obtain an outer envelope sub-area corresponding to the location area;

[0218] Integrate all outer envelope sub-areas to form the outer envelope area of ​​the transmission line; obtain the boundary coordinates of the outer envelope area based on the connected endpoint coordinates;

[0219] The boundary coordinates of the outer envelope area are mapped symmetrically along the horizontal axis of the on-site picture to form the transmission channel area.

[0220] Specifically, the starting ends of the transmission lines are identified in the binary grayscale image through the Hough transform, and the area formed by connecting the starting ends is used as the outer envelope area of ​​the transmission line. Among them, when the identification is based on the Hough transform, when there are two or more transmission lines in each location area, the two outermost transmission lines will be automatically selected during the identification, and the starting ends of these two transmission lines will be projected to the ground to obtain four boundary points, which are connected into a quadrilateral area to obtain the outer envelope sub-area. When mapping to the ground, the mapping area is: the boundary coordinates of the outer envelope sub-area of ​​the transmission line are mapped symmetrically along the X-axis of the image to form a new outer envelope area, and the new outer envelope area is the transmission channel area. Among them, the example of X-axis mirror symmetry mapping is that the starting coordinates are (5, 3), and the X-axis symmetry mapping is (5, -3).

[0221] In order to ensure that the transmission lines on both sides of each location area are symmetrical, have uniform lengths, and have regular transmission channel shapes, after obtaining the endpoints of the two transmission lines located on the outside of each location area, the following steps can be performed:

[0222] Obtain the endpoint coordinates of the two transmission lines located outside of all location areas;

[0223] For the two outer transmission lines in each location area, determine the upper end point and the lower end point of each transmission line;

[0224] Perform coordinate difference processing on the two lower endpoint ordinates and the two upper endpoint ordinates in each position area; the coordinate difference processing is the absolute value of the difference between the two endpoint ordinates, and the endpoint difference threshold is 200 pixels;

[0225] Compare all coordinate difference processing results with the endpoint difference threshold, and give the coordinate difference processing results that exceed the endpoint difference threshold;

[0226] Based on the given coordinate difference processing results, the endpoints of the two transmission lines corresponding to them are obtained;

[0227] Based on the endpoints of the two corresponding transmission lines, the lengths of the two transmission lines corresponding to the endpoints are calculated, and a line segment with the same slope and a certain length as the smaller transmission line is added along its corresponding endpoint; wherein the length of the added line segment is equal to the difference in length between the two transmission lines.

[0228] After collecting on-site pictures of the power transmission line, the present invention can identify the transmission channel area and also identify dynamic targets in the on-site pictures, specifically including:

[0229] Perform target detection on the scene image based on the pre-built target detection model, and provide the detection frame, target type and location information of all targets in the scene image;

[0230] Obtain the target detection result of the scene image collected before the current scene image;

[0231] Combine the target detection results of the current scene image and the previously collected scene image to provide the current trajectory information of the dynamic target;

[0232] The historical trajectory information of the dynamic target is obtained, and combined with the current trajectory information of the dynamic target, the trajectory information of the dynamic target is obtained.

[0233] Each time the embodiment of the present invention collects the current scene picture, it combines it with the target detection result of the scene picture collected the previous time to give the current trajectory information of the dynamic target. That is, after the embodiment of the present invention collects the scene picture at different times, it will obtain the current trajectory information of the dynamic target at the corresponding time. Therefore, when obtaining the historical trajectory information of the dynamic target, the embodiment of the present invention can directly retrieve it from the historical data and use it. Of course, the historical trajectory information of the dynamic target in the embodiment of the present invention can also be obtained when needed according to the process of giving the current trajectory information of the dynamic target in this embodiment.

[0234] As shown in FIG5 , combining the target detection results of the current scene picture and the scene picture collected previously, the current trajectory information of the dynamic target is given, which may include:

[0235] Overlap comparison is performed on all detection frames in the current scene image and all detection frames in the scene image collected previously, and the overlap comparison result is given;

[0236] Based on the overlapping comparison results, each detection frame in the current scene image and the detection frames in the previously collected scene image that overlap are taken as a group of detection frames;

[0237] Each group of detection frames is screened based on the overlap threshold, and the detection frames in the previously collected scene image whose overlap comparison results with the detection frames of the current scene image are less than the overlap threshold are eliminated;

[0238] Compare the geometric centers of the remaining detection frames in the previous scene image in each group of detection frames with the geometric centers of the detection frames in the current scene image, and give a detection frame in the previous scene image that is closest to the geometric center of the detection frame in the current scene image;

[0239] The geometric center of the detection frame in the current scene image within each group of detection frames is connected with the geometric center of a detection frame in the given previous scene image to obtain the trajectory information of the target corresponding to the detection frame in the current scene image.

[0240] Specifically, the overlap threshold is 50%, that is, when the detection frame overlap ratio (IOU) of the target in the previous and next frames of the video (the previous and next frames correspond to the previous captured scene picture and the current scene picture respectively) is higher than 50%, then the two targets in the previous and next frames are judged to be the same object. The geometric midpoints of the detection frames of the same object in adjacent frames are connected to obtain the current trajectory of the moving target.

[0241] In addition, when there is no overlapping area between the detection frames of the previous and subsequent frames, for example, when the target moves too fast or the video drops frames, a target association algorithm based on the distance between the detection frame centers will be applied to ensure continuous tracking of the target. Specifically, before connecting the geometric center of the detection frame in the current scene image within each group of detection frames with the geometric center of a detection frame in the given previous scene image, it also includes:

[0242] Obtain the detection frames in the current scene image where the overlap comparison result shows no overlap;

[0243] Calculate the distance between the geometric center of the non-overlapping detection frame and the geometric center of all detection frames in the previously collected scene image, and give the detection frame in the previously collected scene image with the smallest distance to the non-overlapping detection frame;

[0244] The detection frames without overlap and the detection frames with the smallest distance are divided into one group.

[0245] The target detection model for detecting targets in the scene pictures is pre-built. The target detection model can be pre-built through the following steps:

[0246] Obtain several historical site pictures;

[0247] Mark several historical scene images with detection frames, location information, and target types;

[0248] Based on the marked historical scene pictures, the YOLOv5s model is trained to obtain a trained YOLOv5s model.

[0249] The trained YOLOv5s model is the pre-built target detection model. The target detection model is constructed as follows: Features are extracted from the scene image through the backbone network; the extracted features are processed by the detection head to obtain the predicted target detection box and target type probability; the size and aspect ratio of the predicted detection box are generated through anchor points to obtain the predicted coordinates of the detection box; the YOLOv5 model is trained based on the labeled scene image and the predicted target detection box, target type, and coordinates to obtain the trained YOLOv5 model; when the trained YOLOv5 model makes predictions, after predicting multiple detection boxes for the same target, the YOLOv5 model uses non-maximum suppression to select the target detection box with the highest confidence; the coordinates of the detection box are converted into position information through post-processing, and the target type with the highest predicted probability is assigned to the corresponding detection box.

[0250] Specifically, the construction of the YOLOv5s model may include:

[0251] Backbone network: YOLOv5 uses a deep neural network as its backbone to extract features from the input scene images. The backbone network is based on a convolutional neural network (CNN) architecture, such as CSPDarknet53 or MobileNetV3, which helps capture hierarchical features from the scene images.

[0252] Detection head: The detection head is responsible for predicting the bounding box (detection box) of the detected object and the probability of the object type. YOLOv5 uses a lightweight head composed of several convolutional layers to generate the final detection output results;

[0253] Anchors: YOLOv5 uses anchor boxes to predict bounding boxes of different sizes and aspect ratios. These anchor boxes are predefined boxes that serve as references for the network to predict the final bounding box coordinates.

[0254] Loss function: The loss function is used to train the YOLOv5 model by calculating the difference between the predicted bounding box and the true bounding box. The YOLOv5 model uses a combination of different loss functions, including positioning loss (for bounding box coordinates) and classification loss (for target type probability);

[0255] Non-maximum suppression (NMS): After the YOLOv5 model predicts multiple bounding boxes for the same object, NMS is applied to remove duplicate detections and retain only the most confident detection results;

[0256] Post-processing: The final step involves converting the predicted bounding box coordinates to their respective locations on the scene image and assigning the correct object category to each detected bounding box based on the object type probability.

[0257] After identifying the transmission channel area and the dynamic target in the scene picture, the present invention can determine the interference result between the dynamic target and the transmission channel area based on the identification results of the transmission channel area and the dynamic target, specifically including:

[0258] Overlap and compare the trajectory information of the dynamic target with the transmission channel area and give the overlapping results;

[0259] Based on the overlapping results, dynamic targets whose trajectory information overlaps and interferes with the transmission channel area are screened out.

[0260] After obtaining the interference result, the present invention can determine the alarm information based on the interference result, specifically including:

[0261] According to the location information and transmission channel area of ​​all targets in the current scene picture, and combined with the trajectory information of dynamic targets overlapping with the transmission channel area, the dynamic targets located in the transmission channel area are given;

[0262] Based on the historical alarm information within the predetermined time, the historical dynamic target of the alarm is given;

[0263] Match the dynamic targets in the transmission channel area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched;

[0264] Generates alarm information based on unsuccessful matching of dynamic targets.

[0265] Among them, the scheduled time is pre-set according to the actual transmission line monitoring scenario. Preferably, the scheduled time is 30 seconds. In the actual application scenario, if the same target triggers the alarm multiple times within 30 seconds, only one of the alarms will be reported (that is, the historical dynamic target trajectory information that has been alarmed overlaps with the transmission channel area multiple times, and only one alarm will be issued); if the alarm position does not change within 30 seconds, only one of the alarms will be reported (that is, the historical dynamic target that has been alarmed is always located in the transmission channel area, and only one alarm will be issued).

[0266] In addition, in the process of dynamic target recognition, it is necessary to ignore targets that move too fast and whose movement trajectories briefly pass through the transmission channel, so as to avoid misjudging passing vehicles when the transmission line is close to the road. That is, the present invention can perform high-speed identification of dynamic targets, wherein the high-speed identification of dynamic targets is divided into two cases:

[0267] Too fast: When the distance between the geometric center points of the detection box of the target in adjacent frames changes by more than 200 pixels, it is considered to be too fast and not in operation;

[0268] The moving time in the transmission channel area is too short: when the time interval between the dynamic target entering the transmission channel area and leaving the transmission channel area is shorter than the preset alarm time, it is regarded as the target briefly passing through the area and no alarm is issued.

[0269] Since targets in the transmission channel area can only become threats when they are in motion, as shown in Figure 6, in actual application scenarios, alarm information is generated based on unsuccessful matching of dynamic targets. That is, alarms are generated after high-speed identification of dangerous targets. Specifically, the following may be involved:

[0270] The geometric center of the detection frame at different times is obtained based on the trajectory information of the dynamic target that was not successfully matched; the geometric center of the detection frame at different times is the geometric center of the detection frame in the scene image at different video frames;

[0271] Obtain pixel change values ​​of unmatched dynamic targets based on the geometric center of the detection frame at different times;

[0272] Compare the obtained pixel change value with the pixel change threshold to obtain the dynamic target within the pixel change threshold range; wherein the pixel change threshold range is 10-200 pixels;

[0273] According to the dynamic target within the pixel change threshold range, a historical scene picture containing the dynamic target is obtained;

[0274] Based on the historical and current on-site pictures of the dynamic target within the pixel change threshold range, the historical or current on-site picture with the minimum distance between the dynamic target and the boundary of the transmission channel area is obtained;

[0275] Based on the historical scene picture with the smallest distance or the current scene picture, the corresponding target collection start time is given;

[0276] According to the target acquisition start time and the acquisition time corresponding to the current scene picture, the interference time of the dynamic target entering the transmission channel area is given;

[0277] Based on the intervention time and the preset alarm time, a dynamic target with an intervention time not less than the preset alarm time is given;

[0278] Generate corresponding alarm information based on dynamic targets whose interference time is not less than the preset alarm time.

[0279] In general, the rule for generating alarm information of the present invention is: when the target continues to move in the transmission channel area for more than a certain time (preset alarm time), the target is judged to be a dangerous target. When the target is stationary, regardless of its position, the target is judged to be a non-threatening target. When the target moves outside the transmission channel area, the target is also judged to be a non-threatening target. Among them, the alarm for dangerous targets is specifically: starting from the time the target enters the transmission channel area, that is, the target detection frame overlaps with the transmission channel area. If the target continues to move and its detection frame continues to overlap with the transmission channel area for more than the preset alarm time, the dangerous target is screened out as a dangerous moving target, and an alarm information is generated for it. The early warning information includes the target category, time and corresponding screenshot that triggered the alarm, and may also include the location of the target.

[0280] In addition, the present invention automatically regards the jittering behavior of the target within a small range in a short period of time as static, in order to adapt to the phenomenon of camera jitter caused by strong wind weather conditions or roadbed shaking conditions. Specifically, when the target detection frame continues to move within a range of 10 pixels within the preset alarm time, and the overlapping area of ​​the front and rear frame detection frames (the detection frame of the target in the current scene picture and the detection frame in the scene picture collected last time) continues to exceed 95%, the target is determined to be a stationary target. Among them, the preset alarm time is less than the predetermined time. Specifically, the target detection results of the current scene picture and the scene picture collected last time are combined to give the current trajectory information of the dynamic target, and also include:

[0281] Based on the trajectory information of all targets in the current scene image, the corresponding overlapping comparison results of all target trajectory information within the preset alarm time are given, as well as the distance between the geometric center of the detection frame in the current scene image and the geometric center of a corresponding detection frame in the previously collected scene image within the preset alarm time;

[0282] Based on the dynamic overlap threshold and dynamic distance threshold, the overlap comparison results and the distance to the geometric center corresponding to all target trajectory information are screened to provide the trajectory information of the dynamic target, including:

[0283] Eliminate the target trajectory information whose overlap comparison result is greater than the dynamic overlap threshold and whose geometric center distance is less than the dynamic distance threshold within the preset alarm time, and use the trajectory information of the remaining targets as the trajectory information of the dynamic targets.

[0284] As shown in FIG7 , the present invention further provides a computer vision-based power transmission line safety monitoring and early warning system, which adopts the above-mentioned computer vision-based power transmission line safety monitoring and early warning method. The system includes:

[0285] A data acquisition module, which is used to collect on-site pictures of the transmission line;

[0286] Image recognition module, which is used to identify transmission channel areas and dynamic targets in on-site images;

[0287] An interference judgment module, which is used to determine the interference result between the dynamic target and the transmission channel area based on the identification results of the transmission channel area and the dynamic target;

[0288] The early warning determination module is used to determine the alarm information based on the interference result.

[0289] As shown in FIG8 , the present invention also provides a computer vision-based power transmission line safety monitoring and early warning device, comprising a power transmission channel identification unit 1 , a dynamic target identification unit 2 , and an early warning information generation unit 3 ;

[0290] The transmission channel identification unit is used to collect on-site pictures of the transmission lines;

[0291] The dynamic target recognition unit is used to identify the transmission channel area and dynamic targets in the on-site pictures;

[0292] The early warning information generating unit is used to determine the interference result between the dynamic target and the power transmission channel area based on the identification result of the power transmission channel area and the dynamic target; and is also used to determine the alarm information based on the interference result.

[0293] The dynamic target recognition unit 2 and the warning information generation unit 3 operate in tandem, while the transmission channel recognition unit 1 operates in parallel. The transmission channel recognition unit 1 and the dynamic target recognition unit 2 share the same data source, which can be a directly connected hardware camera or a remote network video source. The transmission channel recognition unit 1 includes an edge detection subunit 4, an edge screening subunit 5, a line segment recognition subunit 6, a transmission line fitting subunit 7, a transmission line boundary recognition subunit 8, and a transmission channel generation subunit 9, with the subunits operating in tandem. The dynamic target recognition unit 2 includes a video stack subunit 10, a target recognition model subunit 11, a target tracking subunit 12, and a dynamic trajectory analysis unit 13. The video stack subunit 10 operates in parallel with the target recognition model subunit 11, the target tracking subunit 12, and the dynamic trajectory analysis unit 13. The warning information generation unit 3 includes a warning information comparison subunit 14, a historical warning cache subunit 15, and a warning information return subunit 16.

[0294] The edge detection subunit 4 converts the input on-site image into a grayscale image and performs edge detection on the grayscale image using the Sobel XY operator. The edge screening subunit 5 filters the results output by the edge detection subunit 4 using grayscale threshold filtering and converts the filtered edge information into binary grayscale. The line segment identification subunit 6 performs a Hough transform on the binary grayscale image output by the edge screening subunit 5 to filter out edges belonging to straight line segments. The transmission line fitting subunit 7 performs line fitting and splicing on the straight line segment edge information and identifies and filters transmission lines based on the orientation, direction, and length of the fitted lines. The transmission line boundary identification subunit 8 determines the outer envelope of the transmission line on the tower to which the camera is attached based on the transmission line information output by the transmission line fitting subunit 7. The transmission channel generation subunit 9 maps the outer envelope of the transmission line to the ground based on the outer envelope information output by the transmission line boundary identification subunit 8 to form a transmission channel area. This embodiment uses a low-power transmission channel identification unit and a dynamic target identification unit to run at the edge, using limited resources to directly analyze and process the data on-site, and only returns the processing results through the 4G / 5G network. Not only does it greatly save the traffic and bandwidth required to transmit the original data, but also due to the high efficiency of the chip's low-power algorithm, there is no need for a high-power power supply on-site to support the operation of the equipment. Existing dynamic anomaly detection requires uninterrupted real-time anomaly recognition, and image processing equipment needs to perform real-time analysis of the streaming data collected on-site, such as video, LiDAR point cloud data, etc. This requires either a real-time data processing unit on the transmission channel or the on-site streaming data can be continuously and real-timely transmitted back to the back-end processing unit.

[0295] The stack length of the video stack subunit 10 is fixed. When new image data (on-site image) is pushed into the stack, if the stack is full, the data at the front of the stack is pushed out and discarded. When a data output instruction is received, the stack pushes out the last pushed data. The target recognition model subunit 11 obtains the latest data from the video stack subunit 10, and sends the data to the target detection model to obtain the target category and location information. The target tracking subunit 12 caches the historical target category and location information output by the target recognition model subunit 11, and tracks the target trajectory of the latest target category monitoring results in combination with the historical information to determine dynamic and static targets. The dynamic trajectory analysis unit 13 determines whether the dynamic target activity range overlaps with the transmission channel area generated by the transmission channel generation subunit 9 based on the dynamic target trajectory information output by the target tracking subunit 12. Finally, dynamic targets that are active in the transmission channel are screened out. The present invention designs a soft connection between the video stack subunit 10 and the target recognition model subunit, so that the target recognition model subunit 11 can be directly connected to the camera for online operation, and can also be connected to the back end for operation through a network transmission channel, which greatly improves the adaptability of the algorithm to the on-site network environment of the power transmission channel.

[0296] The warning information comparison subunit 14 uses the historical warning information stored in the historical warning cache subunit 15 to determine whether the target currently triggering the warning has repeatedly triggered an alarm within a certain period of time. The historical warning cache subunit 15 pushes the latest warning information output by the warning information comparison subunit 14 into the cache for subsequent warning information comparison. The warning information return subunit 16 transmits the latest warning information from the historical warning cache subunit 15 back to the cloud data platform via the 4G / LTE mobile network. The warning information includes the target category, time, and screenshot of the triggering alarm.

[0297] This invention processes and analyzes real-time images of transmission channel sites captured by surveillance cameras into a computing unit with appropriate visual processing capabilities. This system can be deployed and installed in on-site hardware for online operation, or installed in back-end computing equipment to obtain real-time images over the network. The transmission channel identification unit, dynamic target identification unit, and warning information generation unit proposed in this invention perform real-time analysis and processing on the input video. The transmission channel identification unit captures the full-color image captured by the camera. After black-and-white processing, the full-color image is convolved with a Sobel operator to obtain edge information corresponding to the image. The edge screening subunit then analyzes this edge information to determine the distance, clarity, and contrast of the target edge. Edge information with a long distance, poor clarity, and poor contrast with the surrounding area is automatically filtered out. The remaining edge information is binarized and subjected to straight line segment edge screening. This screening process uses the Hough transform algorithm to filter out edges that do not have straight line characteristics, thereby narrowing the target range for the automatic transmission line search. Next, the transmission line fitting subunit will perform orientation discrimination, spacing discrimination, and line fitting discrimination on the straight edge information to identify the location of the transmission line within these straight edges. Based on the location of the transmission line, the transmission line boundary identification subunit generates the envelope of the transmission channel and projects it onto the ground to obtain the transmission channel area. This embodiment deploys surveillance cameras on a large number of towers, obtains camera access rights, and remotely monitors and captures on-site images as training data. Existing dynamic recognition processes do not require a large number of on-site camera resources for data collection, making it difficult to obtain a good dataset for model training.

[0298] The dynamic target recognition unit is responsible for locating and tracking moving targets in real-time on-site images and comparing their trajectory with the aforementioned transmission channel area. If the target's trajectory overlaps with the transmission channel, a warning message is generated for the target. The dynamic target recognition unit is soft-connected to the video input interface and operates in parallel. If field data is interrupted for various reasons, the transmission channel recognition unit will enter hot standby mode and immediately switch to monitoring mode when video is restored, without manual intervention.

[0299] While storing historical alarm information, the warning information generation unit will compare the historical alarm information with the warning alarm time, location, target category, and motion trajectory of the input warning signal after receiving the warning information transmitted by the dynamic target recognition unit to determine whether it is a repeated alarm information. After confirming that the alarm information is correct, the warning information generation unit will generate new alarm information, push it into the information cache, and transmit it back to the background data center through the network.

[0300] Because transmission lines naturally sag due to gravity, the line segment recognition subunit, after performing a Hough transform, filters out excessively long or short line segments by determining their length. The line fitting subunit verifies the connections between line segments by determining the directions and endpoint distances of different line segments, and then determines whether the connected line segments constitute a transmission line. If the line fitting subunit fails to identify sufficient transmission lines in the input image, it outputs an alarm signal to the lower-level units. Upon receiving an alarm signal indicating that the line fitting subunit has failed to identify a transmission line, the line boundary recognition subunit automatically generates a predefined transmission channel region. When outputting the predefined transmission channel region, the line boundary recognition subunit generates an alarm message and transmits it to the backend. Backend transmission line monitoring personnel can visually determine the validity of the predefined transmission channel region based on the imagery fed back. If the backend transmission line monitoring personnel determine that the transmission channel region is unreasonable, they can manually adjust the region parameters and transmit them to the operating equipment. The target tracking subunit automatically treats any short-term, small-scale jittering of a target as static, to accommodate camera jitter caused by windy conditions or shaky roadbeds. When identifying dynamic targets, the target tracking subunit ignores targets moving too quickly or whose trajectories briefly pass through the transmission path, to avoid misidentifying passing vehicles when transmission lines are close to roads.

[0301] The present invention also provides a method for using the visual transmission line detection device, comprising the following steps:

[0302] S1: The surveillance camera and the computing processing unit can be installed on the tower together. The surveillance camera and the computing unit are connected via a USB interface. At the same time, the computing processing unit can also be remotely connected to the camera via a 4G LTE or 5G network. The specific connection method can be selected according to the network conditions of the installation site. In areas with good network conditions, remote connection can be selected to save installation and maintenance costs. In areas with poor network conditions, direct connection with the camera can be selected to save network bandwidth.

[0303] S2: When the equipment is connected, the transmission line monitoring personnel input the camera port address or network streaming address into the transmission channel identification unit to obtain the transmission channel parameter information. If the transmission channel unit issues an alarm message, it means that the preset transmission channel information is output. When the background monitoring personnel receive the alarm message, they can check whether the preset transmission channel meets the monitoring conditions and decide whether to manually intervene to change the transmission channel information.

[0304] S3: After confirming the transmission channel information, the transmission line monitoring personnel enter the camera port address or network streaming address into the dynamic target identification unit and the warning information generation unit. Dynamic moving targets within the transmission channel that meet the dynamic target screening criteria are marked with a warning time and category, and a screenshot is sent back to the backend monitoring platform.

[0305] It can be seen that the above-mentioned embodiments of the present invention achieve the following technical effects: the transmission channel identification unit realizes the automatic identification of the transmission channel in the transmission line, greatly improves the algorithm's adaptability to different conditions and environments, and greatly saves the line operation and maintenance personnel the workload of manually setting the transmission channel monitoring parameters. The dynamic target identification unit realizes the screening of static targets and the determination of non-threatening dynamic targets, which greatly reduces the false alarm rate of dynamic threat targets in various operating environments, thereby reducing the amount of alarm information handled by the background operation and maintenance personnel and the difficulty of handling it. The early warning information generation unit not only reduces duplicate alarms by comparing historical alarm information, but also records the full range of calibration of the alarm triggering target, including trajectory, target category, triggering time, and screenshots of the alarm triggering, so that the background operation and maintenance personnel can accurately and timely grasp the on-site situation and threat level, so as to make the best and fastest disposal.

[0306] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A power transmission line safety monitoring and early warning method based on computer vision, It is characterized in that include: Collect on-site pictures of transmission lines; Identify transmission channel areas and dynamic targets in on-site pictures; Based on the recognition results of the transmission channel area and the dynamic target, the interference result between the dynamic target and the transmission channel area is determined, specifically including: overlapping and comparing the trajectory information of the dynamic target with the transmission channel area to give an overlapping result; based on the overlapping result, screening out the dynamic target whose trajectory information overlaps with the transmission channel area; An alarm message is determined based on the interference result.

2. The power transmission line safety monitoring and early warning method according to claim 1, It is characterized in that Identify the transmission channel area from the on-site pictures, including: Perform edge processing on the scene picture to obtain edge information belonging to the straight line segment; The edge information of the straight line segment is filtered and processed to give the transmission line; The given transmission lines are mapped to the ground to form the transmission channel area.

3. The power transmission line safety monitoring and early warning method according to claim 2, It is characterized in that Perform edge processing on the scene image to obtain edge information belonging to the straight line segment, including: Convert the scene picture into a grayscale picture and perform edge detection to obtain edge information; Based on the grayscale threshold, the edge information is divided to obtain the divided edge information; Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale image; Perform Hough transform on the binary grayscale image to give the polar coordinates of the edge information; The polar coordinates of the edge information are screened to provide multiple sets of edge information that meet predetermined conditions; Treat the edge information of the same group as the edge information of a straight line segment; The predetermined condition is that the number of edge information in the same group is not less than a predetermined number, and the polar coordinates of all edge information in the same group form a straight line segment in the polar coordinate system.

4. The power transmission line safety monitoring and early warning method according to claim 2 or 3, It is characterized in that The edge information of the straight line segment is filtered and processed to give the transmission line, including: Classify the straight line segments with the same slope into the same category; Connect the ends of the two straight line segments with overlapping points in each category to form a Straight line segment; Based on each type of connected straight line segments, the lengths of all straight line segments are obtained; Compare the length of the straight line segments in each category with the length threshold, and give the straight line segments that are greater than the length threshold; The straight line segments in the same category that are greater than the length threshold are regarded as transmission lines in the same location area; Integrate the transmission lines of all location areas to give the transmission lines.

5. The power transmission line safety monitoring and early warning method according to claim 1, It is characterized in that Identify dynamic targets in live images, including: Perform target detection on the scene pictures based on the pre-built target detection model, and provide the detection frames, target types and location information of all targets in the scene pictures; Obtaining the target detection result of the scene picture collected last time relative to the current scene picture; Combine the target detection results of the current scene picture and the scene picture collected last time to give the current trajectory information of the dynamic target; The historical trajectory information of the dynamic target is obtained, and combined with the current trajectory information of the dynamic target, the trajectory information of the dynamic target is obtained.

6. The power transmission line safety monitoring and early warning method according to claim 5, Features Combine the target detection results of the current scene picture and the scene picture collected last time to give the current trajectory information of the dynamic target, including: Overlapping and comparing all detection frames in the current scene image with all detection frames in the scene image collected last time, and providing overlapping comparison results; Based on the overlapping comparison result, each detection frame in the current scene image and the detection frames in the scene image collected last time that overlap are taken as a group of detection frames; Each group of detection frames is screened based on the overlap threshold, and the detection frames in the previous collected scene image whose overlap comparison results with the detection frames of the current scene image are less than the overlap threshold are eliminated; Compare the geometric center of the remaining detection frames in the previous scene picture collected in each group of detection frames with the geometric center of the detection frame in the current scene picture, and give a detection frame in the previous scene picture collected that is closest to the geometric center of the detection frame in the current scene picture; Compare the geometric center of the detection frame in the current scene image within each set of detection frames with the given previous The geometric centers of a detection frame in a scene image collected at one time are connected to obtain the trajectory information of the target corresponding to the detection frame in the current scene image.

7. The power transmission line safety monitoring and early warning method according to claim 6, It is characterized in that Combine the target detection results of the current scene picture and the scene picture collected last time to give the current trajectory information of the dynamic target, including: According to the trajectory information of all targets in the current scene picture, the overlapping comparison results of all target trajectory information within the preset alarm time are given, as well as the distance between the geometric center of the detection frame in the current scene picture and the geometric center of a corresponding detection frame in the scene picture collected last time within the preset alarm time; Based on the dynamic overlap threshold and dynamic distance threshold, the overlap comparison results and the distance of the geometric center corresponding to all target trajectory information are screened to provide the trajectory information of the dynamic target, including: The target trajectory information whose overlap comparison result is greater than the dynamic overlap threshold and whose geometric center distance is less than the dynamic distance threshold within the preset alarm time is eliminated, and the trajectory information of the remaining targets is used as the trajectory information of the dynamic targets.

8. The power transmission line safety monitoring and early warning method according to any one of claims 5 to 7, It is characterized in that Determine alarm information based on interference results, including: According to the location information and transmission channel area of ​​all targets in the current scene picture, and combined with the trajectory information of the dynamic targets overlapping with the transmission channel area, the dynamic targets located in the transmission channel area are given; Based on the historical alarm information within the predetermined time, the historical dynamic target of the alarm is given; Match the dynamic targets in the transmission channel area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched; Generates alarm messages based on unsuccessful matching of dynamic targets.

9. The power transmission line safety monitoring and early warning method according to claim 8, It is characterized in that Generates alarm information based on unsuccessful matching of dynamic targets, including: According to the trajectory information of the dynamic target that was not successfully matched, the geometric center of the detection box at different times is obtained; The unmatched dynamic targets are obtained according to the geometric center of the detection box at different times. Pixel change value; Compare the obtained pixel change value with the pixel change threshold to obtain the dynamic target within the pixel change threshold range; According to the dynamic target within the pixel point change threshold range, a historical scene picture containing the dynamic target is obtained; According to the historical on-site pictures and the current on-site pictures of the dynamic target within the pixel point change threshold range, the historical on-site picture or the current on-site picture with the minimum distance between the dynamic target and the boundary of the transmission channel area is given; Based on the historical scene picture with the smallest distance or the current scene picture, the corresponding target acquisition start time is given; According to the target acquisition start time and the acquisition time corresponding to the current on-site picture, the interference time of the dynamic target entering the power transmission channel area is given; Based on the interference time and preset alarm time, a dynamic target with an interference time not less than the preset alarm time is given; According to the dynamic target whose interference time is not less than the preset alarm time, the corresponding alarm information is generated.

10. A power transmission line safety monitoring and early warning system based on computer vision, using the power transmission line safety monitoring and early warning method based on computer vision according to any one of claims 1 to 9, It is characterized in that The system comprises: A data acquisition module, which is used to collect on-site pictures of the transmission line; Image recognition module, which is used to identify the transmission channel area and dynamic targets in the on-site images; An interference judgment module, which is used to determine the interference result between the dynamic target and the power transmission channel area based on the recognition results of the power transmission channel area and the dynamic target; The early warning determination module is used to determine the alarm information based on the interference result.

Images