A method and system for safety monitoring and early warning of transmission lines based on computer vision

Abstract

The present invention discloses a method and system for safety monitoring and early warning of transmission lines based on computer vision. The method includes: collecting on-site pictures of transmission lines; identifying the transmission channel area and dynamic targets in the on-site pictures; determining the interference result between the dynamic targets and the transmission channel area based on the identification results of the transmission channel area and dynamic targets, specifically including: overlapping and comparing the trajectory information of the dynamic targets with the transmission channel area to give an overlapping result; based on the overlapping result, screening out the dynamic targets whose trajectory information overlaps with the transmission channel area; and determining alarm information based on the interference result. The present invention can automatically define the transmission channel area near transmission towers, greatly reducing the workload of manual definition of the transmission channel by line monitoring personnel and improving the applicability of the algorithm for large-scale applications. It can also improve the accuracy of safety early warning for transmission lines.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision tracking, and particularly relates to a method and system for safety monitoring and early warning of power transmission lines based on computer vision. Background Art

[0002] With the acceleration of the urbanization process, many construction activities have been carried out near power transmission lines. The operations of large construction equipment such as tower cranes, concrete pouring trucks, excavators, and dump trucks pose a threat to the safe operation of power facilities. Therefore, the power industry urgently needs tools that can identify and respond to dynamic abnormal targets online. Dynamic abnormal targets have strong mobility, and their movement trajectories are difficult to predict, which can cause unexpected damage to power transmission lines and towers. Therefore, abnormal detection and identification need to be fast and accurate in order to take timely actions when necessary.

[0003] Currently, the abnormal detection of power transmission channels usually focuses on static target detection, which can be roughly classified into manual inspection or camera inspection according to the detection method. Manual inspection is usually carried out by professional personnel in power operation and maintenance vehicles or helicopters, moving along the power transmission line for visual monitoring, and judging whether there are plants or buildings covering the power transmission channel within the channel according to personal experience. Camera inspection is divided into fixed camera inspection and mobile camera inspection. Fixed camera inspection means that the monitoring camera installed on a specific pole takes pictures of the power transmission channel at a certain time interval, and the captured pictures are transmitted back to the data storage center, and the line operation and maintenance personnel analyze the pictures to judge the safety status of the channel. Mobile cameras usually refer to unmanned aerial vehicles equipped with monitoring cameras cruising along the power transmission line and transmitting the cruising videos back to the operators, who judge the safety status of the line. For example, CN106932688A discloses a power transmission line detector and a power transmission line detection system based on an unmanned aerial vehicle. The power transmission line detector includes: a housing, a camera disposed on the outer wall of the housing, and an image pre-processor, a straight line detector, and a power transmission line identifier disposed inside the housing; wherein, the camera is used to collect image information of the target area; the image pre-processor is used to process the image information; the straight line detector is used to mark the detected straight lines on the processed image information; the power transmission line identifier is used to mark the identified power transmission lines on the image information after marking the straight lines, and output the image information after marking the power transmission lines; the power transmission line detector further includes a communication module for wirelessly transmitting the received image information after marking the power transmission lines to an associated terminal. It can detect power transmission lines more accurately from a complex background environment, bringing convenience to relevant power workers and improving the effectiveness and reliability of power inspection. However, neither manual inspection nor camera inspection can conduct real-time and continuous online monitoring and safety discrimination of power transmission channels, resulting in the inability of the prior art to apply to the monitoring and early warning of on-site dynamic targets.

[0004] The main challenges of dynamic anomaly detection are as follows. Autonomous monitoring of large-scale power grid transmission channels requires algorithms that can adapt to the diversity of transmission scenarios, the complexity of on-site weather, and automatically define transmission channels. Algorithms are also needed to support the tracking of dynamic targets, discriminate their dynamic trajectories and operating states, in order to avoid misjudgment and missed judgment.

[0005] Therefore, how to provide a transmission line monitoring method that is applicable to different transmission scenarios for dynamic target tracking and has high recognition and early warning accuracy is an urgent problem to be solved in this field. Summary of the Invention

[0006] In view of the defects existing in the above-mentioned prior art, the present invention provides a power transmission line safety monitoring and early warning method and system based on computer vision, which can improve the accuracy of power transmission line safety early warning.

[0007] In a first aspect, the present invention provides a power transmission line safety monitoring and early warning method based on computer vision, including:

[0008] Collect on-site pictures of the power transmission line;

[0009] Identify the transmission channel area and dynamic targets in the on-site pictures;

[0010] Based on the recognition results of the transmission channel area and dynamic targets, determine the interference result between the dynamic targets and the transmission channel area;

[0011] Determine the alarm information based on the interference result.

[0012] Further, the identification of the transmission channel area in the on-site pictures includes:

[0013] Perform edge processing on the on-site pictures to obtain edge information belonging to straight line segments;

[0014] Perform screening processing on the edge information of the straight line segments to give the power transmission line;

[0015] Perform ground mapping on the given power transmission line to form a transmission channel area.

[0016] Further, performing edge processing on the on-site pictures to obtain edge information belonging to straight line segments includes:

[0017] Convert the on-site pictures into grayscale pictures and perform edge detection to obtain edge information;

[0018] Based on the grayscale threshold, divide the edge information to obtain the divided edge information;

[0019] Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale picture;

[0020] Perform Hough transform on a binary grayscale image to give the polar coordinates of the edge information;

[0021] Filter the polar coordinates of the edge information to give multiple groups of edge information that meet the predetermined conditions;

[0022] Take the edge information of the same group as the edge information of a straight line segment;

[0023] Among them, the predetermined conditions are that the number of edge information in the same group is not less than the predetermined number, and the polar coordinates of all edge information in the same group form a straight line segment in the polar coordinate system.

[0024] Furthermore, perform Hough transform on the binary grayscale image to give the polar coordinates of the edge information, including:

[0025]

[0026] In the formula, (x, y) are the coordinates of the straight line segment in the rectangular coordinate system of the binary grayscale image, a is the slope of the straight line segment, b is the intercept of the straight line segment, (r, θ) are the polar coordinates of the straight line segment in the polar coordinate system after performing Hough transform, r is the distance from the origin to the straight line segment, and θ is the angle between r and the horizontal axis in the rectangular coordinate system.

[0027] The present invention can screen out the straight line segments existing in the binary grayscale image by performing Hough transform on the binary grayscale image to form polar coordinates, and splice the screened straight line segments in the rectangular coordinate system.

[0028] Furthermore, perform screening processing on the edge information of the straight line segment to give the transmission line, including:

[0029] Divide the straight line segments with the same slope in the straight line segments into the same category;

[0030] Connect the heads and tails of two straight line segments with overlapping points in each category to form a straight line segment;

[0031] Based on each category of straight line segments after connection, obtain the lengths of all straight line segments;

[0032] Compare the length of the straight line segment in each category with the length threshold to give the straight line segments longer than the length threshold;

[0033] Take the straight line segments longer than the length threshold in the same category as the transmission lines in the same position area;

[0034] Integrate the transmission lines in all position areas to give the transmission line.

[0035] Furthermore, dividing the straight line segments with the same slope in the straight line segments into the same category includes:

[0036] Subtract the slopes of all line segments pairwise to obtain the slope difference results;

[0037] Based on the slope difference threshold, screen all the slope difference results to obtain the slope difference results within the slope difference threshold, specifically:

[0038]

[0039] In the formula, k 1 and k 2 are the slopes of two line segments, α is the slope difference threshold, and α is 0.2;

[0040] According to the screened slope difference results, divide the two line segments corresponding to each slope difference result into the same category.

[0041] Further, connect the heads and tails of the two line segments with overlapping points in each category to form a line segment, including:

[0042] Obtain the endpoint coordinates of all line segments in each category;

[0043] Solve the endpoint distances between the endpoint coordinates of any two line segments in each category;

[0044] Based on the distance threshold, screen all the solved endpoint distances to obtain the endpoint distances within the distance threshold, specifically:

[0045]

[0046] In the formula, x 1 , y 1 are the endpoint coordinates of one of the line segments respectively, x 2 , y 2 are the endpoint coordinates of the other line segment respectively, β is the distance threshold, and β is 20 pixels;

[0047] Divide the two line segments corresponding to all the given endpoint distances into two line segments with overlapping points, and connect the corresponding two endpoints.

[0048] Further, perform ground mapping on the given transmission line to form a transmission channel area, including:

[0049] Based on the Hough transform, identify the given transmission line to obtain the endpoints of the two outermost transmission lines in each position area;

[0050] Connect the endpoints of the two outermost transmission lines in all position areas to obtain the outer envelope sub-area corresponding to the position area;

[0051] Integrate all the outer envelope sub-areas to form the outer envelope area of the transmission line;

[0052] Based on the endpoint coordinates of the connection, obtain the boundary coordinates of the outer envelope area;

[0053] Symmetrically map the boundary coordinates of the outer envelope area along the horizontal axis of the on-site picture to form the transmission channel area.

[0054] Further, after obtaining the endpoints of the two outer transmission lines in each position area, it includes:

[0055] Obtain the endpoint coordinates of the two outer transmission lines within all position areas;

[0056] For the two outer transmission lines in each position area, determine the upper endpoint and the lower endpoint of each transmission line;

[0057] Perform coordinate difference processing on the vertical coordinates of the two lower endpoints and the vertical coordinates of the two upper endpoints in each position area respectively;

[0058] Compare all the coordinate difference processing results with the endpoint difference threshold, and give the coordinate difference processing results that exceed the endpoint difference threshold;

[0059] Based on the given coordinate difference processing results, obtain the endpoints of the two transmission lines that match them;

[0060] Based on the endpoints of the two transmission lines that match each other, calculate the lengths of the two transmission lines corresponding to the endpoints, and increase a line segment with the same slope as itself and a certain length along the matching endpoints of the transmission line with the smaller length; wherein, the increased line segment length is equal to the length difference between the two transmission lines.

[0061] Further, the coordinate difference processing is the absolute value of the difference between the vertical coordinates of the two endpoints, and the endpoint difference threshold is 200 pixels.

[0062] Further, perform the recognition of dynamic targets on the on-site picture, including:

[0063] Perform target detection on the on-site picture based on the pre-constructed target detection model, and give the detection frames, target types and position information of all targets in the on-site picture;

[0064] Obtain the target detection results of the on-site picture collected immediately before the current on-site picture;

[0065] Combine the target detection results of the current on-site picture and the on-site picture collected immediately before, and give the current trajectory information of the dynamic target;

[0066] Obtain the historical trajectory information of the dynamic target, and combine it with the current trajectory information of the dynamic target to obtain the trajectory information of the dynamic target.

[0067] Furthermore, combining the current on-site image and the object detection results of the on-site image collected last time, the current trajectory information of the dynamic object is given, including:

[0068] Overlap and compare all the detection boxes in the current on-site image with all the detection boxes in the on-site image collected last time, and give the overlap comparison result;

[0069] Based on the overlap comparison result, each detection box in the current on-site image and the detection box with overlap in the on-site image collected last time are used as a group of detection boxes;

[0070] Based on the overlap threshold, each group of detection boxes is screened, and the detection boxes in the on-site image collected last time with an overlap comparison result less than the overlap threshold with the detection boxes in the current on-site image are removed;

[0071] Compare the geometric centers of the detection boxes in the on-site image collected last time remaining in each group of detection boxes with the geometric centers of the detection boxes in the current on-site image, and give one detection box in the on-site image collected last time that is closest to the geometric center of the detection box in the current on-site image;

[0072] Connect the geometric center of the detection box in the current on-site image in each group of detection boxes with the geometric center of one detection box in the on-site image collected last time given, and obtain the trajectory information of the object corresponding to the detection box in the current on-site image.

[0073] Furthermore, overlap and compare all the detection boxes in the current on-site image with all the detection boxes in the on-site image collected last time, and give the overlap comparison result, specifically:

[0074]

[0075] In the formula, is the overlap comparison result, is the detection box in the on-site image collected last time, is the detection box in the current on-site image.

[0076] Furthermore, before connecting the geometric center of the detection box in the current on-site image in each group of detection boxes with the geometric center of one detection box in the on-site image collected last time given, it also includes:

[0077] Obtain the detection boxes in the current on-site image with no overlap in the overlap comparison result;

[0078] Calculate the distances between the geometric centers of the detection boxes with no overlap and the geometric centers of all the detection boxes in the on-site image collected last time, and give the detection box in the on-site image collected last time with the smallest distance from the detection box with no overlap;

[0079] Divide the detection boxes without overlap and the detection box with the smallest given distance into a group.

[0080] Furthermore, combining the object detection results of the current on-site picture and the previous on-site picture collected, the current trajectory information of the dynamic object is given, and it also includes:

[0081] According to the trajectory information of all objects in the current on-site picture, the overlapping comparison results corresponding to all object trajectory information within the preset alarm time are given, as well as the distance between the geometric center of the detection box in the current on-site picture and the geometric center of a corresponding detection box in the previous on-site picture collected within the preset alarm time;

[0082] Based on the dynamic overlap threshold and the dynamic distance threshold, screen the overlapping comparison results and the distances of the geometric centers corresponding to all object trajectory information, and give the trajectory information of the dynamic object, including:

[0083] Eliminate the object trajectory information with an overlapping comparison result greater than the dynamic overlap threshold and a distance of the geometric center less than the dynamic distance threshold within the preset alarm time, and use the trajectory information of the remaining objects as the trajectory information of the dynamic object.

[0084] Furthermore, based on the dynamic overlap threshold and the dynamic distance threshold, screen the overlapping comparison results and the distances of the geometric centers corresponding to all object trajectory information, and the satisfaction relationship is as follows:

[0085]

[0086] In the formula, a is the time period between the corresponding moment of the current on-site picture and the previous moment separated by the preset alarm time, N is the number of frames of the on-site picture within the preset alarm time, is the detection box in the nth frame of the on-site picture within the time period a, is the abscissa of the geometric center of the detection box in the nth frame of the on-site picture within the time period a, is the ordinate of the geometric center of the detection box in the nth frame of the on-site picture within the time period a, μ is the dynamic overlap threshold, and τ is the dynamic distance threshold.

[0087] Furthermore, the dynamic overlap threshold is 95%, and the dynamic distance threshold is 10 pixels.

[0088] Furthermore, based on the recognition results of the transmission line corridor area and the dynamic object, determine the interference result between the dynamic object and the transmission line corridor area, including:

[0089] Overlap and compare the trajectory information of the dynamic object with the transmission line corridor area, and give the overlap result;

[0090] Based on the overlap result, screen out the dynamic objects whose trajectory information overlaps with the transmission line corridor area.

[0091] Further, determining an alarm message based on the interference result includes:

[0092] Based on the position information of all targets in the current on-site image and the power transmission channel area, and combining the dynamic targets whose trajectory information overlaps with the power transmission channel area, dynamic targets located within the power transmission channel area are given;

[0093] Based on the historical alarm messages within a predetermined time, historical dynamic targets that have been alarmed are given;

[0094] The dynamic targets located within the power transmission channel area and the historical dynamic targets that have been alarmed are matched, and dynamic targets that have not been successfully matched are given;

[0095] An alarm message is generated based on the dynamic targets that have not been successfully matched.

[0096] Further, generating an alarm message based on the dynamic targets that have not been successfully matched includes:

[0097] According to the trajectory information of the dynamic targets that have not been successfully matched, the geometric centers of the detection frames at different times are obtained;

[0098] According to the geometric centers of the detection frames at different times, the pixel change values of the dynamic targets that have not been successfully matched are obtained;

[0099] The obtained pixel change values are compared with a pixel change threshold value to obtain dynamic targets within the pixel change threshold range;

[0100] According to the dynamic targets within the pixel change threshold range, historical on-site images containing the dynamic targets are obtained;

[0101] According to the historical on-site images and the current on-site image of the dynamic targets within the pixel change threshold range, the historical on-site image or the current on-site image with the smallest distance between the dynamic target and the boundary of the power transmission channel area is given;

[0102] Based on the historical on-site image or the current on-site image with the smallest distance, the corresponding target acquisition start time is given;

[0103] According to the target acquisition start time and the acquisition time corresponding to the current on-site image, the interference time when the dynamic target enters the power transmission channel area is given;

[0104] Based on the interference time and a preset alarm time, dynamic targets with an interference time not less than the preset alarm time are given;

[0105] According to the dynamic targets with an interference time not less than the preset alarm time, corresponding alarm messages are generated.

[0106] Second aspect, the present invention further provides a power transmission line safety monitoring and early warning system based on computer vision, adopting the above-mentioned power transmission line safety monitoring and early warning method based on computer vision. The system includes:

[0107] A data acquisition module, which is used to acquire on-site pictures of the power transmission line;

[0108] A picture recognition module, which is used to recognize the power transmission channel area and dynamic targets in the on-site pictures;

[0109] An interference judgment module, which is used to determine the interference result between the dynamic target and the power transmission channel area based on the recognition results of the power transmission channel area and the dynamic target;

[0110] An early warning determination module, which is used to determine the alarm information based on the interference result.

[0111] Furthermore, the picture recognition module is further used for:

[0112] Performing edge processing on the on-site pictures to obtain edge information belonging to straight line segments;

[0113] Performing screening processing on the edge information of the straight line segments to give the power transmission line;

[0114] Performing ground mapping on the given power transmission line to form a power transmission channel area.

[0115] Furthermore, the picture recognition module is further used for:

[0116] Converting the on-site pictures into grayscale pictures and performing edge detection to obtain edge information;

[0117] Based on the grayscale threshold, dividing the edge information to obtain the divided edge information;

[0118] Performing binary grayscale conversion on the divided edge information to obtain a binary grayscale picture;

[0119] Performing Hough transform processing on the binary grayscale picture to give the polar coordinates of the edge information;

[0120] Screening the polar coordinates of the edge information to give multiple groups of edge information that meet the predetermined conditions;

[0121] Regarding the edge information of the same group as the edge information of a straight line segment.

[0122] Furthermore, the picture recognition module is further used for:

[0123] Dividing the straight line segments with the same slope in the straight line segments into the same category;

[0124] Connecting the heads and tails of two straight line segments with overlapping points in each category to form a straight line segment;

[0125] Based on each type of straight line segment after connection, obtain the lengths of all straight line segments;

[0126] Compare the lengths of the straight line segments in each type with the length threshold, and give the straight line segments with lengths greater than the length threshold;

[0127] Take the straight line segments with lengths greater than the length threshold in the same type as the power transmission lines in the same position area;

[0128] Integrate the power transmission lines in all position areas to give the power transmission line.

[0129] Furthermore, the image recognition module is also used for:

[0130] Perform object detection on the on-site image based on a pre-built object detection model, and give the detection frames, object types, and position information of all objects in the on-site image;

[0131] Obtain the object detection results of the on-site image collected immediately before the current on-site image;

[0132] Combine the object detection results of the current on-site image and the on-site image collected immediately before to give the current trajectory information of the dynamic object;

[0133] Obtain the historical trajectory information of the dynamic object, and combine it with the current trajectory information of the dynamic object to obtain the trajectory information of the dynamic object.

[0134] Furthermore, the interference judgment module is also used for:

[0135] Overlap and compare the trajectory information of the dynamic object with the power transmission channel area to give the overlap result;

[0136] Based on the overlap result, filter out the dynamic objects whose trajectory information overlaps with the power transmission channel area.

[0137] Furthermore, the early warning determination module is also used for:

[0138] According to the position information of all objects in the current on-site image and the power transmission channel area, and combine the dynamic objects whose trajectory information overlaps with the power transmission channel area to give the dynamic objects located in the power transmission channel area;

[0139] Based on the historical alarm information within a predetermined time, give the historical dynamic objects that have been alarmed;

[0140] Match the dynamic objects located in the power transmission channel area with the historical dynamic objects that have been alarmed to give the dynamic objects that have not been successfully matched;

[0141] Generate alarm information based on the dynamic objects that have not been successfully matched.

[0142] Further, the early warning determination module is further configured to:

[0143] Based on the position information of all targets in the current on-site image and the transmission channel area, and combining the dynamic targets with overlapping trajectory information and the transmission channel area, give the dynamic targets located within the transmission channel area;

[0144] Based on the historical alarm information within a predetermined time, give the historical dynamic targets that have been alarmed;

[0145] Match the dynamic targets located within the transmission channel area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched;

[0146] Generate alarm information based on the dynamic targets that have not been successfully matched.

[0147] Further, the early warning determination module is further configured to:

[0148] Obtain the geometric center of the detection frame of the dynamic target that has not been successfully matched at different times according to its trajectory information;

[0149] Obtain the pixel change value of the dynamic target that has not been successfully matched according to the geometric center of the detection frame at different times;

[0150] Compare the obtained pixel change value with the pixel change threshold to obtain the dynamic targets within the pixel change threshold range;

[0151] According to the dynamic targets within the pixel change threshold range, obtain the historical on-site images containing the dynamic targets;

[0152] According to the historical on-site images and the current on-site image of the dynamic targets within the pixel change threshold range, give the historical on-site image or the current on-site image with the smallest distance between the dynamic target and the boundary of the transmission channel area;

[0153] Based on the historical on-site image or the current on-site image with the smallest distance, give the corresponding starting time of target acquisition;

[0154] According to the starting time of target acquisition and the acquisition time corresponding to the current on-site image, give the interference time when the dynamic target enters the transmission channel area;

[0155] Based on the interference time and the preset alarm time, give the dynamic targets whose interference time is not less than the preset alarm time;

[0156] Generate the corresponding alarm information according to the dynamic targets whose interference time is not less than the preset alarm time.

[0157] Thirdly, the present invention also provides a power transmission line safety monitoring and early warning device based on computer vision, including a transmission channel recognition unit, a dynamic target recognition unit, and an early warning information generation unit;

[0158] The transmission channel recognition unit is used to collect on-site pictures of the power transmission line;

[0159] The dynamic target recognition unit is used to identify the transmission channel area and dynamic targets in the on-site pictures;

[0160] The early warning information generation unit is used to determine the interference result between the dynamic target and the transmission channel area based on the recognition results of the transmission channel area and the dynamic target; and is also used to determine the alarm information based on the interference result.

[0161] A power transmission line safety monitoring and early warning method and system based on computer vision provided by the present invention has at least the following beneficial effects:

[0162] (1) It can provide a transmission channel area for discriminating hazards. It can also identify dynamic targets, so as to generate alarm information for targets in a moving state, improving the accuracy of power transmission line safety early warning.

[0163] (2) It can automatically define the transmission channel area near the power transmission tower, greatly reducing the workload of manual definition of the transmission channel by line monitoring personnel and improving the applicability of the algorithm for large-scale applications.

[0164] (3) By performing anti-shake processing, trajectory analysis processing, and speed analysis processing on moving targets, static targets and non-threatening dynamic targets are effectively filtered out, greatly reducing the false alarm rate and improving the work efficiency of back-end monitoring personnel. Description of the Drawings

[0165] Figure 1 It is a flowchart of a power transmission line safety monitoring and early warning method based on computer vision provided by the present invention;

[0166] Figure 2 It is a flowchart of obtaining straight-line segment edge information in a certain embodiment provided by the present invention;

[0167] Figure 3 It is a flowchart of giving a power transmission line in a certain embodiment provided by the present invention;

[0168] Figure 4 It is a flowchart of forming a transmission channel area in a certain embodiment provided by the present invention;

[0169] Figure 5 It is a flowchart of giving dynamic target trajectory information in a certain embodiment provided by the present invention;

[0170] Figure 6 Flowchart of generating alarm information for an embodiment provided by the present invention;

[0171] Figure 7 Schematic diagram of a transmission line safety monitoring and early warning system based on computer vision provided by the present invention;

[0172] Figure 8 Schematic diagram of a transmission line safety monitoring and early warning device based on computer vision provided by the present invention.

[0173] Explanation of reference numerals: 1 - Transmission channel identification unit, 2 - Dynamic target identification unit, 3 - Early warning information generation unit, 4 - Edge detection sub - unit, 5 - Edge screening sub - unit, 6 - Line segment identification sub - unit, 7 - Transmission line fitting sub - unit, 8 - Transmission line boundary identification sub - unit, 9 - Transmission channel generation sub - unit, 10 - Video stack sub - unit, 11 - Target identification model sub - unit, 12 - Target tracking sub - unit, 13 - Dynamic trajectory analysis unit, 14 - Early warning information comparison sub - unit, 15 - Historical early warning cache sub - unit, 16 - Early warning information feedback sub - unit. Detailed implementation manners

[0174] In order to better understand the above - mentioned technical solutions, the following will combine the accompanying drawings of the specification and specific implementation manners to make a detailed description of the above - mentioned technical solutions. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0175] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two.

[0176] It should also be noted that the term "comprises", "comprising" or any other variation thereof is intended to cover a non - exclusive inclusion, such that a commodity or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such commodity or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the existence of another identical element in the commodity or device comprising said element.

[0177] As Figure 1 shown, the present invention provides a transmission line safety monitoring and early warning method based on computer vision, including:

[0178] Collect on-site pictures of the transmission line;

[0179] Identify the transmission corridor area and dynamic targets in the on-site pictures;

[0180] Based on the identification results of the transmission corridor area and dynamic targets, determine the interference result between the dynamic targets and the transmission corridor area;

[0181] Determine the alarm information based on the interference result.

[0182] Among them, after collecting the on-site pictures of the transmission line, the identification of the transmission corridor area in the on-site pictures may include:

[0183] Perform edge processing on the on-site pictures to obtain edge information belonging to straight line segments;

[0184] Perform screening processing on the edge information of the straight line segments to give the transmission line;

[0185] Perform ground mapping on the given transmission line to form a transmission corridor area.

[0186] By defining the transmission corridor area near the transmission towers, the workload of manual definition of the transmission corridor by line monitoring personnel is reduced, and the applicability of large-scale applications is improved.

[0187] Such as Figure 2 As shown, in the process of identifying the transmission corridor area in the on-site pictures, the edge processing of the on-site pictures to obtain edge information belonging to straight line segments may include:

[0188] Convert the on-site pictures into grayscale pictures and perform edge detection to obtain edge information;

[0189] Based on the grayscale threshold, divide the edge information to obtain the divided edge information;

[0190] Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale picture;

[0191] Perform Hough transform processing on the binary grayscale picture to give the polar coordinates of the edge information;

[0192] Screen the polar coordinates of the edge information to give multiple groups of edge information that meet the predetermined conditions;

[0193] Take the edge information of the same group as the edge information of a straight line segment;

[0194] Among them, the predetermined condition is that the number of edge information in the same group is not less than the predetermined number, and all the polar coordinates of the edge information in the same group form a straight line segment in the polar coordinate system; the grayscale threshold is 50. Edge information refers to the edge points with corresponding coordinates in the grayscale picture.

[0195] After converting the on-site picture into a grayscale picture, the grayscale picture can be processed by the Sobel operator to complete edge detection and obtain edge information.

[0196] When performing binary grayscale conversion on the divided edge information, the edge information exceeding the conversion threshold is set to 1, and those below the conversion threshold are set to 0.

[0197] After performing the Hough transform on the binary grayscale picture, the edge points corresponding to the edge information are represented by polar coordinates in the polar coordinate system. When screening and giving multiple groups of edge information that meet the predetermined conditions, if more than 100 pieces of edge information can be described by a polar coordinate straight-line formula, then the edge information of that straight-line segment exists in the picture, so these edge information are retained in the edge detection result, and the remaining edge information that does not meet this requirement is deleted.

[0198] Among them, performing the Hough transform on the binary grayscale picture and giving the polar coordinates of the edge information includes:

[0199]

[0200] In the formula, (x, y) are the coordinates of the straight-line segment in the rectangular coordinate system of the binary grayscale picture, a is the slope of the straight-line segment, b is the intercept of the straight-line segment, (r, θ) are the polar coordinates of the straight-line segment in the polar coordinate system after performing the Hough transform, r is the distance from the origin to the straight-line segment, and θ is the angle between r and the horizontal axis in the rectangular coordinate system. The polar coordinate transformation is to screen out which straight-line segments exist in the on-site picture, and all the screened straight-line segments are processed subsequently in the rectangular coordinate system.

[0201] Such as Figure 3 As shown, after obtaining the edge information belonging to the straight-line segment, screening and processing the edge information of the straight-line segment to give the transmission line may include:

[0202] Dividing the straight-line segments with the same slope in the straight-line segment into the same category;

[0203] Connecting the heads and tails of two straight-line segments with overlapping points in each category to form a straight-line segment;

[0204] Based on each category of straight-line segments after connection, obtaining the lengths of all straight-line segments;

[0205] Comparing the length of the straight-line segment in each category with the length threshold to give the straight-line segments longer than the length threshold;

[0206] Taking the straight-line segments longer than the length threshold in the same category as the transmission lines in the same position area;

[0207] Integrating the transmission lines in all position areas to give the transmission line;

[0208] Among them, the straight line segments with the same slope in the straight line segments are divided into the same category, including:

[0209] Subtract the slopes of all the straight line segments in pairs to give the slope difference results;

[0210] Based on the slope difference threshold, screen all the slope difference results to give the slope difference results within the slope difference threshold, specifically:

[0211]

[0212] In the formula, k 1 , k 2 are the slopes of two straight line segments, α is the slope difference threshold, and α is 0.2;

[0213] According to the screened slope difference results, divide the two straight lines corresponding to each slope difference result into the same category.

[0214] Connect the heads and tails of two straight line segments with overlapping points in each category to form a straight line segment, including:

[0215] Obtain the endpoint coordinates of all the straight line segments in each category;

[0216] Solve the endpoint distances between the endpoint coordinates of any two straight line segments in each category;

[0217] Based on the distance threshold, screen all the solved endpoint distances to give the endpoint distances within the distance threshold, specifically:

[0218]

[0219] In the formula, x 1 , y 1 are the endpoint coordinates of one of the straight line segments respectively, x 2 y 2 are the endpoint coordinates of the other straight line segment respectively, β is the distance threshold, and β is 20 pixels;

[0220] Divide the two straight line segments corresponding to all the given endpoint distances into two straight line segments with overlapping points, and connect the corresponding two endpoints.

[0221] In this embodiment, the edge information polar coordinates of all groups in the same category form straight line segments with the same slope in the polar coordinate system; the length threshold is 1 / 20 of the height of the Y-axis of the binary grayscale image, that is, when the length of the straight line segment is greater than 1 / 20 of the height of the Y-axis of the binary grayscale image, then it is determined that the straight line segment is a transmission line.

[0222] Such as Figure 4As shown, after obtaining the transmission line, the given transmission line can be mapped to the ground to form a transmission channel area, specifically including:

[0223] Based on the Hough transform, identify the given transmission line to obtain the starting ends of the two outermost transmission lines in each position area;

[0224] Connect the endpoints of the two outermost transmission lines in all position areas to obtain an outer envelope sub-area corresponding to the position area;

[0225] Integrate all outer envelope sub-areas to form the outer envelope area of the transmission line; based on the connected endpoint coordinates, obtain the boundary coordinates of the outer envelope area;

[0226] Symmetrically map the boundary coordinates of the outer envelope area along the horizontal axis of the field picture to form the transmission channel area.

[0227] Specifically, through the Hough transform, identify the starting ends of the transmission lines in the binary grayscale picture, and use the area formed after connecting each starting end as the outer envelope area of the transmission line. Among them, when performing identification based on the Hough transform, when there are two or more transmission lines in each position area, the two outermost transmission lines will be automatically selected respectively during identification, and the starting ends of these two transmission lines will be projected to the ground to obtain four boundary points, which are connected into a quadrilateral area to obtain the outer envelope sub-area. When performing ground mapping, the mapping area is: the new outer envelope area formed by symmetrically mapping the boundary coordinates of the outer envelope sub-area of the transmission line along the X axis of the picture, and this new outer envelope area is the transmission channel area. Among them, an example of X-axis mirror symmetry mapping is that the starting coordinate is (5, 3), and after X-axis symmetry mapping, it is (5, -3).

[0228] In order to ensure the symmetry of the transmission lines on both sides of each position area, uniform length, and regular shape of the transmission channel, after obtaining the endpoints of the two outermost transmission lines in each position area, it includes:

[0229] Obtain the endpoint coordinates of the two outermost transmission lines in all position areas;

[0230] For the two outermost transmission lines in each position area, determine the upper endpoint and the lower endpoint of each transmission line;

[0231] Perform coordinate difference processing on the vertical coordinates of the two lower endpoints and the vertical coordinates of the two upper endpoints in each position area respectively; among them, the coordinate difference processing is the absolute value of the difference between the vertical coordinates of the two endpoints, and the endpoint difference threshold is 200 pixels;

[0232] Compare all coordinate difference processing results with the endpoint difference threshold, and give the coordinate difference processing results that exceed the endpoint difference threshold;

[0233] Based on the processed result of the given coordinate difference, obtain the endpoints of two power transmission lines that match and correspond to it;

[0234] Based on the endpoints of two power transmission lines that match and correspond to each other, calculate the lengths of the two power transmission lines corresponding to the endpoints, and increase a line segment with the same slope as itself and a certain length along the endpoints that match and correspond to the power transmission line with the smaller length; wherein, the increased length of the line segment is equal to the length difference between the two power transmission lines.

[0235] After collecting the on-site pictures of the power transmission line, while identifying the power transmission channel area, the present invention can also identify dynamic targets in the on-site pictures, specifically including:

[0236] Perform object detection on the on-site pictures based on a pre-constructed object detection model, and give the detection frames, object types, and position information of all objects in the on-site pictures;

[0237] Obtain the object detection result of the on-site picture collected in the previous time relative to the current on-site picture;

[0238] Combine the object detection results of the current on-site picture and the on-site picture collected in the previous time to give the current trajectory information of the dynamic target;

[0239] Obtain the historical trajectory information of the dynamic target, and combine it with the current trajectory information of the dynamic target to obtain the trajectory information of the dynamic target.

[0240] In each embodiment of the present invention, after collecting the current on-site picture each time, it combines with the object detection result of the on-site picture collected in the previous time to give the current trajectory information of the dynamic target, that is, in each embodiment of the present invention, after collecting the on-site pictures at different times, it will obtain the current trajectory information of the dynamic target corresponding to each time. Therefore, when obtaining the historical trajectory information of the dynamic target in the embodiments of the present invention, it can be directly retrieved and used from the historical data. Of course, the historical trajectory information of the dynamic target in the embodiments of the present invention can also be obtained when needed according to the process of giving the current trajectory information of the dynamic target in this embodiment.

[0241] Among them, as Figure 5 shown, combining the object detection results of the current on-site picture and the on-site picture collected in the previous time to give the current trajectory information of the dynamic target may include:

[0242] Overlap and compare all the detection frames in the current on-site picture with all the detection frames in the on-site picture collected in the previous time, and give the overlap comparison result;

[0243] Based on the overlap comparison result, take each detection frame in the current on-site picture and the detection frames that overlap in the on-site picture collected in the previous time as a group of detection frames;

[0244] Filter each group of detection boxes based on the overlap threshold, and eliminate the detection boxes in the on-site pictures collected last time whose overlap comparison results with the detection boxes in the current on-site picture are less than the overlap threshold;

[0245] Compare the geometric centers of the detection boxes in the on-site pictures collected last time remaining in each group of detection boxes with the geometric centers of the detection boxes in the current on-site picture, and give one detection box in the on-site picture collected last time that is closest to the geometric center of the detection box in the current on-site picture;

[0246] Connect the geometric centers of the detection boxes in the current on-site picture in each group of detection boxes with the geometric center of one detection box in the on-site picture collected last time to obtain the trajectory information of the target corresponding to the detection box in the current on-site picture.

[0247] Specifically, the overlap threshold is 50%. That is, when the detection box overlap ratio (IOU) of the target in the front and back frames of the video (the front and back frames correspond to the on-site pictures collected last time and the current on-site picture respectively) is higher than 50%, it is determined that the two targets in the front and back frames are the same object. Connecting the geometric midpoints of the detection boxes of the same object in adjacent frames can obtain the current trajectory of the moving target.

[0248] In addition, when there is no overlapping area between the detection boxes of the front and back frames, such as when the target moves too fast or the video has dropped frames, the target association algorithm based on the distance between the detection box centers will be applied to ensure continuous tracking of the target. Specifically, before connecting the geometric centers of the detection boxes in the current on-site picture in each group of detection boxes with the geometric center of one detection box in the on-site picture collected last time, it also includes:

[0249] Obtain the detection boxes in the current on-site picture with non-existent overlap in the overlap comparison results;

[0250] Calculate the distances between the geometric centers of the detection boxes with non-existent overlap and the geometric centers of all detection boxes in the on-site picture collected last time, and give the detection box in the on-site picture collected last time that has the smallest distance from the detection box with non-existent overlap;

[0251] And divide the detection boxes with non-existent overlap and the given detection box with the smallest distance into one group.

[0252] The target detection model for performing target detection on the on-site picture is pre-constructed. This target detection model can be pre-constructed through the following steps:

[0253] Obtain a number of historical on-site pictures;

[0254] Mark detection boxes, position information, and target types for a number of historical on-site pictures;

[0255] Based on the marked historical site pictures, the YOLOv5s model is trained to obtain the trained YOLOv5s model.

[0256] Among them, the trained YOLOv5s model is the pre-constructed object detection model. The construction of this object detection model is specifically as follows: features are extracted from the site pictures through the backbone network; the extracted features are processed by the detection head to obtain the detection boxes and object type probabilities of the predicted objects; the size and aspect ratio of the predicted detection boxes are generated through the anchor points to obtain the predicted coordinates of the detection boxes; based on the marked site pictures and the detection boxes, object types, and coordinates of the predicted objects, the YOLOv5 model is trained to obtain the trained YOLOv5 model; when predicting through the trained YOLOv5 model, after the YOLOv5 model predicts multiple detection boxes for the same object, the non-maximum suppression is used to screen out the object detection box with the highest confidence; through post-processing, the coordinates of the detection box are converted into position information, and the object type with the highest predicted probability is assigned to the corresponding detection box.

[0257] Specifically, the construction of the YOLOv5s model can include:

[0258] Backbone network: YOLOv5 uses a deep neural network as the backbone to extract features from the input site pictures. The backbone network is based on the convolutional neural network (CNN) architecture, such as CSPDarknet53 or MobileNetV3, which helps to capture hierarchical features from the site pictures;

[0259] Detection head: The detection head is responsible for predicting the bounding boxes (detection boxes) and object type probabilities of the detected objects. YOLOv5 uses a lightweight head composed of several convolutional layers to generate the final detection output results;

[0260] Anchor points: YOLOv5 uses anchor boxes to predict bounding boxes of different sizes and aspect ratios. These anchor boxes are predefined boxes that serve as references for the network to predict the final bounding box coordinates;

[0261] Loss function: The loss function is used to train the YOLOv5 model by calculating the difference between the predicted bounding boxes and the ground truth bounding boxes. The YOLOv5 model uses a combination of different loss functions, including localization loss (for bounding box coordinates) and classification loss (for object type probabilities);

[0262] Non-maximum suppression (NMS): After the YOLOv5 model predicts multiple bounding boxes for the same object, NMS is applied to remove duplicate detections and only retain the most confident detection results;

[0263] Post - processing: The last step involves converting the predicted bounding box coordinates to their respective positions on the on - site image and assigning the correct object category to each detected bounding box based on the target type probability.

[0264] After the present invention identifies the transmission line corridor area and dynamic targets in the on - site image, based on the identification results of the transmission line corridor area and dynamic targets, it can determine the interference result between the dynamic target and the transmission line corridor area, specifically including:

[0265] Overlap and compare the trajectory information of the dynamic target with the transmission line corridor area to give an overlap result;

[0266] Based on the overlap result, filter out the dynamic targets whose trajectory information overlaps and interferes with the transmission line corridor area.

[0267] After the present invention obtains the interference result, it can determine the alarm information based on the interference result, specifically including:

[0268] According to the position information of all targets in the current on - site image and the transmission line corridor area, and combining the dynamic targets whose trajectory information overlaps with the transmission line corridor area, give the dynamic targets located within the transmission line corridor area;

[0269] Based on the historical alarm information within a predetermined time, give the historical dynamic targets that have been alarmed;

[0270] Match the dynamic targets located within the transmission line corridor area with the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched;

[0271] Generate alarm information based on the dynamic targets that have not been successfully matched.

[0272] Among them, the predetermined time is preset according to the actual transmission line monitoring scenario. Preferably, the predetermined time is 30 seconds. In the actual application scenario, if the same target triggers an alarm multiple times within 30 seconds, only one alarm will be reported (that is, if the trajectory information of the historical dynamic target that has been alarmed overlaps with the transmission line corridor area multiple times, only one alarm will be made); if the alarm location remains unchanged within 30 seconds, only one alarm will be reported (that is, if the historical dynamic target that has been alarmed is always located within the transmission line corridor area, only one alarm will be made).

[0273] In addition, during the identification process of dynamic targets, it is necessary to ignore the targets with too fast moving speed and whose moving trajectories briefly pass through the transmission line corridor to avoid misjudgment in situations such as passing vehicles when the transmission line is close to the road. That is, the present invention can perform high - speed discrimination on dynamic targets. Among them, the high - speed discrimination of dynamic targets is divided into two situations:

[0274] Too fast speed: When the change in the geometric center point distance of the detection box of the target between adjacent frames exceeds 200 pixel points, it is regarded as a non - operating state with too fast speed;

[0275] The moving time within the power transmission channel area is too short: When the time interval between when a dynamic target enters the power transmission channel area and when it leaves the power transmission channel area is shorter than the preset alarm time, it is regarded that the target briefly passes through the area and no alarm is given.

[0276] Since the targets within the power transmission channel area can only pose a threat when in a moving state, therefore, as Figure 6 shown, in the actual application scenario, alarm information is generated based on the dynamic targets that have not been successfully matched, that is, after quickly discriminating dangerous targets, alarms are generated. Specifically, it can include:

[0277] Obtain the geometric centers of the detection frames of the dynamic targets that have not been successfully matched at different times according to their trajectory information; the geometric centers of the detection frames at different times are the geometric centers of the detection frames in the on-site pictures at different video frames;

[0278] Obtain the pixel change values of the dynamic targets that have not been successfully matched according to the geometric centers of the detection frames at different times;

[0279] Compare the obtained pixel change values with the pixel change threshold to obtain the dynamic targets within the pixel change threshold range; among them, the pixel change threshold range is 10 - 200 pixel points;

[0280] According to the dynamic targets within the pixel change threshold range, obtain the historical on-site pictures containing the dynamic targets;

[0281] According to the historical on-site pictures and the current on-site pictures of the dynamic targets within the pixel change threshold range, give the historical on-site picture or the current on-site picture with the smallest distance between the dynamic target and the boundary of the power transmission channel area;

[0282] Based on the historical on-site picture or the current on-site picture with the smallest distance, give the corresponding starting time of target acquisition;

[0283] According to the starting time of target acquisition and the acquisition time corresponding to the current on-site picture, give the interference time when the dynamic target enters the power transmission channel area;

[0284] Based on the interference time and the preset alarm time, give the dynamic targets with the interference time not less than the preset alarm time;

[0285] According to the dynamic targets with the interference time not less than the preset alarm time, generate the corresponding alarm information.

[0286] Generally speaking, the rule for the present invention to generate alarm information is as follows: When the target continuously moves within the power transmission corridor area for more than a certain time (preset alarm time), the target is determined to be a dangerous target. When the target is stationary, regardless of its position, the target is determined to be a non-threatening target. When the target moves outside the power transmission corridor area, the target is also determined to be a non-dangerous target. Among them, for dangerous targets, an alarm is given. Specifically, starting from the time when the target enters the power transmission corridor area, that is, when the target detection frame overlaps with the power transmission corridor area, if the target continuously moves and the overlapping time between its detection frame and the power transmission corridor area exceeds the preset alarm time, then the dangerous target is screened out as a dangerous moving target, and alarm information is generated for it. The warning information includes the target category, time, and corresponding screenshots that trigger the alarm, and may also include the position of the target.

[0287] In addition, the present invention automatically regards the jitter behavior of the target within a small range in a short time as static to adapt to the phenomenon of camera jitter caused by strong wind weather conditions or roadbed shaking conditions. Specifically, when the target detection frame continuously moves within a range of 10 pixel points within the preset alarm time, and the overlapping area of the front and rear frame detection frames (the detection frame of the target in the current on-site picture and the detection frame in the on-site picture collected last time) continuously exceeds 95%, then the target is determined to be a stationary target. Among them, the preset alarm time is less than the predetermined time. Specifically, combining the target detection results of the current on-site picture and the on-site picture collected last time, the current trajectory information of the dynamic target is given, and it also includes:

[0288] According to the trajectory information of all targets in the current on-site picture, the overlapping comparison results corresponding to all target trajectory information within the preset alarm time, and the distance between the geometric center of the detection frame in the current on-site picture and the geometric center of a corresponding detection frame in the on-site picture collected last time within the preset alarm time are given;

[0289] Based on the dynamic overlapping threshold and the dynamic distance threshold, the overlapping comparison results and the distances of the geometric centers corresponding to all target trajectory information are screened, and the trajectory information of the dynamic target is given, including:

[0290] The target trajectory information with an overlapping comparison result greater than the dynamic overlapping threshold and a geometric center distance less than the dynamic distance threshold within the preset alarm time is eliminated, and the trajectory information of the remaining targets is used as the trajectory information of the dynamic target.

[0291] As Figure 7 shown, the present invention also provides a power transmission line safety monitoring and warning system based on computer vision, which adopts the above-mentioned power transmission line safety monitoring and warning method based on computer vision. The system includes:

[0292] A data acquisition module, which is used to acquire on-site pictures of the power transmission line;

[0293] An image recognition module for recognizing the transmission line corridor area and dynamic targets in on-site images;

[0294] An interference judgment module for determining the interference result between the dynamic target and the transmission line corridor area based on the recognition results of the transmission line corridor area and the dynamic target;

[0295] An early warning determination module for determining alarm information based on the interference result.

[0296] As Figure 8 shown, the present invention also provides a transmission line safety monitoring and early warning device based on computer vision, including a transmission line corridor recognition unit 1, a dynamic target recognition unit 2, and an early warning information generation unit 3;

[0297] The transmission line corridor recognition unit is used to collect on-site images of the transmission line;

[0298] The dynamic target recognition unit is used to recognize the transmission line corridor area and dynamic targets in on-site images;

[0299] The early warning information generation unit is used to determine the interference result between the dynamic target and the transmission line corridor area based on the recognition results of the transmission line corridor area and the dynamic target; and is also used to determine alarm information based on the interference result.

[0300] Among them, the dynamic target recognition unit 2 and the early warning information generation unit 3 operate in series, the transmission line corridor recognition unit 1 operates in parallel, the transmission line corridor recognition unit 1 and the dynamic target recognition unit 2 share the same data source, and the data source can be a directly connected hardware camera device or a remote network video source. The transmission line corridor recognition unit 1 includes an edge detection subunit 4, an edge screening subunit 5, a line segment recognition subunit 6, a transmission line fitting subunit 7, a transmission line boundary recognition subunit 8, and a transmission line corridor generation subunit 9, and the subunits operate in series. The dynamic target recognition unit 2 includes a video stack subunit 10, a target recognition model subunit 11, a target tracking subunit 12, and a dynamic trajectory analysis unit 13. The video stack subunit 10 operates in parallel with the target recognition model subunit 11, the target tracking subunit 12, and the dynamic trajectory analysis unit 13. The early warning information generation unit 3 includes an early warning information comparison subunit 14, a historical early warning cache subunit 15, and an early warning information feedback subunit 16.

[0301] The edge detection subunit 4 converts the input on-site picture into a grayscale picture, and uses the Sobel XY operator to perform edge detection on the grayscale picture. The edge screening subunit 5 performs screening on the result output by the edge detection subunit 4 based on grayscale threshold filtering, and performs binary grayscale conversion on the screened edge information. The line segment recognition subunit 6 performs a Hough transform operation on the binary grayscale picture output by the edge screening subunit 5, and screens out the edges belonging to straight line segments in the edge information. The power transmission line fitting subunit 7 performs straight line fitting and splicing on the straight line segment edge information, and judges and screens out the power transmission line according to information such as the orientation, direction, and length of the fitted straight line. The power transmission line boundary recognition subunit 8 judges the outer envelope area of the power transmission line on the tower where the camera is located according to the power transmission line information output by the power transmission line fitting subunit 7. The power transmission channel generation subunit 9 forms a power transmission channel area by performing ground mapping on the outer envelope area of the power transmission line according to the outer envelope area information of the power transmission line output by the power transmission line boundary recognition subunit 8. In this embodiment, a low-power power transmission channel recognition unit and a dynamic target recognition unit are used to run at the edge, directly analyze and process the data on-site with limited resources, and only return the processing results through the 4G / 5G network. This not only greatly saves the traffic and bandwidth required for transmitting the original data, but also, due to the high efficiency of the low-power algorithm of the chip, does not require a high-power power supply on-site to support the operation of the device. Existing dynamic anomaly detection requires continuous real-time anomaly recognition. The image processing device needs to perform real-time analysis on the streaming data collected on-site, such as videos, LiDAR point cloud data, etc. This requires either a real-time data processing unit on the power transmission channel or the on-site streaming data to be continuously and real-time transmitted back to the backend processing unit.

[0302] The stack length of the video stack subunit 10 is fixed. When new picture data (on-site picture) is pushed into the stack, if the stack is full, the earliest data in the stack is pushed out and discarded. When a data output instruction is received, the stack pushes out the last pushed data. The target recognition model subunit 11 obtains the latest data from the video stack subunit 10, and sends the data into the target detection model to obtain the target category and location information. The target tracking subunit 12 caches the historical target category and location information output by the target recognition model subunit 11, and combines the historical information to perform target trajectory tracking on the latest target category monitoring result to judge dynamic and static targets. The dynamic trajectory analysis unit 13 judges whether the activity range of the dynamic target overlaps with the power transmission channel area generated by the power transmission channel generation subunit 9 according to the dynamic target trajectory information output by the target tracking subunit 12. Finally, the dynamic targets active in the power transmission channel are screened out. By designing a soft connection between the video stack subunit 10 and the target recognition model subunit, the present invention enables the target recognition model subunit 11 to either be directly connected to the camera for online operation or be connected to the backend through the network transmission channel, greatly improving the adaptability of the algorithm to the on-site network environment of the power transmission channel.

[0303] The early warning information comparison sub-unit 14 determines whether the target triggering the current early warning has repeated alarms within a certain period of time according to the historical early warning information in the historical early warning cache sub-unit 15. The historical early warning cache sub-unit 15 pushes the latest early warning information output by the early warning information comparison sub-unit 14 into the cache for subsequent early warning information comparison. The early warning information feedback sub-unit 16 transmits the latest early warning information pushed out by the historical early warning cache sub-unit 15 back to the cloud data platform through the 4G / LTE mobile network. The early warning information includes the target category, time, and screenshot triggering the alarm.

[0304] In the present invention, the real-time on-site pictures of the power transmission channel obtained by the monitoring camera are imported into a computing unit with corresponding visual processing capabilities for processing and analysis. It can be operated online in the on-site hardware device deployed and installed, or installed in the backend computing device to obtain real-time on-site pictures through the network. The power transmission channel recognition unit, dynamic target recognition unit, and early warning information generation unit proposed in the present invention will perform real-time analysis and disposal on the input video. Among them, the power transmission channel recognition unit obtains the full-color image captured by the camera. After the full-color picture is processed into black and white, the Sobel operator performs convolution operation on it to obtain the edge information corresponding to the image. Subsequently, the edge screening sub-unit will analyze the edge information to judge the distance, clarity, and contrast of the target edge. The edge information with a relatively far distance, poor clarity, and poor contrast with the surroundings will be automatically filtered. The remaining edge information will be binaryzied and subjected to straight-line segment edge screening. This screening process uses the Hough transform algorithm to filter out the edges without straight-line features in the edge information to narrow the target range for automatically searching for the power transmission line. Next, the power transmission line fitting sub-unit will perform azimuth discrimination, mutual spacing discrimination, and fitting straight-line discrimination on the straight-edge information to identify the location of the power transmission line among these straight edges. According to the position of the power transmission line, the power transmission line boundary recognition sub-unit gives the envelope line of the power transmission channel and obtains the power transmission channel area range through ground projection. In this embodiment, monitoring cameras are deployed on a large number of poles and towers, and access rights to the cameras are obtained. Through remote monitoring, on-site pictures are intercepted as training data. In the existing dynamic recognition process, there are not many camera resources set on-site for data collection, so it is difficult to obtain a good data set for model training.

[0305] The dynamic target recognition unit is responsible for positioning and tracking the moving targets in the real-time on-site pictures, and comparing their movement trajectories with the above-mentioned power transmission channel area. If the movement trajectory of the moving target coincides with the power transmission channel, early warning information for this moving target will be generated. The dynamic target recognition unit is soft-connected to the video input interface during operation and runs in parallel. When the on-site data is interrupted for various reasons, the power transmission channel recognition unit will enter the hot standby mode and can immediately switch to the monitoring mode when the video resumes without manual restart intervention.

[0306] While storing the historical alarm information, after receiving the warning information passed in by the dynamic target recognition unit, the warning information generation unit will compare the historical alarm information with the warning alarm time, location, target category, and movement trajectory of the input warning signal to determine whether it is duplicate alarm information. After confirming that the alarm information is correct, the warning information generation unit will generate new alarm information, push it into the information cache, and send it back to the background data center through the network.

[0307] Since the power transmission line presents a natural drooping state due to gravity, the line segment recognition subunit filters out the line segments with too long or too short lengths by judging the length of the straight line segments after Hough transformation. The power transmission line fitting subunit confirms the connection between line segments by judging the directions of different line segments and the distances between endpoints, and then judges whether the connected line segment group constitutes a power transmission line. When the power transmission line fitting subunit fails to find enough power transmission lines from the input image, it will output an alarm signal to the lower-level unit. After receiving the alarm signal that the power transmission line fitting subunit fails to find the power transmission line, the power transmission line boundary recognition subunit will automatically generate a preset power transmission channel area. When outputting the preset power transmission channel area, the power transmission line boundary recognition subunit will generate an alarm message and send it to the background. The background power transmission line monitoring personnel can judge whether the preset power transmission channel area is reasonable according to the visually transmitted image. When the background power transmission line monitoring personnel recognize that the range of the power transmission channel is unreasonable, they can manually adjust the area parameters and input them into the operating equipment. The target tracking subunit automatically regards the jitter behavior of the target within a small range in a short time as static to adapt to the camera jitter caused by strong wind weather conditions or roadbed shaking conditions. During the process of dynamic target recognition, the target tracking subunit will ignore the targets with too fast moving speed and short moving trajectories passing through the power transmission channel to avoid misjudgment of passing vehicles by the algorithm when the power transmission line is close to the road.

[0308] The present invention also provides a usage method of a visual power transmission line detection device, including the following steps:

[0309] S1: The monitoring camera and the computing and processing unit can be installed on the pole tower together. Then, connect the monitoring camera and the computing unit through the USB interface. At the same time, the computing and processing unit can also be remotely interconnected with the camera through 4G LTE or 5G network. The specific connection method can be selected according to the network conditions of the installation site. In areas with good network conditions, the remote interconnection method can be selected to save installation costs and maintenance costs. In areas with poor network conditions, the direct connection method of the camera can be selected to save network bandwidth.

[0310] S2: Under the condition that the device connection is completed, the power transmission line monitor inputs the camera port address or the network streaming address into the power transmission channel recognition unit to obtain the power transmission channel parameter information. If the power transmission channel unit issues an alarm message, it means that the preset power transmission channel information is output. The background monitor can check whether the preset power transmission channel meets the monitoring conditions when receiving the alarm message and decide whether to manually intervene to change the power transmission channel information.

[0311] S3: After the power transmission channel information is confirmed, the power transmission line monitor inputs the camera port address or the network streaming address into the dynamic target recognition unit and the early warning information generation unit. The dynamic moving targets in the power transmission channel will be calibrated with the early warning time, category, screenshot and sent back to the background monitoring platform when they meet the dynamic target screening conditions.

[0312] It can be seen that the above embodiments of the present invention achieve the following technical effects: The power transmission channel recognition unit realizes the automatic recognition of the power transmission channels in the power transmission lines, greatly improving the adaptive ability of the algorithm to different conditional environments and saving a great deal of the workload of manual setting of the power transmission channel monitoring parameters by the line maintenance personnel. The dynamic target recognition unit realizes the screening of static targets and the determination of non-threatening dynamic targets, greatly reducing the false alarm rate of dynamic threat targets in various operating environments, thus reducing the disposal volume and difficulty of the alarm information for the background maintenance personnel. The early warning information generation unit not only reduces repeated alarms by comparing historical alarm information, but also comprehensively calibrates and records the triggered alarm targets, including the trajectory, target category, trigger time, and screenshot at the time of triggering the alarm, enabling the background maintenance personnel to accurately and timely grasp the on-site situation and threat level, so as to make the best disposal as quickly as possible.

[0313] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the present invention. Obviously, those skilled in the art can make various changes and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. A method for safety monitoring and early warning of transmission lines based on computer vision, characterized in that, it includes: Collect on-site pictures of transmission lines; Identify the transmission channel area and dynamic targets in the on-site pictures; Based on the recognition results of the transmission channel area and dynamic targets, determine the interference result between the dynamic targets and the transmission channel area, specifically including: comparing the trajectory information of the dynamic targets with the transmission channel area to give an overlap result; based on the overlap result, screening out the dynamic targets whose trajectory information overlaps with the transmission channel area; Determine the alarm information based on the interference result; Among them, identifying the transmission channel area in the on-site pictures includes: Perform edge processing on the on-site pictures to obtain edge information belonging to straight line segments; Perform screening processing on the edge information of the straight line segments to give the transmission lines; Perform ground mapping on the given transmission lines to form a transmission channel area, including: based on the Hough transform, identify the given transmission lines to obtain the starting ends of the two outer transmission lines in each position area; connect the endpoints of the two outer transmission lines in all position areas to obtain the corresponding outer envelope sub-areas of the position areas; integrate all the outer envelope sub-areas to form the outer envelope area of the transmission line; based on the connected endpoint coordinates, obtain the boundary coordinates of the outer envelope area; symmetrically map the boundary coordinates of the outer envelope area along the horizontal axis of the on-site pictures to form the transmission channel area; Identify the dynamic targets in the on-site pictures, including: perform object detection on the on-site pictures based on a pre-constructed object detection model to give the detection frames, object types and position information of all objects in the on-site pictures; obtain the object detection results of the on-site pictures collected immediately before the current on-site pictures; combine the object detection results of the current on-site pictures and the on-site pictures collected immediately before to give the current trajectory information of the dynamic targets; obtain the historical trajectory information of the dynamic targets, and combine it with the current trajectory information of the dynamic targets to obtain the trajectory information of the dynamic targets.

2. The method for safety monitoring and early warning of transmission lines according to claim 1, characterized in that, Performing edge processing on the on-site pictures to obtain edge information belonging to straight line segments includes: Convert the on-site pictures into grayscale pictures and perform edge detection to obtain edge information; Based on the grayscale threshold, divide the edge information to obtain the divided edge information; Perform binary grayscale conversion on the divided edge information to obtain a binary grayscale picture; Perform Hough transform processing on the binary grayscale picture to give the polar coordinates of the edge information; Screen the polar coordinates of the edge information to give multiple groups of edge information that meet the predetermined conditions; Take the edge information of the same group as the edge information of a straight line segment; Among them, the predetermined condition is that the number of edge information in the same group is not less than the predetermined number, and all the polar coordinates of the edge information in the same group form a straight line segment in the polar coordinate system.

3. The method for safety monitoring and early warning of transmission lines according to claim 1 or 2, characterized in that, Performing screening processing on the edge information of the straight line segments to give the transmission lines includes: Divide the straight line segments with the same slope in the straight line segments into the same category; Connect the start and end points of two line segments with overlapping points in each category to form a single line segment; Based on each category of line segments after connection, obtain the lengths of all line segments; Compare the length of each line segment in each category with a length threshold, and output the line segments with lengths greater than the length threshold; Use the line segments with lengths greater than the length threshold in the same category as the power transmission lines in the same position area; Integrate the power transmission lines in all position areas to output the power transmission line; 4. The power transmission line safety monitoring and early warning method according to claim 1, characterized in that Combined with the object detection results of the current on-site picture and the previous on-site picture, output the current trajectory information of the dynamic object, including: Compare all detection frames in the current on-site picture with all detection frames in the previous on-site picture for overlap, and output the overlap comparison result; Based on the overlap comparison result, take each detection frame in the current on-site picture and the detection frames with overlap in the previous on-site picture as a group of detection frames; Based on an overlap threshold, screen each group of detection frames, and eliminate the detection frames in the previous on-site picture whose overlap comparison result with the detection frames in the current on-site picture is less than the overlap threshold; Compare the geometric centers of the detection frames in the previous on-site picture remaining in each group of detection frames with the geometric centers of the detection frames in the current on-site picture, and output one detection frame in the previous on-site picture that is closest to the geometric center of the detection frame in the current on-site picture; Connect the geometric center of the detection frame in the current on-site picture in each group of detection frames with the geometric center of one detection frame in the previous on-site picture that is output, to obtain the trajectory information of the object corresponding to the detection frame in the current on-site picture.

5. The power transmission line safety monitoring and early warning method according to claim 4, characterized in that, Combined with the object detection results of the current on-site picture and the previous on-site picture, output the current trajectory information of the dynamic object, further including: According to the trajectory information of all objects in the current on-site picture, output the overlap comparison result corresponding to all object trajectory information within a preset alarm time, and the distance between the geometric center of the detection frame in the current on-site picture and the geometric center of one corresponding detection frame in the previous on-site picture within the preset alarm time; Based on a dynamic overlap threshold and a dynamic distance threshold, screen the overlap comparison results and the distances of the geometric centers corresponding to all object trajectory information, and output the trajectory information of the dynamic object, including: Eliminate the object trajectory information with an overlap comparison result greater than the dynamic overlap threshold and a geometric center distance less than the dynamic distance threshold within the preset alarm time, and use the trajectory information of the remaining objects as the trajectory information of the dynamic object.

6. The power transmission line safety monitoring and early warning method according to claim 4, characterized in that, Before connecting the geometric center of the detection frame in the current on-site picture in each group of detection frames with the geometric center of one detection frame in the previous on-site picture that is output, further including: Obtain the detection frames in the current on-site picture with a non-overlapping overlap comparison result; Calculate the distances between the geometric centers of the detected bounding boxes without overlap and the geometric centers of all detected bounding boxes in the previously captured on-site image, and give the detected bounding box in the previously captured on-site image with the minimum distance from the detected bounding box without overlap; And group the detected bounding boxes without overlap and the detected bounding box with the minimum given distance into one group.

7. The transmission line safety monitoring and early warning method according to claim 1, 4 or 5, characterized in that, Determining alarm information based on the interference result, including: According to the position information of all targets in the current on-site image and the transmission channel area, and combining the dynamic targets whose trajectories overlap with the transmission channel area, give the dynamic targets located in the transmission channel area; Based on the historical alarm information within a predetermined time, give the historical dynamic targets that have been alarmed; Match the dynamic targets located in the transmission channel area and the historical dynamic targets that have been alarmed, and give the dynamic targets that have not been successfully matched; Generate alarm information based on the dynamic targets that have not been successfully matched.

8. The transmission line safety monitoring and early warning method according to claim 7, characterized in that, Generating alarm information based on the dynamic targets that have not been successfully matched, including: Obtain the geometric centers of the detected bounding boxes of the dynamic targets that have not been successfully matched at different times according to their trajectory information; Obtain the pixel change value of the dynamic targets that have not been successfully matched according to the geometric centers of the detected bounding boxes at different times; Compare the obtained pixel change value with the pixel change threshold to obtain the dynamic targets within the pixel change threshold range; According to the dynamic targets within the pixel change threshold range, obtain the historical on-site images containing the dynamic targets; According to the historical on-site images and the current on-site image of the dynamic targets within the pixel change threshold range, give the historical on-site image or the current on-site image with the minimum distance from the dynamic targets to the boundary of the transmission channel area; Based on the historical on-site image or the current on-site image with the minimum distance, give the corresponding starting time of target acquisition; According to the starting time of target acquisition and the acquisition time corresponding to the current on-site image, give the interference time when the dynamic target enters the transmission channel area; Based on the interference time and the preset alarm time, give the dynamic targets whose interference time is not less than the preset alarm time; Generate the corresponding alarm information according to the dynamic targets whose interference time is not less than the preset alarm time.

9. A transmission line safety monitoring and early warning system based on computer vision, adopting the transmission line safety monitoring and early warning method based on computer vision described in any one of claims 1-8, characterized in that, The system includes: A data acquisition module, which is used to acquire on-site images of the transmission line; An image recognition module, which is used to recognize the transmission channel area and dynamic targets in the on-site image; An interference judgment module, which is used to determine the interference result between the dynamic targets and the transmission channel area based on the recognition results of the transmission channel area and the dynamic targets; An early warning determination module, which is used to determine alarm information based on the interference result.

Images