A power transmission line anti-external damage automatic early warning method and device based on deep learning

Abstract

This invention discloses an automatic early warning method and device for external damage prevention of transmission lines based on deep learning. It constructs a 3D image of the external damage source corresponding to each external damage source image using the IGEV algorithm, obtains and predicts the feature image sequence to obtain a predicted 3D image sequence of external damage sources. It acquires first 3D point cloud data corresponding to each actual external damage source and second 3D point cloud data corresponding to each actual transmission line in the external damage source 3D image sequence, and calculates a first distance between the actual external damage source and the actual transmission line. It acquires third 3D point cloud data corresponding to each predicted external damage source and fourth 3D point cloud data corresponding to each predicted transmission line in the predicted external damage source 3D image sequence, and calculates a second distance between the predicted external damage source and the predicted transmission line. Based on the first and second distances, a target early warning signal is triggered. Compared with existing technologies, the technical solution of this invention can achieve timely prediction of external damage source behavior and realize real-time early warning.

Description

Technical Field

[0001] This invention relates to the technical field of external damage prevention for power transmission lines, and in particular to an automatic early warning method and device for external damage prevention of power transmission lines based on deep learning. Background Technology

[0002] With the rapid development of the national economy and the gradual expansion of the State Grid's scale, the damage caused by external forces to its operation has also gradually increased, becoming a major obstacle to the stable operation of the power grid system.

[0003] Currently, the main method for monitoring external damage sources is to install visual monitoring devices near transmission lines and build a visual information system for transmission lines. Through visual monitoring, data transmission, and image recognition, external damage hazards such as construction vehicles under and around transmission lines can be qualitatively identified. Although the above method can qualitatively analyze and obtain information about external damage sources, it cannot perceive specific information such as the size, orientation, and location of external damage hazards, nor can it calculate their clearance distance to the conductor in real time. In addition, the above monitoring system is prone to generating a large amount of redundant data and requires a large amount of manual labor, which brings difficulties to the monitoring work. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for automatic early warning of external damage to transmission lines based on deep learning, so as to realize real-time prediction and real-time early warning of external damage source behavior.

[0005] To address the aforementioned technical problems, this invention provides an automatic early warning method for external damage to transmission lines based on deep learning, comprising:

[0006] Based on the IGEV algorithm, each external impact source image in the external impact source image database is subjected to binocular stereo matching processing to obtain the image depth data corresponding to each external impact source image. Based on the image depth data, an external impact source 3D image corresponding to each external impact source image is constructed to generate an external impact source 3D image sequence.

[0007] Feature extraction is performed on each external 3D image in the external 3D image sequence to obtain a feature image sequence. The feature image sequence is then predicted to obtain a predicted external 3D image sequence.

[0008] Obtain the actual external fault source and actual transmission line corresponding to each external fault source 3D image in the external fault source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external fault source and the second three-dimensional point cloud data corresponding to the actual transmission line;

[0009] Acquire the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line;

[0010] Based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, calculate the first distance between the actual external damage source and the actual transmission line; based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data, calculate the second distance between the predicted external damage source and the predicted transmission line; based on the first distance and the second distance, trigger a target early warning signal.

[0011] In one possible implementation, before performing binocular stereo matching processing on each external damage source image in the external damage source image database based on the IGEV algorithm, the method further includes:

[0012] Real-time transmission line images are acquired using binocular cameras, and target recognition is performed on the real-time transmission line images using the YOLOv7 algorithm. When it is determined that there is an external damage source in the real-time transmission line images, the real-time transmission line images are used as external damage source images and stored in the external damage source image database.

[0013] In one possible implementation, acquiring the actual external fault source and the actual transmission line corresponding to each 3D image of the external fault source in the 3D image sequence, as well as the first 3D point cloud data corresponding to the actual external fault source and the second 3D point cloud data corresponding to the actual transmission line, specifically includes:

[0014] The external source image is obtained for each external source 3D image in the external source 3D image sequence. A two-dimensional detection box is generated for the external source image based on a two-dimensional target detection algorithm. Based on the image depth data of the external source image, the two-dimensional detection box is converted into a three-dimensional detection box to generate a three-dimensional candidate region.

[0015] Based on the back projection technique of the truncated cone, the pixel coordinates in the external destruction source image are mapped to the point cloud space to obtain the three-dimensional point cloud data corresponding to the three-dimensional candidate region;

[0016] The three-dimensional point cloud data is subjected to binary classification to divide the three-dimensional point cloud data into target three-dimensional point clouds and non-target three-dimensional point clouds. Based on the target three-dimensional point clouds and the non-target three-dimensional point clouds, the three-dimensional candidate region is segmented into instances to generate a three-dimensional detection box corresponding to each target. The targets include actual external damage sources and actual power transmission lines.

[0017] Based on the three-dimensional detection frame and the three-dimensional point cloud data, the first three-dimensional point cloud data corresponding to the actual external damage source and the second three-dimensional point cloud data corresponding to the actual transmission line are obtained.

[0018] In one possible implementation, binocular stereo matching is performed on each external damage source image in the external damage source image database based on the IGEV algorithm to obtain the image depth data corresponding to each external damage source image, specifically including:

[0019] Feature extraction is performed on the left and right outer source images of each outer source image in the source image database to obtain the first multi-scale feature corresponding to the left outer source image and the second multi-scale feature corresponding to the right outer source image.

[0020] Feature point matching is performed on the first multi-scale feature and the second multi-scale feature to obtain multiple feature point matching pairs, and the pixel spacing between the multiple feature point matching pairs is calculated.

[0021] Based on the binocular disparity algorithm, disparity is calculated for each feature point in the first multi-scale feature and the second multi-scale feature to obtain the first disparity corresponding to each feature point, and based on the first disparity, a disparity map corresponding to each external damage source image is generated.

[0022] The disparity map is depth-transformed to obtain a depth map. Based on the depth map, the image depth data corresponding to each external damage source image is obtained.

[0023] In one possible implementation, feature extraction is performed on each external impact source 3D image in the external impact source 3D image sequence to obtain a feature image sequence, and prediction is performed on the feature image sequence to obtain a predicted external impact source 3D image sequence, specifically including:

[0024] Based on the CNN model, feature extraction is performed on each external destruction source 3D image in the external destruction source 3D image sequence to obtain the image feature vector corresponding to each external destruction source 3D image. All image feature vectors are integrated to generate a feature image sequence.

[0025] The feature image sequence is input into a pre-trained long short-term memory neural network, so that the long short-term memory neural network can predict the feature image sequence and output the predicted feature vector sequence at the target time.

[0026] The predicted feature vector sequence is converted into a binary sequence, and the binary sequence is subjected to inverse transformation to obtain the predicted external destruction source 3D image sequence.

[0027] In one possible implementation, triggering a target warning signal based on the first distance and the second distance specifically includes:

[0028] Based on the first distance corresponding to each external 3D image in the external 3D image sequence, an actual distance sequence is generated; based on the second distance corresponding to each predicted external 3D image in the predicted external 3D image sequence, a predicted distance sequence is generated.

[0029] Obtain the first minimum distance value in the actual distance sequence, and simultaneously obtain the second minimum distance value in the predicted distance sequence;

[0030] The target warning signal is obtained and triggered by matching the first minimum distance value and the second minimum distance value with a preset warning signal.

[0031] In one possible implementation, based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, the first distance between the actual external fault source and the actual transmission line is calculated, specifically including:

[0032] The second three-dimensional point cloud data is fitted to obtain a fitted straight line for the transmission line.

[0033] Calculate the shortest distance from the first three-dimensional point cloud data to the fitted straight line of the transmission line, and use the shortest distance as the first distance between the actual external damage source and the actual transmission line.

[0034] The present invention also provides an automatic early warning device for external damage to transmission lines based on deep learning, comprising: an external damage source 3D image sequence generation module, an external damage source 3D image sequence prediction module, an actual point cloud data acquisition module, a predicted point cloud data acquisition module, and a distance early warning module;

[0035] The external impact source 3D image sequence generation module is used to perform binocular stereo matching processing on each external impact source image in the external impact source image database based on the IGEV algorithm to obtain the image depth data corresponding to each external impact source image, and to construct an external impact source 3D image corresponding to each external impact source image based on the image depth data, thereby generating an external impact source 3D image sequence.

[0036] The external source 3D image sequence prediction module is used to extract features from each external source 3D image in the external source 3D image sequence to obtain a feature image sequence, and to predict the feature image sequence to obtain a predicted external source 3D image sequence.

[0037] The actual point cloud data acquisition module is used to acquire the actual external source and actual transmission line corresponding to each external source 3D image in the external source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external source and the second three-dimensional point cloud data corresponding to the actual transmission line.

[0038] The predicted point cloud data acquisition module is used to acquire the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line.

[0039] The distance warning module is used to calculate a first distance between the actual external damage source and the actual transmission line based on the first three-dimensional point cloud data and the second three-dimensional point cloud data; calculate a second distance between the predicted external damage source and the predicted transmission line based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data; and trigger a target warning signal based on the first distance and the second distance.

[0040] The present invention provides an automatic early warning device for preventing external damage to transmission lines based on deep learning, which further includes: an external damage source image acquisition module;

[0041] The external damage source image acquisition module is used to acquire real-time transmission line images based on a binocular camera, perform target recognition on the real-time transmission line images based on the YOLOv7 algorithm, and when it is determined that there is an external damage source in the real-time transmission line images, the real-time transmission line images are used as external damage source images and the external damage source images are stored in the external damage source image database.

[0042] In one possible implementation, the actual point cloud data acquisition module is used to acquire the actual external fault source and the actual transmission line corresponding to each 3D image of the external fault source in the external fault source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external fault source and the second three-dimensional point cloud data corresponding to the actual transmission line, specifically including:

[0043] The external source image is obtained for each external source 3D image in the external source 3D image sequence. A two-dimensional detection box is generated for the external source image based on a two-dimensional target detection algorithm. Based on the image depth data of the external source image, the two-dimensional detection box is converted into a three-dimensional detection box to generate a three-dimensional candidate region.

[0044] Based on the back projection technique of the truncated cone, the pixel coordinates in the external destruction source image are mapped to the point cloud space to obtain the three-dimensional point cloud data corresponding to the three-dimensional candidate region;

[0045] The three-dimensional point cloud data is subjected to binary classification to divide the three-dimensional point cloud data into target three-dimensional point clouds and non-target three-dimensional point clouds. Based on the target three-dimensional point clouds and the non-target three-dimensional point clouds, the three-dimensional candidate region is segmented into instances to generate a three-dimensional detection box corresponding to each target. The targets include actual external damage sources and actual power transmission lines.

[0046] Based on the three-dimensional detection frame and the three-dimensional point cloud data, the first three-dimensional point cloud data corresponding to the actual external damage source and the second three-dimensional point cloud data corresponding to the actual transmission line are obtained.

[0047] In one possible implementation, the external impact source 3D image sequence generation module is used to perform binocular stereo matching processing on each external impact source image in the external impact source image database based on the IGEV algorithm to obtain image depth data corresponding to each external impact source image, specifically including:

[0048] Feature extraction is performed on the left and right outer source images of each outer source image in the source image database to obtain the first multi-scale feature corresponding to the left outer source image and the second multi-scale feature corresponding to the right outer source image.

[0049] Feature point matching is performed on the first multi-scale feature and the second multi-scale feature to obtain multiple feature point matching pairs, and the pixel spacing between the multiple feature point matching pairs is calculated.

[0050] Based on the binocular disparity algorithm, disparity is calculated for each feature point in the first multi-scale feature and the second multi-scale feature to obtain the first disparity corresponding to each feature point, and based on the first disparity, a disparity map corresponding to each external damage source image is generated.

[0051] The disparity map is depth-transformed to obtain a depth map. Based on the depth map, the image depth data corresponding to each external damage source image is obtained.

[0052] In one possible implementation, the external impact source 3D image sequence prediction module is used to extract features from each external impact source 3D image in the external impact source 3D image sequence to obtain a feature image sequence, and to predict the feature image sequence to obtain a predicted external impact source 3D image sequence, specifically including:

[0053] Based on the CNN model, feature extraction is performed on each external destruction source 3D image in the external destruction source 3D image sequence to obtain the image feature vector corresponding to each external destruction source 3D image. All image feature vectors are integrated to generate a feature image sequence.

[0054] The feature image sequence is input into a pre-trained long short-term memory neural network, so that the long short-term memory neural network can predict the feature image sequence and output the predicted feature vector sequence at the target time.

[0055] The predicted feature vector sequence is converted into a binary sequence, and the binary sequence is subjected to inverse transformation to obtain the predicted external destruction source 3D image sequence.

[0056] In one possible implementation, the distance warning module is used to trigger a target warning signal based on the first distance and the second distance, specifically including:

[0057] Based on the first distance corresponding to each external 3D image in the external 3D image sequence, an actual distance sequence is generated; based on the second distance corresponding to each predicted external 3D image in the predicted external 3D image sequence, a predicted distance sequence is generated.

[0058] Obtain the first minimum distance value in the actual distance sequence, and simultaneously obtain the second minimum distance value in the predicted distance sequence;

[0059] The target warning signal is obtained and triggered by matching the first minimum distance value and the second minimum distance value with a preset warning signal.

[0060] In one possible implementation, the distance warning module is used to calculate a first distance between the actual external damage source and the actual transmission line based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, specifically including:

[0061] The second three-dimensional point cloud data is fitted to obtain a fitted straight line for the transmission line.

[0062] Calculate the shortest distance from the first three-dimensional point cloud data to the fitted straight line of the transmission line, and use the shortest distance as the first distance between the actual external damage source and the actual transmission line.

[0063] The present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the deep learning-based automatic early warning method for preventing external damage to transmission lines as described in any of the preceding claims.

[0064] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the deep learning-based automatic early warning method for preventing external damage to transmission lines as described in any of the preceding claims.

[0065] This invention provides an automatic early warning method and device for preventing external damage to transmission lines based on deep learning, which has the following advantages compared with the prior art:

[0066] The IGEV algorithm is used to perform binocular stereo matching on each external fault source image in the external fault source image database to obtain image depth data corresponding to each external fault source image. Based on the image depth data, a 3D image of the external fault source corresponding to each external fault source image is constructed, generating a 3D image sequence of the external fault source. Features are extracted from each external fault source 3D image in the external fault source 3D image sequence to obtain a feature image sequence. The feature image sequence is then used for prediction to obtain a predicted external fault source 3D image sequence. The actual external fault source and the actual transmission line corresponding to each external fault source 3D image in the external fault source 3D image sequence, as well as the first three-dimensional point cloud number corresponding to the actual external fault source, are obtained. Based on the second three-dimensional point cloud data corresponding to the actual transmission line; the predicted external fault source and predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line; based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, the first distance between the actual external fault source and the actual transmission line is calculated; based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data, the second distance between the predicted external fault source and the predicted transmission line is calculated; based on the first distance and the second distance, a target warning signal is triggered. Compared with the prior art, the technical solution of the present invention improves the situation where the previous external fault source alarm system could not provide real-time alarm by predicting the behavior of external fault sources, and realizes real-time prediction and real-time warning of external fault source behavior. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating an embodiment of an automatic early warning method for preventing external damage to transmission lines based on deep learning provided by the present invention.

[0068] Figure 2 This is a schematic diagram of an embodiment of an automatic early warning device for preventing external damage to power transmission lines based on deep learning, provided by the present invention. Detailed Implementation

[0069] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Example 1, see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of an automatic early warning method for preventing external damage to transmission lines based on deep learning provided by the present invention. Figure 1As shown, the method includes steps 101-105, as detailed below:

[0071] Step 101: Perform binocular stereo matching processing on each external impact source image in the external impact source image database based on the IGEV algorithm to obtain the image depth data corresponding to each external impact source image, and construct an external impact source 3D image corresponding to each external impact source image based on the image depth data to generate an external impact source 3D image sequence.

[0072] In one embodiment, real-time transmission line images are acquired based on binocular cameras, and target recognition is performed on the real-time transmission line images based on the YOLOv7 algorithm. When it is determined that there is an external damage source in the real-time transmission line images, the real-time transmission line images are used as external damage source images, and the external damage source images are stored in the external damage source image database.

[0073] Preferably, the external sources of damage include large construction machinery and easily entangled objects, wherein the large construction machinery includes excavators and cranes, and the easily entangled objects include kites, advertising banners, balloons, and ribbons.

[0074] In one embodiment, a binocular camera captures real-time images near the power transmission line, and the real-time image data is transmitted via 5G communication. The YOLOv7 algorithm is used on the computer at the transmission terminal to identify external damage targets in the obtained real-time images.

[0075] In one embodiment, after acquiring real-time transmission line images using a binocular camera, image processing is performed on the real-time transmission line images to obtain a first real-time transmission line image. The image processing includes image scaling, image normalization, and image cropping. The first transmission line image is then input into a pre-trained target recognition model loaded with the YOLOv7 algorithm. This allows the target recognition model to extract features from the first transmission line image, obtain image features, and output detection boxes, confidence scores, and categories for each object category based on these image features. Then, a non-maximum suppression (NMS) algorithm is used to remove redundant detection boxes, generating the final target detection result. Based on the target detection result, it is confirmed whether there are external damage sources in the real-time transmission line image.

[0076] Specifically, from an overall perspective, the YOLO algorithm uses a single convolutional neural network model to achieve end-to-end object detection. The process is as follows: first, the input image is adjusted to a size of 448×448, then it is input into the CNN network, and finally the network prediction results are processed to obtain the detected target. Compared with other algorithms, the YOLO algorithm is a unified framework, which is faster and the training process is also end-to-end.

[0077] The YOLO algorithm's CNN network first segments the input image into... Each cell in a grid detects targets whose center point falls within that cell. For example, when detecting a vehicle near a power transmission line, the cell containing the vehicle's center point is responsible for predicting the vehicle's location. Each cell predicts B bounding boxes and their confidence scores. The confidence score has two aspects: the probability that the bounding box contains the target and the accuracy of the bounding box. The former is denoted as B. When the bounding box is the background, i.e., the grid does not contain the target, However, when the bounding box contains a target, The accuracy of bounding boxes is characterized by the intersection-over-union (IOU) ratio between the predicted bounding box and the actual ground truth bounding box, denoted as . Thus, the definition of confidence level is obtained: The size and position of the bounding box can be represented by four values: (x, y, w, h), where (x, y) is the offset of each cell relative to the top-left corner coordinate point, and its unit is relative to the cell size. The predicted values ​​of w and h are the ratio of the width and height of the entire image. Theoretically, the size of these four elements is in the range of [0, 1]. In addition to the above four elements, the predicted value of each bounding box also includes its confidence value c, denoted as (x, y, w, h, c).

[0078] In addition, considering that a cell may contain C categories of objects and their corresponding probability values, these probability values ​​are actually conditional probabilities under the confidence levels of each bounding box, i.e. Based on this, the class-specific confidence scores of the bounding boxes for each category are calculated:

[0079] ;

[0080] The bounding box category confidence score represents the probability that the bounding box belongs to each category and the quality of the bounding box matching the target. Generally, the predicted boxes of the network are filtered based on the category confidence score.

[0081] Therefore, each cell needs to be predicted. A value, if the input image is divided into If the grid is such that the final predicted value is... The size of the tensor. YOLO uses convolutional network layers to extract features from each frame of the image, and then uses a fully connected layer to calculate and output the predicted values. Based on the output predicted values, it determines the objects present in the image and marks their bounding boxes, thus realizing the recognition of objects in real-time transmission line images.

[0082] In one embodiment, target recognition is performed on the real-time transmission line image based on the YOLOv7 algorithm. If it is determined that there is no external damage source in the real-time transmission line image, the real-time transmission line image is discarded.

[0083] In one embodiment, the IGVE-Stereo algorithm framework consists of a multi-scale feature extractor, a combined geometry encoding volume, an update operator based on a convolutional gated recurrent unit neural network (ConvGRU), and a spatial upsampling module.

[0084] In one embodiment, when performing binocular stereo matching processing on each external source image in the external source image database based on the IGEV algorithm to obtain the image depth data corresponding to each external source image, feature extraction is performed on the left and right external source images of each external source image in the database to obtain a first multi-scale feature corresponding to the left external source image and a second multi-scale feature corresponding to the right external source image; feature point matching is performed on the first and second multi-scale features to obtain multiple feature point matching pairs, and the pixel spacing between the multiple feature point matching pairs is calculated; based on the binocular disparity algorithm, disparity is calculated on each feature point in the first and second multi-scale features to obtain a first disparity corresponding to each feature point, and a disparity map corresponding to each external source image is generated based on the first disparity; the disparity map is depth-converted to obtain a depth map, and the image depth data corresponding to each external source image is obtained based on the depth map.

[0085] Specifically, the disparity map is depth-transformed based on a depth transformation formula, which is as follows:

[0086] ;

[0087] In the formula, For depth, For the focal length of the binocular camera, For the baseline of the binocular camera, For parallax.

[0088] Specifically, an IGVE-Stereo model is constructed based on the IGVE-Stereo algorithm framework. After the left and right lateral damage source images acquired by the binocular camera are input into the IGVE-Stereo model, they first enter the multi-scale feature extractor. The multi-scale feature extractor consists of two parts: a feature network, which is used to extract multi-scale features relative to the left and right sides to establish the cost volume and guide cost aggregation; and a context network, which is used to extract multi-scale context features to initialize and update the hidden state of ConvGRU.

[0089] The feature network uses MobileNet V2 pre-trained on ImageNet to downsample the input to 1 / 32, and then upsamples it to obtain multi-scale features:

[0090] ;

[0091] in, and Used to build cost volume This is the first multi-scale feature corresponding to the left external rupture source image. This is the second multi-scale feature corresponding to the right lateral rupture source image.

[0092] The context network consists of a series of residual blocks and downsampling layers, generating multi-scale context features at 1 / 4, 1 / 8, and 1 / 16 of the input channel image resolution. These multi-scale context features are used to initialize the hidden state of the ConvGRU-based update operator and are inserted into the ConvGRU at each iteration.

[0093] In obtaining and After two multi-scale features, Eye channel dimensions are divided into For each group, correlation maps are calculated to construct a 4-dimensional group-wise correlation volume, calculated using the following formula:

[0094] .

[0095] The above results are further processed using a lightweight 3D regularized network to obtain a better ability to capture the global set and structure, resulting in a geometric coding quantity. The calculation is as follows:

[0096] .

[0097] The 3D regularized network R is a lightweight 3D UNet consisting of three downsampling blocks and three upsampling blocks. The IGVE algorithm calculates weights from the feature map of the left reference image in the relative position to activate the cost channels and performs cost aggregation to obtain the Guided Cost Volume, which is calculated as follows:

[0098] ;

[0099] In the formula, For sigmoid activation function, This is the Hadamard Product. and When combined, they form a set of combinations and a coding quantity.

[0100] Using soft parameters (soft argmin) from Regress an initial disparity The calculation process is as follows:

[0101] ;

[0102] Where d is the parallax index at 1 / 4 resolution.

[0103] From the initial input parallax Initially, a three-level ConvGRU is used to iteratively update the disparity, and the hidden state of the three-level ConvGRU is initialized using multi-scale contextual features. In each iteration, the current disparity is used... By indexing the combination set and the coding quantity through linear interpolation, a set of features is generated. The calculation is expressed as:

[0104] ;

[0105] In the formula, r is the index radius, and p represents the pooling operation.

[0106] These geometric features and current disparity prediction After passing through two coding layers and Connection formation The formula is as follows:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] In the formula, , , Context features generated from the context, based on the hidden state Disparity residuals obtained through two convolutional layers Then update the current parallax:

[0113] ;

[0114] Finally, using the predicted disparity at 1 / 4 resolution A resolution disparity map is generated by weighted combination, and finally the disparity map is converted into a depth map based on the relationship between the disparity map and the depth map.

[0115] In one embodiment, based on the image depth data, a 3D image of the external source corresponding to each external source image is constructed, and a sequence of 3D images of the external source is generated based on the acquisition time corresponding to each external source image.

[0116] Step 102: Extract features from each external 3D image in the external 3D image sequence to obtain a feature image sequence, and predict the feature image sequence to obtain a predicted external 3D image sequence.

[0117] In one embodiment, features are extracted from each external impact source 3D image in the external impact source 3D image sequence based on a CNN model to obtain an image feature vector corresponding to each external impact source 3D image. All image feature vectors are integrated to generate a feature image sequence. The feature image sequence is input into a pre-trained long short-term memory neural network so that the long short-term memory neural network can predict the feature image sequence and output a predicted feature vector sequence at the target time. The predicted feature vector sequence is converted into a binary sequence, and the binary sequence is subjected to inverse transformation to obtain the predicted external impact source 3D image sequence.

[0118] Specifically, the Convolutional Neural Network (CNN) model can be used to compress impact data and extract features. When processing depth maps, in addition to the RGB information of the image itself, depth information needs to be compressed as a fourth channel. A typical CNN includes: convolutional layers, subsampling layers, fully connected layers, and activation function layers.

[0119] The purpose of convolutional layers is to extract different features from the input. Some convolutional layers may only extract low-level features such as edges, lines, and corners. Multi-layered networks can iterate from low-level features to extract more complex features. A convolutional layer includes the following main parameters: kernel size, padding, and stride. During convolution, the kernel slides across the image according to the stride. Within the image region covered by each kernel, the original pixel values ​​are multiplied by the kernel value and summed. The result is the image compression and feature extraction achieved through convolution. Padding is designed to be the pixel values ​​at image boundaries, preventing the loss of edge pixels during feature extraction. When there are multiple channels, the convolutional kernel needs to have the same number of channels as the original image. Each channel's kernel is calculated with the corresponding input data for each channel. The formulas involved in the convolution process are as follows:

[0120] ;

[0121] In the formula, W is the input dimension, that is, the input image is W. W, padding value P, kernel size F F, where the output data dimension is N. N.

[0122] Pooling layers are essentially a form of downsampling. There are various forms of pooling layers, with max pooling being the most common. It divides the input image into several rectangular regions and outputs the maximum value for each sub-region. This mechanism is effective because, after discovering a feature, its precise location is far less important than its relative position to other features. Pooling layers continuously reduce the spatial size of the data, thus decreasing the number of parameters and computational cost, which to some extent controls overfitting. Typically, pooling layers are periodically inserted between convolutional layers in a CNN.

[0123] The activation function layer and fully connected layer are placed after the max pooling layer. These two layers then output the compressed image features of the convolutional neural network. By stacking multiple convolutional neural network layers with the above structure, the input image can be compressed into the required image features.

[0124] In this embodiment, a convolutional neural network (CNN) is used to compress the input image into a one-dimensional vector. The resulting one-dimensional feature vectors are then merged in sequence to obtain a feature vector time series. In other words, all image feature vectors are integrated in chronological order to generate a feature image sequence.

[0125] In one embodiment, a Long Short-Term Memory (LSTM) neural network is a type of neural network developed based on a Recurrent Neural Network (RNN). Compared to RNNs, it has more hidden neurons, solving the problems of gradient vanishing and gradient exploding during long-term training. The LSTM network calculates the output and current state of the current neuron by inputting the output state of the previous neuron along with the input data of the current neuron. This current state is then input into the neuron at the next time step, and multiple layers are stacked for prediction.

[0126] An LSTM neural network contains a hidden layer line that transmits the previous time step's hidden state within a neuron. The current state consists of a conveyor belt and three gates: a forget gate, an input gate, and a transmission gate, all composed of activation functions. These three gates together determine the input neuron state at the next time step. The changes occur. The forget gate is used to filter redundant information, modifying the input of the current hidden layer neuron. Input to the neuron at the previous moment Compress to [0, 1], where 1 represents accepting all information and 0 represents forgetting all information. The calculation formula is as follows:

[0127] ;

[0128] In the formula, This represents the neuron input at the previous moment. This represents the input at the current moment. This represents the forget gate weight matrix. This is a bias term.

[0129] In and After the current neuron receives the input, the input gate uses the Sigmoid activation function to determine the required information, and together with the tanh function, calculates an updated candidate state value. Then, the information retained by the forget gate, along with the neuron's state information from the previous moment and the update candidate value, are added to obtain the new... The calculation process is as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] in, This represents the amount of neuron state updates. This represents the update coefficient.

[0134] The output layer determines the output portion of the state using the Sigmod function, compresses the cell state to [0, 1] using the tanh function, and then multiplies it by the Sigmod result to obtain the cell output. The calculation process is as follows:

[0135] ;

[0136] .

[0137] Through a series of calculations involving the forget gate, input gate, and output gate, and repeated training, the functional relationship between the data can be derived. Finally, the input values ​​of the hidden layer are... The predicted value is obtained by performing a fully connected layer:

[0138] .

[0139] The time series of image feature vectors extracted by the convolutional neural network is input into the LSTM network. After repeated training, the feature vectors of the next time step can be obtained. After deconvolution and upsampling, the feature vectors can be restored to the image.

[0140] Step 103: Obtain the actual external fault source and actual transmission line corresponding to each external fault source 3D image in the external fault source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external fault source and the second three-dimensional point cloud data corresponding to the actual transmission line.

[0141] In one embodiment, three-dimensional point cloud data is generated from the existing external damage source 3D image sequence. Specifically, the Frustum PointNets algorithm is used to identify and segment the external damage source and transmission line corresponding to each external damage source 3D image in the external damage source 3D image sequence, and generate three-dimensional point clouds for the external damage source and the transmission line separately, thereby reducing the computation time required.

[0142] In one embodiment, the model structure using the Frustum PointNets algorithm performs 3D detection using 2D images and point cloud data. First, a detection box is generated in the 2D image. Then, combined with the image depth information, a 3D candidate region is mapped onto the point cloud using a truncated cone backprojection, generating 3D point cloud data. In the second stage, the 3D point cloud data is used to segment the candidate region found in the previous stage, and finally, a 3D detection box is generated, realizing the generation and detection segmentation of the 3D point cloud.

[0143] In one embodiment, the Frustum PointNets model mainly consists of three parts. The first part processes the RGB and depth data of the input 2D image, using truncated cone back projection to generate 3D candidate regions and 3D point cloud data. Specifically, the Frustum PointNets model uses a 2D CNN to obtain 2D detection boxes of target objects on the 2D RGB image, i.e., performing object detection on the 2D RGB image, and then combining the depth information to map the 2D detection boxes into a 3D detection box. The 2D object detection uses an FPN architecture, which mainly consists of stacked CNN layers with semantic information, constructing a pyramid structure to detect target objects and generate detection boxes. The input image is passed to the bottom of the pyramid structure, and then compressed layer by layer from bottom to top through the stacked CNN layers, finally obtaining the 2D detection boxes of the target objects. Using a projection matrix, the 2D detection boxes can be projected into a 3D detection box. Combining the depth information and the 2D information inherent in the RGB image, the target object points distributed in three-dimensional space can be obtained.

[0144] Specifically, the external source image corresponding to each external source 3D image in the external source 3D image sequence is obtained, a two-dimensional detection box of the external source image is generated based on a two-dimensional target detection algorithm, and the two-dimensional detection box is converted into a three-dimensional detection box based on the image depth data of the external source image to generate a three-dimensional candidate region.

[0145] Specifically, based on the back projection technique of the truncated cone, the pixel coordinates in the external destruction source image are mapped to the point cloud space to obtain the three-dimensional point cloud data corresponding to the three-dimensional candidate region.

[0146] In one embodiment, the second part of the Frustum PointNets model is an instance segmentation network. The Frustum PointNets model uses the PointNet network to perform binary classification on 3D point cloud data in 3D space, that is, to divide the target point cloud into target points and non-target points, thus obtaining the target point cloud. The PointNet network takes point cloud coordinate data as input. The position of the stereo camera is known, and the distance from the point cloud to the stereo camera can be obtained from the depth information. Using two adjacent stereo cameras, the coordinates of the point cloud can be calculated based on the principle of lateral intersection. The PointNet network takes point cloud coordinate data as input, first uses a multilayer perceptron neural network (i.e., a multilayer feedforward neural network) to perform dimensionality increase on the input data, then uses a max pooling layer to obtain global features, and combines local features to classify the point cloud. Finally, the target point cloud is retained and the non-target point cloud is deleted to obtain the 3D point cloud of the target object.

[0147] In one embodiment, the final part of the Frustum PointNets model utilizes a PointNet-based TNet network to correct and align the input points, and obtains the 3D detection box through a regression prediction network.

[0148] Specifically, the three-dimensional point cloud data is subjected to binary classification to divide the three-dimensional point cloud data into target three-dimensional point clouds and non-target three-dimensional point clouds. Based on the target three-dimensional point clouds and the non-target three-dimensional point clouds, the three-dimensional candidate region is segmented into instances to generate a three-dimensional detection box corresponding to each target. The targets include actual external damage sources and actual power transmission lines.

[0149] Specifically, based on the three-dimensional detection frame and the three-dimensional point cloud data, the first three-dimensional point cloud data corresponding to the actual external damage source and the second three-dimensional point cloud data corresponding to the actual transmission line are obtained.

[0150] In one embodiment, by pre-identifying targets and segmenting targets in 3D point cloud generation, the problems of traditional external damage source alarm systems, such as the need for long-term manual monitoring and a large amount of redundant useless data, are solved.

[0151] Step 104: Obtain the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line.

[0152] In one embodiment, the predicted external fault source 3D image sequence is used to generate three-dimensional point cloud data. Specifically, the Frustum PointNets algorithm is used to identify and segment the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, and generate three-dimensional point clouds for the predicted external fault source and the predicted transmission line separately, thereby reducing the computation time required.

[0153] In one embodiment, the predicted external source image corresponding to each predicted external source 3D image in the predicted external source 3D image sequence is obtained, a two-dimensional detection box of the predicted external source image is generated based on a two-dimensional target detection algorithm, and the two-dimensional detection box is converted into a three-dimensional detection box based on the predicted image depth data of the predicted external source image to generate a predicted three-dimensional candidate region.

[0154] In one embodiment, based on the back projection technique of a truncated cone, the pixel coordinates in the predicted external source image are mapped to the point cloud space to obtain the predicted three-dimensional point cloud data corresponding to the predicted three-dimensional candidate region.

[0155] In one embodiment, the predicted 3D point cloud data is subjected to binary classification processing to divide the predicted 3D point cloud data into predicted target 3D point clouds and predicted non-target 3D point clouds. Based on the predicted target 3D point clouds and the predicted non-target 3D point clouds, the predicted 3D candidate region is segmented into instances to generate a predicted 3D detection box corresponding to each predicted target. The predicted targets include predicted external damage sources and predicted transmission lines.

[0156] In one embodiment, based on the predicted three-dimensional detection frame and the predicted three-dimensional point cloud data, the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted actual transmission line are obtained.

[0157] In one embodiment, by pre-identifying targets and segmenting targets in 3D point cloud generation, the problems of traditional external damage source alarm systems, such as the need for long-term manual monitoring and a large amount of redundant useless data, are solved.

[0158] Step 105: Based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, calculate the first distance between the actual external damage source and the actual transmission line; based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data, calculate the second distance between the predicted external damage source and the predicted transmission line; based on the first distance and the second distance, trigger a target warning signal.

[0159] In one embodiment, the second three-dimensional point cloud data is fitted to obtain a fitted straight line for the transmission line; the shortest distance from the first three-dimensional point cloud data to the fitted straight line for the transmission line is calculated, and the shortest distance is used as the first distance between the actual external damage source and the actual transmission line.

[0160] Specifically, after obtaining the three-dimensional point cloud data corresponding to the actual external damage source and the actual transmission line, the spatial straight line equation of the actual transmission line can be fitted using the second three-dimensional point cloud data of the actual transmission line, so as to convert the actual transmission line into a spatial straight line and obtain the fitted straight line of the transmission line. The shortest distance between the first three-dimensional point cloud data corresponding to the actual external damage source and the fitted straight line of the transmission line can be directly calculated.

[0161] In one embodiment, the fourth three-dimensional point cloud data is fitted to obtain a predicted transmission line fitting line; the predicted shortest distance from the third three-dimensional point cloud data to the predicted transmission line fitting line is calculated, and the predicted shortest distance is used as the second distance between the predicted external fault source and the predicted transmission line.

[0162] Specifically, after obtaining the three-dimensional point cloud data corresponding to the predicted external damage source and the predicted transmission line, the spatial straight line equation of the predicted transmission line can be fitted using the fourth three-dimensional point cloud data of the predicted transmission line, so as to convert the predicted transmission line into a spatial straight line and obtain the fitted straight line of the predicted transmission line. The predicted shortest distance between the third three-dimensional point cloud data corresponding to the predicted external damage source and the fitted straight line of the predicted transmission line can be directly calculated.

[0163] In one embodiment, based on a preset formula for the distance from a spatial point to a straight line, the shortest distance from the first three-dimensional point cloud data to the fitted straight line of the transmission line is calculated, and the predicted shortest distance from the third three-dimensional point cloud data to the predicted fitted straight line of the transmission line is calculated; wherein, the formula for the distance from a spatial point to a straight line is as follows:

[0164] ;

[0165] In the formula, ( , , ) represents the spatial location of the external source point cloud, and A, B, C, and D are the coefficients of the fitted straight line of the transmission line.

[0166] In one embodiment, an actual distance sequence is generated based on the first distance corresponding to each external 3D image in the external 3D image sequence.

[0167] In one embodiment, a predicted distance sequence is generated based on the second distance corresponding to each predicted external hazard 3D image in the predicted external hazard 3D image sequence.

[0168] In one embodiment, a first minimum distance value in the actual distance sequence is obtained, and a second minimum distance value in the predicted distance sequence is also obtained; the first minimum distance value and the second minimum distance value are matched with a preset warning signal to obtain and trigger a target warning signal.

[0169] Specifically, by using the latest distance values ​​and future predicted values ​​recorded in the obtained actual distance sequence and predicted distance sequence, different levels of early warning are triggered according to the distance. The early warning is issued simultaneously at the site and at the control terminal. The control terminal will display the specific location of the early warning site and the current shortest distance and the predicted shortest distance at the next moment.

[0170] In one embodiment, the preset warning signals include a yellow warning signal, an orange warning signal, and a red warning signal; wherein, the yellow warning signal is when the shortest distance between the external damage source and the transmission line is greater than 3 meters and less than or equal to 5 meters; the orange warning signal is when the shortest distance between the external damage source and the transmission line is greater than 1 meter and less than or equal to 3 meters; and the red warning signal is when the shortest distance between the external damage source and the transmission line is less than or equal to 1 meter.

[0171] Specifically, when the shortest distance between the external fault source and the transmission line is greater than 3 meters and less than or equal to 5 meters, a yellow alarm will be sounded simultaneously at the site and the control terminal; when the shortest distance between the external fault source and the transmission line is greater than 1 meter and less than or equal to 3 meters, an orange alarm will be sounded simultaneously at the site and the control terminal; and when the shortest distance between the external fault source and the transmission line is less than or equal to 1 meter, a red alarm will be sounded simultaneously at the site and the control terminal.

[0172] Example 2, see Figure 2 , Figure 2 This is a schematic diagram of an embodiment of an automatic early warning device for external damage to power transmission lines based on deep learning, provided by the present invention. Figure 2 As shown, the device includes an external destruction source 3D image sequence generation module 201, an external destruction source 3D image sequence prediction module 202, an actual point cloud data acquisition module 203, a predicted point cloud data acquisition module 204, and a distance warning module 205, as detailed below:

[0173] The external impact source 3D image sequence generation module 201 is used to perform binocular stereo matching processing on each external impact source image in the external impact source image database based on the IGEV algorithm to obtain the image depth data corresponding to each external impact source image, and to construct an external impact source 3D image corresponding to each external impact source image based on the image depth data, thereby generating an external impact source 3D image sequence.

[0174] The external impact source 3D image sequence prediction module 202 is used to extract features from each external impact source 3D image in the external impact source 3D image sequence to obtain a feature image sequence, and to predict the feature image sequence to obtain a predicted external impact source 3D image sequence.

[0175] The actual point cloud data acquisition module 203 is used to acquire the actual external damage source and the actual transmission line corresponding to each external damage source 3D image in the external damage source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external damage source and the second three-dimensional point cloud data corresponding to the actual transmission line.

[0176] The predicted point cloud data acquisition module 204 is used to acquire the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line.

[0177] The distance warning module 205 is used to calculate the first distance between the actual external damage source and the actual transmission line based on the first three-dimensional point cloud data and the second three-dimensional point cloud data; calculate the second distance between the predicted external damage source and the predicted transmission line based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data; and trigger a target warning signal based on the first distance and the second distance.

[0178] The automatic early warning device for preventing external damage to transmission lines based on deep learning provided in this embodiment also includes: an external damage source image acquisition module.

[0179] In one embodiment, the external damage source image acquisition module is used to acquire real-time transmission line images based on a binocular camera, perform target recognition on the real-time transmission line images based on the YOLOv7 algorithm, and when it is determined that there is an external damage source in the real-time transmission line images, the real-time transmission line images are used as external damage source images, and the external damage source images are stored in the external damage source image database.

[0180] In one embodiment, the actual point cloud data acquisition module 203 is used to acquire the actual external fault source and the actual transmission line corresponding to each external fault source 3D image in the external fault source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external fault source and the second three-dimensional point cloud data corresponding to the actual transmission line. Specifically, this includes: acquiring the external fault source image corresponding to each external fault source 3D image in the external fault source 3D image sequence; generating a two-dimensional detection box for the external fault source image based on a two-dimensional target detection algorithm; converting the two-dimensional detection box into a three-dimensional detection box based on the image depth data of the external fault source image to generate a three-dimensional candidate region; and performing back projection based on a truncated cone. The technique maps the pixel coordinates in the external damage source image to a point cloud space to obtain the 3D point cloud data corresponding to the 3D candidate region; performs binary classification processing on the 3D point cloud data to divide it into target 3D point clouds and non-target 3D point clouds; performs instance segmentation on the 3D candidate region based on the target 3D point clouds and the non-target 3D point clouds to generate a 3D detection box corresponding to each target, wherein the targets include actual external damage sources and actual transmission lines; and obtains the first 3D point cloud data corresponding to the actual external damage source and the second 3D point cloud data corresponding to the actual transmission line based on the 3D detection box and the 3D point cloud data.

[0181] In one embodiment, the external impact source 3D image sequence generation module 201 is used to perform binocular stereo matching processing on each external impact source image in the external impact source image database based on the IGEV algorithm to obtain image depth data corresponding to each external impact source image. Specifically, it includes: extracting features from the left and right external impact source images of each external impact source image in the impact source image database to obtain a first multi-scale feature corresponding to the left external impact source image and a second multi-scale feature corresponding to the right external impact source image; performing feature point matching on the first and second multi-scale features to obtain multiple feature point matching pairs, and calculating the pixel spacing between the multiple feature point matching pairs; calculating the disparity for each feature point in the first and second multi-scale features based on the binocular disparity algorithm to obtain a first disparity corresponding to each feature point, and generating a disparity map corresponding to each external impact source image based on the first disparity; performing depth transformation on the disparity map to obtain a depth map, and obtaining image depth data corresponding to each external impact source image based on the depth map.

[0182] In one embodiment, the external impact source 3D image sequence prediction module 202 is used to extract features from each external impact source 3D image in the external impact source 3D image sequence to obtain a feature image sequence, and to predict the feature image sequence to obtain a predicted external impact source 3D image sequence. Specifically, this includes: extracting features from each external impact source 3D image in the external impact source 3D image sequence based on a CNN model to obtain an image feature vector corresponding to each external impact source 3D image; integrating all image feature vectors to generate a feature image sequence; inputting the feature image sequence into a pre-trained long short-term memory neural network to enable the long short-term memory neural network to predict the feature image sequence and output a predicted feature vector sequence at a target time; converting the predicted feature vector sequence into a binary sequence; and performing an inverse transformation on the binary sequence to obtain the predicted external impact source 3D image sequence.

[0183] In one embodiment, the distance warning module 205 is used to trigger a target warning signal based on the first distance and the second distance, specifically including: generating an actual distance sequence based on the first distance corresponding to each external 3D image in the external 3D image sequence; generating a predicted distance sequence based on the second distance corresponding to each predicted external 3D image in the predicted external 3D image sequence; obtaining the minimum value of the first distance in the actual distance sequence and simultaneously obtaining the minimum value of the second distance in the predicted distance sequence; matching the minimum value of the first distance and the minimum value of the second distance with a preset warning signal to obtain and trigger the target warning signal.

[0184] In one embodiment, the distance warning module 205 is used to calculate the first distance between the actual external damage source and the actual transmission line based on the first three-dimensional point cloud data and the second three-dimensional point cloud data. Specifically, it includes: performing fitting processing on the second three-dimensional point cloud data to obtain a fitted straight line of the transmission line; calculating the shortest distance from the first three-dimensional point cloud data to the fitted straight line of the transmission line, and using the shortest distance as the first distance between the actual external damage source and the actual transmission line.

[0185] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0186] It should be noted that the above-described embodiment of the automatic early warning device for external damage to transmission lines based on deep learning is merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Based on the above-described embodiments of the automatic early warning method for external damage prevention of transmission lines based on deep learning, another embodiment of the present invention provides an automatic early warning terminal device for external damage prevention of transmission lines based on deep learning. This automatic early warning terminal device for external damage prevention of transmission lines based on deep learning includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the automatic early warning method for external damage prevention of transmission lines based on deep learning according to any embodiment of the present invention.

[0188] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the deep learning-based automatic early warning terminal device for preventing external damage to power transmission lines.

[0189] The deep learning-based automatic early warning terminal device for external damage to transmission lines can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and a memory.

[0190] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the deep learning-based automatic early warning terminal device for external damage to power transmission lines, connecting all parts of the device via various interfaces and lines.

[0191] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the deep learning-based automatic early warning terminal device for external damage prevention of power transmission lines by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0192] Based on the above embodiments of the automatic early warning method for external damage prevention of transmission lines based on deep learning, another embodiment of the present invention provides a storage medium, the storage medium including a stored computer program, wherein, when the computer program is running, the device where the storage medium is located executes the automatic early warning method for external damage prevention of transmission lines based on deep learning of any embodiment of the present invention.

[0193] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0194] In summary, this invention provides an automatic early warning method and device for external damage prevention of transmission lines based on deep learning. It uses the IGEV algorithm to perform binocular stereo matching processing on each external damage source image in the external damage source image database to obtain image depth data corresponding to each external damage source image. Based on the image depth data, it constructs a 3D image of the external damage source corresponding to each external damage source image, generating a 3D image sequence of the external damage source. Features are extracted from each 3D image of the external damage source in the 3D image sequence to obtain a feature image sequence. The feature image sequence is then used for prediction to obtain a predicted 3D image sequence of the external damage source. Finally, the actual external damage source and the actual transmission line corresponding to each 3D image of the external damage source in the 3D image sequence are obtained. The system acquires the first three-dimensional point cloud data corresponding to the actual external fault source and the second three-dimensional point cloud data corresponding to the actual transmission line in the predicted external fault source 3D image sequence; it also acquires the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line; based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, it calculates the first distance between the actual external fault source and the actual transmission line; based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data, it calculates the second distance between the predicted external fault source and the predicted transmission line; and based on the first distance and the second distance, it triggers a target warning signal. Compared with the prior art, the technical solution of the present invention improves the situation where the previous external fault source alarm system could not provide real-time alarms by predicting the behavior of external fault sources, and realizes real-time prediction and real-time warning of external fault source behavior.

[0195] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based automatic early warning method for external damage to transmission lines, characterized in that, include: Based on the IGEV algorithm, each external impact source image in the external impact source image database is subjected to binocular stereo matching processing to obtain the image depth data corresponding to each external impact source image. Based on the image depth data, an external impact source 3D image corresponding to each external impact source image is constructed to generate an external impact source 3D image sequence. Feature extraction is performed on each external 3D image in the external 3D image sequence to obtain a feature image sequence. The feature image sequence is then predicted to obtain a predicted external 3D image sequence. The actual external fault source and actual transmission line corresponding to each 3D image of the external fault source in the 3D image sequence are obtained, as well as the first 3D point cloud data corresponding to the actual external fault source and the second 3D point cloud data corresponding to the actual transmission line, specifically as follows: The external source image is obtained for each external source 3D image in the external source 3D image sequence. A two-dimensional detection box is generated for the external source image based on a two-dimensional target detection algorithm. Based on the image depth data of the external source image, the two-dimensional detection box is converted into a three-dimensional detection box to generate a three-dimensional candidate region. Based on the back projection technique of the truncated cone, the pixel coordinates in the external destruction source image are mapped to the point cloud space to obtain the three-dimensional point cloud data corresponding to the three-dimensional candidate region; The three-dimensional point cloud data is subjected to binary classification to divide the three-dimensional point cloud data into target three-dimensional point clouds and non-target three-dimensional point clouds. Based on the target three-dimensional point clouds and the non-target three-dimensional point clouds, the three-dimensional candidate region is segmented into instances to generate a three-dimensional detection box corresponding to each target. The targets include actual external damage sources and actual power transmission lines. Based on the three-dimensional detection frame and the three-dimensional point cloud data, the first three-dimensional point cloud data corresponding to the actual external damage source and the second three-dimensional point cloud data corresponding to the actual transmission line are obtained. Acquire the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line; Based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, calculate the first distance between the actual external damage source and the actual transmission line; based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data, calculate the second distance between the predicted external damage source and the predicted transmission line; based on the first distance and the second distance, trigger a target early warning signal.

2. The method for automatic early warning of external damage to transmission lines based on deep learning as described in claim 1, characterized in that, Before performing binocular stereo matching processing on each external damage source image in the external damage source image database based on the IGEV algorithm, the following steps are also included: Real-time transmission line images are acquired using binocular cameras, and target recognition is performed on the real-time transmission line images using the YOLOv7 algorithm. When it is determined that there is an external damage source in the real-time transmission line images, the real-time transmission line images are used as external damage source images and stored in the external damage source image database.

3. The method for automatic early warning of external damage to transmission lines based on deep learning as described in claim 1, characterized in that, Based on the IGEV algorithm, binocular stereo matching is performed on each external damage source image in the external damage source image database to obtain the image depth data corresponding to each external damage source image, specifically including: Feature extraction is performed on the left and right outer source images of each outer source image in the source image database to obtain the first multi-scale feature corresponding to the left outer source image and the second multi-scale feature corresponding to the right outer source image. Feature point matching is performed on the first multi-scale feature and the second multi-scale feature to obtain multiple feature point matching pairs, and the pixel spacing between the multiple feature point matching pairs is calculated. Based on the binocular disparity algorithm, disparity is calculated for each feature point in the first multi-scale feature and the second multi-scale feature to obtain the first disparity corresponding to each feature point, and based on the first disparity, a disparity map corresponding to each external damage source image is generated. The disparity map is depth-transformed to obtain a depth map. Based on the depth map, the image depth data corresponding to each external damage source image is obtained.

4. The method for automatic early warning of external damage to transmission lines based on deep learning as described in claim 1, characterized in that, Feature extraction is performed on each external impact source 3D image in the external impact source 3D image sequence to obtain a feature image sequence. Prediction is then performed on the feature image sequence to obtain a predicted external impact source 3D image sequence, specifically including: Based on the CNN model, feature extraction is performed on each external destruction source 3D image in the external destruction source 3D image sequence to obtain the image feature vector corresponding to each external destruction source 3D image. All image feature vectors are integrated to generate a feature image sequence. The feature image sequence is input into a pre-trained long short-term memory neural network, so that the long short-term memory neural network can predict the feature image sequence and output the predicted feature vector sequence at the target time. The predicted feature vector sequence is converted into a binary sequence, and the binary sequence is subjected to inverse transformation to obtain the predicted external destruction source 3D image sequence.

5. The method for automatic early warning of external damage to transmission lines based on deep learning as described in claim 1, characterized in that, Based on the first distance and the second distance, triggering a target warning signal specifically includes: Based on the first distance corresponding to each external 3D image in the external 3D image sequence, an actual distance sequence is generated; based on the second distance corresponding to each predicted external 3D image in the predicted external 3D image sequence, a predicted distance sequence is generated. Obtain the first minimum distance value in the actual distance sequence, and simultaneously obtain the second minimum distance value in the predicted distance sequence; The target warning signal is obtained and triggered by matching the first minimum distance value and the second minimum distance value with a preset warning signal.

6. The method for automatic early warning of external damage to transmission lines based on deep learning as described in claim 1, characterized in that, Based on the first three-dimensional point cloud data and the second three-dimensional point cloud data, the first distance between the actual external damage source and the actual transmission line is calculated, specifically including: The second three-dimensional point cloud data is fitted to obtain a fitted straight line for the transmission line. Calculate the shortest distance from the first three-dimensional point cloud data to the fitted straight line of the transmission line, and use the shortest distance as the first distance between the actual external damage source and the actual transmission line.

7. A deep learning-based automatic early warning device for external damage to power transmission lines, characterized in that, include: The module includes a 3D image sequence generation module for external sources, a 3D image sequence prediction module for external sources, a real point cloud data acquisition module, a predicted point cloud data acquisition module, and a distance warning module. The external impact source 3D image sequence generation module is used to perform binocular stereo matching processing on each external impact source image in the external impact source image database based on the IGEV algorithm to obtain the image depth data corresponding to each external impact source image, and to construct an external impact source 3D image corresponding to each external impact source image based on the image depth data, thereby generating an external impact source 3D image sequence. The external source 3D image sequence prediction module is used to extract features from each external source 3D image in the external source 3D image sequence to obtain a feature image sequence, and to predict the feature image sequence to obtain a predicted external source 3D image sequence. The actual point cloud data acquisition module is used to acquire the actual external source and actual transmission line corresponding to each external source 3D image in the external source 3D image sequence, as well as the first three-dimensional point cloud data corresponding to the actual external source and the second three-dimensional point cloud data corresponding to the actual transmission line. The predicted point cloud data acquisition module is used to acquire the predicted external fault source and the predicted transmission line corresponding to each predicted external fault source 3D image in the predicted external fault source 3D image sequence, as well as the third three-dimensional point cloud data corresponding to the predicted external fault source and the fourth three-dimensional point cloud data corresponding to the predicted transmission line. The distance warning module is used to calculate a first distance between the actual external damage source and the actual transmission line based on the first three-dimensional point cloud data and the second three-dimensional point cloud data; calculate a second distance between the predicted external damage source and the predicted transmission line based on the third three-dimensional point cloud data and the fourth three-dimensional point cloud data; and trigger a target warning signal based on the first distance and the second distance.

8. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the deep learning-based automatic early warning method for preventing external damage to transmission lines as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the automatic early warning method for external damage prevention of transmission lines based on deep learning as described in any one of claims 1 to 6.

Images