A power transmission channel construction machinery intrusion hidden end-to-end early warning method and system

Abstract

The application discloses a power transmission channel construction machinery invasion hidden danger end-to-end early warning method and system, and relates to the technical field of high-voltage power transmission line safety monitoring.The three-dimensional point cloud data and the monitored visible light images of the power transmission corridor are collected, the visible light images and the three-dimensional point cloud data are automatically matched and camera pose estimation is carried out based on an image-point cloud intelligent registration model based on an attention mechanism; the construction machinery target in the visible light images of the power transmission corridor is subjected to instance segmentation, and the category and two-dimensional position information of the hidden danger target in the images are obtained; based on the image-point cloud registration result and the construction machinery instance segmentation result, the three-dimensional space position of the construction machinery target in the visible light images is reconstructed, the relative position of the construction machinery hidden danger target and the power transmission corridor is obtained; the online monitoring device pose information is obtained, and the visible light images captured in real time by the online monitoring device are used to early warn the dangerous target in the power transmission corridor.

Description

Technical Field

[0001] This invention relates to the field of high-voltage transmission line safety monitoring technology, and in particular to an end-to-end early warning method and system for potential intrusion hazards of construction machinery in transmission channels. Background Technology

[0002] As a critical infrastructure for long-distance power transmission in the power system, the operating status of transmission lines directly affects the overall stability and reliability of the power grid. However, because transmission lines often traverse complex terrains such as suburbs, farmland, and mountains, and operate in open environments for extended periods, they are highly susceptible to external factors and damage, especially interference from unauthorized construction machinery operations and unauthorized intrusion by personnel. These external damage hazards are characterized by their suddenness, destructiveness, and difficulty in early warning, and have become a significant contributing factor to transmission line tripping, equipment failures, and even regional power outages.

[0003] Currently, the investigation of external damage hazards on transmission lines still mainly relies on manual inspections and fixed-point video surveillance. Although some regions have deployed online monitoring devices for transmission lines, the massive amount of data generated by these devices necessitates manual image analysis, leading to problems such as delayed response, limited coverage, and low accuracy in hazard identification. This traditional approach is insufficient to meet the real-time monitoring and early warning needs for external damage risks across the entire transmission line. With the rapid development of artificial intelligence and computer vision technologies, deep learning-based methods for identifying external damage risk targets and providing dynamic early warning are gradually becoming an important research direction for intelligent operation and maintenance of transmission lines, and are of great significance for improving the level of intelligent power grid safety protection and management.

[0004] In practical applications, power transmission lines vulnerable to external damage are typically located in and around towns and cities, and are significantly affected by frequent construction activities and the movement of large machinery. Due to the long, narrow, and widely distributed spatial characteristics of transmission corridors, images collected by monitoring equipment often cover large areas outside the corridors, leading to significant errors in locating potential external damage targets using existing detection methods. Consequently, some construction machinery or moving targets that have not intruded into the transmission corridor may be misjudged as potential threats by the system, resulting in a high false alarm rate and impacting the practicality and reliability of the monitoring system.

[0005] Laser point cloud data, as a high-precision spatial information carrier, can accurately reflect the three-dimensional spatial structure of towers, conductors, and surrounding features within power transmission channels, providing a potential advantage for spatial location of external damage hazards. However, point cloud data has limited semantic expressive capabilities, making it difficult to effectively identify and distinguish different types of targets; at the same time, its acquisition and processing require high equipment performance and cost, limiting its promotion and application in large-scale power transmission channels.

[0006] To address the practical needs of detecting intrusion of construction machinery into power transmission lines, this paper proposes an end-to-end early warning method for potential intrusion hazards of construction machinery based on online monitoring images and offline laser point cloud registration, thereby improving the effectiveness of online monitoring of power transmission channels. Summary of the Invention

[0007] In view of the above-mentioned prior art, the present invention provides an end-to-end early warning method and system for potential intrusion hazards of construction machinery in power transmission channels, which mainly solves the technical problems existing in the background art.

[0008] To achieve the above objectives, the technical solution of this invention is implemented as follows: In a first aspect, the present invention provides an end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels, the method comprising the following steps: Step S1: Collect 3D point cloud data and monitored visible light images of the power transmission corridor, and automatically match the visible light images and 3D point cloud data and estimate the camera pose using an image-point cloud intelligent registration model based on an attention mechanism. Step S2: Perform instance segmentation on the construction machinery targets in the visible light image of the power transmission corridor, and obtain the category and two-dimensional location information of the potential hazards in the image; Step S3: Based on the image-point cloud registration results and the construction machinery instance segmentation results, reconstruct the three-dimensional spatial position of the construction machinery target in the visible light image to obtain the relative position of the construction machinery hazard target and the power transmission corridor; Step S4: Obtain the position and orientation information of the online monitoring device, and issue early warnings for dangerous targets in the power transmission corridor through the visible light images captured in real time by the online monitoring device.

[0009] As a preferred embodiment of the present invention, the image-point cloud intelligent registration model based on the attention mechanism in step S1 automatically matches the visible light image and the 3D point cloud data and estimates the camera pose, specifically including: Acquire visible light image data and 3D point cloud data to be registered; Features are extracted from the visible light image data and the three-dimensional point cloud data respectively to obtain image features and point cloud features; The image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features. Based on the fused image consistency features and point cloud consistency features, the similarity between the point cloud features and the image features is calculated, and corresponding point matching is performed according to the similarity to obtain a set of point cloud-pixel matching pairs. Based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are obtained using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose.

[0010] As a preferred embodiment of the present invention, the step of extracting features from the visible light image data and the three-dimensional point cloud data respectively to obtain image features and point cloud features specifically includes: Image features are extracted using a ResNet-34 convolutional neural network, and deep and shallow semantic features are fused across connections to obtain a feature map with half the resolution of the original image data as the image features. After downsampling the point cloud data, the KPConv network is used to extract point cloud features. For each point in the point cloud, points within a sphere with a radius of m are defined as neighboring points. Kernel functions are used to characterize the features of all neighboring points, and the weighted sum of the features of all neighboring points is used as the feature of the current point to obtain the point cloud features.

[0011] As a preferred embodiment of the present invention, the image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features, including: The input image features and point cloud features are positionally encoded using a self-attention module to obtain image feature matrices and point cloud feature matrices, respectively. These feature matrices are then multiplied by learnable parameter matrices to obtain the corresponding results. vector, Vector sum vector; The cross-attention module is used to calculate the following respectively. Vector and the The dot product of the transposes of the vectors is normalized using the softmax function to generate an attention weight distribution; this attention weight distribution is then compared with the... Vector multiplication and weighted summation yield image attention features and point cloud attention features, respectively. The image attention features and the image feature matrix are summed by residual addition and normalized to obtain the image consistency features; the point cloud attention features and the point cloud feature matrix are summed by residual addition and normalized to obtain the point cloud consistency features.

[0012] As a preferred embodiment of the present invention, corresponding point matching is performed based on the cosine similarity to obtain a set of point cloud-pixel matching pairs, including: Bidirectional nearest neighbor matching retrieval is performed based on the cosine similarity between the point cloud features and the image features. Specifically, for the nearest neighbor retrieval from point cloud features to image features, for each point in the point cloud, the corresponding row is searched in the similarity matrix, and the image feature pixel with the highest similarity value is found in that row. This pixel is then marked as the optimal image matching candidate point for that point cloud. For the nearest neighbor retrieval from image features to point cloud features, for each pixel in the image features, the corresponding column is searched in the similarity matrix, and the point cloud feature with the highest similarity value is found in that column. This point cloud feature is then marked as the optimal point cloud matching candidate point for that pixel. The highest confidence matching pairs are selected from the optimal image matching candidate points and the optimal point cloud matching candidate points to determine the corresponding points of the same name, thus obtaining a set of point cloud-pixel pairs.

[0013] As a preferred embodiment of the present invention, based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are solved using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose, including: Based on the point cloud-pixel matching pair set, obtain the coordinates of a 3D point cloud point and its corresponding 2D pixel coordinates in the image for each matching pair; A pinhole camera imaging model is established, wherein the model uses the camera intrinsic parameter matrix, rotation matrix, and translation vector to convert the rotation matrix into a Rodrigues vector for representation; Based on the point cloud-pixel matching pair set, the EPnP algorithm is used to perform initial fitting and solution of the simplified camera intrinsic and extrinsic parameters to obtain the initial camera intrinsic and extrinsic parameter estimates. Starting from the initial estimated value, a loss function based on the reprojection error of all matching pairs is constructed, and a nonlinear least squares algorithm is used to iteratively adjust the camera intrinsic and extrinsic parameters until the convergence condition is met, thereby obtaining the camera pose.

[0014] As a preferred embodiment of the present invention, to avoid interference from abnormal matching point pairs, the RANSAC algorithm is used to optimize the point cloud-pixel matching pair set, including: From the current set of point cloud-pixel matching pairs, select no more than N point pairs with the highest similarity as the interior point set, and solve the camera intrinsic and extrinsic parameters for the current iteration; Using the obtained camera intrinsic and extrinsic parameters, calculate the projection positions of all point cloud points in the intrinsic point set onto the image plane; Calculate the offset error between the projected position of each point cloud point and the coordinates of its matching pixel point, and remove point pairs with the largest offset error in the inner point set by a specified proportion, until the average projection error of the point cloud points in the inner point set meets the preset requirements. The camera intrinsic and extrinsic parameters obtained at this time are used as the final camera pose estimation results.

[0015] As a preferred embodiment of the present invention, step S2 specifically employs the CM-SAM model to perform instance segmentation of construction machinery targets in the visible light image of the power transmission corridor, and obtains the category and two-dimensional location information of the potential hazards in the image, including: Acquire visible light images of the power transmission channel containing construction machinery targets; The visible light image is input into an image encoder to extract image features; The image features are input into an automatic prompt generation network, and at least one prompt message for indicating potential hazard targets is automatically generated based on the image features. The prompt message includes a target category and a location bounding box. The prompt information is input into the prompt encoder for encoding to obtain the prompt feature vector; The image features and the cue feature vector are input into the mask decoder. The image features and the cue feature vector are fused through a cross-attention mechanism to decode and generate an instance segmentation mask for the construction machinery target, thus completing the instance segmentation of external damage hazards. Based on the instance segmentation mask of the construction machinery target, the category and two-dimensional location information of the potential hazard target in the image are obtained.

[0016] As a preferred embodiment of the present invention, step S3 involves reconstructing the three-dimensional spatial position of the construction machinery target in the visible light image based on the image-point cloud registration result and the construction machinery instance segmentation result, to obtain the relative position of the construction machinery hazard target and the power transmission corridor, including: Based on the segmentation mask, the depth information of the construction machinery target relative to the imaging plane is extracted; Based on the camera intrinsic parameters and the depth information, the pixel position of the construction machinery target in the image is reconstructed in three dimensions to obtain its coordinates in three-dimensional space; The minimum spatial distance between the three-dimensional coordinates of the construction machinery target and the point cloud of the transmission line conductor is calculated to obtain the relative position of the construction machinery hazard target and the transmission corridor, so as to assess the safety risk.

[0017] Secondly, the present invention also provides an end-to-end early warning system for potential intrusion hazards of construction machinery in power transmission channels, applicable to any of the methods described above, wherein the system includes: The image-point cloud registration module is used to acquire 3D point cloud data and monitored visible light images of the power transmission corridor. It automatically matches the visible light images and 3D point cloud data and estimates the camera pose by using an image-point cloud intelligent registration model based on an attention mechanism. The instance segmentation module is used to segment construction machinery targets in visible light images of power transmission corridors and obtain the category and two-dimensional location information of potential hazards in the image. The ranging module is used to reconstruct the three-dimensional spatial position of construction machinery targets in visible light images based on image-point cloud registration results and construction machinery instance segmentation results, and to obtain the relative position of construction machinery hazard targets and power transmission corridors; The early warning module is used to acquire the position and orientation information of the online monitoring device and to issue early warnings for dangerous targets in the power transmission corridor through the visible light images captured in real time by the online monitoring device.

[0018] The beneficial effects of this invention are as follows: The end-to-end early warning method for construction machinery intrusion hazards in power transmission channels proposed in this invention can obtain the three-dimensional spatial coordinates of construction machinery by fusing online monitoring images with offline point clouds, thereby realizing refined alarm of external damage intrusion hazards, reducing false alarms caused by distant hazard targets, and providing technical support for intelligent inspection of power transmission channels. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the steps of an end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels according to the present invention; Figure 2 This is a structural diagram of the RPN extraction network provided by the present invention; Figure 3 This is a diagram illustrating the monitoring effect of construction machinery according to an embodiment of the present invention; Figure 4 The present invention provides a structural diagram of an end-to-end early warning system for potential intrusion hazards of construction machinery in power transmission channels. Detailed Implementation

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. In the following description, the expression "some embodiments" refers to a subset of all possible embodiments; however, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0021] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0022] It should be understood that the present invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Furthermore, the terminology used herein is intended only to describe particular embodiments and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “compose” and / or “comprising,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0023] It should also be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "inner," "outer," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0024] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0025] Firstly, this invention provides an end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels, in conjunction with the attached... Figure 1 The method includes the following steps: Step S1: Collect 3D point cloud data and monitored visible light images of the power transmission corridor, and automatically match the visible light images and 3D point cloud data and estimate the camera pose using an image-point cloud intelligent registration model based on an attention mechanism.

[0026] As a preferred embodiment of the present invention, the image-point cloud intelligent registration model based on the attention mechanism in step S1 automatically matches the visible light image and the 3D point cloud data and estimates the camera pose, specifically including: Step S1.1: Obtain the visible light image data and 3D point cloud data to be registered; In this embodiment, the laser point cloud data of the power transmission corridor provided by the power supply bureau is used as the data source. The laser point cloud is collected by a drone. The point cloud of different environmental parts such as towers, conductors, vegetation, ground and buildings is classified by algorithm. In this embodiment, the cross-film data registration of images and point clouds is mainly achieved by using environmental elements that are not easily changed, such as towers, buildings and ground.

[0027] In this embodiment, the visible light images are captured by an online monitoring device for the power transmission channel of a power grid company. Typically, images of the power transmission corridor are captured at 15-minute intervals to identify potential targets for construction machinery intrusion.

[0028] Step S1.2: Extract the features from the visible light image data and the three-dimensional point cloud data respectively to obtain image features and point cloud features; Step S1.3: The image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features; Step S1.4: Based on the fused image consistency features and point cloud consistency features, calculate the similarity between the point cloud features and the image features, and perform corresponding point matching according to the similarity to obtain a set of point cloud-pixel matching pairs; Step S1.5: Based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are obtained using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose.

[0029] Step S1.2: Extract the features from the visible light image data and the three-dimensional point cloud data respectively to obtain image features and point cloud features.

[0030] As a preferred embodiment of the present invention, the step of extracting features from the visible light image data and the three-dimensional point cloud data respectively to obtain image features and point cloud features specifically includes: Image features are extracted using a ResNet-34 convolutional neural network, and deep and shallow semantic features are fused across connections to obtain a feature map with half the resolution of the original image data as the image features. After downsampling the point cloud data, the KPConv network is used to extract point cloud features. For each point in the point cloud, points within a sphere with a radius of m are defined as neighboring points. Kernel functions are used to characterize the features of all neighboring points, and the weighted sum of the features of all neighboring points is used as the feature of the current point to obtain the point cloud features.

[0031] In this embodiment, ResNet-34 and KPConv are used as backbone networks to extract image and point cloud features, respectively. First, the image size is uniformly adjusted to 1200×800. The ResNet-34 convolutional neural network is used to extract features from the image. At the same time, the model fuses deep and shallow semantic features through cross-connections, and finally obtains a feature map with half the resolution of the original data. For the point cloud data, the point cloud is first downsampled by 1 / 4 to reduce the number of points. Then, the KPConv network is used to extract features from each point. Points within a sphere with a radius of m are defined as neighborhood points. All neighborhood points are represented by kernel functions, and the features of all neighborhood points are weighted and summed to represent the current point cloud features.

[0032] Step S1.3: The image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features; As a preferred embodiment of the present invention, the image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features, including: The input image features and point cloud features are positionally encoded using a self-attention module to obtain image feature matrices and point cloud feature matrices, respectively. These feature matrices are then multiplied by learnable parameter matrices to obtain the corresponding results. vector, Vector sum vector; The cross-attention module is used to calculate the following respectively. Vector and the The dot product of the transposes of the vectors is normalized using the softmax function to generate an attention weight distribution; this attention weight distribution is then compared with the... Vector multiplication and weighted summation yield image attention features and point cloud attention features, respectively. The image attention features and the image feature matrix are summed by residual addition and normalized to obtain the image consistency features; the point cloud attention features and the point cloud feature matrix are summed by residual addition and normalized to obtain the point cloud consistency features.

[0033] In this embodiment, the present invention constructs an image-point cloud feature fusion module composed of self-attention and cross-attention. For the self-attention module, given an input feature map... First, the input vector is encoded to obtain vector, Vector sum Vectors, as shown below:

[0034] Among them, the input feature map The feature matrix is ​​the result of location encoding of image or point cloud features. , , Given a learnable parameter matrix, the output of the self-attention module can be expressed as follows:

[0035] Extracted global features The data is then fed into a feedforward neural network to further integrate cross-channel semantic features.

[0036] For image features and point cloud features, the process of extracting consistency features by the cross-attention module is as follows:

[0037] in, This is a point cloud cross-attention feature map. , , These are point cloud features. vector, Vector sum vector, For image cross-attention feature maps, , , Image features vector, Vector sum vector.

[0038] Step S1.4: Based on the fused image consistency features and point cloud consistency features, calculate the similarity between the point cloud features and the image features, and perform corresponding point matching according to the similarity to obtain a set of point cloud-pixel matching pairs; In a preferred embodiment of the present invention, based on the fused image consistency features and point cloud consistency features, the cosine similarity between the point cloud features and the image features is calculated, as follows: The image consistency features and point cloud consistency features are converted into image feature vectors and point cloud feature vectors, and the dot product operation is performed on the point cloud feature vectors and the image feature vectors to obtain a dot product scalar value. Calculate the magnitude of the point cloud feature vector and the magnitude of the image feature vector, respectively; Dividing the dot product scalar value by the product of the magnitudes of the two vectors yields the cosine similarity between the point cloud features and the image features.

[0039] In this embodiment, the cosine similarity between the filtered point cloud features and the image features is calculated. As shown in the following formula:

[0040] in, Represents the dot product of two vectors. Represents the magnitude of a vector. For point cloud coordinates, These are the pixel coordinates.

[0041] As a preferred embodiment of the present invention, corresponding point matching is performed based on the cosine similarity to obtain a set of point cloud-pixel matching pairs, including: Bidirectional nearest neighbor matching retrieval is performed based on the cosine similarity between the point cloud features and the image features. Specifically, for the nearest neighbor retrieval from point cloud features to image features, for each point in the point cloud, the corresponding row is searched in the similarity matrix, and the image feature pixel with the highest similarity value is found in that row. This pixel is then marked as the optimal image matching candidate point for that point cloud. For the nearest neighbor retrieval from image features to point cloud features, for each pixel in the image features, the corresponding column is searched in the similarity matrix, and the point cloud feature with the highest similarity value is found in that column. This point cloud feature is then marked as the optimal point cloud matching candidate point for that pixel. The highest confidence matching pairs are selected from the optimal image matching candidate points and the optimal point cloud matching candidate points to determine the corresponding points of the same name, thus obtaining a set of point cloud-pixel pairs.

[0042] Step S1.5: Based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are obtained using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose.

[0043] As a preferred embodiment of the present invention, based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are solved using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose, including: Based on the point cloud-pixel matching pair set, obtain the coordinates of a 3D point cloud point and its corresponding 2D pixel coordinates in the image for each matching pair; A pinhole camera imaging model is established, wherein the model uses the camera intrinsic parameter matrix, rotation matrix, and translation vector to convert the rotation matrix into a Rodrigues vector for representation; In this embodiment, the pinhole camera imaging model is shown in the following formula:

[0044] in, The depth corresponding to the point cloud. For the camera intrinsic parameter matrix, For the camera rotation matrix, As the translation vector, and considering that transmission line monitoring cameras are typically telephoto lenses, this invention simplifies the camera intrinsic parameters, which can be expressed as the following formula:

[0045] in, Indicates the camera in the image Equivalent focal length on the axis (horizontal direction), Indicates the principal point of the camera in the image. Coordinates of points on the axis.

[0046] Based on the point cloud-pixel matching pair set, the EPnP algorithm is used to perform initial fitting and solution of the simplified camera intrinsic and extrinsic parameters to obtain the initial camera intrinsic and extrinsic parameter estimates. Starting from the initial estimated value, a loss function based on the reprojection error of all matching pairs is constructed, and a nonlinear least squares algorithm is used to iteratively adjust the camera intrinsic and extrinsic parameters until the convergence condition is met, thereby obtaining the camera pose.

[0047] In this embodiment, the loss function is as follows:

[0048]

[0049] in,( , () represents the pixel coordinates. For point cloud coordinates, , The intrinsic and extrinsic parameters are obtained from the solution, and then a nonlinear least squares algorithm is used to adjust the intrinsic and extrinsic parameters to determine the camera pose, as follows: Calculate the sum of squared distances between the theoretical pixel position of the 3D point projected onto the image according to the current parameters and its actual matching pixel position for all matching point pairs, and use this as the loss function.

[0050] Calculate the Jacobian matrix, and in each iteration, calculate the loss function with respect to all parameters to be optimized. , , r, The partial derivatives of each component of the error are used to construct the Jacobian matrix. Combining the Jacobian matrix, the current error, and an adaptive damping factor, the parameter adjustment direction and step size that reduce the expected error are obtained. Parameter updates are then performed.

[0051] The updated error is then recalculated. If the error decreases, the parameter update is accepted, and the damping factor is reduced to allow for more aggressive step sizes in subsequent iterations. If the error does not decrease, the update is rejected, the damping factor is increased to make the search more conservative, and the increment is recalculated.

[0052] Repeat the above iterative loop until the convergence condition is met. When the iteration converges, transform the Rodriguez vector r in the obtained optimal parameter vector back into the rotation matrix R. Finally, the optimized intrinsic parameters, the rotation matrix R, and the translation vector t are used together as the camera pose output.

[0053] As a preferred embodiment of the present invention, to avoid interference from abnormal matching point pairs, the RANSAC algorithm is used to optimize the point cloud-pixel matching pair set, including: From the current set of point cloud-pixel matching pairs, select no more than N point pairs with the highest similarity as the interior point set, and solve the camera intrinsic and extrinsic parameters for the current iteration; Using the obtained camera intrinsic and extrinsic parameters, calculate the projection positions of all point cloud points in the intrinsic point set onto the image plane; Calculate the offset error between the projected position of each point cloud point and the coordinates of its matching pixel point, and remove point pairs with the largest offset error in the inner point set by a specified proportion, until the average projection error of the point cloud points in the inner point set meets the preset requirements. The camera intrinsic and extrinsic parameters obtained at this time are used as the final camera pose estimation results.

[0054] Step S2: Perform instance segmentation on the construction machinery targets in the visible light image of the power transmission corridor, and obtain the category and two-dimensional location information of the potential hazards in the image.

[0055] As a preferred embodiment of the present invention, step S2 specifically employs the CM-SAM model to perform instance segmentation of construction machinery targets in the visible light image of the power transmission corridor, and obtains the category and two-dimensional location information of the potential hazards in the image, including: Acquire visible light images of the power transmission channel containing construction machinery targets; The visible light image is input into an image encoder to extract image features; The image features are input into an automatic prompt generation network, and at least one prompt message for indicating potential hazard targets is automatically generated based on the image features. The prompt message includes a target category and a location bounding box. The prompt information is input into the prompt encoder for encoding to obtain the prompt feature vector; The image features and the cue feature vector are input into the mask decoder. The image features and the cue feature vector are fused through a cross-attention mechanism to decode and generate an instance segmentation mask for the construction machinery target, thus completing the instance segmentation of external damage hazards. Based on the instance segmentation mask of the construction machinery target, the category and two-dimensional location information of the potential hazard target in the image are obtained.

[0056] In this embodiment, the construction machinery targets to be monitored mainly include the following types: cranes, cement pump trucks, pile drivers, bulldozers, excavators, dump trucks, and soil compactors. By filtering historical monitoring data, 5,000 sample images containing construction machinery targets are obtained. Instance segmentation and labeling are performed on the construction machinery targets in these images, and the model is trained.

[0057] In this embodiment, as shown in the appendix Figure 2 As shown, the CM-SAM model extracts semantic information through the RPN network and automatically generates rectangular prompt boxes containing category and location information. End-to-end segmentation of external damage hazard targets is achieved by encoding the prompt boxes. The CM-SAM model can be specifically represented by the following formula:

[0058] in, As an image encoder, in this embodiment, the image encoder uses a ViT network as its backbone to extract features from the input image. , To generate automatic suggestion networks, rectangular boxes are generated using features extracted from image encoding. and categories , For the cue encoder, the bounding box is encoded into a feature vector. , As a mask decoder, it fuses the correlation between image features and cue encoding through a cross-attention mechanism to decode and obtain the image segmentation result. .

[0059] In one embodiment, the RPN network is used to extract potential external damage targets from an image, as shown in the attached figure. Figure 2 As shown, for the input multi-scale feature map, the RPN network is used to extract target candidate regions in the image, and the subsequent region features are fused with the location encoding (PE) and then input into the classification network. and location regression network To obtain effective hazard target categories and position rectangle Finally, for After encoding, the information is used as a prompt feature and input into the mask decoder to guide the model to complete the segmentation of external damage hazard instances.

[0060] Step S3: Based on the image-point cloud registration results and the construction machinery instance segmentation results, the three-dimensional spatial position of the construction machinery target in the visible light image is reconstructed to obtain the relative position of the construction machinery hazard target and the power transmission corridor.

[0061] As a preferred embodiment of the present invention, step S3 involves reconstructing the three-dimensional spatial position of the construction machinery target in the visible light image based on the image-point cloud registration result and the construction machinery instance segmentation result, to obtain the relative position of the construction machinery hazard target and the power transmission corridor, including: Based on the segmentation mask, the depth information of the construction machinery target relative to the imaging plane is extracted; Based on the camera intrinsic parameters and the depth information, the pixel position of the construction machinery target in the image is reconstructed in three dimensions to obtain its coordinates in three-dimensional space; The minimum spatial distance between the three-dimensional coordinates of the construction machinery target and the point cloud of the transmission line conductor is calculated to obtain the relative position of the construction machinery hazard target and the transmission corridor, so as to assess the safety risk.

[0062] In this embodiment, the segmentation mask for each construction machinery target is: Using the bottom edge of the mask as a reference, the depth information of the construction machinery target relative to the imaging plane is extracted, as shown in the following formula:

[0063]

[0064] in, Outline of the construction machinery target. C b The bottom outline, For the pixels on the outline of the construction machinery, The vertical axis coordinates of the outline pixels. This is for the bottom margin. This refers to the depth corresponding to the construction machinery.

[0065] The 3D position of the construction machinery is reconstructed based on camera intrinsic parameters and depth information to obtain the relative distance between the construction machinery and the power transmission line. The 3D spatial position of the construction machinery target is shown in the following formula:

[0066] in,( , ( ) is a mask for construction machinery. pixel coordinates in The coordinates of the reconstructed construction machinery are in three-dimensional space.

[0067] The minimum distance between the coordinates of the construction machinery and the traverse point cloud is shown in the following formula:

[0068] in, For the coordinates of the traverse point cloud, This refers to the minimum distance between the construction machinery and the guide wire.

[0069] Step S4: Obtain the position and orientation information of the online monitoring device, and issue early warnings for dangerous targets in the power transmission corridor through the visible light images captured in real time by the online monitoring device.

[0070] In this embodiment, typical construction machinery hazards near the power transmission channel were selected for measurement to evaluate the effectiveness of the method. The test results are shown in Table 1. Table 1 Test Results

[0071] The test results above show that the method proposed in this invention can identify potential hazards of construction machinery near power transmission corridors and measure the minimum distance between them and the conductors. The relative errors are all within 10%, which can meet the application requirements of intelligent online monitoring of power transmission lines. Specifically, this invention was used to test actual monitoring images, and one recognition example is shown in the attached figure. Figure 3 As shown.

[0072] In this embodiment, the distance between the construction machinery and the conductor is used to determine whether the construction machinery has intruded into the power transmission corridor area. According to the power transmission line operation and maintenance standards, in a 220kV power transmission corridor, a distance greater than 50 meters between the construction machinery and the conductor is considered no risk, a distance of 25-50 meters is considered moderate risk, and a distance less than 25 meters is considered severe risk. In this embodiment, warnings are issued for moderate and severe risks to reduce invalid alarms.

[0073] Secondly, the present invention also provides an end-to-end early warning system for potential intrusion hazards of construction machinery in power transmission channels, applicable to any of the aforementioned end-to-end early warning methods for potential intrusion hazards of construction machinery in power transmission channels, as shown in the appendix. Figure 4 As shown, the system includes: The image-point cloud registration module is used to acquire 3D point cloud data and monitored visible light images of the power transmission corridor. It automatically matches the visible light images and 3D point cloud data and estimates the camera pose by using an image-point cloud intelligent registration model based on an attention mechanism. The instance segmentation module is used to segment construction machinery targets in visible light images of power transmission corridors and obtain the category and two-dimensional location information of potential hazards in the image. The ranging module is used to reconstruct the three-dimensional spatial position of construction machinery targets in visible light images based on image-point cloud registration results and construction machinery instance segmentation results, and to obtain the relative position of construction machinery hazard targets and power transmission corridors; The early warning module is used to acquire the position and orientation information of the online monitoring device and to issue early warnings for dangerous targets in the power transmission corridor through the visible light images captured in real time by the online monitoring device.

[0074] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels.

[0075] In this embodiment, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0076] Fourthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any of the end-to-end early warning methods for potential intrusion hazards of construction machinery in power transmission channels provided in embodiments of this application.

[0077] Fifthly, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any of the end-to-end early warning methods for potential intrusion hazards of construction machinery in power transmission channels provided in embodiments of this application.

[0078] In this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be accomplished by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0079] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the end-to-end early warning methods for potential intrusion hazards of construction machinery in power transmission channels provided in embodiments of this application.

[0080] It should be noted that, through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0081] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels, characterized in that, The method includes the following steps: Step S1: Collect 3D point cloud data and monitored visible light images of the power transmission corridor, and automatically match the visible light images and 3D point cloud data and estimate the camera pose using an image-point cloud intelligent registration model based on an attention mechanism. Step S2: Perform instance segmentation on the construction machinery targets in the visible light image of the power transmission corridor, and obtain the category and two-dimensional location information of the potential hazards in the image; Step S3: Based on the image-point cloud registration results and the construction machinery instance segmentation results, reconstruct the three-dimensional spatial position of the construction machinery target in the visible light image to obtain the relative position of the construction machinery hazard target and the power transmission corridor; Step S4: Obtain the position and orientation information of the online monitoring device, and issue early warnings for dangerous targets in the power transmission corridor through the visible light images captured in real time by the online monitoring device.

2. The end-to-end early warning method for potential intrusion hazards during power transmission channel construction, as described in claim 1, is characterized in that... In step S1, the image-point cloud intelligent registration model based on the attention mechanism automatically matches visible light images and 3D point cloud data and estimates camera pose, specifically including: Acquire visible light image data and 3D point cloud data to be registered; Features are extracted from the visible light image data and the three-dimensional point cloud data respectively to obtain image features and point cloud features; The image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features. Based on the fused image consistency features and point cloud consistency features, the similarity between the point cloud features and the image features is calculated, and corresponding point matching is performed according to the similarity to obtain a set of point cloud-pixel matching pairs. Based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are obtained using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose.

3. The end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels according to claim 2, characterized in that, The step of extracting features from the visible light image data and the 3D point cloud data respectively to obtain image features and point cloud features specifically includes: Image features are extracted using a ResNet-34 convolutional neural network, and deep and shallow semantic features are fused across connections to obtain a feature map with half the resolution of the original image data as the image features. After downsampling the point cloud data, the KPConv network is used to extract point cloud features. For each point in the point cloud, points within a sphere with a radius of m are defined as neighboring points. Kernel functions are used to characterize the features of all neighboring points, and the weighted sum of the features of all neighboring points is used as the feature of the current point to obtain the point cloud features.

4. The end-to-end early warning method for potential intrusion hazards of construction machinery in power transmission channels according to claim 3, characterized in that, The image features and the point cloud features are fused using a self-attention module and a cross-attention module to obtain fused image consistency features and point cloud consistency features, including: The input image features and point cloud features are positionally encoded using a self-attention module to obtain image feature matrices and point cloud feature matrices, respectively. These feature matrices are then multiplied by learnable parameter matrices to obtain the corresponding results. vector, Vector sum vector; The cross-attention module is used to calculate the following respectively. Vector and the The dot product of the transposes of the vectors is normalized using the softmax function to generate an attention weight distribution; this attention weight distribution is then compared with the... Vector multiplication and weighted summation yield image attention features and point cloud attention features, respectively. The image attention features and the image feature matrix are summed by residual addition and normalized to obtain the image consistency features; the point cloud attention features and the point cloud feature matrix are summed by residual addition and normalized to obtain the point cloud consistency features.

5. The end-to-end early warning method for potential intrusion hazards during power transmission channel construction, as described in claim 4, is characterized in that... Based on the cosine similarity, corresponding points are matched to obtain a set of point cloud-pixel matching pairs, including: Bidirectional nearest neighbor matching retrieval is performed based on the cosine similarity between the point cloud features and the image features. Specifically, for the nearest neighbor retrieval from point cloud features to image features, for each point in the point cloud, the corresponding row is searched in the similarity matrix, and the image feature pixel with the highest similarity value is found in that row. This pixel is then marked as the optimal image matching candidate point for that point cloud. For the nearest neighbor retrieval from image features to point cloud features, for each pixel in the image features, the corresponding column is searched in the similarity matrix, and the point cloud feature with the highest similarity value is found in that column. This point cloud feature is then marked as the optimal point cloud matching candidate point for that pixel. The highest confidence matching pairs are selected from the optimal image matching candidate points and the optimal point cloud matching candidate points to determine the corresponding points of the same name, thus obtaining a set of point cloud-pixel pairs.

6. The end-to-end early warning method for potential intrusion hazards during power transmission channel construction machinery, as described in claim 5, is characterized in that... Based on the point cloud-pixel matching pair set, the initial estimates of the camera intrinsic and extrinsic parameters are obtained using the EPnP algorithm, and the intrinsic and extrinsic parameters are adjusted using a nonlinear least squares algorithm to obtain the camera pose, including: Based on the point cloud-pixel matching pair set, obtain the coordinates of a 3D point cloud point and its corresponding 2D pixel coordinates in the image for each matching pair; A pinhole camera imaging model is established, wherein the model uses the camera intrinsic parameter matrix, rotation matrix, and translation vector to convert the rotation matrix into a Rodrigues vector for representation; Based on the point cloud-pixel matching pair set, the EPnP algorithm is used to perform initial fitting and solution of the simplified camera intrinsic and extrinsic parameters to obtain the initial camera intrinsic and extrinsic parameter estimates. Starting from the initial estimated value, a loss function based on the reprojection error of all matching pairs is constructed, and a nonlinear least squares algorithm is used to iteratively adjust the camera intrinsic and extrinsic parameters until the convergence condition is met, thereby obtaining the camera pose.

7. The end-to-end early warning method for potential intrusion hazards during power transmission channel construction machinery, as described in claim 6, is characterized in that... To avoid interference from abnormal matching point pairs, the RANSAC algorithm is used to optimize the point cloud-pixel matching pair set, including: From the current set of point cloud-pixel matching pairs, select no more than N point pairs with the highest similarity as the interior point set, and solve the camera intrinsic and extrinsic parameters for the current iteration; Using the obtained camera intrinsic and extrinsic parameters, calculate the projection positions of all point cloud points in the intrinsic point set onto the image plane; Calculate the offset error between the projected position of each point cloud point and the coordinates of its matching pixel point, and remove point pairs with the largest offset error in the inner point set by a specified proportion, until the average projection error of the point cloud points in the inner point set meets the preset requirements. The camera intrinsic and extrinsic parameters obtained at this time are used as the final camera pose estimation results.

8. The end-to-end early warning method for potential intrusion hazards during power transmission channel construction machinery, as described in claim 7, is characterized in that... Step S2 specifically employs the CM-SAM model to perform instance segmentation of construction machinery targets in the visible light image of the power transmission corridor, and obtains the category and two-dimensional location information of potential hazards in the image, including: Acquire visible light images of the power transmission channel containing construction machinery targets; The visible light image is input into an image encoder to extract image features; The image features are input into an automatic prompt generation network, and at least one prompt message for indicating potential hazard targets is automatically generated based on the image features. The prompt message includes a target category and a location bounding box. The prompt information is input into the prompt encoder for encoding to obtain the prompt feature vector; The image features and the cue feature vector are input into the mask decoder. The image features and the cue feature vector are fused through a cross-attention mechanism to decode and generate an instance segmentation mask for the construction machinery target, thus completing the instance segmentation of external damage hazards. Based on the instance segmentation mask of the construction machinery target, the category and two-dimensional location information of the potential hazard target in the image are obtained.

9. The end-to-end early warning method for potential intrusion hazards during power transmission channel construction machinery, as described in claim 8, is characterized in that... In step S3, based on the image-point cloud registration results and the construction machinery instance segmentation results, the three-dimensional spatial position of the construction machinery target in the visible light image is reconstructed to obtain the relative position of the construction machinery hazard target and the power transmission corridor, including: Based on the segmentation mask, the depth information of the construction machinery target relative to the imaging plane is extracted; Based on the camera intrinsic parameters and the depth information, the pixel position of the construction machinery target in the image is reconstructed in three dimensions to obtain its coordinates in three-dimensional space; The minimum spatial distance between the three-dimensional coordinates of the construction machinery target and the point cloud of the transmission line conductor is calculated to obtain the relative position of the construction machinery hazard target and the transmission corridor, so as to assess the safety risk.

10. An end-to-end early warning system for potential intrusion hazards during power transmission channel construction, applied to the method described in any one of claims 1-9, characterized in that, The system includes: The image-point cloud registration module is used to acquire 3D point cloud data and monitored visible light images of the power transmission corridor. It automatically matches the visible light images and 3D point cloud data and estimates the camera pose by using an image-point cloud intelligent registration model based on an attention mechanism. The instance segmentation module is used to segment construction machinery targets in visible light images of power transmission corridors and obtain the category and two-dimensional location information of potential hazards in the image. The ranging module is used to reconstruct the three-dimensional spatial position of construction machinery targets in visible light images based on image-point cloud registration results and construction machinery instance segmentation results, and to obtain the relative position of construction machinery hazard targets and power transmission corridors; The early warning module is used to acquire the position and orientation information of the online monitoring device and to issue early warnings for dangerous targets in the power transmission corridor through the visible light images captured in real time by the online monitoring device.

Images