A Multimodal Point Cloud-Based Tower Tilt Detection Method Based on Lie Group Structure Self-Attention

Abstract

This invention discloses a multimodal point cloud tower tilt detection method based on Lie group structure self-attention, comprising the following steps: S1, multimodal input data and spatial registration of the tower; S2, mapping of the tower's multimodal features to the Lie group; S3, constructing a Lie group structure self-attention mechanism; S4, constructing the overall tower structure; S5, segmentation of the tower's main structure and output of the tilt angle. This invention maps tower point cloud and image features to a unified Lie group space. By introducing Lie group structure position encoding and a self-attention mechanism, it achieves explicit modeling of rigid transformation invariant features and efficient multimodal fusion. This effectively compensates for the shortcomings of existing monitoring methods, such as insufficient utilization of the geometric consistency of tower multimodal data, weak noise resistance, and insufficient spatial structure modeling. It fundamentally improves the accuracy, generalization ability, and adaptability to complex environments of tower tilt detection, greatly enhancing the intelligent level of power distribution network tower health monitoring.

Description

Technical Field

[0001] This invention discloses a multimodal point cloud tower tilt detection method based on Lie group structure self-attention, which belongs to the application of image detection technology in power distribution network tower maintenance. Background Technology

[0002] Currently, the industry mainly adopts two technical approaches for detecting the tilt of power distribution network towers. One approach is based on visual image analysis, acquiring two-dimensional images of the towers using cameras, and then extracting features such as the tower outline and main axis using traditional image processing methods (e.g., edge detection, Hough transform) or convolutional neural network (CNN) models to determine the tilt. This method is relatively simple to implement, but in actual power distribution network environments, it is easily affected by various uncertainties such as changes in lighting, image resolution, viewing angle, and on-site occlusion, making it difficult to guarantee the robustness and accuracy of the detection results. The other mainstream approach is based on three-dimensional point cloud data acquired by sensors such as LiDAR, using deep learning networks such as PointNet and Point Transformer to extract features and segment the structure of the point cloud, thereby achieving automatic identification of the tower's spatial structure and its main axis direction. This approach overcomes the dependence of image-based methods on lighting and viewing angle, but in practical applications, point cloud data often suffers from noise, sparsity, occlusion, and partial structural loss. Furthermore, since most existing methods only deal with a single modality and fail to fully explore and integrate multi-source information, the generalization ability and robustness of the models to complex environments remain limited.

[0003] While the aforementioned methods for detecting tower tilt have contributed to the development of intelligent operation and maintenance in power distribution networks, their limitations are becoming increasingly apparent as the complexity of real-world application environments increases. Firstly, two-dimensional image-based analysis methods inherently rely on external environmental conditions and cannot effectively mitigate information loss caused by changes in lighting, equipment obstruction, and varying viewing angles, leading to significant instability in detection results under non-ideal scenarios. Simultaneously, while single-modal point cloud analysis methods can improve the understanding of spatial structures to some extent, they often overlook the incompleteness and high noise levels of data in the actual operating environment of towers. More critically, these methods typically focus only on the single feature space of point clouds or images, lacking deep fusion and modeling of the inherent geometric relationships and physical constraints between multimodal data. This makes it difficult for the models to capture the essential invariance characteristics of tower structures under rigid transformations such as rotation and translation. Summary of the Invention

[0004] The technical problem solved by this invention is to provide a multimodal point cloud tower tilt detection method based on Lie group structure self-attention, which addresses the problems of low accuracy, poor robustness, and insufficient modeling of rigid transformation invariance in traditional single-modal or simple fusion methods for tower tilt detection.

[0005] This invention is achieved using the following technical solution:

[0006] This invention first discloses a multimodal point cloud tower tilt detection method based on Lie group structure self-attention, comprising the following steps:

[0007] S1. Multimodal input data and spatial registration of the tower, wherein the multimodal input data includes point cloud data and RGB image data of the tower, and the multimodal input data of the tower is spatially registered at the point cloud and image feature level and stitched together to obtain the multimodal features of the tower;

[0008] S2. Multimodal features of the tower are mapped to Lie group. The multimodal features of the tower are mapped to six-dimensional Lie algebra parameters. The six-dimensional Lie algebra parameters are mapped to the Lie group space of rigid body motion to obtain the spatial rigid body transformation matrix of each feature point in the tower point cloud data.

[0009] S3. Construct a self-attention mechanism for Lie group structures. Obtain the relative structural quantities between feature points in the Lie group space of the tower point cloud data through the spatial rigid body transformation matrix. Construct a self-attention mechanism for Lie group structures to fuse and output the initial multimodal feature matrix of all feature points in the tower point cloud data.

[0010] S4. Construct the overall structure of the tower. The initial multimodal feature matrix is ​​fitted by a feedforward neural network to output the fused features of all feature points of the tower point cloud data.

[0011] S5. Segmentation and tilt angle output of the main pole structure: The probability of each feature point in the pole point cloud data belonging to the main pole is predicted by feature segmentation through feature point fusion. The main pole point set is selected by setting a probability threshold. The main pole direction is obtained through the center of the main pole. The tilt angle of the pole is obtained according to the angle between the main pole direction and the vertical direction.

[0012] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, in step S1, a UAV equipped with a lidar and a high-definition camera is used to acquire images of the tower, obtain the tower's point cloud data and RGB image data, and through lidar-camera extrinsic parameter calibration, the point cloud spatial coordinates of the tower's point cloud data are matched one-to-one with the pixel coordinates of the RGB image data, thereby achieving feature-level spatial registration between the point cloud and the image.

[0013] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, the local point cloud features of each feature point in the tower point cloud data and the corresponding image features after registration are extracted and stitched together to obtain the multimodal features of the tower.

[0014] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, further, in step S2, the multimodal features of the tower are first mapped into a six-dimensional Lie algebra vector through a linear layer:

[0015] ,

[0016] Where W and b are the learnable parameters of the linear layer. d represents the dimension of the multimodal feature, and R represents the real number field. Let be the rotation component in a six-dimensional vector of a Lie algebra. For the translation component in a six-dimensional vector of a Lie algebra, The multimodal feature of the i-th feature point in the tower point cloud data;

[0017] For each feature point in the tower point cloud data, the corresponding Lie algebra six-dimensional vector is mapped to the Lie group space of rigid body motion, and the Lie algebra matrix is ​​constructed by the following formula. :

[0018] ,

[0019] in, for The antisymmetric matrix is ​​used to obtain the spatial rigid body transformation matrix of each feature point in the tower point cloud data through Lie group exponential mapping.

[0020] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, further, in step S3, adjacent feature points are selected from the tower point cloud data by local adjacency sampling, and the relative structural quantity of adjacent feature points in the Lie group space is calculated by the following formula:

[0021] ,

[0022] in, , Let i be the Lie group element of the spatial rigid body transformation matrix of adjacent feature points i and j. It is a six-dimensional vector of Lie algebra;

[0023] The location structure is encoded as follows: , To construct a learnable multilayer perceptron, a self-attention mechanism based on a Lie group structure is as follows:

[0024] ,

[0025] in, , , , , and It projects the original features from the attention mechanism onto the weights of the query, key, and value space; softmax is the normalization function. For the learnable structural weights of the attention mechanism, The scaling factor for the attention mechanism. It is the initial multimodal feature matrix, which is the sequential stacking of the multimodal features of all feature points in the tower point cloud data. N represents the number of feature points in the tower point cloud data.

[0026] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, further, in step S4, the feedforward neural network is cascaded with the Lie group structure self-attention layer in step S3 through multiple feedforward neural network layers, and the fused feature expression of all feature points of the output tower point cloud data is as follows:

[0027] ,

[0028] This is the initial multimodal feature matrix for all feature points in the tower point cloud data. The fused features are the output of the l-th layer feedforward neural network. This represents the self-attention mechanism of the l-th layer Lie group structure. Let L be the l-th layer of the feedforward neural network, L be the number of self-attention layers in the Lie group structure, and F be the fused feature of all feature points in the tower point cloud data output by the last feedforward neural network layer.

[0029] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, further, in step S5, the probability of each feature point belonging to the main pole of the tower is predicted by the following formula:

[0030] ,

[0031] This represents the predicted probability that the i-th feature point in the tower point cloud data belongs to the main pole of the tower. This represents the fused feature vector of the nth feature point in the tower point cloud data, which originates from the output of step S4. This represents the learnable weight vector used for classification. For bias terms, This is the activation function, used to compress the result into probabilities between 0 and 1;

[0032] By setting a probability threshold Select the main pole point set of the tower :

[0033] ,

[0034] The center of the main tower is calculated using the following formula. :

[0035] ,

[0036] Main club point concentration main club point Spatial position coordinates

[0037] Obtain the covariance matrix of the main pole center, and perform eigenvalue decomposition on the covariance matrix of the main pole center. The eigenvector corresponding to the largest eigenvalue is the direction of the main pole.

[0038] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, further, when there are multiple towers, spatial clustering, connected component analysis, template matching or point cloud neighborhood growth are first performed on all feature points in the tower point cloud data to separate the point sets corresponding to different main towers, and then the main tower direction of each tower is output respectively.

[0039] In the multimodal point cloud tower tilt detection method based on Lie group structure self-attention of the present invention, further, in step S5, the vertical direction is defined as... The angle θ between the direction of the main pole of the tower and the vertical direction is:

[0040] ,

[0041] The direction of the main pole of the tower.

[0042] This invention also discloses a multimodal point cloud tower tilt detection system based on Lie group structure self-attention using the above method, comprising:

[0043] The multimodal feature generation module takes multimodal input data of the tower as input and performs spatial registration. The multimodal input data includes point cloud data and RGB image data of the tower. The multimodal input data of the tower is spatially registered at the point cloud and image feature level and then stitched together to obtain the multimodal features of the tower.

[0044] The Lie group structure self-attention module constructs a Lie group structure self-attention mechanism, maps the multimodal features of the tower to a Lie group, maps the multimodal features of the tower to six-dimensional Lie algebra parameters, maps the six-dimensional Lie algebra parameters to the Lie group space of rigid body motion, obtains the spatial rigid body transformation matrix of each feature point in the tower point cloud data, obtains the structural relative quantities between feature points in the Lie group space through the spatial rigid body transformation matrix, and constructs the Lie group structure self-attention mechanism to fuse and output the initial multimodal feature matrix of all feature points in the tower point cloud data;

[0045] The feedforward neural network module is a cascaded combination of a multi-layer feedforward neural network and a Lie group structure self-attention module. The feedforward neural network fits the initial multimodal feature matrix to output the fused features of all feature points of the tower point cloud data, thus constructing the overall structure of the tower.

[0046] The segmentation and detection module segments the main pole structure and outputs the tilt angle. It predicts the probability that each feature point in the pole point cloud data belongs to the main pole by using feature point fusion segmentation. It selects the main pole point set by setting a probability threshold, obtains the main pole direction through the center of the main pole, and obtains the pole tilt angle based on the angle between the main pole direction and the vertical direction.

[0047] The proposed multimodal point cloud tower tilt detection method based on Lie group structure self-attention embeds both point cloud and image modal data into a Lie group space. This theoretically characterizes and utilizes the invariance of data under rigid transformations. By introducing Lie group structure position encoding and self-attention mechanism, the method explicitly models and transmits the structural consistency of point cloud and image under spatial transformations during feature fusion and weight calculation. This not only significantly enhances the expressiveness and robustness of multimodal feature fusion but also maintains accurate identification of the main tower structure and tilt calculation capability even when faced with sparse, occluded, and noise-contaminated tower data. This fundamentally overcomes the limitations of traditional techniques and provides a solid theoretical foundation and effective engineering path for improving the accuracy and stability of tower tilt detection in complex scenarios.

[0048] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0049] (1) Using Lie group mathematical tools, multimodal data are embedded into the same Lie group space. The multimodal features of tower point cloud data and image data are mapped to six-dimensional parameters of Lie algebra through linear layer mapping, which correspond to the Lie group describing the rigid body rotational motion + translational motion in three-dimensional space. The Lie group elements, i.e. rigid body transformation matrix, are obtained through exponential mapping, which gives each feature point of the tower point cloud data the physical property of "spatial rigid motion". It has natural spatial invariance and data registration ability. Through Lie group mapping, it has natural robustness to spatial transformations such as rotation, translation, and viewpoint disturbance. It effectively addresses the problem of multi-view, multi-platform, and multimodal data fusion of tower data in actual inspection, fully characterizes the rotational and translational properties of the tower spatial structure, and realizes a unified expression at the physical level.

[0050] (2) A structured position encoding based on Lie groups is designed to provide a high-dimensional structured description of the spatial relationships between feature points in the tower point cloud data. For feature point pairs, the logarithmic mapping of the Lie group is calculated to obtain the minimum spatial transformation relationship, which is then mapped to a structured position encoding via MLP. A weighted term is introduced into the multi-layer structured self-attention mechanism to explicitly integrate the semantic content with the spatial structure consistency, enabling the subsequent attention mechanism to effectively capture the spatial constraints and consistency of the tower's main structure. The structured position encoding directly incorporates spatial constraints into feature fusion, significantly improving the accuracy of segmenting core structures such as the main pole and main axis. It performs exceptionally well, especially in extreme cases such as sparse, occluded, and missing data in the original point cloud data, thus enhancing the discriminability and robustness of subsequent segmentation of the tower's main pole, branches, and other structures.

[0051] (3) Introduce Lie group structure information into the self-attention module. Through the dual weighting of content and structure, the accurate identification and segmentation of key structures such as the main pole of the tower can be achieved. The segmentation of the main pole adopts structural consistency loss to ensure that the distribution of the main pole point set is highly consistent with the physical structure. After the covariance matrix eigenvalue decomposition, the error of the main axis direction is controlled within 0.8 degrees, which ultimately improves the stability and anti-interference ability of tilt angle detection.

[0052] (4) Through the overall multimodal feature generation module, the Lie group structure self-attention module, the feedforward neural network module, and the segmentation detection module, the automatic calculation and output of the tower tilt angle is finally realized. The modules work together to realize the accurate modeling of the tower spatial structure and the intelligent detection of the tilt state. It can be adapted to various sensor platforms and actual working conditions. Without manual intervention, it can automatically complete a series of tasks such as multimodal data fusion, spatial structure recognition and tilt detection. It provides the power operation and maintenance department with an efficient, stable and intelligent structural health monitoring tool, which significantly improves the inspection efficiency and detection reliability.

[0053] In summary, the multimodal point cloud tower tilt detection method provided by this invention is based on the self-attention mechanism of Lie group structure. It maps tower point cloud and image features to a unified Lie group space. By introducing Lie group structure position encoding and self-attention mechanism, it achieves explicit modeling of rigid transformation invariant features and efficient multimodal fusion. It is particularly suitable for monitoring the structural health of power distribution network towers in complex outdoor environments. It effectively makes up for the shortcomings of existing monitoring methods, such as insufficient utilization of geometric consistency of tower multimodal data, weak noise resistance, and insufficient spatial structure modeling. It fundamentally improves the accuracy, generalization ability, and adaptability to complex environments of tower tilt detection, and greatly enhances the intelligence level of tower health detection.

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the process for detecting the tilt of a multimodal point cloud tower based on self-attention of Lie group structure in the embodiments.

[0056] Figure 2 This is a schematic diagram of the self-attention mechanism of the Lie group structure in the embodiment.

[0057] Figure 3 This is a schematic diagram of a multimodal point cloud tower tilt detection system based on Lie group structure self-attention in the embodiment. Detailed Implementation

[0058] Example

[0059] In practical scenarios of daily inspection and structural health monitoring of power distribution network towers, taking the inspection of towers by drones equipped with lidar and high-definition cameras as an example, the on-site equipment can simultaneously collect 3D point cloud and image data to form multimodal raw detection data. After data acquisition, a one-to-one correspondence and physical space pairing between point cloud data and image data is achieved. All spatial sampling points are transformed into information units that integrate multimodal features. These multimodal features are mapped to the parameter space of Lie group Lie algebras with rigid body spatial physical meaning. After deep fusion of the self-attention mechanism of multi-layer Lie group structure and feedforward network, the complex spatial relationships of the tower's main structure and connecting components are effectively captured. Then, the main structure in the point cloud is segmented, and the spatial point set belonging to the tower's main pole is selected. The main axis direction is extracted through statistical analysis. Combined with the spatial vertical direction, the system can directly output the tilt angle of the tower's main axis, realizing automatic assessment of the tower's structural state.

[0060] See Figure 1 The diagram illustrates the flowchart of the multimodal point cloud tower tilt detection method based on Lie group structure self-attention, which includes the following steps:

[0061] S1. Multimodal input data and spatial registration of the tower. The multimodal input data includes point cloud data and RGB image data of the tower. The multimodal input data of the tower is spatially registered at the point cloud and image feature level and then stitched together to obtain the multimodal features of the tower.

[0062] A drone equipped with a LiDAR and a high-definition camera was used to collect images of power distribution network towers during inspection, obtaining point cloud data and RGB image data of the towers. The input tower point cloud data... for:

[0063] ,

[0064] N represents the three-dimensional coordinates of each feature point in the point cloud in global space, and N represents the number of point clouds.

[0065] The synchronously acquired RGB images are:

[0066] ,

[0067] Let h represent the nth RGB image, h be the image height, and w be the image width. Through LiDAR-camera extrinsic parameter calibration, the spatial coordinates of the tower point cloud data are mapped one-to-one with the pixel coordinates of the RGB image data, thus achieving feature-level spatial registration between the point cloud and the image.

[0068] The local point cloud features of each feature point in the tower point cloud data are extracted and stitched together with the corresponding registered image features to obtain the multimodal features of the tower. Specifically, for each feature point in the tower point cloud data... Extract its corresponding local point cloud features This includes the normal, density, and spatial location of points, and extracts image features from the corresponding image data. The high-dimensional semantic features of the image data output by the convolutional neural network are concatenated to output the multimodal features of the tower. ,

[0069] ,

[0070] in , representing the dimension of the multimodal features. The dimension representing the feature of a point cloud. The dimensions of image features.

[0071] S2. Mapping the multimodal features of the tower to a Lie group: The multimodal features of the tower are mapped to six-dimensional Lie algebra parameters. The six-dimensional Lie algebra parameters are then mapped to the Lie group space of rigid body motion to obtain the spatial rigid body transformation matrix of each feature point in the tower point cloud data.

[0072] To express the multimodal features of the tower as parameters with rigid body space meaning, we first map the multimodal features of the tower into a six-dimensional vector of Lie algebra through a linear layer. The mathematical expression for the linear layer is as follows:

[0073] ,

[0074] Where W and b are the learnable parameters of the linear layer. d represents the dimension of the multimodal feature, and R represents the real number field. Let be the rotation component in a six-dimensional vector of a Lie algebra. For the translation component in a six-dimensional vector of a Lie algebra, The multimodal feature of the i-th feature point in the tower point cloud data, where the rotation component Describe the orientation or minute torsion of a tower's local structure in three-dimensional space, including translational components. The corresponding actual spatial position or relative displacement of the tower.

[0075] For each feature point in the tower point cloud data, the corresponding six-dimensional Lie algebra vector is mapped to the Lie group space of rigid body motion. For each feature point, the Lie algebra six-dimensional vector ( Construct the Lie algebra matrix using the following formula :

[0076] ,

[0077] in, for An antisymmetric matrix describing the rotational part; each feature point in the tower point cloud data is mapped via a Lie group exponent. This yields the spatial rigid body transformation matrix for all feature points of the tower. This step ensures that all feature points of the tower are uniformly aligned through rotation and translation, regardless of the data source mode or acquisition angle, and can be represented using Lie group elements. Accurately characterize the spatial relationships of corresponding feature points.

[0078] S3. Construct a self-attention mechanism for Lie group structures. Obtain the relative structural quantities between feature points in the Lie group space through the spatial rigid body transformation matrix. Construct the self-attention mechanism for Lie group structures to fuse and output the initial multimodal feature matrix of all feature points in the tower point cloud data.

[0079] From the tower point cloud data, adjacent feature points are selected using local adjacency sampling. For each reference feature point, its nearest neighbor points are selected in its point cloud or fused feature space based on Euclidean distance or feature similarity to form a structural relationship feature point pair. The relative structural quantity of adjacent feature points in the Lie group space is calculated using the following formula:

[0080] ,

[0081] in, , Let i be the Lie group element of the spatial rigid body transformation matrix of adjacent feature points i and j. Let be a six-dimensional vector of Lie algebra, which reflects the minimum rigid body motion of adjacent feature points i and j in rotation and translation.

[0082] The location structure is encoded as follows , As a learnable multilayer perceptron, it adaptively mines spatial structural patterns. Based on structural position encoding, it incorporates spatial rigid structures into attention allocation. Instead of relying solely on point-to-point distance or feature similarity, it deeply integrates spatial physical constraints, fundamentally improving the scientific rigor and effectiveness of tower structure recognition and spatial invariance modeling.

[0083] The self-attention mechanism for Lie group structures that integrates content and structural information is constructed as follows:

[0084] ,

[0085] This represents a structure-enhanced self-attention mechanism. Based on the original attention mechanism, it introduces the Lie group structure position encoding PE₍ᵢⱼ₎ as a structural constraint term, enhancing the spatial structure modeling capability. , , , , and It projects the original features in the attention mechanism onto the weights of the query, key, and value space. Softmax is a well-known normalization function that maps a set of real numbers to a probability distribution. For the learnable structural weights of the attention mechanism, The scaling factor for the attention mechanism. It is the initial multimodal feature matrix, which is the sequential stacking of the multimodal features of all feature points in the tower point cloud data. N represents the number of feature points in the tower point cloud data.

[0086] The self-attention mechanism of the Lie group structure constructed in steps S2 and S3 above is as follows: Figure 2 As shown, the structural information attention mechanism ensures that the attention weights reflect both content similarity and spatial structural constraints, enabling structural units such as the main pole, branches, and base of the tower to naturally "recognize" each other in a multimodal space. This method outperforms the ordinary Transformer, which only uses feature similarity weighting, and is particularly suitable for power pole scenarios with strong structural consistency and diverse spatial transformations.

[0087] S4. Construct the overall structure of the tower. The initial multimodal feature matrix is ​​fitted by a feedforward neural network to output the fused features of all feature points of the tower point cloud data.

[0088] The feedforward neural network is cascaded with a multi-layer feedforward neural network and a Lie group structure self-attention layer in step S3. The output of each Lie group structure self-attention layer is used as the input of the feedforward neural network layer. The fused feature expression of all feature points of the output tower point cloud data is as follows:

[0089] ,

[0090] This is the initial multimodal feature matrix for all feature points in the tower point cloud data. The fused features are the output of the l-th layer feedforward neural network. This represents the self-attention mechanism of the l-th layer Lie group structure, including the calculation of Query, Key, and Value, and the weighted encoding of structural positions. The l-th layer is a feedforward neural network layer, which is a standard feedforward network module, consisting of two MLP layers and one activation function layer, and includes normalization and residual connections. L is the number of self-attention layers in the Lie group structure, and F is the fusion feature of all feature points in the tower point cloud data output by the last feedforward neural network layer, which serves as the basis for the global high-dimensional features of the tower for main pole segmentation and main axis fitting.

[0091] Each layer of the Lie group self-attention mechanism combines the structural relationships and content information of the Lie group in the tower features, continuously strengthening the spatial physical constraints between feature points. This multi-layer stacking enables the model to capture the main structure of the tower, complex connectors, and the global spatial context, effectively improving the discriminative power and robustness of the main structure. The feedforward network performs nonlinear transformations and enriches the feature space, while normalization and residual connections ensure convergence and generalization.

[0092] The number of layers L in the self-attention mechanism layer and the feedforward neural network layer can be flexibly set according to the complexity of the actual point cloud data, the structural representation requirements, and the availability of computing resources. Typically, it ranges from 2 to 6 layers, ensuring the accuracy of main tower structure recognition while balancing model training stability and inference efficiency. L=4 is the optimal configuration used in the experiments of this embodiment.

[0093] S5. Segmentation and tilt angle output of the main pole structure: The probability of each feature point in the pole point cloud data belonging to the main pole is predicted by feature segmentation through feature point fusion. The main pole point set is selected by setting a probability threshold. The main pole direction is obtained through the center of the main pole. The tilt angle of the pole is obtained according to the angle between the main pole direction and the vertical direction.

[0094] Output high-dimensional fusion features The input is fed into the segmentation branch, and the probability that each feature point belongs to the main pole of the tower is predicted using the following formula:

[0095] This represents the predicted probability that the i-th feature point in the tower point cloud data belongs to the main pole of the tower. This represents the fused feature vector of the nth feature point in the tower point cloud data, which originates from the output of step S4. This represents the learnable weight vector used for classification. For bias terms, This is the activation function, used to compress the result into probabilities between 0 and 1;

[0096] By setting a probability threshold Select the main pole point set of the tower :

[0097] ,

[0098] The center of the main tower is calculated using the following formula:

[0099] ,

[0100] Main club point concentration main club point Spatial location coordinates, The center of the main pole of the tower, that is, the centroid or center point of the main pole of the tower, refers to the spatial average position of all points in the tower's point set.

[0101] The covariance matrix of the tower's main pole center is calculated as follows:

[0102] ,

[0103] The eigenvalue decomposition of the covariance matrix of the main pole center is as follows:

[0104] ,

[0105] The characteristic value of the decomposition of the main pole center of the tower is... Let k be the eigenvector corresponding to the eigenvalue, and k be the number of the nth eigencomponent in the eigenvalue decomposition of the covariance matrix of the pole point cloud. The largest eigenvalue is the direction in which the distribution of the pole point cloud set has the greatest extensibility in three-dimensional space, which is the longest structural direction. This conforms to the actual definition of the pole's main axis. Therefore, the eigenvector corresponding to the largest eigenvalue is chosen as the main axis direction of the pole point set, i.e., the main pole direction in the pole space.

[0106] Define the vertical direction as The angle between the direction of the main pole of the tower and the vertical direction is:

[0107] ,

[0108] The angle between the direction of the main pole of the tower and the vertical direction. The direction of the main pole of the tower.

[0109] When multiple towers exist, first perform spatial clustering, connected component analysis, template matching, or point cloud neighborhood growth on all feature points in the tower point cloud data to separate the point sets corresponding to different main towers, denoted as follows: , …and then output the main pole direction of each tower separately. , …Finally, calculate the angle between the direction of each main rod and the vertical direction. , …

[0110] The structural segmentation branch for predicting the main pole structure is trained and tested through the above process. Specifically, the structural segmentation branch uses a lightweight multilayer perceptron (MLP) classifier based on fused features, supplemented by structural consistency loss for optimization. During training, real main pole labels are used as supervision signals, and the network parameters are optimized by fusing BCE loss and Lie group spatial structural consistency loss, thereby achieving high-precision segmentation of the main pole structure and extraction of the principal axis direction.

[0111] Specifically, the multimodal fusion features of the tower with marked tilt angles are input into the structural segmentation branch to predict the tilt angle of the main tower. During training, the segmentation accuracy and tilt angle error are evaluated using the marked tilt angles and the predicted tilt angles. Additionally, a structural consistency loss is introduced, namely a structural loss function based on the logarithmic mapping of Lie groups, which directly optimizes the geometric consistency of the segmentation point set in the Lie group space using the following formula:

[0112] .

[0113] , This represents the true Lie group pose transformation matrix (SE(3) element) corresponding to feature points i and j in the tower point cloud data in the training data, used to provide supervision signals, and compared with the model predictions. , In contrast, the constraint model maintains the spatial structural consistency of the master point set.

[0114] By using a structural consistency loss function based on the logarithmic mapping of Lie groups, the geometric consistency of the split point set in the Lie group space can be directly optimized. Unlike the traditional Euclidean space point set loss, this method can effectively ensure the physical rationality of principal axis extraction and spatial structure.

[0115] like Figure 3As shown in the figure, a multimodal point cloud tower tilt detection system based on Lie group structure self-attention using the above method is disclosed, including a multimodal feature generation module 100, a Lie group structure self-attention module 200, a feedforward neural network module 300, and a segmentation detection module 400.

[0116] The multimodal feature generation module 100 is used to input multimodal input data of the tower and perform spatial registration. The multimodal input data includes point cloud data and RGB image data of the tower. The multimodal input data of the tower is spatially registered at the point cloud and image feature level and then stitched together to obtain the multimodal features of the tower.

[0117] The Lie group structure self-attention module 200 constructs a Lie group structure self-attention mechanism, mapping the multimodal features of the tower to a Lie group. The multimodal features of the tower are mapped to six-dimensional Lie algebra parameters, which are then mapped to the Lie group space of rigid body motion. This yields the spatial rigid body transformation matrix of each feature point in the tower point cloud data. The structural relative quantities between feature points in the tower point cloud data in the Lie group space are obtained through the spatial rigid body transformation matrix. Finally, the Lie group structure self-attention mechanism is constructed to fuse and output the initial multimodal feature matrix of all feature points in the tower point cloud data.

[0118] The feedforward neural network module 300 adopts a multi-layer feedforward neural network layer and a multi-layer Lie group structure self-attention layer set by the Lie group structure self-attention module. The output of each Lie group structure self-attention layer is used as the input of the feedforward neural network layer. The feedforward neural network fits the initial multimodal feature matrix to output the fused features of all feature points of the tower point cloud data, and constructs the overall structure of the tower.

[0119] The segmentation and detection module 400 segments the main pole structure and outputs the tilt angle. It predicts the probability that each feature point in the pole point cloud data belongs to the main pole by using feature point fusion segmentation. It selects the main pole point set by setting a probability threshold, obtains the main pole direction through the center of the main pole, and obtains the pole tilt angle based on the angle between the main pole direction and the vertical direction.

[0120] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0121] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multi-modal point cloud tower tilt detection based on Lie group structure self-attention, characterized in that: Comprising the following steps: S1, the multi-modal input data of the tower and the spatial registration, the multi-modal input data of the tower including point cloud data and RGB image data of the tower, the multi-modal input data of the tower is registered and spliced to obtain the multi-modal feature of the tower. S2, the multi-modal feature of the tower is mapped to the Lie group, the multi-modal feature of the tower is mapped to a six-dimensional Lie algebra parameter, the six-dimensional Lie algebra parameter is mapped to the Lie group space of rigid body motion, and the spatial rigid transformation matrix of each feature point in the tower point cloud data is obtained. S3, construct Lie group structure self-attention mechanism, obtain the structure relative quantity of feature points in tower point cloud data in Lie group space through spatial rigid transformation matrix, construct Lie group structure self-attention mechanism to fuse the initial multi-modal feature matrix of all feature points in tower point cloud data, the process is as follows: From the tower point cloud data, the adjacent feature points are selected by local adjacent sampling, and the structure relative quantity of the adjacent feature points in the Lie group space is calculated by the following formula: ， wherein, , is the Lie group element of the spatial rigid body transformation matrix of neighboring feature points i and j, is the Lie algebra six-dimensional vector; The positioning structure position encoding is: , is a learnable multi-layer perceptron, and the Lie group structure self-attention mechanism is constructed as follows: ， wherein, , , , , and are weights that project the original features in the attention mechanism to the query, key, value space, softmax is a normalization function, is a learnable structure weight of the attention mechanism, is a scaling factor of the attention mechanism, is an initial multi-modal feature matrix of all multi-modal features of the feature points in the tower-pole point cloud data stacked in order, , N represents the number of feature points in the tower-pole point cloud data. S4, construct the overall structure of the tower, the initial multi-modal feature matrix is fitted and output by the feedforward neural network to obtain the fusion feature of all feature points in the tower point cloud data. S5, tower main rod structure segmentation and inclination angle output, the probability that each feature point in the tower point cloud data belongs to the tower main rod is segmented and predicted through the fusion feature of the feature point, the tower main rod point set is selected by setting the probability threshold, the main rod direction is obtained through the tower main rod center, and the inclination angle of the tower is obtained according to the included angle between the main rod direction and the vertical direction.

2. The multi-modal point cloud tower leaning detection method based on Lie group structure self-attention according to claim 1, characterized in that: In the step S1, the unmanned aerial vehicle is used to carry out image acquisition on the tower by laser radar and high-definition camera, and the point cloud data and RGB image data of the tower are obtained. The point cloud space coordinates of the tower point cloud data are corresponded with the pixel coordinates of the RGB image data one by one through laser radar-camera external parameter calibration, so as to realize the point cloud and image feature level spatial registration.

3. The multi-modal point cloud tower leaning detection method based on Lie group structure self-attention according to claim 2, characterized in that: The local point cloud feature of each feature point in the tower point cloud data and the corresponding image feature after registration are spliced to obtain the multi-modal feature of the tower.

4. The multi-modal point cloud tower leaning detection method based on Lie group structure self-attention according to claim 1, characterized in that: In the step S2, the multi-modal feature of the tower is first mapped to a six-dimensional Lie algebra vector through a linear layer: ， wherein W, b are learnable parameters of the linear layer, , d represents the dimension of the multi-modal feature, R represents the real number field, is the rotation component in the Lie algebra six-dimensional vector, is the translation component in the Lie algebra six-dimensional vector, is the multi-modal feature of the i-th feature point in the tower point cloud data; For each feature point in the tower point cloud data, the corresponding Lie algebra six-dimensional vector is mapped to the Lie group space of rigid body motion, and a Lie algebra matrix is constructed by the following formula : ， wherein is antisymmetric matrix, through Lie group exponential mapping to get the spatial rigid body transformation matrix of each feature point in the tower point cloud data.

5. The Lie group structure self-attention-based multi-modal point cloud tower leaning detection method according to claim 1, characterized in that: In the step S4, the feedforward neural network is composed of multiple feedforward neural network layers and Lie group structure self-attention layers in step S3, and the fusion feature expression of all feature points in the tower point cloud data is as follows: ， is an initial multi-modal feature matrix of all feature points in the tower point cloud data, is a fusion feature output by the l-th feedforward neural network, represents an l-th layer Lie group structure self-attention mechanism, is an l-th layer feedforward neural network layer, L is the number of Lie group structure self-attention layers, and F is a fusion feature of all feature points in the tower point cloud data output by the last layer feedforward neural network layer.

6. The Lie group structure self-attention-based multi-modal point cloud tower leaning detection method according to claim 1, characterized in that: In the step S5, the probability that each feature point belongs to the tower main rod is predicted by the following formula: ， denotes the predicted probability that the i-th feature point in the tower point cloud data belongs to the tower main pole, denotes the fusion feature vector of the i-th feature point in the tower point cloud data, which is derived from the output of step S4, denotes the learnable weight vector for classification, is a bias term, is an activation function used to compress the result into a probability between 0 and 1; By a set probability threshold Selecting a set of main pole points of a tower : ， The center of the main pole of the tower is calculated by the following formula : ， the spatial position coordinates of the main rod point the main rod point The covariance matrix of the tower main rod center is calculated, and the eigenvalue decomposition is performed on the covariance matrix of the tower main rod center, wherein the eigenvector corresponding to the maximum eigenvalue is the main rod direction of the tower.

7. The Lie group structure self-attention-based multi-modal point cloud tower leaning detection method according to claim 1, characterized in that: When there are multiple towers, first, the spatial clustering or connected domain analysis or template matching or point cloud field growth is performed on all feature points in the tower point cloud data, the point sets corresponding to different main rods are separated, and then the main rod directions of each tower are output.

8. The Lie group structure self-attention-based multi-modal point cloud tower leaning detection method according to claim 1, characterized in that: In the step S5, the vertical direction is defined as , and the angle θ between the main rod direction of the tower and the vertical direction is: ， is the main pole direction of the tower.

9. A multi-modal point cloud tower tilt detection system based on Lie group structure self-attention, characterized in that: Comprising: The multi-modal feature generation module inputs the multi-modal input data of the tower and performs spatial registration, the multi-modal input data including point cloud data and RGB image data of the tower, and performs point cloud and image feature level spatial registration on the multi-modal input data of the tower to obtain multi-modal features of the tower by splicing; The Lie group structure self-attention module constructs a Lie group structure self-attention mechanism, maps the multi-modal features of the tower to a Lie group, maps the multi-modal features of the tower to six-dimensional Lie algebra parameters, maps the six-dimensional Lie algebra parameters to the Lie group space of rigid body motion, obtains the spatial rigid body transformation matrix of each feature point in the tower point cloud data, obtains the structural relative quantity of the feature points in the tower point cloud data in the Lie group space through the spatial rigid body transformation matrix, constructs the Lie group structure self-attention mechanism to fuse the initial multi-modal feature matrix of all feature points in the tower point cloud data, selects adjacent feature points from the tower point cloud data by local adjacent sampling, and calculates the structural relative quantity of the adjacent feature points in the Lie group space by the following formula: ， wherein, , is the Lie group element of the spatial rigid body transformation matrix of neighboring feature points i and j, is the Lie algebra six-dimensional vector; The positioning structure position encoding is: , is a learnable multi-layer perceptron, and the Lie group structure self-attention mechanism is constructed as follows: ， wherein, , , , , and are weights that project the original features in the attention mechanism to the query, key, value space, softmax is a normalization function, is a learnable structure weight of the attention mechanism, is a scaling factor of the attention mechanism, is an initial multi-modal feature matrix of the multi-modal features of all feature points in the tower-pole point cloud data stacked in order, , N represents the number of feature points in the tower-pole point cloud data; The feedforward neural network module adopts a multi-layer feedforward neural network layer and a Lie group structure self-attention layer of the Lie group structure self-attention module to cascade combination, fits the initial multi-modal feature matrix to output the fusion features of all feature points in the tower point cloud data through the feedforward neural network, and constructs the overall structure of the tower; The segmentation detection module segments the tower main rod structure and outputs the inclination angle, segments the probability of each feature point in the tower point cloud data belonging to the tower main rod through the fusion features of the feature points, selects the tower main rod point set by setting a probability threshold, obtains the main rod direction through the tower main rod center, and obtains the inclination angle of the tower according to the included angle between the main rod direction and the vertical direction.

Images