Power transmission line multi-mode inspection data fusion method based on deep learning
By constructing a unified spatiotemporal coordinate system and using a dedicated encoder and mutual attention mechanism to fuse multimodal inspection data, the problem of heterogeneous data fusion was solved, and efficient and accurate transmission line inspection data fusion and intelligent decision-making were achieved.
Patent Information
- Application Number
- CN202610184138.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies struggle to effectively integrate heterogeneous information from multiple sources, such as UAV images, lidar point cloud data, online monitoring data, satellite remote sensing images, and historical inspection data. This results in significant differences in data resolution, structure, and physical meaning during power transmission line inspections, and the lack of a unified labeling and alignment mechanism makes it difficult to support inferences about complex fault mechanisms.
A unified spatiotemporal coordinate system is constructed to align multimodal inspection data spatiotemporally. A dedicated encoder is used to extract features and project them onto a unified dimensional feature space. A mutual attention mechanism is used for advanced semantic feature fusion. A cross-modal semantic collaboration module is introduced for dynamic weighted fusion to optimize model parameters.
It has achieved efficient and accurate fusion of multimodal inspection data, improved the comprehensive perception and intelligent decision-making capabilities of transmission lines, and enhanced the ability to perceive anomalies and defects in complex environments.
Smart Images
Figure CN122046246A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to substation inspection, specifically to a deep learning-based method for multimodal inspection data fusion of transmission lines. Background Technology
[0002] Transmission lines are critical infrastructure of the power system, and their operational status directly affects the safety and stability of the power system. In recent years, various automated methods, including drone inspections, online monitoring devices, and satellite remote sensing, have been deployed for transmission line inspections. These inspection methods can acquire a large amount of sensory data about transmission line equipment and its surrounding environment, providing important information for defect detection, condition assessment, and risk prediction.
[0003] However, the data collected by existing transmission line inspection methods mainly have the following problems: 1) The data from drone inspections mainly includes image data and point cloud data. Image data can be used to identify external defects such as insulator damage and hardware corrosion, but it is easily affected by factors such as lighting, weather, and viewing angle, and it is difficult to reflect the internal operating status of the equipment. Point cloud data can accurately reconstruct the three-dimensional structure of transmission lines, but it lacks texture and color information and is prone to introducing noise in complex terrain or vegetation-covered environments. 2) Online monitoring devices can collect thermal, electrical and mechanical status parameters of equipment, but the data dimensions are limited and lack spatial correlation, making it difficult to fully perceive the environment; 3) Satellite remote sensing can provide macro-environmental information along the route, such as topographic relief, vegetation distribution, and external risk sources, but its spatial resolution and timeliness are limited, and it is easily affected by clouds and fog, making it difficult to identify minor defects. 4) Historical inspection data contains rich expert experience and defect evolution patterns, and is an important source of information. However, it is mostly structured text or scattered images, lacking a unified data format and semantic organization, making it difficult to associate and utilize with newly collected data.
[0004] To improve the intelligence level of power transmission line inspection, existing research has attempted joint analysis of multi-source data, such as fusing UAV images and point cloud data to achieve 3D defect detection, or combining online monitoring data and historical inspection data for trend prediction. However, traditional multimodal data fusion methods still face many challenges when dealing with heterogeneous data in power transmission line scenarios: different modal inspection data vary significantly in resolution, structure, and physical meaning; spatiotemporal inconsistencies in data acquisition lead to a lack of unified annotation and alignment mechanisms; existing methods mostly rely on shallow feature stitching, making it difficult to explicitly model equipment topology, component relationships, and maintenance knowledge, thus failing to effectively support complex fault mechanism reasoning. Therefore, there is an urgent need to develop a deep learning data fusion method that can unify and integrate multi-source information such as UAV images, LiDAR point cloud data, online monitoring data, satellite remote sensing images, and historical inspection data to achieve multimodal deep semantic understanding and improve the comprehensive perception and intelligent decision-making capabilities of power transmission lines. Summary of the Invention
[0005] (a) Technical problems to be solved In view of the above-mentioned shortcomings of the existing technology, the present invention provides a deep learning-based method for fusing multimodal inspection data of transmission lines, which can effectively overcome the shortcomings of the existing technology in that it is difficult to effectively fuse multimodal inspection data of transmission lines.
[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A deep learning-based method for fusing multimodal inspection data of transmission lines includes the following steps: S1. Construct a unified spatiotemporal coordinate system and map the multimodal inspection data to this coordinate system to perform spatiotemporal alignment of the multimodal inspection data; S2. Based on the characteristics of different modalities, a dedicated encoder is used to extract features from the inspection data of each modality and project them onto a feature space of a unified dimension. S3. Utilize the mutual attention mechanism to dynamically weight and fuse the high-level semantic features between different modal inspection data to obtain a global perception vector; S4. Input the global perception vector into the downstream task network and optimize the model parameters using the corresponding loss function.
[0007] Preferably, in S1, a unified spatiotemporal coordinate system is constructed, and the multimodal inspection data is mapped to this coordinate system. Spatiotemporal alignment of the multimodal inspection data is performed, including: For drone image sets I t Let E be a single frame image captured by the UAV at time t, where T is the time step. The extrinsic parameter matrix E is estimated using a dense reconstruction algorithm based on SfM.t This yields the pose (R) of each image frame in a unified spatiotemporal coordinate system. t ,t t ), R t Let be the rotation matrix at time t, where t t Let be the translation vector at time t; For lidar point cloud data sets , Let T be the coordinates of the i-th 3D point. Each 3D point consists of three real coordinates, determined by the extrinsic calibration matrix T. Li Complete the coordinate transformation to a unified spacetime coordinate system: ; in, Let i be the coordinates of the i-th 3D point in the unified spacetime coordinate system. The coordinates of the i-th 3D point Homogeneous coordinate representation; For online monitoring data sets and satellite remote sensing image collection S k (t) represents the data collected by the k-th sensor at time t, R j For the j-th satellite remote sensing image, it is bound to the line node in a unified spatiotemporal coordinate system through GIS geographic coordinates, and time alignment is achieved using linear interpolation: ; ; in, To bind online monitoring data to relevant data S on line nodes in a unified spatiotemporal coordinate system t Data corresponding to time t in a unified spatiotemporal coordinate system Indicates the relevant data S t Perform linear interpolation at time t; Binding satellite remote sensing images to relevant data R on line nodes in a unified spatiotemporal coordinate system t Data corresponding to time t in a unified spatiotemporal coordinate system This indicates the relevant data R t Linear interpolation is performed at time t.
[0008] Preferably, in S2, based on the characteristics of different modalities, a dedicated encoder is used to extract features from the inspection data of each modality and project them onto a feature space of a unified dimension, including: S21, For UAV image I t and satellite remote sensing images R jFeature extraction is performed using the Vision Transformer (ViT). ViT captures long-range dependencies and global semantic information in images through a self-attention mechanism. Compared with traditional convolutional networks, it can better model complex textures and spatial relationships, making the global feature representation more robust and distinguishable. ; ; Among them, f v To obtain images from drones t The global features extracted from f r To obtain R from satellite remote sensing images j The global features extracted are PatchEmbed, which represents the segmentation of the input image into B×B image patches and the linear mapping of these image patches into an embedding vector sequence for input into the Vision Transformer (ViT) for processing. CLS_token is a classification label, which is a learnable vector that is added to the beginning of the embedding vector sequence. After processing by the Vision Transformer (ViT), its final state can be used as the global feature representation of the entire image. For lidar point cloud data sets The PointNet feature encoding network is used for feature encoding. PointNet uses a multilayer perceptron (MLP) and global pooling to extract geometric structural features from point cloud data. It is insensitive to the order of the input point cloud data, maintains rotation and translation invariance, and ensures a complete representation of the target shape information. ; Among them, f p MaxPool represents the global features extracted from the lidar point cloud data set P, where MaxPool represents the aggregation of global features for all points. For online monitoring data sets Feature encoding is performed using a recurrent neural network (GRU). The GRU captures long-short-term dependencies through a gating mechanism, effectively modeling the dynamic changes in time-series data. It can memorize and predict continuous state changes, enhancing the model's ability to understand temporal information. ; Among them, f s These are global features extracted from the online monitoring dataset S; For historical inspection data H, a knowledge graph is first constructed, and then a graph neural network (GNN) is used for feature extraction. By embedding node and neighborhood information through graph convolution operations, the structured knowledge of historical inspection data H can be transformed into semantic features that can be used for fusion, while preserving complex relationship information between data, and providing prior knowledge support for decision-making. ; Among them, f h KG(H) represents the global features extracted from historical inspection data H, where KG(H) represents the knowledge graph constructed from historical inspection data H, and GraphConv represents the graph convolution operation. S22. Map the global features of each modal inspection data to a feature space of a unified dimension through a linear projection layer: ; ; ; ; ; Among them, f v 'For images from drones I t The global features f extracted v The feature vectors W are mapped to a feature space of uniform dimension via linear projection. v b v These are the global features f v The corresponding weights and biases of the linear projection layer; f r 'For satellite remote sensing image R j The global features f extracted r The feature vectors W are mapped to a feature space of uniform dimension via linear projection. r b r These are the global features f r The corresponding weights and biases of the linear projection layer; f p 'The global feature f extracted from the lidar point cloud data set P' p The feature vectors W are mapped to a feature space of uniform dimension via linear projection. p b p These are the global features f p The corresponding weights and biases of the linear projection layer; f s 'The global feature f extracted from the online monitoring dataset S' s The feature vectors W are mapped to a feature space of uniform dimension via linear projection. s b s These are the global features f s The corresponding weights and biases of the linear projection layer; f h 'The global feature f extracted from historical inspection data H' hThe feature vectors W are mapped to a feature space of uniform dimension via linear projection. h b h These are the global features f h The corresponding weights and biases of the linear projection layer; S23. Finally, the multimodal joint feature F is obtained: .
[0009] Preferably, in S3, a mutual attention mechanism is used to dynamically weight and fuse high-level semantic features between inspection data from different modalities to obtain a global perception vector, including: Based on a unified feature space, a cross-modal semantic collaboration module (CMSAM) is introduced. It utilizes a cross-attention mechanism to achieve high-level semantic interaction and collaborative modeling between inspection data from different modalities. CMSAM can correct semantic deviations between features of different modalities, aligning features of different modalities in a unified semantic space. It can also dynamically mine complementary information between inspection data from different modalities. In addition, it can effectively suppress redundant information and noise interference, improve the quality of the fused global perception vector, and provide more robust and reliable feature inputs for downstream tasks.
[0010] Preferably, in S3, a mutual attention mechanism is used to dynamically weight and fuse high-level semantic features between inspection data from different modalities to obtain a global perception vector, including: S31. For each modal feature in the multimodal joint feature F , Let m be the query modality and n be the key modality. A corresponding query vector Q is generated through linear transformation. m Key vector K n Sum vector V n : ; ; ; Among them, W Q W K W V All are weight matrices of linear transformation; S32. Calculate the cross-modal weight matrix A between the query modality m and the key modality n using the cross-attention mechanism. mn : ; Where d is the feature dimension and softmax is the activation function; S33, Using the cross-modal weight matrix Amn Obtain the cross-modal fusion features of query modality m : ; S34. Fuse the cross-modal fusion features of all query modalities to obtain the global perception vector F. fuse : ; in, To query the cross-modal fusion features of modality m The corresponding attention weights .
[0011] Preferably, in S4, the global perception vector is input to the downstream task network, and the model parameters are optimized using the corresponding loss function, including: The global perception vector F fuse The input is fed into a downstream task network comprising three tasks: defect detection, risk assessment, and inspection decision-making. During model training, the model parameters are optimized using a multi-task joint loss function L. ; Among them, L det For target detection loss, L risk For the risk score regression loss, L cls For state classification loss, , , All of these are adaptive weight coefficients, which can be dynamically adjusted during model training to balance the convergence speed of each task.
[0012] (III) Beneficial Effects Compared with existing technologies, the deep learning-based multimodal inspection data fusion method for transmission lines provided by this invention has the following advantages: 1) It can simultaneously process multi-source heterogeneous information such as UAV images, LiDAR point cloud data, online monitoring data, satellite remote sensing images, and historical inspection data. By unifying the spatiotemporal coordinate system and feature space, it aligns and normalizes data of different modalities, eliminating differences in timestamps, spatial references, and scales, thereby improving the efficiency and accuracy of multi-source information fusion. 2) Design dedicated encoders for data of different modalities (such as Vision Transformer ViT, PointNet, GRU, and GNN), and combine them with an adaptive cross-modal fusion feature attention weight mechanism. The model can dynamically adjust the feature weights according to the contribution of each modal data during training, thereby enhancing the expressive power of different modal features in a unified space and improving the richness and discriminative power of multimodal representation. 3) Introducing the cross-modal semantic collaboration module CMSAM and the cross-attention mechanism enables high-level semantic interaction and collaborative modeling between different modal data. By dynamically calculating the attention weights between cross-modal fusion features of each modality, the semantic deviation between features of each modality can be effectively corrected, the semantic consistency of the global perception vector can be enhanced, and the model's ability to perceive anomalies, defects and risks in complex environments can be improved. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0014] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram illustrating the use of a dedicated encoder in this invention to extract features from the inspection data of each modality and project them onto a feature space of a unified dimension. Figure 3 This is a schematic diagram illustrating the dynamic weighted fusion of high-level semantic features between different modal inspection data using a mutual attention mechanism in this invention. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0016] The following describes the specific process of the deep learning-based multimodal inspection data fusion method for transmission lines provided by this invention, using concrete examples (e.g.) Figure 1 (As shown).
[0017] To address the issue of inconsistencies in timestamps, spatial references, and resolutions among multi-source data, step S1 is first adopted: S1. Construct a unified spatiotemporal coordinate system and map the multimodal inspection data to this coordinate system. Perform spatiotemporal alignment on the multimodal inspection data, including: For drone image sets I tLet E be a single frame image captured by the UAV at time t, where T is the time step. The extrinsic parameter matrix E is estimated using a dense reconstruction algorithm based on SfM (Structure-from-Motion). t This yields the pose (R) of each image frame in a unified spatiotemporal coordinate system. t ,t t ), R t Let be the rotation matrix at time t, where t t Let be the translation vector at time t; For lidar point cloud data sets , Let T be the coordinates of the i-th 3D point. Each 3D point consists of three real coordinates, determined by the extrinsic calibration matrix T. Li Complete the coordinate transformation to a unified spacetime coordinate system: ; in, Let i be the coordinates of the i-th 3D point in the unified spacetime coordinate system. The coordinates of the i-th 3D point Homogeneous coordinate representation; For online monitoring data sets and satellite remote sensing image collection S k (t) represents the data collected by the k-th sensor at time t, R j For the j-th satellite remote sensing image, it is bound to the line node in a unified spatiotemporal coordinate system through GIS geographic coordinates, and time alignment is achieved using linear interpolation: ; ; in, To bind online monitoring data to relevant data S on line nodes in a unified spatiotemporal coordinate system t Data corresponding to time t in a unified spatiotemporal coordinate system Indicates the relevant data S t Perform linear interpolation at time t; Binding satellite remote sensing images to relevant data R on line nodes in a unified spatiotemporal coordinate system t Data corresponding to time t in a unified spatiotemporal coordinate system This indicates the relevant data R t Linear interpolation is performed at time t.
[0018] In this invention, the inspection data of different modalities are heterogeneous, including the two-dimensional texture and spatial structure information of image data, the three-dimensional geometric shape of point cloud data, the dynamic changes of time-series sensor data, and the knowledge structure of historical inspection data. To address the characteristics of different modalities, a dedicated encoder is used to extract features from the inspection data of each modality, thereby fully mining the semantic information within each modality and reducing the differences between modalities.
[0019] S2. Based on the characteristics of different modalities, a dedicated encoder is used to extract features from the inspection data of each modality and project them onto a feature space of a unified dimension, such as... Figure 2 As shown, it includes: S21, For UAV image I t and satellite remote sensing images R j Feature extraction is performed using the VisionTransformer (ViT). VisionTransformer captures long-range dependencies and global semantic information in images through a self-attention mechanism. Compared with traditional convolutional networks, it can better model complex textures and spatial relationships, making the global feature representation more robust and distinguishable. ; ; Among them, f v To obtain images from drones t The global features extracted from f r To obtain R from satellite remote sensing images j The global features extracted are PatchEmbed, which represents the segmentation of the input image into B×B image patches and the linear mapping of these image patches into an embedding vector sequence for input into the Vision Transformer (ViT) for processing. CLS_token is a classification label, which is a learnable vector that is added to the beginning of the embedding vector sequence. After processing by the Vision Transformer (ViT), its final state can be used as the global feature representation of the entire image. For lidar point cloud data sets The PointNet feature encoding network is used for feature encoding. PointNet uses a multilayer perceptron (MLP) and global pooling to extract geometric structural features from point cloud data. It is insensitive to the order of the input point cloud data, maintains rotation and translation invariance, and ensures a complete representation of the target shape information. ; Among them, f p MaxPool represents the global features extracted from the lidar point cloud data set P, where MaxPool represents the aggregation of global features for all points. For online monitoring data sets Feature encoding is performed using a recurrent neural network (GRU). The GRU captures long-short-term dependencies through a gating mechanism, effectively modeling the dynamic changes in time-series data. It can memorize and predict continuous state changes, enhancing the model's ability to understand temporal information. ; Among them, f s These are global features extracted from the online monitoring dataset S; For historical inspection data H, a knowledge graph is first constructed, and then a graph neural network (GNN) is used for feature extraction. By embedding node and neighborhood information through graph convolution operations, the structured knowledge of historical inspection data H can be transformed into semantic features that can be used for fusion, while preserving complex relationship information between data, and providing prior knowledge support for decision-making. ; Among them, f h KG(H) represents the global features extracted from historical inspection data H, where KG(H) represents the knowledge graph constructed from historical inspection data H, and GraphConv represents the graph convolution operation. S22. Map the global features of each modal inspection data to a feature space of a unified dimension through a linear projection layer: ; ; ; ; ; Among them, f v 'For images from drones I t The global features f extracted v The feature vectors W are mapped to a feature space of uniform dimension via linear projection. v b v These are the global features f v The corresponding weights and biases of the linear projection layer; f r 'For satellite remote sensing image R j The global features f extracted r The feature vectors W are mapped to a feature space of uniform dimension via linear projection. r b r These are the global features f r The corresponding weights and biases of the linear projection layer; f p 'The global feature f extracted from the lidar point cloud data set P' pThe feature vectors W are mapped to a feature space of uniform dimension via linear projection. p b p These are the global features f p The corresponding weights and biases of the linear projection layer; f s 'The global feature f extracted from the online monitoring dataset S' s The feature vectors W are mapped to a feature space of uniform dimension via linear projection. s b s These are the global features f s The corresponding weights and biases of the linear projection layer; f h 'The global feature f extracted from historical inspection data H' h The feature vectors W are mapped to a feature space of uniform dimension via linear projection. h b h These are the global features f h The corresponding weights and biases of the linear projection layer; S23. Finally, the multimodal joint feature F is obtained: .
[0020] S3. Utilize a mutual attention mechanism to dynamically weight and fuse high-level semantic features among inspection data from different modalities to obtain a global perception vector, including: Based on a unified feature space, a cross-modal semantic alignment module (CMSAM) is introduced. Utilizing cross-attention, this module enables high-level semantic interaction and collaborative modeling between inspection data from different modalities. CMSAM corrects semantic deviations between features of different modalities, aligning them within a unified semantic space. It also dynamically mines complementary information between inspection data from different modalities (e.g., image textures can supplement the 3D geometry of point cloud data, and historical inspection data can assist in interpreting time-series sensor data). Furthermore, it effectively suppresses redundant information and noise interference, improving the quality of the fused global perception vector and providing more robust and reliable feature inputs for downstream tasks. like Figure 3 As shown in Figure S31, for each modal feature in the multimodal joint feature F... , Let m be the query modality and n be the key modality. A corresponding query vector Q is generated through linear transformation. m Key vector K n Sum vector V n : ; ; ; Among them, W Q W K W V All are weight matrices of linear transformation; S32. Calculate the cross-modal weight matrix A between the query modality m and the key modality n using the cross-attention mechanism. mn : ; Where d is the feature dimension and softmax is the activation function; S33, Using the cross-modal weight matrix A mn Obtain the cross-modal fusion features of query modality m : ; S34. Fuse the cross-modal fusion features of all query modalities to obtain the global perception vector F. fuse : ; in, To query the cross-modal fusion features of modality m The corresponding attention weights .
[0021] S4. Input the global perception vector into the downstream task network and optimize the model parameters using the corresponding loss function, including: The global perception vector F fuse The input is fed into a downstream task network comprising three tasks: defect detection, risk assessment, and inspection decision-making (the downstream task network will not be elaborated upon in this application, as it is not the focus of this invention). During model training, a multi-task joint loss function L is used to optimize the model parameters. ; Among them, L det For target detection loss, L risk For the risk score regression loss, L cls For state classification loss, , , All of these are adaptive weight coefficients, which can be dynamically adjusted during model training to balance the convergence speed of each task.
[0022] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A deep learning-based method for fusing multimodal inspection data of transmission lines, characterized in that: Includes the following steps: S1. Construct a unified spatiotemporal coordinate system and map the multimodal inspection data to this coordinate system to perform spatiotemporal alignment of the multimodal inspection data; S2. Based on the characteristics of different modalities, a dedicated encoder is used to extract features from the inspection data of each modality and project them onto a feature space of a unified dimension. S3. Utilize the mutual attention mechanism to dynamically weight and fuse the high-level semantic features between different modal inspection data to obtain a global perception vector; S4. Input the global perception vector into the downstream task network and optimize the model parameters using the corresponding loss function.
2. The method for fusing multimodal inspection data of transmission lines based on deep learning according to claim 1, characterized in that: In S1, a unified spatiotemporal coordinate system is constructed, and multimodal inspection data is mapped to this coordinate system. Spatiotemporal alignment of the multimodal inspection data is performed, including: For drone image sets I t Let E be a single frame image captured by the UAV at time t, where T is the time step. The extrinsic parameter matrix E is estimated using a dense reconstruction algorithm based on SfM. t This yields the pose (R) of each image frame in a unified spatiotemporal coordinate system. t ,t t ), R t Let be the rotation matrix at time t, where t t Let be the translation vector at time t; For lidar point cloud data sets , Let T be the coordinates of the i-th 3D point. Each 3D point consists of three real coordinates, determined by the extrinsic calibration matrix T. Li Complete the coordinate transformation to a unified spacetime coordinate system: ； in, Let i be the coordinates of the i-th 3D point in the unified spacetime coordinate system. The coordinates of the i-th 3D point Homogeneous coordinate representation; For online monitoring data sets and satellite remote sensing image collection S k (t) represents the data collected by the k-th sensor at time t, R j For the j-th satellite remote sensing image, it is bound to the line node in a unified spatiotemporal coordinate system through GIS geographic coordinates, and time alignment is achieved using linear interpolation: ； ； in, To bind online monitoring data to relevant data S on line nodes in a unified spatiotemporal coordinate system t Data corresponding to time t in a unified spatiotemporal coordinate system Indicates the relevant data S t Perform linear interpolation at time t; Binding satellite remote sensing images to relevant data R on line nodes in a unified spatiotemporal coordinate system t Data corresponding to time t in a unified spatiotemporal coordinate system This indicates the relevant data R t Linear interpolation is performed at time t.
3. The method for fusing multimodal inspection data of transmission lines based on deep learning according to claim 2, characterized in that: In S2, based on the characteristics of different modalities, a dedicated encoder is used to extract features from the inspection data of each modality and project them onto a feature space of a unified dimension, including: S21, For UAV image I t and satellite remote sensing images R j Feature extraction is performed using the Vision Transformer (ViT). ViT captures long-range dependencies and global semantic information in images through a self-attention mechanism. Compared with traditional convolutional networks, it can better model complex textures and spatial relationships, making the global feature representation more robust and distinguishable. ； ； Among them, f v To obtain images from drones t The global features extracted from f r To obtain R from satellite remote sensing images j The global features extracted are PatchEmbed, which represents the segmentation of the input image into B×B image patches and the linear mapping of these image patches into an embedding vector sequence for input into the Vision Transformer (ViT) for processing. CLS_token is a classification label, which is a learnable vector that is added to the beginning of the embedding vector sequence. After processing by the Vision Transformer (ViT), its final state can be used as the global feature representation of the entire image. For lidar point cloud data sets The PointNet feature encoding network is used for feature encoding. PointNet uses a multilayer perceptron (MLP) and global pooling to extract geometric structural features from point cloud data. It is insensitive to the order of the input point cloud data, maintains rotation and translation invariance, and ensures a complete representation of the target shape information. ； Among them, f p MaxPool represents the global features extracted from the lidar point cloud data set P, where MaxPool represents the aggregation of global features for all points. For online monitoring data sets Feature encoding is performed using a recurrent neural network (GRU). The GRU captures long-short-term dependencies through a gating mechanism, effectively modeling the dynamic changes in time-series data. It can memorize and predict continuous state changes, enhancing the model's ability to understand temporal information. ； Among them, f s These are global features extracted from the online monitoring dataset S; For historical inspection data H, a knowledge graph is first constructed, and then a graph neural network (GNN) is used for feature extraction. By embedding node and neighborhood information through graph convolution operations, the structured knowledge of historical inspection data H can be transformed into semantic features that can be used for fusion, while preserving complex relationship information between data, and providing prior knowledge support for decision-making. ； Among them, f h KG(H) represents the global features extracted from historical inspection data H, where KG(H) represents the knowledge graph constructed from historical inspection data H, and GraphConv represents the graph convolution operation. S22. Map the global features of each modal inspection data to a feature space of a unified dimension through a linear projection layer: ； ； ； ； ； Among them, f v 'For images from drones I t The global features f extracted v The feature vectors W are mapped to a feature space of uniform dimension via linear projection. v b v These are the global features f v The corresponding weights and biases of the linear projection layer; f r 'For satellite remote sensing image R j The global features f extracted r The feature vectors W are mapped to a feature space of uniform dimension via linear projection. r b r These are the global features f r The corresponding weights and biases of the linear projection layer; f p 'The global feature f extracted from the lidar point cloud data set P' p The feature vectors W are mapped to a feature space of uniform dimension via linear projection. p b p These are the global features f p The corresponding weights and biases of the linear projection layer; f s 'The global feature f extracted from the online monitoring dataset S' s The feature vectors W are mapped to a feature space of uniform dimension via linear projection. s b s These are the global features f s The corresponding weights and biases of the linear projection layer; f h 'The global feature f extracted from historical inspection data H' h The feature vectors W are mapped to a feature space of uniform dimension via linear projection. h b h These are the global features f h The corresponding weights and biases of the linear projection layer; S23. Finally, the multimodal joint feature F is obtained: 。 4. The method for fusing multimodal inspection data of transmission lines based on deep learning according to claim 3, characterized in that: In S3, a mutual attention mechanism is used to dynamically weight and fuse high-level semantic features between different modal inspection data to obtain a global perception vector, including: Based on a unified feature space, a cross-modal semantic collaboration module (CMSAM) is introduced. It utilizes a cross-attention mechanism to achieve high-level semantic interaction and collaborative modeling between inspection data from different modalities. CMSAM can correct semantic deviations between features of different modalities, aligning features of different modalities in a unified semantic space. It can also dynamically mine complementary information between inspection data from different modalities. In addition, it can effectively suppress redundant information and noise interference, improve the quality of the fused global perception vector, and provide more robust and reliable feature inputs for downstream tasks.
5. The deep learning-based multimodal inspection data fusion method for transmission lines according to claim 4, characterized in that: In S3, a mutual attention mechanism is used to dynamically weight and fuse high-level semantic features between different modal inspection data to obtain a global perception vector, including: S31. For each modal feature in the multimodal joint feature F , Let m be the query modality and n be the key modality. A corresponding query vector Q is generated through linear transformation. m Key vector K n Sum vector V n : ； ； ； Among them, W Q W K W V All are weight matrices of linear transformation; S32. Calculate the cross-modal weight matrix A between the query modality m and the key modality n using the cross-attention mechanism. mn : ； Where d is the feature dimension and softmax is the activation function; S33, Using the cross-modal weight matrix A mn Obtain the cross-modal fusion features of query modality m : ； S34. Fuse the cross-modal fusion features of all query modalities to obtain the global perception vector F. fuse : ； in, To query the cross-modal fusion features of modality m The corresponding attention weights .
6. The method for fusing multimodal inspection data of transmission lines based on deep learning according to claim 5, characterized in that: In S4, the global perception vector is input to the downstream task network, and the model parameters are optimized using the corresponding loss function, including: The global perception vector F fuse The input is fed into a downstream task network comprising three tasks: defect detection, risk assessment, and inspection decision-making. During model training, the model parameters are optimized using a multi-task joint loss function L. ； Among them, L det For target detection loss, L risk For the risk score regression loss, L cls For state classification loss, , , All of these are adaptive weight coefficients, which can be dynamically adjusted during model training to balance the convergence speed of each task.