A power transmission line monitoring method and apparatus

Abstract

The present application relates to the technical field of power transmission line monitoring, and discloses a power transmission line monitoring method and device, which completes whole-process processing by fusing multi-modal perception and language semantic guidance and relying on a feature fusion model. A task vector is generated through text semantics to realize accurate selection of visual word units; combined with intra-modal self-attention and cross-modal gating fusion mechanism, effective features are screened by local reliability. The present application fully plays the complementary advantages of different modal data, suppresses noise and invalid information, strengthens cross-modal feature association, improves the precision and robustness of power transmission line monitoring in complex environments, and can stably output high-quality global fusion feature maps.

Description

Technical Field

[0001] This invention relates to the field of power transmission line monitoring technology, specifically to a power transmission line monitoring method and device. Background Technology

[0002] As a critical infrastructure for power transmission, power transmission lines are erected in complex and variable environments, often subject to interference from high-altitude environments, which can easily lead to faults such as conductor breakage, insulator damage, and foreign object adhesion. Traditional inspection methods mainly rely on manual visual inspection or analysis of images collected by single-type sensors, making it difficult to achieve both real-time performance and accuracy for all-weather monitoring. Therefore, a new technical solution is needed that improves the system's detection, diagnosis, and tracking performance in complex environments by uniformly encoding different visual and linguistic information across multiple modalities and introducing an intelligent cross-attention mechanism.

[0003] Existing power transmission line detection systems mostly employ single-modal devices such as visible light cameras, infrared thermal imagers, or lidar for observation. However, each type of sensor has its own limitations: visible light cameras offer high resolution but are easily affected by changes in lighting conditions; infrared thermal imagers are sensitive to high-temperature faults but have lower spatial resolution; and lidar provides three-dimensional spatial information but is significantly affected by adverse weather conditions such as rain, snow, fog, and haze. Furthermore, there are challenges in time synchronization and spatial alignment between different modal data. Direct fusion of multimodal information can easily introduce redundant noise or information loss, limiting the effectiveness of joint analysis.

[0004] Existing technologies have attempted to introduce multimodal fusion to improve monitoring performance. For example, simply overlaying visible light and infrared images can enhance fault characteristics, but these methods are mostly limited to dual-modal fusion and pixel-level alignment, lacking a unified cross-modal collaborative mechanism. Furthermore, most current methods focus on a single task or single objective, such as only focusing on heat source detection of abnormal conductors or target segmentation of static images, lacking comprehensive consideration of multiple tasks such as detection, early warning, and tracking. The overall system efficiency and responsiveness need improvement. Simultaneously, existing research makes limited use of deep or semantic information during inspection processes; for example, fault descriptions and inspection instructions have not been fully integrated into the monitoring model, thus limiting the system's accurate understanding and execution of task requirements in complex scenarios. In summary, existing technologies have significant shortcomings in the efficient fusion, real-time processing, and multi-task collaboration of multimodal data, making it difficult to adapt to the complex requirements of multiple scenarios and environments in power transmission and transformation line monitoring. Summary of the Invention

[0005] This invention provides a method and apparatus for monitoring power transmission lines, which addresses the significant shortcomings of existing technologies in terms of efficient fusion, real-time processing, and multi-task collaboration of multimodal data, making it difficult to adapt to the complex requirements of multiple scenarios and environments in power transmission and transformation line monitoring.

[0006] In a first aspect, the present invention provides a method for monitoring transmission lines, the method comprising: Multiple modal data of the transmission line are acquired, and the modal data are superimposed with additional embedding information to generate multimodal lexical units, which are then input into a preset feature fusion model. Based on each word of the input text sequence of the feature fusion model, a role-enhanced language word is constructed, and power transmission inspection terms are superimposed on the role-enhanced language word to generate a sentence-level task vector; The sentence-level task vector is used to filter the multimodal lexical units and superimpose spatial position, temporal position and modal position to generate multimodal lexical units with positional encoding; Intramodal self-attention computation is performed on multimodal lexical units with positional encoding to extract corresponding single-modal features, and high-confidence features are selected from all the single-modal features based on local reliability scores. High-confidence features from each modality are fused across modalities, and a gating mechanism is used to optimize the cross-modal fusion results. By integrating the results of single-modal self-attention and the cross-modal fusion features after gating optimization, a global fusion feature map is obtained.

[0007] This invention integrates multimodal perception with language semantic guidance, relying on a feature fusion model to complete the entire process. It generates task vectors through text semantics, achieving precise selection of visual lexical units; combining intramodal self-attention and cross-modal gating fusion mechanisms with local reliability filtering of effective features. This invention fully leverages the complementary advantages of different modal data, suppresses noise and invalid information, strengthens cross-modal feature correlation, improves the accuracy and robustness of transmission line monitoring in complex environments, and can stably output high-quality global fusion feature maps.

[0008] In one optional implementation, the step of acquiring multiple modal data of the transmission line, superimposing additional embedding information on each modal data to generate multimodal lexical units, and inputting them into a preset feature fusion model includes: Acquire multiple modal data of the transmission line; wherein, the modal data includes image modes, point cloud modes, and electric field signal modes; Preprocess the modal data as described above; Based on the preprocessed image modalities, a defect guidance map is constructed and the initial image corresponding to the image modalities is fused together; The fused image is subjected to separate patch embedding processing to obtain image terms; The point cloud data corresponding to the preprocessed point cloud mode is set as the point set, and the voxel side length is determined according to the preset target distance. By combining the voxel side lengths and voxel statistical vectors, the features of the point set are extracted to obtain point cloud lexical units; A dual-branch coding system in both the time and frequency domains is used to encode the preprocessed electric field signal modes, resulting in electric field signal terms. The image lexical units, the point cloud lexical units, and the electric field signal lexical units are respectively superimposed with auxiliary embedding information to generate multimodal lexical units, which are then input into a preset feature fusion model; wherein, the auxiliary embedding information includes quality-aware modality identifier embedding, location embedding, and local reliability embedding.

[0009] This invention employs differentiated feature extraction methods for three modalities: image, point cloud, and electric field data. It combines defect-guided generation, voxelization, and time-frequency dual-branch encoding to generate various lexical units. By overlaying auxiliary embedded information such as quality, location, and local reliability, it can distinguish modality origins and data quality, preserving the unique details of each modality and avoiding information loss and interference caused by early fusion.

[0010] In one optional implementation, the step of constructing role-enhanced language lexical units from each lexical unit of the input text sequence based on the feature fusion model, and superimposing power transmission inspection terms onto the role-enhanced language lexical units to generate sentence-level task vectors, includes: Obtain the input text sequence for the feature fusion model; Add role tags to each lexical unit in the input text sequence according to semantic tags to construct role-enhanced language lexical units; Based on the soft attention mechanism, the terminology of power transmission inspection is fused into the role-enhanced language lexical and then subjected to task semantic pooling to obtain sentence-level task vectors.

[0011] This invention adds role tags to text lexical units to achieve fine-grained semantic differentiation, incorporates inspection terminology with soft attention, and generates sentence-level task vectors through semantic pooling. It can fully mine semantic information from the text, strengthen semantic expressions such as faults, locations, and actions, and form precise task guidance features.

[0012] In one optional implementation, the step of filtering the multimodal lexical units using the sentence-level task vector and superimposing spatial, temporal, and modal positions to generate multimodal lexical units with positional encoding includes: Based on the sentence-level task vector, irrelevant feature words are removed from the multimodal word units; The removed multimodal lexical units are superimposed with their spatial, temporal, and modal positions to generate multimodal lexical units with positional encoding.

[0013] This invention utilizes sentence-level task vectors to perform lexical filtering, effectively eliminating irrelevant features and reducing redundant information interference. By superimposing multiple types of positional encoding, lexical units simultaneously carry spatial, temporal, and modal attributes, improving the feature representation dimensions and enhancing the model's ability to perceive target location, temporal changes, and modal origins.

[0014] In one optional implementation, the step of performing intramodal self-attention computation on multimodal lexical units with positional encoding, extracting corresponding single-modal features, and selecting high-confidence features from all the single-modal features based on local reliability scores includes: Perform intramodal self-attention operations on each modal lexical unit with positional encoding to generate corresponding single-modal features; Calculate the local reliability score of the single-modal features for each modality; High-confidence features are selected from all the single-modal features based on the local reliability score.

[0015] This invention fully mines the internal correlation features of a single modality through intramodal self-attention and evaluates the effectiveness of features by combining local reliability scores. Based on the scores, high-confidence features are selected, while low-quality and invalid features are filtered out, which significantly reduces the amount of subsequent cross-modal computation, reduces noise interference, and improves the overall feature quality and computational efficiency.

[0016] In one optional implementation, the step of performing cross-modal fusion on the high-confidence features of each modality and optimizing the cross-modal fusion result using a gating mechanism includes: Based on the high-confidence features corresponding to each modality, construct cross-modal queries, keys, and values; Under the preset physical consistency bias constraint of transmission lines, cross-attention calculation is performed on the high-confidence features corresponding to each mode based on the cross-modal query, key and value to obtain cross-modal fusion features; Based on the aforementioned cross-modal fusion features, the mode pair gating coefficients are calculated; Based on the modality pair gating coefficients, the cross-modal fusion features are subjected to adaptive weighted correction processing.

[0017] This invention introduces a physical consistency bias constraint to perform cross-attention calculation, which aligns with the characteristic association patterns of transmission line scenarios. By using modal gating coefficients to adaptively weight and correct the fusion results, redundant cross-modal information can be dynamically suppressed, effective feature interactions can be enhanced, and the accuracy and anti-interference capability of multimodal fusion can be improved.

[0018] In one optional implementation, the integration of the single-modal self-attention results and the gated optimized cross-modal fusion features to obtain a global fusion feature map includes: By combining the unimodal features from the unimodal self-attention results with the gated optimized cross-modal fusion features, aggregated features are generated. The aggregated features are subjected to layer normalization and inter-layer update processing to obtain a global fused feature map.

[0019] This invention integrates single-modal features and cross-modal fusion features, taking into account both local details and intermodal correlations. By optimizing feature representation through layer normalization and inter-layer updates, it unifies feature distribution, improves overall representation, and ultimately obtains a global fusion feature map that is complete in information and stable in features.

[0020] In one optional implementation, the method further includes: The global fusion feature map is input into the parallel convolution head, and the target center heatmap, bounding regression, class probability and infrared fault heatmap are output. Based on the target center heatmap, the bounding box regression value, the category probability, and the infrared fault heatmap, a multi-task loss function is constructed. According to the multi-task loss function, the feature fusion model and the parallel convolution head are jointly optimized in terms of parameters; Based on the target center heat map, the bounding box regression value, the category probability, the infrared fault heat map, the historical feature template and the preset dynamic matching threshold, the multiple devices, line faults and foreign objects attached to the line of the transmission line are continuously located and tracked. The feature template is dynamically updated based on the trajectory tracking status.

[0021] This invention utilizes parallel convolutional heads to simultaneously perform localization, classification, and fault area detection, and combines multi-task loss to achieve joint model optimization, thereby improving overall prediction accuracy. Based on detection results, historical templates, and dynamic thresholds, it continuously tracks line equipment, faults, and attached foreign objects, and dynamically updates the templates, effectively ensuring the stability and continuity of long-term monitoring.

[0022] In one optional implementation, the method further includes: Construct the mass vector corresponding to each mode; Input the quality vector corresponding to each mode into a preset quality score network to calculate the quality score of the corresponding mode. The quality scores of all modes are normalized using a normalization function to obtain the global weights for each mode. By combining the global weights and word-level statistical features corresponding to each modality, the local reliability gating coefficient is calculated. Based on the global weights corresponding to each modality and the local reliability gating coefficients, the multimodal lexical units and their corresponding single-modal features are subjected to double-layer weighting. Construct modal quality pseudo-labels; By combining the quality scores of each modality and the pseudo-labels of the modality quality, a quality supervision loss function and a weight entropy regularization term are constructed. Based on the quality supervision loss function and the weight entropy regularization term, the network parameters of the quality score network are iteratively optimized.

[0023] This invention employs a dual-layer weighting mechanism of modality and lexical units, dynamically allocating weights based on quality scores to effectively suppress interference from low-quality data and invalid features. Combined with quality pseudo-labels, supervised loss, and entropy regularization constraints, the quality score network parameters are continuously optimized, improving the accuracy of modality quality assessment and enhancing the model's adaptability and feature representation stability under complex conditions.

[0024] In a second aspect, the present invention provides a power transmission line monitoring device, the device comprising: The acquisition module is used to acquire multiple modal data of the transmission line, and superimpose additional embedding information on each modal data to generate multimodal lexical units and input them into a preset feature fusion model; The construction module is used to construct role-enhanced language lexical units based on each lexical unit of the input text sequence of the feature fusion model, and to superimpose power transmission inspection terms onto the role-enhanced language lexical units to generate sentence-level task vectors; The overlay module is used to filter the multimodal lexical units through the sentence-level task vector and overlay spatial position, temporal position and modal position to generate multimodal lexical units with positional encoding; The filtering module is used to perform intramodal self-attention calculation on multimodal lexical units with positional encoding, extract the corresponding single-modal features, and filter out high-confidence features from all the single-modal features based on the local reliability score. The optimization module is used to perform cross-modal fusion of high-confidence features from each modality and to optimize the cross-modal fusion results using a gating mechanism. The fusion feature module is used to integrate the results of single-modal self-attention and the gated optimized cross-modal fusion features to obtain a global fusion feature map. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a schematic flowchart of a first method for monitoring transmission lines according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a second process for a transmission line monitoring method according to an embodiment of the present invention; Figure 3 This is an overall flowchart of the transmission line monitoring method according to an embodiment of the present invention; Figure 4This is a schematic diagram of multimodal data acquisition and fusion according to an embodiment of the present invention, wherein each modal data is preprocessed and embedded into a Transformer network; Figure 5 This is a structural block diagram of a power transmission line monitoring device according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0029] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0030] This invention provides a method for monitoring transmission lines, which integrates multimodal perception with language semantic guidance and completes the entire process using a feature fusion model. It generates task vectors through text semantics to achieve accurate selection of visual lexical units; and combines intramodal self-attention and cross-modal gating fusion mechanisms with local reliability to select effective features. This invention fully leverages the complementary advantages of different modal data, suppresses noise and invalid information, strengthens cross-modal feature correlation, and improves the accuracy and robustness of transmission line monitoring in complex environments, stably outputting high-quality global fusion feature maps.

[0031] According to an embodiment of the present invention, a method for monitoring transmission lines is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a method for monitoring power transmission lines. Figure 1This is a flowchart of a transmission line monitoring method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain multiple modal data of the transmission line, and superimpose the auxiliary embedding information on each modal data to generate multimodal lexical units and input them into the preset feature fusion model.

[0033] It should be noted that modal data refers to different types of data sources collected during transmission line monitoring, including images, point clouds, electric field signals, etc., reflecting the line status from different dimensions; attached embedded information refers to the identification and attribute-class encoding information added to the modal data, used to characterize attributes such as data quality, location, and reliability; multimodal lexical refers to the basic processing unit obtained after structured decomposition and encoding of various types of modal data, which is the basic input for model operation; feature fusion model refers to the deep learning network model used to complete multimodal feature extraction, correlation calculation and fusion processing.

[0034] In this embodiment of the invention, multiple modal data such as images, point clouds, and electric field signals from the power transmission line site are collected. Quality perception, location, local reliability, and other auxiliary embedded information are added to each type of modal data in turn. The processed modal data is converted into standardized multimodal lexical units, and the generated multimodal lexical units are then fed into a pre-built feature fusion model for subsequent calculations.

[0035] Step S102: Based on each word of the input text sequence of the feature fusion model, construct role-enhanced language words and overlay power transmission inspection terms onto the role-enhanced language words to generate sentence-level task vectors.

[0036] It should be noted that the input text sequence refers to the character sequence composed of textual information related to power transmission inspection, which is used to provide semantic guidance; the role-enhanced lexical refers to the text lexical with added semantic role labels, which strengthens the text's semantic discrimination ability; the power transmission inspection terminology refers to the professional vocabulary specifically used in the field of line inspection, which is used to clarify the task orientation and business semantics; and the sentence-level task vector refers to the vector obtained by integrating and encoding the text semantics, which is used to guide the selection and processing of multimodal features.

[0037] In this embodiment of the invention, the input text sequence corresponding to the feature fusion model is extracted, semantic role labels are added to each basic word in the sequence to construct role-enhanced language words, and then power transmission inspection professional terms are incorporated into them. After semantic integration processing, a sentence-level task vector is generated to guide feature selection.

[0038] Step S103: Multimodal lexical units are filtered and their spatial, temporal, and modal positions are superimposed using sentence-level task vectors to generate multimodal lexical units with positional encoding.

[0039] It should be noted that location encoding refers to the encoding information that represents spatial location, collection time, and the modality to which the data belongs, supplementing the feature dimension; multimodal lexical with location encoding refers to multimodal lexical after superimposing location information.

[0040] In this embodiment of the invention, invalid content in multimodal lexical units is removed by using sentence-level task vectors. After retaining valid lexical units, spatial, temporal, and modal positional information are superimposed sequentially to complete the encoding, and finally multimodal lexical units carrying complete positional attributes are generated.

[0041] Step S104: Perform intramodal self-attention calculation on multimodal lexical units with positional encoding, extract the corresponding single-modal features, and select high-confidence features from all single-modal features based on local reliability scores.

[0042] It should be noted that intramodal self-attention refers to the operation of calculating the correlation degree of elements within a single modality to mine the inherent features of single-modal data; single-modal features refer to feature information extracted through intramodal operations that corresponds to only a single data source; local reliability score is used to evaluate the quantitative score of the effectiveness of a single feature and distinguish the quality of features; high confidence features refer to high-quality features that meet the reliability score and have high effectiveness.

[0043] In this embodiment of the invention, intramodal self-attention calculation is performed on multimodal lexical units with positional encoding to fully mine the internal correlation information of each modal data to extract the corresponding single modal features. The local reliability score of each single modal feature is calculated simultaneously. Based on the score, high-confidence features with stronger effectiveness are selected and retained, while low-quality interference features are eliminated.

[0044] Step S105: Perform cross-modal fusion on the high-confidence features of each modality, and use a gating mechanism to optimize the cross-modal fusion results.

[0045] It should be noted that cross-modal fusion refers to the process of associating and integrating features from different modalities; gating mechanism refers to a control strategy that optimizes the fusion effect by adaptively adjusting feature weights through coefficients.

[0046] In this embodiment of the invention, the high-confidence features corresponding to each modality obtained by screening are subjected to cross-modal association fusion to fully explore the complementary information between different modal features. At the same time, a gating mechanism is introduced to adaptively adjust the fusion weight of each modality feature, suppress redundant and interference features, and refine the cross-modal fusion results to improve the overall representation quality of the fused features.

[0047] Step S106: Integrate the single-modal self-attention results and the gated optimized cross-modal fusion features to obtain the global fusion feature map.

[0048] It should be noted that the global fusion feature map refers to the complete feature map formed after integrating single-modal and cross-modal information, which carries comprehensive features.

[0049] In this embodiment of the invention, the extracted single-modal features are integrated with the cross-modal fusion features optimized by the gating mechanism, taking into account both single-modal detailed information and multimodal correlation features. The two types of features are aggregated and optimized to finally generate a global fusion feature map that is comprehensive in information and stable in representation.

[0050] This embodiment provides a method for monitoring power transmission lines. Figure 2 This is a flowchart of a transmission line monitoring method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain multiple modal data of the transmission line, and superimpose the auxiliary embedding information on each modal data to generate multimodal lexical units and input them into the preset feature fusion model.

[0051] Specifically, step S201 includes: Step S2011: Acquire multiple modal data of the transmission line; wherein, the modal data includes image mode, point cloud mode and electric field signal mode.

[0052] It should be noted that image mode refers to visible light image data of transmission lines acquired through image acquisition equipment, used to characterize the visual features of line appearance, defects, and foreign objects; point cloud mode refers to three-dimensional spatial point set data of transmission lines acquired through lidar, used to reflect the spatial structure, size, and location information of the equipment; electric field signal mode refers to the collected time-series signal data of the operating electric field of transmission lines, used to characterize the electrical operating status and latent fault characteristics of the line.

[0053] In embodiments of the present invention, such as Figure 3 As shown, multiple modal data of the transmission line are acquired, including image modal, point cloud modal, and electric field signal modal. Specifically, these include RGB (visible light color image), RGB-T (visible light and thermal infrared image fusion mode), RGB-D (visible light and depth image fusion mode), RGB-E (visible light and electric field sensing data fusion mode), and RGB-L (visible light and lidar point cloud fusion mode).

[0054] Step S2012: Preprocess the data for each modality.

[0055] In this embodiment of the invention, the data of each modality is preprocessed separately. Specifically, the image modality (visible light, infrared thermal imaging, depth image) is normalized and denoised; the point cloud modality (LiDAR) is filtered and downsampled; and the sensor signals such as electric field sensing (e.g., one-dimensional time series data) are filtered and normalized.

[0056] Step S2013: Based on the preprocessed image modalities, construct a defect guidance map and fuse the initial image corresponding to the image modalities.

[0057] It should be noted that the defect-guided map refers to the auxiliary guided feature map generated for the defect features of the transmission line image; the split patch embedding refers to the feature transformation method of dividing the complete image into independent patch units and completing the encoding embedding.

[0058] In this embodiment of the invention, the first The preprocessed input of each modality is denoted as ,in In the formula, m is the mode index, and vis, ir, dep, lidar and ele represent the visible light image mode, infrared thermal imaging mode, depth image mode, lidar point cloud mode and electric field sensing mode, respectively.

[0059] For image modalities, first construct the defect guidance map:

[0060] In the formula, This is the defect guidance diagram; Norm is the normalization operator; α1, α2, and α3 are the weighting coefficients for gradient features, linear structure features, and thermal anomaly features, respectively. For gradient operators; It exhibits a linear structure response; This is for thermal contrast or localized abnormal response.

[0061] The defect-guided image is fused with the initial image corresponding to the image modality.

[0062] Step S2014: Perform separate patch embedding processing on the fused image to obtain image words.

[0063] In this embodiment of the invention, a split patch embedding is used to obtain the first... Image terms:

[0064] In the formula, For the first Modal input; Element-wise multiplication; This is a defect guidance diagram; Weights are integrated into the defect guidance graph; For the first Patch extraction operator; For depthwise convolution; It is a point-by-point mapping matrix; For image lexical bias terms; Let i be the i-th image word.

[0065] Step S2015: Set the point cloud data corresponding to the preprocessed point cloud mode as a point set, and determine the voxel side length according to the preset target distance.

[0066] It should be noted that the voxel side length refers to the cube side length parameter when performing spatial cube partitioning on the point cloud point set, which is the reference size for point cloud voxelization.

[0067] In this embodiment of the invention, for lidar point clouds, the point cloud data is set as a point set. Based on target distance Adaptive determination of voxel side length

[0068] In the formula, P is the point cloud set; For the first j The target distance corresponding to the voxel; The side length of the basic voxel; This is the distance adjustment coefficient; For the first j Voxel side length; V(·) is the voxelization operator; For the first j Individualized results.

[0069] Step S2016: Combine voxel side length and voxel statistical vector to extract features of the point set and obtain point cloud lexical units.

[0070] It should be noted that the voxel statistical vector refers to the set of statistical features of point cloud data within a single voxel, which is used to refine the spatial distribution and structural features of point clouds.

[0071] In this embodiment of the invention, the voxel side length is used as the basis, and a voxel statistical vector is introduced. In the formula, j is the voxel index; j , j and j Let ρ be the mean x-coordinate, mean y-coordinate, and mean z-coordinate of the point cloud point within the j-th voxel, respectively; j σ is the point density; j The discreteness of the spatial coordinates of the point cloud; ι j δ is the statistical measure of reflection intensity. j This refers to statistics on the offset of the railway corridor. The corresponding terminology is:

[0072] In the formula, For the first j Each point cloud word element; is the point cloud feature mapping matrix; GAP is the global average pooling operator; Conv3D is the 3D convolution operator; For the first j Individual sizing results; The voxel statistical vector mapping matrix; For voxel statistics vectors; For point cloud lexical bias terms; These are point density, dispersion, reflection intensity statistics, and line corridor offset, respectively.

[0073] Step S2017: Using time-domain and frequency-domain dual-branch coding, the preprocessed electric field signal modes are encoded to obtain electric field signal terms.

[0074] It should be noted that time-domain and frequency-domain dual-branch coding refers to a feature extraction method that synchronously encodes electric field signals from both time and frequency dimensions, comprehensively mining signal features.

[0075] In this embodiment of the invention, for electric field signals It employs a dual-branch coding system in both the time and frequency domains:

[0076]

[0077] In the formula, For the first i Input of segment electric field signal; For time-domain branching lexical units; For frequency domain branching terms; and These are time-domain one-dimensional convolution and frequency-domain one-dimensional convolution, respectively; STFT stands for Short-Time Fourier Transform. For time-frequency fusion gating coefficients; For the Sigmoid function; This is the gated weight matrix; For gating bias terms; ⊙ represents element-wise multiplication; These are the terms for the fused electric field signal.

[0078] Step S2018: Superimpose auxiliary embedding information onto image lexical units, point cloud lexical units, and electric field signal lexical units respectively to generate multimodal lexical units and input them into a preset feature fusion model; wherein, the auxiliary embedding information includes quality-aware modal identifier embedding, location embedding, and local reliability embedding.

[0079] It should be noted that quality-aware modality identification embedding refers to dedicated coding information used to mark modality types and characterize the quality of single-modality data acquisition; local reliability embedding refers to coding information used to characterize the validity and confidence level of local modality data.

[0080] In this embodiment of the invention, the basic mode vector is first constructed. Mass vector and acquisition state vector And generate quality-aware modality identifier embeddings:

[0081] In the formula, Basic mode vector; It is a mass vector; To collect the state vector; Embed for quality-perceived modality identification; This involves concatenating vectors. The weight matrix for the quality-perceived modality embedding transformation; b m The bias vector for the quality-aware modality identifier embedding transformation.

[0082] The lexical units of each modality are respectively superimposed with quality-aware modality identifier embedding, position embedding, and local reliability embedding to obtain multimodal lexical units, which are then input into a pre-defined feature fusion model. The unified lexical expression for each modality is as follows:

[0083] In the formula, For word elements; Embed for quality-perceived modality identification; For location embedding; Embedded for local reliability; For layer normalization.

[0084] By employing the above methods, each modal term simultaneously contains modality source, quality state, spatial location information, and local reliability information, thereby improving the stability of the characterization of fine-grained defects, sparse point clouds, and weak signal anomalies in the subsequent fusion stage.

[0085] Step S202: Based on each word of the input text sequence of the feature fusion model, construct role-enhanced language words and superimpose power transmission inspection terms onto the role-enhanced language words to generate sentence-level task vectors.

[0086] Specifically, step S202 includes: Step S2021: Obtain the input text sequence for the feature fusion model.

[0087] In this embodiment of the invention, the input text sequence of the feature fusion model is obtained, and the input text sequence is represented as... ,in, A text sequence; For the first Each word element.

[0088] Step S2022: Add role labels to each lexical in the input text sequence according to semantic labels to construct role-enhanced language lexical units.

[0089] It should be noted that semantic tags refer to classification tags used to distinguish the semantic attributes and business functions of text words, and are adapted to power transmission inspection business scenarios; role tags refer to attribute tags bound to text words, used to define the semantic roles of words, and to achieve semantic differentiation enhancement of words.

[0090] In this embodiment of the invention, each word is assigned a role label according to semantics such as equipment, location, fault, degree, action, and orientation. And construct the initial representation of the role-enhanced language lexical:

[0091] In the formula, This is the degree weighting coefficient; Tag the character; These are word embedding, position embedding, role embedding, and degree embedding, respectively. This represents the initial language lexical representation.

[0092] Step S2023: Based on the soft attention mechanism, the terminology of power transmission inspection is fused into the role-enhanced language lexical units and task semantic pooling is performed to obtain the sentence-level task vector.

[0093] It should be noted that soft attention mechanism refers to an attention algorithm that adaptively allocates attention through probability weights; task semantic pooling refers to an operation method that aggregates and compresses refined semantic features, integrating scattered word semantics into a holistic task semantic representation.

[0094] In this embodiment of the invention, a prototype set of power transmission inspection terminology is set. Soft attention is used to inject domain terms (i.e., power transmission inspection terms) into lexical representations (i.e., role-enhanced language lexical initial representations):

[0095] Based on this, task semantic pooling is performed on all language lexical units to obtain sentence-level task vectors:

[0096] In the formula, For the initial language lexical representation; For the first A terminology prototype; The soft attention weights of lexical units to their prototypes; These are the lexical units after the domain semantics have been injected. This is a sentence-level task vector; For pooling weights; For sentence-level pooling parameters; For the j-th injected domain semantics, it represents the language lexical unit.

[0097] The preferred terminology prototypes include target semantic templates such as localized conductor heating, insulator cracks, foreign object attachment, abnormal sag, icing, and discharge traces. Through this encoding method, the degree of fault, location references, and action instructions in the text can participate in subsequent visual fusion in a fine-grained manner, and prior enhancement can be achieved for task-related visual regions.

[0098] Step S203: Multimodal lexical units are filtered and their spatial, temporal, and modal positions are superimposed using sentence-level task vectors to generate multimodal lexical units with positional encoding.

[0099] Specifically, step S203 includes: Step S2031: Based on the sentence-level task vector, remove irrelevant feature words from the multimodal word units.

[0100] It should be noted that irrelevant feature terms refer to redundant multimodal terms that are not related to the power transmission inspection and detection task and have no effective characterization function, and should be removed and filtered.

[0101] In this embodiment of the invention, sentence-level task vectors are used for visual lexical units. (i.e., pre-screening and gating of lexical units for each modality):

[0102] In the formula, For the j-th visual word element; For language-guided gating coefficients; Language-guided gating weight matrix; For language-guided gating bias terms; For the Sigmoid function; For sentence-level task vectors; ⊙ represents element-wise multiplication; after gating and weighting These are the pre-selected visual lexical units.

[0103] Irrelevant feature words are removed from multimodal word groups through pre-screening gating.

[0104] Step S2032: The removed multimodal lexical units are superimposed with their spatial, temporal, and modal positions to generate multimodal lexical units with positional encoding.

[0105] It should be noted that spatial location refers to the attribute information that characterizes the coordinates and relative position of modal data in the real-world space; temporal location refers to the attribute information that characterizes the timing and time nodes of modal data acquisition; and modal location refers to the specific location attribute information used to distinguish the dimensional affiliation of different modal data.

[0106] In this embodiment of the invention, let the first... The modality at time... The The unified word unit is By jointly injecting spatial location, temporal location, and modal location, the encoder input is obtained, which is a multimodal lexical unit with positional encoding.

[0107] In the formula, For a moment The input word units; These are spatial, temporal, and modal positional codes, respectively.

[0108] Step S204: Perform intramodal self-attention calculation on multimodal lexical units with positional encoding, extract the corresponding single-modal features, and select high-confidence features from all single-modal features based on local reliability scores.

[0109] Specifically, step S204 includes: Step S2041: Perform intramodal self-attention operation on each modal lexical with positional encoding to generate corresponding single-modal features.

[0110] In this embodiment of the invention, self-attention modeling is first performed on lexical units within the same modality:

[0111] In the formula, For the first Layer Modal characteristics; This is the output of the intramodal self-attention function.

[0112] By performing intramodal self-attention operations on each modal lexical unit with positional encoding (i.e., lexical units within the same modality), the single-modal features corresponding to each modality can be obtained.

[0113] Step S2042: Calculate the local reliability score of the single-mode features of each modality.

[0114] In this embodiment of the invention, based on the single-modal features of each modality, the local reliability of each single-modal feature is quantitatively evaluated from the dimensions of feature integrity, effective information content, and noise interference level, and the corresponding local reliability score is calculated.

[0115] Step S2043: Select high-confidence features from all single-modal features based on the local reliability score results.

[0116] In this embodiment of the invention, to reduce cross-modal computational load, a local reliability score is used. Choosing a sparse set:

[0117] Only for Cross-attention is performed on the lexical units.

[0118] In the formula, For a Top-K sparse set, K K represents the number of lexical units selected and retained in each modality based on the local reliability score, where K is a positive integer.

[0119] Based on the local reliability scores of each single-mode feature, a reasonable screening threshold is set to retain high-quality features with scores higher than the threshold and remove invalid and noisy features with low scores, thereby obtaining high-confidence features suitable for power transmission inspection tasks.

[0120] Step S205: Perform cross-modal fusion on the high-confidence features of each modality, and use a gating mechanism to optimize the cross-modal fusion results.

[0121] Specifically, step S205 includes: Step S2051: Based on the high confidence features corresponding to each modality, construct cross-modality queries, keys, and values.

[0122] It should be noted that cross-modal query, key and value refer to the three sets of feature vectors in cross-modal attention computation, which are used for feature retrieval, association matching and feature information output, respectively.

[0123] In this embodiment of the invention, for modality and modality ,remember

[0124] In the formula, These are the query, key, and value, respectively.

[0125] Step S2052: Under the preset physical consistency bias constraint of transmission lines, cross-attention calculation is performed on the high-confidence features corresponding to each mode based on the cross-modal query, key and value to obtain cross-modal fusion features.

[0126] It should be noted that the physical consistency bias constraint of transmission lines refers to the prior physical rule constraints that conform to the structure, electrical characteristics, and spatial distribution of transmission lines; cross-attention calculation refers to the attention operation that realizes the mutual correlation and interactive matching of different modal features.

[0127] In this embodiment of the invention, the cross-attention with physical consistency bias is:

[0128] in,

[0129] In the formula, For the first Cross-modal attention features from mode b to mode a in the layer; , , These represent the query, key, and value, respectively; d is the feature dimension scaling factor. For physical consistency bias; and σ represents the position of the word segment projected onto the unified reference space; σ is the scale parameter. , , These are the weighting coefficients for geometric proximity, line corridor consistency, and occlusion penalty, respectively. For line corridor consistency indication; To obscure the indicator quantity.

[0130] Cross-modal fusion features can be obtained by performing cross-attention calculation on the high-confidence features corresponding to each modality.

[0131] Step S2053: Calculate the mode pair gating coefficients based on cross-modal fusion features.

[0132] It should be noted that the modality pair gating coefficient refers to the adaptive weighting coefficient calculated based on the pairing relationship of two different sets of modal features, which is used to adjust the modality fusion intensity.

[0133] In this embodiment of the invention, based on the cross-modal fusion characteristics, the mode pair gating coefficients are further calculated:

[0134] In the formula, For the first Mode pair gating coefficients between layer modes a and b; σ is the Sigmoid function; This is the gated weight matrix; The term is the gated bias; Pool is the pooling operator. , The first Intramodal characteristics of layer modes a and b; , ... This refers to the cross-modal fusion features after gating correction.

[0135] Step S2054: Based on the modality pair gating coefficients, perform adaptive weighted correction processing on the cross-modal fusion features.

[0136] It should be noted that adaptive weighted correction processing refers to a feature correction method that adjusts feature weights in real time based on dynamic gating coefficients to optimize the fusion effect.

[0137] In this embodiment of the invention, the fusion weights of different modal features are adaptively adjusted based on the modality pair gating coefficients, and the cross-modal fusion features are weighted and corrected.

[0138] Step S206: Integrate the single-modal self-attention results and the gated optimized cross-modal fusion features to obtain a global fusion feature map.

[0139] Specifically, step S206 includes: Step S2061: Combine the single-modal features of the single-modal self-attention result with the gated optimized cross-modal fusion features to generate aggregated features.

[0140] It should be noted that aggregated features refer to composite integrated features obtained by fusing single-modal fine features with optimized cross-modal correlation features, taking into account both single-modal details and multimodal correlation information.

[0141] In this embodiment of the invention, single-modal features obtained by intramodal self-attention calculation and high-quality cross-modal fusion features after adaptive correction of the gating coefficient by modality are aggregated and fused to integrate single-modal detail information and cross-modal correlation information to generate aggregated features with more complete information dimensions.

[0142] Step S2062: Perform layer normalization and inter-layer update processing on the aggregated features to obtain the global fused feature map.

[0143] It should be noted that layer normalization refers to a deep learning feature optimization technique used to unify the data distribution of aggregated features, scale feature values, and avoid model training bias caused by differences in feature values; inter-layer update processing refers to the optimization operation of iteratively updating and filling in gaps in feature information at different layers of the network.

[0144] In this embodiment of the invention, the aggregated features are sequentially subjected to layer normalization to unify the distribution of feature data and eliminate differences in numerical scale. Then, the network layer feature association is iteratively optimized through inter-layer update processing to refine the feature representation details, and finally a global fusion feature map with both detailed information and global association information is obtained.

[0145] Specifically, complete the inter-layer update:

[0146] In the formula, For the first The update characteristics of layer mode a; LN is the layer normalization; For the first Intramodal self-attention features of layer mode a; The sum of cross-modal fusion features obtained by converging other modalities to modality a; FFN is a feedforward network.

[0147] This approach enables intramodal structural relationships, intermodal geometric consistency, and quality status to participate in feature fusion simultaneously, thereby suppressing misaligned attention and enhancing the long-range semantic tracking capability of transmission line targets.

[0148] Step S207: Input the global fusion feature map into the parallel convolution head and output the target center heatmap, bounding box regression, class probability and infrared fault heatmap.

[0149] It should be noted that the parallel convolution head refers to the multi-path parallel convolution output branch at the end of the model; the target center heat map refers to the feature map that represents the confidence level of the center position of the transmission target, fault and foreign object in the form of a heat map; the bounding box regression quantity refers to the offset parameter used to correct the offset deviation of the target detection box; the class probability refers to the confidence probability value of each class output by the model; and the infrared fault heat map refers to the heat feature map generated by focusing on the abnormal heating area of ​​the transmission line.

[0150] In this embodiment of the invention, the fused feature map output by the feature fusion network is denoted as... The parallel convolutional head outputs the target center heatmap, bounding box regression, class probability, and infrared fault heatmap, respectively.

[0151]

[0152] in, For a moment fusion feature map; Infrared feature map; These are the center heatmap, bounding box regression, category probability, and infrared heat map, respectively. Used to represent the central response of targets such as conductors, insulators, and foreign objects. Used to represent the target position and size parameters. Used to represent the probability of a category or fault category. Used to represent the probability distribution of anomalous hot areas in infrared thermal images.

[0153] Step S208: Construct a multi-task loss function based on the target center heatmap, bounding box regression, category probability, and infrared fault heatmap.

[0154] It should be noted that the multi-task loss function refers to the comprehensive loss function that integrates the errors of multiple tasks such as localization, classification, and fault detection.

[0155] In this embodiment of the invention, based on the target center heatmap, bounding box regression value, category probability, and infrared fault heatmap, corresponding sub-loss functions are constructed for target localization, bounding box correction, fault classification, and infrared anomaly detection tasks, respectively. These sub-loss functions are then weighted and fused to construct a unified multi-task loss function adapted to power transmission line inspection scenarios. The multi-task loss function used during the training phase is as follows:

[0156] In the formula, These are the center heatmap, bounding box regression, category probability, and infrared heat map, respectively. For the corresponding truth value; These are focal loss, L1 loss, generalized IoU loss, and binary cross-entropy loss, respectively.

[0157] Step S209: Perform joint parameter optimization on the feature fusion model and the parallel convolution head according to the multi-task loss function.

[0158] It should be noted that joint parameter optimization refers to the training optimization method of synchronously iteratively updating the parameters of the feature fusion model and the parallel convolutional head network.

[0159] In this embodiment of the invention, the multi-task loss function is used as the basis for supervision and constraint. The network parameters of the feature fusion model and the parallel convolution head are updated synchronously through the backpropagation algorithm to complete the joint iterative optimization of the two structures, reduce the multi-task detection error, and achieve the overall model's collaborative optimization.

[0160] Step S210: Based on the target center heat map, bounding box regression, category probability, infrared fault heat map, historical feature template and preset dynamic matching threshold, continuously locate and track multiple devices, line faults and foreign objects attached to the transmission line.

[0161] It should be noted that historical feature templates refer to the standard target and fault feature templates saved in previous frames; dynamic matching thresholds refer to feature matching thresholds that can be adaptively adjusted according to the scene conditions to meet the tracking and judgment requirements of complex inspection environments; trajectory tracking refers to the detection process that relies on continuous frame feature matching to continuously lock the position of power transmission equipment, faults and attached foreign objects and generate motion trajectories.

[0162] In this embodiment of the invention, the target center heat map, border regression value, category probability and infrared fault heat map are combined, and the feature similarity is verified by relying on historical feature templates and matching preset dynamic matching thresholds. This enables accurate positioning of various equipment, line faults and foreign objects attached to the line of the transmission line. At the same time, the continuous tracking and trajectory inference of the target are achieved by combining the feature correlation of continuous frames.

[0163] During the online tracking phase, let the current candidate feature be... The historical abstract template is The overall score is:

[0164] The dynamic threshold is defined as:

[0165] In the formula, These are the current candidate features; As a historical abstract template; For a moment The prediction box; This is a dynamic threshold.

[0166] Step S211: Dynamically update the feature template based on the trajectory tracking status.

[0167] It should be noted that the target trajectory tracking is real-time and dynamically iterates and updates the feature parameters and representation content of the historical feature templates to adapt to the real-time changes in the target shape, position and environment in the power transmission inspection scenario, ensuring the continuous accuracy and stability of subsequent feature matching and trajectory tracking.

[0168] when At that time, perform a template update with an update coefficient of 1.

[0169] Among them, the first The template has been updated to

[0170] Weighted aggregation of multi-template memories

[0171] In the formula, These are the current candidate features; This represents the current average modal reliability. This represents the time interval since the last update. For drift measurement; For the first One template; Aggregate weights for templates; Template lifespan; For feature splicing; The temperature coefficient is used to control the smoothness of the template aggregation weight distribution; μ is the decay coefficient for template lifespan.

[0172] The above methods enable integrated modeling of detection, hotspot localization, and long-term template memory, improving tracking stability in scenarios with occlusion, scale changes, and slow drift.

[0173] In some alternative implementations, the method further includes: Step a1: Construct the mass vector corresponding to each mode.

[0174] It should be noted that the quality vector refers to the feature vector generated for each mode of image, point cloud, electric field signal, and statistical data such as integrity, sharpness, and signal-to-noise ratio.

[0175] In this embodiment of the invention, a mass vector is constructed for each mode. It can be composed of at least some of the following: brightness, contrast, blur, occlusion rate, temperature difference peak, point density, outlier ratio, signal-to-noise ratio, and spectral entropy.

[0176] Step a2: Input the quality vector corresponding to each mode into the preset quality score network and calculate the quality score of the corresponding mode.

[0177] It should be noted that the quality scoring network refers to a lightweight neural network specifically designed to learn the data quality characteristics of each modality and output a quantified quality score; the modality quality score refers to the quantized score output by the quality scoring network.

[0178] In this embodiment of the invention, a mass vector is constructed for each mode. Modal quality scores are calculated using a quality scoring network:

[0179] In the formula, For the first Modal mass vector; For quality score; It is a non-linear activation function; These are the modal global weights.

[0180] Step a3: Normalize the quality scores of all modes using a normalization function to obtain the global weights for each mode.

[0181] It should be noted that global weight refers to the weight coefficient obtained after normalizing the original quality scores of each modality.

[0182] In this embodiment of the invention, all modal quality scores are normalized to global weights using Softmax:

[0183] In the formula, For quality score; This refers to the temperature parameter.

[0184] Step a4: Combine the global weights and word-level statistical features corresponding to each modality to calculate the local reliability gating coefficient.

[0185] It should be noted that lexical-level statistical features refer to the micro-statistical features such as distribution, amplitude, and effectiveness obtained by statistically analyzing single-modal lexical units; local reliability gating coefficient refers to the dynamic adjustment coefficient generated by combining global modal weights and local lexical features.

[0186] In this embodiment of the invention, based on the global weight of each modality, combined with the lexical-level statistical features corresponding to the multimodal lexical, and taking into account both the overall quality ratio of the modality and the effectiveness of the local features of each lexical, the local reliability gating coefficient corresponding to each lexical is calculated.

[0187] Specifically, for the first Modal first Introducing local reliability gating for each lexical element:

[0188] In the formula, For the first One input word element; The reliability gating coefficient is at the word level. The weighted lexical units; Let be a local statistical vector, where, These are word-level statistics such as local attention entropy, local variance, local occlusion rate, or point density. b is the weight matrix for local reliability gating; This is a bias term for local reliability gating.

[0189] Step a5: Based on the global weights and local reliability gating coefficients corresponding to each modality, perform double-layer weighting on the multimodal lexical units and their corresponding single-modal features.

[0190] It should be noted that the dual-layer weighted processing refers to a processing method that performs dual weight correction on multimodal lexical units and single-modal features from two dimensions: global modality weight and local lexical unit reliability.

[0191] In this embodiment of the invention, by combining the global weights of each modality with the local reliability gating coefficients at the word level, the multimodal words are first weighted at the modality level with the global weights, and then the words and corresponding single-modal features are corrected at the word level with the local reliability gating coefficients. This completes the two-layer weighting process from the overall modality to the local words, suppressing low-quality information and strengthening the representation of effective features.

[0192] Step a6: Construct modal quality pseudo-labels.

[0193] It should be noted that modal quality pseudo-labels refer to virtual modal quality labels generated based on the prior rules of power transmission inspection.

[0194] In this embodiment of the invention, in order to constrain the interpretability of the quality network, pseudo-label quality scores are constructed:

[0195] In the formula, These are contrast ratio, signal-to-noise ratio, structural integrity, and occlusion distortion. This represents the quality score of the pseudo-label.

[0196] Step a7: Combine the quality scores of each modality and the modality quality pseudo-labels to construct the quality supervision loss function and the weight entropy regularization term.

[0197] It should be noted that the quality supervision loss function refers to the loss function constructed using modal quality scores as predicted values ​​and modal quality pseudo-labels as true values; the weight entropy regularization term refers to the constraint term constructed based on the modal weight entropy values.

[0198] In this embodiment of the invention, the error between each modal quality score and the pre-constructed modal quality pseudo-label is calculated, and a quality supervision loss function is constructed to constrain the prediction accuracy of the quality score. At the same time, an entropy regularization term for the modal weights is introduced to prevent the weight distribution from becoming extreme, thus forming a complete loss constraint system.

[0199] Step a8: Based on the quality supervision loss function and the weight entropy regularization term, the network parameters of the quality score network are iteratively optimized.

[0200] It should be noted that parameter iterative optimization refers to the training process that relies on backpropagation of the loss function to continuously correct and update the network parameters, so that the model output results continuously approach the optimal state.

[0201] In this embodiment of the invention, the network parameters of the quality score network are iteratively updated through the backpropagation algorithm based on the quality supervision loss function and the weight entropy regularization term. This continuously reduces the quality score prediction error and optimizes the weight distribution, so that the network output gradually approaches the preset optimal state, thereby achieving accurate training and optimization of the quality score network.

[0202] Specifically, quality monitoring and regularization loss are employed:

[0203] in, For quality supervision losses, This is the weight entropy regularization term. Because...

[0204] Therefore, when nighttime or shading causes the visible light mode to... Lower and When it rises, there is When the infrared modal temperature difference contrast increases or the point cloud integrity improves, a corresponding mode will appear. This dual-weighting mechanism can simultaneously suppress low-quality information and highlight high-reliability observations at both the modality and lexical levels. and These represent a decrease and an increase in value, respectively.

[0205] Through the steps described above, this scheme integrates multi-source visual and linguistic information into a unified model. Modal embedding and patch embedding schemes enable the model to fully utilize the characteristics of each modality, fine-grained language coding enhances contextual semantic understanding, and gated cross-attention makes cross-modal fusion more selective. By comprehensively applying these technologies, the system can maintain accurate detection of targets such as conductors and cables even in various complex environments (such as low light, strong winds and rain, and partial obstruction), achieving high-precision monitoring and tracking of transmission line targets in complex environments.

[0206] This invention addresses heterogeneous multimodal inputs such as visible light (RGB), thermal infrared (T), depth (D), electric field sensing (E), and lidar (L). It introduces a learnable modality identifier (Embedding) for each modality, ensuring that features from different sources carry modal identity information before fusion, thus avoiding feature confusion. Simultaneously, it employs modality-specific feature embedding branches for different data types (e.g., independent patch embedding for 2D image modalities like visible light and infrared, voxelization for lidar 3D point cloud modalities, and 1D convolution for electric field 1D signal modalities). While preserving the unique details of each modality and avoiding information loss due to early mixing, it uniformly transforms these modalities into sequential tokens, providing high-quality feature input for subsequent Transformer fusion.

[0207] This invention utilizes a pre-trained language model to perform multi-token semantic encoding on texts such as inspection instructions and fault descriptions, forming a fine-grained language feature sequence. This sequence, along with visual lexical units, is then input into a Transformer network to achieve semantic guidance and task alignment of linguistic information with visual targets, thereby enhancing the model's ability to understand intent and discriminate targets in complex scenarios.

[0208] This invention introduces learnable gating parameters into the Transformer fusion layer to dynamically adjust the cross-attention output at the channel or token level. By generating a gating vector through Sigmoid activation and multiplying it with features, redundant or low-quality information is suppressed, achieving selective enhancement of cross-modal features and improving the fusion quality and anti-interference capability of the system in extreme environments.

[0209] This invention introduces a template update strategy based on confidence and time interval during the tracking process to construct a memory feature library with temporal consistency. Simultaneously, it utilizes the real-time quality assessment results of each modal input in the fusion network to dynamically adjust and allocate confidence to the temporal memory features, prioritizing the use of high-quality modal information to update the template and suppressing temporal interference from low-quality modalities. This effectively addresses issues such as drastic changes in target appearance, occlusion, and model drift that may occur during long-term tracking.

[0210] like Figure 4 As shown, the system corresponding to this invention deeply integrates modules such as multimodal preprocessing, language guidance, gated cross-attention fusion, adaptive weighting and memory template updating to build a general framework applicable to platforms such as drones and inspection robots. It can achieve high-precision, robust and semantically interpretable real-time perception and tracking of key targets such as wires, insulators and foreign objects in various complex environments such as daytime, nighttime, rain, fog and strong light.

[0211] This embodiment also provides a transmission line monitoring device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0212] This embodiment provides a power transmission line monitoring device, such as... Figure 5 As shown, it includes: The acquisition module 301 is used to acquire multiple modal data of the transmission line, and superimpose the auxiliary embedding information on each modal data to generate multimodal lexical units and input them into a preset feature fusion model; Module 302 is used to construct role-enhanced language lexical units based on each lexical unit of the input text sequence based on the feature fusion model, and to overlay power transmission inspection terms onto the role-enhanced language lexical units to generate sentence-level task vectors; The overlay module 303 is used to filter multimodal lexical units through sentence-level task vectors and overlay spatial, temporal, and modal positions to generate multimodal lexical units with positional encoding. The filtering module 304 is used to perform intramodal self-attention calculation on multimodal lexical units with positional encoding, extract the corresponding single-modal features, and filter out high-confidence features from all single-modal features based on local reliability scores. The optimization module 305 is used to perform cross-modal fusion of high-confidence features of each modality and to optimize the cross-modal fusion results using a gating mechanism. The fusion feature module 306 is used to integrate the single-modal self-attention results and the gated optimized cross-modal fusion features to obtain a global fusion feature map.

[0213] In some optional implementations, the acquisition module 301 includes: The data acquisition unit is used to acquire multiple modal data of the transmission line; the modal data includes image modes, point cloud modes, and electric field signal modes. The preprocessing unit is used to preprocess the data of each modality; Image units are constructed to build defect guidance maps based on preprocessed image modalities and fuse the initial images corresponding to the image modalities; The execution unit is used to perform separate patch embedding processing on the fused image to obtain image words; Point set unit, used to set the point cloud data corresponding to the preprocessed point cloud mode as a point set, and determine the voxel side length according to the preset target distance; The feature unit is used to extract features of the point set by combining the voxel side length and voxel statistical vector to obtain point cloud words; The coding unit is used to encode the preprocessed electric field signal modes using a dual-branch coding system in the time and frequency domains to obtain electric field signal terms. The input unit is used to superimpose auxiliary embedding information on image terms, point cloud terms, and electric field signal terms respectively to generate multimodal terms and input them into a preset feature fusion model; wherein, the auxiliary embedding information includes quality-aware modality identifier embedding, location embedding, and local reliability embedding.

[0214] In some alternative implementations, the construction module 302 includes: The sequence acquisition unit is used to acquire the input text sequence for the feature fusion model. Adding units are used to add role labels to each lexical unit in the input text sequence according to semantic labels, and constructing role-enhanced language lexical units; The pooling processing unit is used to fuse power transmission inspection terms into role-enhanced language lexical units based on a soft attention mechanism and perform task semantic pooling processing to obtain sentence-level task vectors.

[0215] In some alternative implementations, the overlay module 303 includes: The elimination unit is used to eliminate irrelevant feature words in multimodal word units based on sentence-level task vectors; The overlay unit is used to overlay the spatial, temporal, and modal positions of the removed multimodal lexical units to generate multimodal lexical units with positional encoding.

[0216] In some alternative implementations, the filtering module 304 includes: The computation unit is used to perform intramodal self-attention operations on each pair of modal lexical units with positional encoding to generate corresponding single-modal features; The calculation unit is used to calculate the local reliability score of the single-modal features of each type of modality. The filtering unit is used to select high-confidence features from all single-modal features based on the local reliability score.

[0217] In some alternative implementations, the optimization module 305 includes: Construct key units to build cross-modal queries, keys, and values ​​based on high-confidence features corresponding to each modality; The constraint unit is used to perform cross-attention calculation on the high-confidence features corresponding to each mode based on the cross-modal query, key and value under the preset transmission line physical consistency bias constraint to obtain cross-modal fusion features; The coefficient calculation unit is used to calculate the mode pair gating coefficients based on cross-modal fusion features; The correction processing unit is used to perform adaptive weighted correction processing on cross-modal fusion features based on the modality gating coefficients.

[0218] In some alternative implementations, the feature fusion module 306 includes: The aggregation unit is used to combine the single-modal features of the single-modal self-attention result and the gated optimized cross-modal fusion features to generate aggregated features; The update processing unit is used to perform layer normalization and inter-layer update processing on the aggregated features to obtain the global fused feature map.

[0219] In some alternative embodiments, the device further includes: The convolutional unit is used to input the global fused feature map into the parallel convolutional head and output the target center heatmap, bounding box regression, class probability and infrared fault heatmap. A function unit is constructed to build a multi-task loss function based on the target center heatmap, bounding box regression, class probability, and infrared fault heatmap. The optimization unit is used to jointly optimize the parameters of the feature fusion model and the parallel convolution head according to the multi-task loss function; The tracking unit is used to continuously locate and track multiple devices, line faults, and foreign objects attached to the transmission line based on the target center heat map, border regression, category probability, infrared fault heat map, historical feature template and preset dynamic matching threshold. The update unit is used to dynamically update the feature template based on the trajectory tracking status.

[0220] In some alternative embodiments, the device further includes: Construct a mass vector unit to build the mass vector corresponding to each mode; The scoring unit is used to input the quality vector corresponding to each mode into a preset quality scoring network and calculate the quality score of the corresponding mode. The normalization processing unit is used to normalize the quality scores corresponding to all modes through a normalization function to obtain the global weights corresponding to each mode. The joint unit is used to combine the global weights and word-level statistical features corresponding to each modality to calculate the local reliability gating coefficient. The weighted processing unit is used to perform two-layer weighted processing on multimodal lexical units and their corresponding single-modal features based on the global weights and local reliability gating coefficients corresponding to each modality. Construct label units to build modal quality pseudo-labels; Construct regularization term units to combine the quality scores of each modality and the modality quality pseudo-labels to construct the quality supervision loss function and the weight entropy regularization term; The iterative optimization unit is used to iteratively optimize the network parameters of the quality score network based on the quality supervision loss function and the weight entropy regularization term.

[0221] The transmission line monitoring device provided in this embodiment of the invention can execute the transmission line monitoring method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0222] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for monitoring transmission lines, characterized in that, The method includes: Multiple modal data of the transmission line are acquired, and the modal data are superimposed with additional embedding information to generate multimodal lexical units, which are then input into a preset feature fusion model. Based on each word of the input text sequence of the feature fusion model, a role-enhanced language word is constructed, and power transmission inspection terms are superimposed on the role-enhanced language word to generate a sentence-level task vector; The sentence-level task vector is used to filter the multimodal lexical units and superimpose spatial position, temporal position and modal position to generate multimodal lexical units with positional encoding; Intramodal self-attention computation is performed on multimodal lexical units with positional encoding to extract corresponding single-modal features, and high-confidence features are selected from all the single-modal features based on local reliability scores. High-confidence features from each modality are fused across modalities, and a gating mechanism is used to optimize the cross-modal fusion results. By integrating the results of single-modal self-attention and the cross-modal fusion features after gating optimization, a global fusion feature map is obtained.

2. The method according to claim 1, characterized in that, The process of acquiring multiple modal data of the transmission line, superimposing additional embedding information on each modal data to generate multimodal lexical units, and inputting them into a preset feature fusion model includes: Acquire multiple modal data of the transmission line; wherein, the modal data includes image modes, point cloud modes, and electric field signal modes; Preprocess the modal data as described above; Based on the preprocessed image modalities, a defect guidance map is constructed and the initial image corresponding to the image modalities is fused together; The fused image is subjected to separate patch embedding processing to obtain image terms; The point cloud data corresponding to the preprocessed point cloud mode is set as the point set, and the voxel side length is determined according to the preset target distance. By combining the voxel side lengths and voxel statistical vectors, the features of the point set are extracted to obtain point cloud lexical units; A dual-branch coding system in both the time and frequency domains is used to encode the preprocessed electric field signal modes, resulting in electric field signal terms. The image lexical units, the point cloud lexical units, and the electric field signal lexical units are respectively superimposed with auxiliary embedding information to generate multimodal lexical units, which are then input into a preset feature fusion model; wherein, the auxiliary embedding information includes quality-aware modality identifier embedding, location embedding, and local reliability embedding.

3. The method according to claim 1, characterized in that, The input text sequence based on the feature fusion model is used to construct role-enhanced language lexical units, and power transmission inspection terminology is superimposed onto these lexical units to generate sentence-level task vectors, including: Obtain the input text sequence for the feature fusion model; Add role tags to each lexical unit in the input text sequence according to semantic tags to construct role-enhanced language lexical units; Based on the soft attention mechanism, the terminology of power transmission inspection is fused into the role-enhanced language lexical and then subjected to task semantic pooling to obtain sentence-level task vectors.

4. The method according to claim 1, characterized in that, The process of filtering the multimodal lexical units using the sentence-level task vector and superimposing spatial, temporal, and modal positions to generate multimodal lexical units with positional encoding includes: Based on the sentence-level task vector, irrelevant feature words are removed from the multimodal word units; The removed multimodal lexical units are superimposed with their spatial, temporal, and modal positions to generate multimodal lexical units with positional encoding.

5. The method according to claim 1, characterized in that, The step involves performing intramodal self-attention computation on multimodal lexical units with positional encoding, extracting corresponding single-modal features, and selecting high-confidence features from all the single-modal features based on local reliability scores, including: Perform intramodal self-attention operations on each modal lexical unit with positional encoding to generate corresponding single-modal features; Calculate the local reliability score of the single-modal features for each modality; High-confidence features are selected from all the single-modal features based on the local reliability score.

6. The method according to claim 1, characterized in that, The process of fusing high-confidence features from each modality across modalities and optimizing the cross-modal fusion results using a gating mechanism includes: Based on the high-confidence features corresponding to each modality, construct cross-modal queries, keys, and values; Under the preset physical consistency bias constraint of transmission lines, cross-attention calculation is performed on the high-confidence features corresponding to each mode based on the cross-modal query, key and value to obtain cross-modal fusion features; Based on the aforementioned cross-modal fusion features, the mode pair gating coefficients are calculated; Based on the modality pair gating coefficients, the cross-modal fusion features are subjected to adaptive weighted correction processing.

7. The method according to claim 1, characterized in that, The integration of the single-modal self-attention results and the gated optimized cross-modal fusion features yields a global fusion feature map, including: By combining the unimodal features from the unimodal self-attention results with the gated optimized cross-modal fusion features, aggregated features are generated. The aggregated features are subjected to layer normalization and inter-layer update processing to obtain a global fused feature map.

8. The method according to claim 7, characterized in that, The method further includes: The global fusion feature map is input into the parallel convolution head, and the target center heatmap, bounding regression, class probability and infrared fault heatmap are output. Based on the target center heatmap, the bounding box regression value, the category probability, and the infrared fault heatmap, a multi-task loss function is constructed. According to the multi-task loss function, the feature fusion model and the parallel convolution head are jointly optimized in terms of parameters; Based on the target center heat map, the bounding box regression value, the category probability, the infrared fault heat map, the historical feature template and the preset dynamic matching threshold, the multiple devices, line faults and foreign objects attached to the line of the transmission line are continuously located and tracked. The feature template is dynamically updated based on the trajectory tracking status.

9. The method according to claim 1, characterized in that, The method further includes: Construct the mass vector corresponding to each mode; Input the quality vector corresponding to each mode into a preset quality score network to calculate the quality score of the corresponding mode. The quality scores of all modes are normalized using a normalization function to obtain the global weights for each mode. By combining the global weights and word-level statistical features corresponding to each modality, the local reliability gating coefficient is calculated. Based on the global weights corresponding to each modality and the local reliability gating coefficients, the multimodal lexical units and their corresponding single-modal features are subjected to double-layer weighting. Construct modal quality pseudo-labels; By combining the quality scores of each modality and the pseudo-labels of the modality quality, a quality supervision loss function and a weight entropy regularization term are constructed. Based on the quality supervision loss function and the weight entropy regularization term, the network parameters of the quality score network are iteratively optimized.

10. A transmission line monitoring device, characterized in that, The device includes: The acquisition module is used to acquire multiple modal data of the transmission line, and superimpose additional embedding information on each modal data to generate multimodal lexical units and input them into a preset feature fusion model; The construction module is used to construct role-enhanced language lexical units based on each lexical unit of the input text sequence of the feature fusion model, and to superimpose power transmission inspection terms onto the role-enhanced language lexical units to generate sentence-level task vectors; The overlay module is used to filter the multimodal lexical units through the sentence-level task vector and overlay spatial position, temporal position and modal position to generate multimodal lexical units with positional encoding; The filtering module is used to perform intramodal self-attention calculation on multimodal lexical units with positional encoding, extract the corresponding single-modal features, and filter out high-confidence features from all the single-modal features based on the local reliability score. The optimization module is used to perform cross-modal fusion of high-confidence features from each modality and to optimize the cross-modal fusion results using a gating mechanism. The fusion feature module is used to integrate the results of single-modal self-attention and the gated optimized cross-modal fusion features to obtain a global fusion feature map.

Images