Method, apparatus and electronic device for determining component instance information

Abstract

The application discloses a kind of determination method, device and electronic equipment of component instance information.Therein, the method includes: obtaining multi-modal sensor data;According to multi-modal sensor data, determine a plurality of scale feature sets;According to multi-scale visual features, perform power component target detection and instance segmentation operation, obtain the preliminary identification information corresponding to multiple power component instances respectively;According to the preliminary identification information of multiple power component instances respectively and a plurality of scale feature sets, determine and the component state information corresponding to multiple power component instances respectively;According to the preliminary identification information of multiple power component instances respectively and component state information, perform instance information fusion operation, obtain component instance information.The application solves the technical problems that power component implicit defects are difficult to identify and component state diagnosis reliability is poor in the related art.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method, apparatus, and electronic device for determining component instance information. Background Technology

[0002] 10kV overhead lines are widely distributed in urban and rural power distribution networks. Their operating environment is complex, with long-distance lines and diverse component types. Long-term exposure to external conditions such as wind, rain, icing, and vegetation growth makes them prone to hidden dangers such as insulator damage, foreign object hanging on conductors, overheating of clamps, and insufficient safety distances. Existing line inspection methods mainly rely on manual inspection or visible light image detection based on drones, which have the following shortcomings: First, existing technologies mostly rely on a single visible light mode, making it difficult to effectively identify latent defects such as infrared thermal anomalies and partial discharges; second, for targets with significant scale differences (such as long-distance conductors and small fittings), the detection accuracy and stability are insufficient; furthermore, multi-sensor data is usually processed independently, lacking a unified modeling and deep fusion mechanism, resulting in limited diagnostic reliability and poor reliability of component condition diagnosis.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for determining component instance information, in order to at least solve the technical problems in the related art, such as the difficulty in identifying latent defects in power components and the poor reliability of component status diagnosis.

[0005] According to one aspect of the present invention, a method for determining component instance information is provided, comprising: acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on a UAV, and the multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; determining a multi-scale feature set based on the multimodal sensor data, wherein the multi-scale feature set includes multi-scale visual features, global geometric features, acoustic spectrum features, and a global attitude feature vector; and performing power component target detection and instance segmentation operations based on the multi-scale visual features to obtain... The system obtains preliminary identification information corresponding to multiple power component instances, wherein the corresponding preliminary identification information includes: bounding box coordinates of the corresponding power component instance, component category, and contour mask, wherein the component category includes: conductor and tower; based on the preliminary identification information of the multiple power component instances and the multi-scale feature set, the system determines the component status information corresponding to the multiple power component instances, wherein the corresponding component status information includes at least one of the following: defect status information and risk parameter status information; based on the preliminary identification information and component status information of the multiple power component instances, the system performs an instance information fusion operation to obtain component instance information.

[0006] Optionally, based on the preliminary identification information of the plurality of power component instances and the multi-scale feature set, determining the component state information corresponding to each of the plurality of power component instances includes: inputting the preliminary identification information of the plurality of power component instances and the multi-scale feature set into a shared feature encoding model to obtain the component state information corresponding to each of the plurality of power component instances. The shared feature encoding model includes a target loss function, which is a weighted sum of the classification loss of the defect classification task, the regression loss of the risk parameter regression task, and the segmentation loss of the instance segmentation task. The classification loss measures the difference between the predicted category and the true defect category in the defect classification task; the regression loss measures the difference between the predicted risk parameter and the true risk parameter in the risk parameter regression task; and the segmentation loss measures the difference between the predicted contour mask and the true contour mask in the instance segmentation task.

[0007] Optionally, determining a multi-scale feature set based on the multimodal sensor data includes: inputting the visual imaging data into a feature extraction model to obtain the multi-scale visual features, wherein the feature extraction model includes a backbone network and a feature pyramid structure network, the backbone network including any one of the following: a convolutional neural network, a visual transformer network, the feature pyramid structure network being used to determine the multi-scale visual features based on the hierarchical features output by different levels of the backbone network, the backbone network including at least a first level and a second level, the first level being used to extract detail information features, the second level being used to extract semantic information features, and the resolution corresponding to the first level being higher than the resolution corresponding to the second level.

[0008] Optionally, based on the multimodal sensor data, a multi-scale feature set is determined, including: for each target point in the spatial depth point cloud data, determining a neighborhood point set centered on the target point; extracting local geometric features of the neighborhood point set through a mapping function based on the relative positional relationship between each neighborhood point in the neighborhood point set and the target point; aggregating the local geometric features through an aggregation function to obtain target point features corresponding to the target point; and obtaining global geometric features corresponding to the spatial depth point cloud data based on the target point features corresponding to each target point, wherein the global geometric features are used to characterize at least one of the following: the suspension shape of the conductor, the spatial configuration of the tower, and the safe distance between the conductor and surrounding objects.

[0009] Optionally, based on the multimodal sensor data, determining a multi-scale feature set includes: performing a Fourier transform operation on the acoustic signal data to convert the acoustic signal data from a time-domain signal to a time-frequency domain to obtain acoustic time-frequency domain features; inputting the acoustic time-frequency domain features into a convolutional neural network for encoding to obtain the acoustic spectrum features, wherein the acoustic spectrum features are used to characterize at least one of the following: partial discharge features, corona discharge features, and mechanical anomaly features.

[0010] Optionally, based on the multimodal sensor data, a multi-scale feature set is determined, including: constructing a multi-dimensional temporal signal based on the attitude inertial sequence data and the position coordinate data, wherein the multi-dimensional temporal signal includes the three-dimensional position coordinate vector, three-axis attitude angle vector, three-axis acceleration vector, and three-axis angular velocity vector of the UAV at multiple time steps; sequentially inputting the vectors corresponding to the multiple time steps into a stacked long short-term memory network to obtain hidden state vectors corresponding to the multiple time steps, wherein the hidden state vectors represent the temporal feature information of the corresponding time step and all previous time steps; performing an average pooling operation on the hidden state vectors corresponding to the multiple time steps to obtain a global attitude feature vector, and then processing the global attitude feature vector.

[0011] Optionally, after performing an instance information fusion operation based on the preliminary identification information and component status information of the plurality of power component instances to obtain component instance information, the method further includes: determining the plurality of power component instances in the component instance information as nodes in a graph structure; determining the connection relationship between each node in the graph structure based on the geometric proximity between nodes, and using the connection relationship as an edge in the graph structure; and constructing a line space topology graph based on the nodes and the edges.

[0012] According to one aspect of the present invention, a device for determining component instance information is provided, comprising: an acquisition module for acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on a UAV, and the multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; a first determination module for determining a multi-scale feature set based on the multimodal sensor data, wherein the multi-scale feature set includes multi-scale visual features, global geometric features, acoustic spectrum features, and a global attitude feature vector; and a second determination module for performing power component target detection and instance segmentation based on the multi-scale visual features. The operation obtains preliminary identification information corresponding to multiple power component instances, wherein the corresponding preliminary identification information includes: the bounding box coordinates of the corresponding power component instance, the component category, and the contour mask, wherein the component category includes: conductor, tower; a third determining module is used to determine the component status information corresponding to the multiple power component instances based on the preliminary identification information of the multiple power component instances and the multi-scale feature set, wherein the corresponding component status information includes at least one of the following: defect status information, risk parameter status information; a fourth determining module is used to perform an instance information fusion operation based on the preliminary identification information and component status information of the multiple power component instances to obtain component instance information.

[0013] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method for determining component instance information as described above.

[0014] According to one aspect of the present invention, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method for determining component instance information as described above.

[0015] In this embodiment of the invention, multimodal sensor data is acquired, which is collected by a sensor system mounted on a UAV. The multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Based on the multimodal sensor data, a multi-scale feature set is determined, which includes multi-scale visual features, global geometric features, acoustic spectrum features, and a global attitude feature vector. Based on the multi-scale visual features, power component target detection and instance segmentation operations are performed to obtain preliminary identification information corresponding to multiple power component instances. The corresponding preliminary identification information includes: the bounding box coordinates of the corresponding power component instance, the component category, and the contour mask. The component category includes: conductors and towers. Based on the preliminary identification information and the multi-scale feature set of the multiple power component instances, component status information corresponding to the multiple power component instances is determined. The corresponding component status information includes at least one of the following: defect status information and risk parameter status information. Based on the preliminary identification information and component status information of the multiple power component instances, an instance information fusion operation is performed to obtain component instance information. This invention employs a combination of multimodal perception and multi-scale feature extraction, along with power component detection, segmentation, and condition diagnosis. A UAV-mounted sensor system collects multimodal sensor data, which is then used to determine a multi-scale feature set. Based on these data, target detection and instance segmentation of the power components are performed using the multi-scale visual features to obtain preliminary identification information. This preliminary identification information is then combined with the multi-scale feature set to determine the component's condition information. Finally, instance information fusion is performed on the preliminary identification information and the component condition information. This approach achieves the goal of identifying latent defects in power components and improving the reliability of component condition diagnosis. Consequently, it achieves the technical effects of accurately identifying power components, reliably diagnosing component condition, and effectively identifying latent defects, thus solving the technical problems of difficulty in identifying latent defects in power components and poor reliability of component condition diagnosis in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of a method for determining component instance information according to an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the overall architecture provided by an optional embodiment of the present invention;

[0019] Figure 3 This is a structural block diagram of a device for determining component instance information according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example 1

[0023] According to an embodiment of the present invention, an embodiment of a method for determining component instance information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] Figure 1 This is a flowchart of a method for determining component instance information according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0025] Step S102: Acquire multimodal sensor data. The multimodal sensor data is collected by the sensor system carried by the UAV. The multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data.

[0026] Among them, multimodal sensor data refers to the heterogeneous data set collected simultaneously by various types of sensors carried by UAVs in the same inspection mission. It can provide multi-dimensional and complementary panoramic perception information for power line inspection, and overcome the problem of insufficient perception capability of single sensors in complex environments such as night, fog, haze, and backlight. For example, visual imaging data provides equipment appearance information, and spatial depth point cloud data provides three-dimensional geometric structure information.

[0027] Among them, visual imaging data refers to image data collected by high-definition visible light cameras and infrared thermal imaging cameras, which can clearly present the appearance, color, markings and temperature distribution information of power equipment, such as visible abnormalities such as insulator damage and thermal abnormalities such as wire clamp overheating.

[0028] Among them, acoustic signal data refers to audio signals collected by acoustic imagers and ultrasonic sensors, which can capture abnormal acoustic features such as partial discharge, corona discharge, and mechanical loosening, providing a non-contact detection method for discovering hidden defects.

[0029] Among them, spatial depth point cloud data refers to a set of three-dimensional spatial points generated by lidar by emitting laser beams and receiving reflected echoes. It can accurately measure the distance between drones and power lines, towers, and vegetation, providing centimeter-level precision information for three-dimensional mapping and spatial geometric feature extraction.

[0030] Among them, attitude inertial sequence data refers to the time series of three-axis acceleration and three-axis angular velocity of the UAV output in real time by the inertial measurement unit. It can provide prior information on the UAV's body motion and assist in attitude estimation and flight status monitoring.

[0031] Among them, location coordinate data refers to the global latitude, longitude and elevation coordinates of the UAV obtained by the GPS receiver, which can provide absolute geographical reference for the UAV and support the spatial coordinate association of component instances.

[0032] In this step, the sensor system onboard the UAV simultaneously collects raw data in six modalities during the inspection flight. This step provides a temporally aligned and spatially unified multimodal raw data foundation for subsequent multi-scale feature extraction and power component identification, enabling the system to comprehensively utilize the complementary sensing capabilities of different sensors and effectively overcome the limitations of insufficient sensing information from a single modality in complex environments.

[0033] Step S104: Based on the multimodal sensor data, determine the multi-scale feature set, which includes multi-scale visual features, global geometric features, acoustic spectrum features, and global pose feature vector.

[0034] Among them, the multi-scale feature set refers to the structured data set containing multiple modal and multi-scale hierarchical features obtained after feature extraction from multi-modal sensor data. It can transform heterogeneous raw sensor data into a unified and standardized feature representation, providing a feature foundation for subsequent target detection, instance segmentation and state recognition. It includes multi-scale visual features, global geometric features, acoustic spectrum features and global pose feature vectors.

[0035] Among them, multi-scale visual features refer to the sequence of visual feature maps with different resolution levels obtained after multi-layer feature extraction of visual imaging data. It can simultaneously retain the spatial detail information in the low-level feature map and the semantic information in the high-level feature map, supporting the accurate detection of power components of different scales. For example, the low-level feature map is used to detect small-scale components such as small fittings, while the high-level feature map is used to detect large-scale components such as conductors.

[0036] Among them, global geometric features refer to the features that represent the overall spatial structure information of the point cloud after extracting features from spatial depth point cloud data using point convolutional networks. These features can preserve three-dimensional spatial geometric information such as conductor suspension morphology, tower spatial configuration, and safety distance.

[0037] Among them, acoustic spectrum features refer to the time-frequency domain features obtained by encoding acoustic signal data after time-frequency transformation. They can characterize the distribution features of acoustic signals in the frequency and time dimensions, and provide feature basis for identifying acoustic abnormality patterns such as partial discharge, corona discharge and mechanical anomalies.

[0038] Among them, the global attitude feature vector refers to the feature vector obtained after performing sequence modeling on the attitude inertial sequence data, which represents the stability of the UAV's flight attitude and the characteristics of trajectory changes, and can assist in positioning and abnormal state judgment.

[0039] In this step, targeted feature extraction is performed on the raw data of various modalities obtained in step S102. Optionally, visual imaging data is processed to extract multi-scale visual features through a feature pyramid network, spatial depth point cloud data is processed to extract global geometric features through a point convolutional network, acoustic signal data is processed to extract acoustic spectral features after time-frequency transformation, and attitude inertial sequence data is processed to extract global attitude feature vectors through a time-series network. Through this step, heterogeneous raw sensor data is transformed into a structured multi-scale, multi-modal feature set, providing a unified feature input for target detection, instance segmentation, and multi-task state recognition, enabling power components of different scales to be effectively detected at their corresponding feature levels.

[0040] Step S106: Based on multi-scale visual features, perform power component target detection and instance segmentation operations to obtain preliminary identification information corresponding to multiple power component instances. The corresponding preliminary identification information includes: bounding box coordinates of the corresponding power component instance, component category, and contour mask. The component category includes: conductor and tower.

[0041] Among them, the power component target detection and instance segmentation operation refers to the location and classification of power components by a detection network based on multi-scale visual features. It can be a joint recognition process that outputs pixel-level contour masks by a segmentation network, which can simultaneously obtain the spatial location, category and precise contour of the component, providing a geometric and semantic basis for subsequent state diagnosis and topology analysis.

[0042] The preliminary identification information refers to the set of basic identification results output after target detection and instance segmentation for each power component instance. It can provide the position and category basis of the component for subsequent state identification and instance information fusion, including bounding box coordinates, component category and contour mask.

[0043] Among them, the bounding box coordinates refer to the positional parameters of the rectangular border surrounding the power component instance in the image. They can determine the approximate spatial range of the component in the image and provide the region of interest for subsequent state recognition.

[0044] Among them, component category refers to the equipment type label to which the identified power component belongs, which can distinguish different types of power equipment. For example, the category label for conductor is conductor, and the category label for pole is pole.

[0045] Among them, the contour mask refers to the binary mask image output after pixel-level segmentation of power component instances. It can accurately depict the shape contour of the component at the pixel level, providing a precise component area for refined geometric measurement.

[0046] Among them, power component instances refer to specific equipment individuals with spatial location and individual attributes that are actually identified by target detection and instance segmentation algorithms in specific inspection scenarios, and can provide independent equipment information units for subsequent status diagnosis and topology relationship analysis.

[0047] In this step, the multi-scale visual features generated in step S104 are used to locate the power components at each scale level and output the bounding box coordinates and component categories. Simultaneously, a pixel-level contour mask is used to form preliminary identification information for each power component instance. This step achieves the transformation from raw visual data to structured component recognition results. The system can accurately identify the location, category, and precise contour of each power component, such as each tower and each conductor segment, in the image, providing accurate target objects and spatial boundaries for subsequent multi-task state recognition and topology construction.

[0048] Step S108: Based on the preliminary identification information of multiple power component instances and the multi-scale feature set, determine the component status information corresponding to the multiple power component instances respectively, wherein the corresponding component status information includes at least one of the following: defect status information, risk parameter status information;

[0049] Among them, component status information refers to the status description information output after defect judgment and risk quantification for each power component instance. It can upgrade the component identification results into status diagnostic information with operation and maintenance value, including at least one of defect status information and risk parameter status information.

[0050] Among them, the defect status information refers to the classification results of whether a power component instance has a specific type of defect. It can directly guide operation and maintenance personnel to determine whether the component needs to be repaired and the urgency of the repair, such as defect types like insulator damage, broken conductor strands, and overheating of wire clamps.

[0051] Among them, risk parameter status information refers to the continuous numerical parameters output after quantitative calculation of the operational risks of power component instances. It can provide objective quantitative basis for assessing the safety status of components, such as conductor sag, the distance between conductors and surrounding objects, and infrared hot spot temperature gradient.

[0052] In this step, the preliminary identification information of each power component instance serves as the basis for spatial positioning. Multimodal features of the corresponding component's location are extracted from a multi-scale feature set. Optionally, a multi-task learning framework with a shared feature encoding layer can be used to simultaneously perform defect classification and risk parameter regression, outputting defect status information and risk parameter status information respectively. Through this step, the system can not only identify the component but also further diagnose its status, achieving a progressive analysis from appearance recognition to defect assessment and risk quantification, providing both quantitative and qualitative status information for inspection decisions.

[0053] Step S110: Based on the preliminary identification information and component status information of multiple power component instances, perform instance information fusion operation to obtain component instance information.

[0054] Among them, the instance information fusion operation refers to the process of integrating the preliminary identification information of the same power component instance with the component status information, which can unify the spatial location information, category information, outline information and status information of the component into a structured data record.

[0055] Among them, component instance information refers to the complete set of structured information output after fusing preliminary identification information and component status information for each power component instance. It can provide a one-stop component-level information input for downstream topology relationship analysis and inspection report generation, including the component's three-dimensional spatial location, contour mask, component category and status attributes.

[0056] In this step, the preliminary identification information and component status information of each power component instance are associated and integrated to generate component instance information containing all dimensions of component attributes. Through this step, the spatial, geometric, semantic, and status information of the component are output in a unified manner, so that each power component instance carries complete operation and maintenance attributes, providing a structured and semantic data foundation for subsequent line topology map construction and digital operation and maintenance decision-making.

[0057] Through steps S102-S110 above, multimodal sensor data is acquired. This multimodal sensor data is collected by the sensor system mounted on the UAV and includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Based on the multimodal sensor data, a multi-scale feature set is determined, which includes multi-scale visual features, global geometric features, acoustic spectrum features, and global attitude feature vectors. Based on the multi-scale visual features, power component target detection and instance segmentation operations are performed to obtain preliminary identification information corresponding to multiple power component instances. This preliminary identification information includes the bounding box coordinates of the corresponding power component instance, the component category, and the contour mask. The component category includes: conductors and towers. Based on the preliminary identification information and the multi-scale feature set of multiple power component instances, component status information corresponding to each power component instance is determined. This component status information includes at least one of the following: defect status information and risk parameter status information. Based on the preliminary identification information and component status information of multiple power component instances, an instance information fusion operation is performed to obtain component instance information. This invention employs a combination of multimodal perception and multi-scale feature extraction, along with power component detection, segmentation, and condition diagnosis. A UAV-mounted sensor system collects multimodal sensor data, which is then used to determine a multi-scale feature set. Based on these data, target detection and instance segmentation of the power components are performed using the multi-scale visual features to obtain preliminary identification information. This preliminary identification information is then combined with the multi-scale feature set to determine the component's condition information. Finally, instance information fusion is performed on the preliminary identification information and the component condition information. This approach achieves the goal of identifying latent defects in power components and improving the reliability of component condition diagnosis. Consequently, it achieves the technical effects of accurately identifying power components, reliably diagnosing component condition, and effectively identifying latent defects, thus solving the technical problems of difficulty in identifying latent defects in power components and poor reliability of component condition diagnosis in related technologies.

[0058] As an optional embodiment, based on the preliminary identification information and multi-scale feature sets of multiple power component instances respectively, the component state information corresponding to each of the multiple power component instances is determined, including: inputting the preliminary identification information and multi-scale feature sets of multiple power component instances respectively into a shared feature encoding model to obtain the component state information corresponding to each of the multiple power component instances. The shared feature encoding model includes an objective loss function, which is composed of a weighted sum of the classification loss of the defect classification task, the regression loss of the risk parameter regression task, and the segmentation loss of the instance segmentation task. The classification loss is used to measure the difference between the predicted category and the true defect category of the defect classification task, the regression loss is used to measure the difference between the predicted risk parameter and the true risk parameter of the risk parameter regression task, and the segmentation loss is used to measure the difference between the predicted contour mask and the true contour mask of the instance segmentation task.

[0059] Among them, the shared feature encoding model refers to a neural network model consisting of a feature encoding backbone network and multiple task head networks. Multiple task head networks share the output of the same set of feature encoding layers, which enables different tasks to promote each other during training and extract common features that have discriminative power for multiple tasks.

[0060] The objective loss function refers to the overall optimization objective adopted when training the shared feature encoding model, which is composed of a weighted sum of multiple task loss terms. It can unify the optimization objectives of multiple related tasks into a joint training framework and improve the generalization performance of each task through joint optimization.

[0061] Among them, the defect classification task refers to the classification task of determining whether a power component instance has a preset type of defect, and can directly output the judgment result of whether the component has a specific defect and the type of defect.

[0062] Among them, the risk parameter regression task refers to the regression task that continuously predicts the operational risk parameters of power component instances, and can output quantitative risk indicators.

[0063] Among them, the instance segmentation task refers to the segmentation task of outputting pixel-level contour masks for power component instances, which can provide accurate component region boundaries and provide accurate spatial constraints for defect classification and risk parameter regression tasks.

[0064] Among them, classification loss refers to the loss term that measures the difference between the predicted category and the true defect category in the defect classification task, and can guide the model to improve the accuracy of defect classification during the training process.

[0065] Among them, regression loss refers to the loss term that measures the difference between the predicted risk parameter and the actual risk parameter in the risk parameter regression task, and can guide the model to improve the accuracy of risk parameter prediction during the training process.

[0066] Segmentation loss refers to the loss term that measures the difference between the predicted contour mask and the real contour mask in the instance segmentation task. It can guide the model to improve the accuracy of component contour segmentation during training.

[0067] In this embodiment, the preliminary identification information and multi-scale feature set of each power component instance are input into a shared feature encoding model. Multiple task head networks of this model share the output of the same feature encoding layer, and jointly optimize the three tasks of defect classification, risk parameter regression, and instance segmentation during the training phase using a target loss function. In this way, the shared feature encoding layer can simultaneously receive gradient supervision signals from the three tasks of defect classification, risk parameter regression, and instance segmentation, learning robust feature representations with universal discriminative power across multiple tasks. The accurate contour mask provided by the instance segmentation task provides precise component regions for defect classification and risk parameter regression, eliminating interference from complex backgrounds, making defect classification more accurate and risk parameter calculation more reliable. The three tasks promote and constrain each other during joint training. Compared to independent training of each task, this effectively mitigates the risk of overfitting, improves the generalization performance of each task, and ultimately enables the system to achieve both high classification accuracy and risk parameter prediction accuracy in component state diagnosis.

[0068] As an optional embodiment, determining a multi-scale feature set based on multimodal sensor data includes: inputting visual imaging data into a feature extraction model to obtain multi-scale visual features. The feature extraction model includes a backbone network and a feature pyramid structure network. The backbone network includes any one of the following: a convolutional neural network, a visual transformer network. The feature pyramid structure network is used to determine multi-scale visual features based on the hierarchical features output by different levels of the backbone network. The backbone network includes at least a first level and a second level. The first level is used to extract detail information features, and the second level is used to extract semantic information features. The resolution corresponding to the first level is higher than the resolution corresponding to the second level.

[0069] Among them, the feature extraction model refers to the neural network model that performs multi-layer feature encoding on visual imaging data, including the backbone network and the feature pyramid structure network, which can extract multi-scale visual features containing different resolutions and semantic levels from the original image.

[0070] The backbone network refers to the deep convolutional or transformer network in the feature extraction model, which is used to extract image features layer by layer and can output feature maps with different resolutions and semantic levels at different depth levels.

[0071] Among them, the feature pyramid structure network refers to a network structure that connects to the backbone network and fuses the feature maps output from different levels of the backbone network. It can fuse deep high semantic features with shallow high-resolution detail features, so that feature maps at each scale level have strong semantic expression capabilities and spatial positioning accuracy.

[0072] The first layer, which refers to the shallow network layer near the input end in the backbone network, can extract high-resolution feature maps and retain detailed information features such as edges, contours and textures, providing accurate spatial positioning information for small-scale component detection.

[0073] The second layer, which refers to the deep network layer near the output end in the backbone network, can extract low-resolution feature maps, has a large receptive field and rich semantic information features, and provides a basis for class judgment for the identification of large-scale components.

[0074] In this embodiment, visual imaging data is input into a feature extraction model. Multi-scale visual features are obtained through layer-by-layer feature extraction via a backbone network and multi-layer feature fusion via a feature pyramid structure network. The backbone network contains at least a first level and a second level. The first level outputs high-resolution detail features, and the second level outputs high-semantic semantic features. The feature pyramid structure network fuses the feature maps from each level. In this way, the feature pyramid structure network fuses deep high-semantic features with shallow high-resolution features, enabling feature maps at each scale to possess both strong semantic discrimination capabilities and spatial positioning accuracy. The high-resolution features of the first level ensure high-precision spatial positioning of small-scale components, while the high-semantic features of the second level ensure reliable category recognition of large-scale components. For power components with significant size differences, such as large-scale conductors spanning the screen and small-scale insulators at long distances, the system can effectively detect them at the corresponding feature levels, solving the problem that single-scale features cannot simultaneously handle the detection of targets of different sizes, and significantly improving the comprehensiveness and accuracy of power component detection.

[0075] As an optional embodiment, a multi-scale feature set is determined based on multimodal sensor data, including: for each target point in the spatial depth point cloud data, determining a neighborhood point set centered on the target point; extracting local geometric features of the neighborhood point set through a mapping function based on the relative positional relationship between each neighborhood point and the target point; aggregating the local geometric features through an aggregation function to obtain the target point features corresponding to the target point; and obtaining global geometric features corresponding to the spatial depth point cloud data based on the target point features corresponding to each target point, wherein the global geometric features are used to characterize at least one of the following: the suspension shape of the conductor, the spatial configuration of the tower, and the safe distance between the conductor and surrounding objects.

[0076] The target point refers to a spatial point in the spatial depth point cloud data where local geometric features are currently being calculated, and can serve as the basic processing unit for point cloud feature extraction.

[0077] Among them, the neighborhood point set refers to the set of nearby spatial points selected within a preset spatial range with the target point as the center, which can provide spatial context information for the extraction of local geometric features of the target point.

[0078] The mapping function refers to a mathematical function that maps the relative positional relationship between neighboring points and the target point into a feature representation. It can extract local geometric structure information from the spatial distribution of neighboring points and is usually implemented using a multilayer perceptron.

[0079] Aggregation functions are mathematical functions that summarize and aggregate the mapping features of each neighboring point in a neighborhood point set. They can compress the features of multiple neighboring points into a feature vector that represents the local geometric environment of the target point. They are usually implemented using max pooling or average pooling.

[0080] Among them, target point features refer to the feature vectors that represent the local geometric structure of the target point's location after processing the neighborhood point set of the target point through mapping and aggregation functions, and can describe the spatial structure morphology around the target point.

[0081] In this embodiment, for each target point in the spatial depth point cloud data, a neighborhood point set centered on the target point is first determined. Based on the relative positional relationship between the neighborhood points and the target point, the local geometric features of the neighborhood point set are extracted through a mapping function. Then, the target point features are summarized through an aggregation function. Finally, the features of each target point are summarized into global geometric features representing the overall spatial structure. In this way, the point convolutional network can directly process unordered point cloud data. Through the extraction and aggregation of local neighborhood geometric features, it learns key three-dimensional geometric information such as the sag curve shape of the conductor, the spatial configuration of the tower, and the safety distance point by point, preserving the three-dimensional spatial structural characteristics of the original point cloud and providing accurate spatial geometric constraints for subsequent cross-modal fusion with visual features.

[0082] As an optional embodiment, a multi-scale feature set is determined based on multimodal sensor data, including: performing a Fourier transform operation on the acoustic signal data to convert the acoustic signal data from a time-domain signal to a time-frequency domain to obtain acoustic time-frequency domain features; inputting the acoustic time-frequency domain features into a convolutional neural network for encoding to obtain acoustic spectral features, wherein the acoustic spectral features are used to characterize at least one of the following: partial discharge features, corona discharge features, and mechanical anomaly features.

[0083] The Fourier transform operation refers to the mathematical transformation operation that converts acoustic signal data into time-frequency domain data. It can convert time-domain audio signals into a two-dimensional spectral feature map that retains both time and frequency information, reflecting the frequency component distribution of the audio signal in different time periods.

[0084] The time-frequency domain refers to the signal representation domain that includes both time and frequency dimensions. It can show the changes in the signal spectrum over time, which is different from the pure frequency domain representation that only retains frequency information but loses time information.

[0085] Among them, acoustic time-frequency domain features refer to the two-dimensional spectral feature map obtained after Fourier transform of acoustic signal data, which can be used as input for convolutional neural network encoding to further extract higher-level acoustic spectral features.

[0086] Among them, partial discharge characteristics refer to the characteristic modes in the acoustic spectrum that are related to the partial discharge phenomenon on the surface of insulators or conductors, which can help detect and locate partial discharge faults.

[0087] Among them, corona discharge characteristics refer to the characteristic patterns in the acoustic spectrum that are related to the corona discharge phenomenon on the surface of the conductor or fitting, which can help detect abnormal corona discharge.

[0088] Among them, mechanical anomaly features refer to the characteristic patterns in the acoustic spectrum that are related to the loosening of the mechanical structure and abnormal vibration of the equipment, which can help detect mechanical faults.

[0089] In this embodiment, a Fourier transform operation is performed on the acoustic signal data to convert the original time-domain acoustic signal from the time domain to the time-frequency domain, obtaining acoustic time-frequency domain features that retain two-dimensional information of time and frequency. These acoustic time-frequency domain features are then input into a convolutional neural network for deep encoding to obtain acoustic spectral features. In this way, the Fourier transform converts the one-dimensional time-domain audio signal into a two-dimensional time-frequency domain feature map, enabling the acoustic signal to be effectively processed by the convolutional neural network. The convolutional neural network further learns high-level acoustic spectral features from the time-frequency domain features, effectively capturing the unique spectral patterns of typical anomalies such as partial discharge, corona discharge, and mechanical loosening. This provides non-contact acoustic anomaly perception capabilities for multimodal condition diagnosis, compensating for the shortcomings of visual modalities in detecting internal defects and latent faults such as partial discharge.

[0090] As an optional embodiment, a multi-scale feature set is determined based on multimodal sensor data, including: constructing a multi-dimensional temporal signal based on attitude inertial sequence data and position coordinate data, wherein the multi-dimensional temporal signal includes the three-dimensional position coordinate vector, three-axis attitude angle vector, three-axis acceleration vector, and three-axis angular velocity vector of the UAV at multiple time steps; sequentially inputting the vectors corresponding to multiple time steps into a stacked long short-term memory network to obtain hidden state vectors corresponding to multiple time steps, wherein the hidden state vectors represent the temporal feature information of the corresponding time step and all previous time steps; performing an average pooling operation on the hidden state vectors corresponding to multiple time steps to obtain a global attitude feature vector, and then...

[0091] Among them, multidimensional time-series signals refer to structured time-series data composed of attitude inertial sequence data and position coordinate data at multiple time steps, with each time step containing multiple dimensional parameters, which can completely describe the spatial position changes and attitude motion state of the UAV over a period of time.

[0092] Among them, stacked long short-term memory network refers to a deep recurrent neural network structure composed of multiple layers of long short-term memory network units connected in series. It can extract features by abstracting time series data layer by layer. The shallow network captures short-term motion patterns, and the deep network learns long-term motion laws.

[0093] The hidden state vector refers to the feature vector output by the Long Short-Term Memory network at each time step, which contains historical information of the current time step and previous time steps. It can represent the cumulative temporal feature information of the corresponding time step and all previous time steps.

[0094] Average pooling is an operation that calculates the average value of the hidden state vector at each time step along the time dimension. It can compress a variable-length hidden state sequence into a fixed-dimensional global feature vector, which comprehensively reflects the motion characteristics of the entire time series.

[0095] In this embodiment, a multi-dimensional temporal signal is constructed based on attitude inertial sequence data and position coordinate data. The multi-dimensional vectors corresponding to each time step are sequentially input into a stacked long short-term memory network. The shallow network extracts short-term motion patterns, and the deep network learns long-term motion patterns to obtain hidden state vectors for each time step. These hidden state vectors are then aggregated into a global attitude feature vector through average pooling. In this way, the multi-layer structure of the stacked long short-term memory network can perform hierarchical modeling of the UAV's motion state. The shallow layer captures instantaneous changes in acceleration and angular velocity to reflect attitude stability and vibration characteristics, while the deep layer learns the long-term variation of the trajectory to reflect flight smoothness. Average pooling compresses the variable-length sequence into a fixed-dimensional global attitude feature vector, providing a compact and information-rich representation of the UAV's motion state for subsequent assisted localization and abnormal state judgment.

[0096] As an optional embodiment, after performing an instance information fusion operation based on the preliminary identification information and component status information of multiple power component instances to obtain component instance information, the method further includes: identifying multiple power component instances in the component instance information as nodes in a graph structure; determining the connection relationship between nodes in the graph structure based on the geometric proximity between nodes, and using the connection relationship as an edge in the graph structure; and constructing a line space topology graph based on the nodes and edges.

[0097] Among them, graph structure refers to a mathematical structure composed of nodes and edges, which can express the spatial connection relationship between power component instances in a structured way.

[0098] In this context, a node refers to a data unit in the graph structure that represents an instance of a power component. It can carry the device attribute information of the corresponding power component instance. For example, when a tower instance is used as a node, it carries its spatial coordinates and device attitude information.

[0099] Geometric proximity refers to the degree of closeness between two nodes in three-dimensional space, which can serve as the basic basis for determining whether the power component instances corresponding to the two nodes have a connection relationship in physical space.

[0100] In this context, an edge refers to a line segment in a graph structure that connects two nodes, representing a physical connection between two power component instances.

[0101] Among them, the connection relationship refers to the physical spatial connectivity between power component instances, which can provide a connection basis for the construction of the topology graph.

[0102] In this embodiment, after obtaining the component instance information, each power component instance in the component instance information is identified as a node in the graph structure. The geometric proximity between nodes is calculated based on their spatial coordinates. Node pairs that satisfy the geometric proximity condition are identified as having a connection relationship, and these connections are used as edges in the graph structure. A line spatial topology graph is then constructed based on the nodes and edges. In this way, the system can automatically infer the spatial connection relationships between components from the discrete component instance information output from the perception stage, organizing the independent component identification results into a topology graph with structured connections. This achieves a leap from component-level perception to line-level topology understanding, providing structured line model input for UAV autonomous inspection path planning and topology updates of the power grid geographic information system.

[0103] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0104] In related technologies, 10kV overhead lines are widely distributed in urban and rural power distribution networks. Their operating environment is complex, with large line spans and diverse component types. Long-term exposure to external conditions such as wind, rain, icing, and vegetation growth makes them prone to hidden dangers such as insulator damage, foreign object hanging on conductors, wire clamp overheating, and insufficient safety distances. Existing line inspection methods mainly rely on manual inspection or visible light image detection based on UAVs, which have the following shortcomings: First, existing technologies mostly rely on a single visible light mode, making it difficult to effectively identify latent defects such as infrared thermal anomalies and partial discharges; second, for targets with significant scale differences (such as long-distance conductors and small fittings), the detection accuracy and stability are insufficient; furthermore, multi-sensor data is usually processed independently, lacking a unified modeling and deep fusion mechanism, resulting in limited diagnostic reliability; existing methods mostly remain at the component level of identification, making it difficult to automatically construct line-level spatial topology relationships, and failing to provide structured environmental information for UAV autonomous tracking and decision-making. Therefore, there is an urgent need for a technical solution that can perform unified feature extraction, intelligent identification, and state diagnosis on multi-modal inspection data acquired by UAVs, and automatically analyze the spatial topology relationships of the lines.

[0105] In view of this, an optional embodiment of the present invention provides a method for determining component instance information, which can also be called a method and system for 10kV overhead line UAV inspection based on multimodal feature extraction, intelligent recognition and topology relationship analysis, so as to improve the automation level and decision reliability of UAV inspection operations. Figure 2 This is a schematic diagram of the overall architecture provided by an optional embodiment of the present invention, such as... Figure 2 As shown, the system adopts a layered design, comprising five main parts: a multimodal data acquisition layer, a multimodal feature encoding and hierarchical extraction layer, an intelligent recognition layer, a topology parsing layer, and an application decision layer. Specifically, it includes:

[0106] 1. Multimodal data acquisition layer:

[0107] Core function: As the system's perception entry point, it completes the collection of raw data from all dimensions of 10kV overhead line inspections through multimodal sensors carried by drones.

[0108] Components: Optical and thermal imaging sensing unit (high-definition visible light, infrared thermal imager, acoustic imager); Acoustic and ultrasonic sensing unit (acoustic imager, ultrasonic sensor); Spatial perception and positioning unit (LiDAR, GPS receiver, IMU).

[0109] Data flow: Multi-source heterogeneous data such as visible light images, infrared thermal images, acoustic signals, laser point clouds, and GPS / IMU spatiotemporal attitude data (same as the above-mentioned visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data) are output to the multimodal feature encoding and hierarchical extraction layer as the original input for feature extraction.

[0110] 2. Multimodal feature encoding and hierarchical extraction layer:

[0111] Core function: Preprocessing and deep encoding raw data to transform heterogeneous data into a unified multidimensional feature vector, providing support for intelligent recognition.

[0112] Components include: visible light and infrared modal feature extraction (Convolutional Neural Network (CNN) + Feature Pyramid to extract visual and infrared physical features); laser point cloud modal feature extraction (PointNet++ to extract spatial structure features); acoustic modal feature extraction (Short Time Fourier Transform + MobileNet to extract acoustic features); and spatiotemporal attitude sequence feature extraction (Long Short Time Memory Network (LSTM) to extract UAV attitude and trajectory features).

[0113] Data flow: The feature vectors of various modalities are fused to generate a multidimensional feature vector, which is then output to the intelligent recognition layer as input for target detection and state diagnosis.

[0114] 3. Intelligent Recognition Layer:

[0115] Core functions: Based on multi-dimensional feature vectors, it completes power component identification, status assessment and anomaly diagnosis, and outputs structured component and fault information.

[0116] Components include: object detection and instance segmentation (Faster R-CNN / YOLO localization and classification, Mask R-CNN output pixel-level mask); state recognition and anomaly diagnosis (multi-task learning framework to achieve defect classification and risk parameter regression); and multimodal evidence fusion diagnosis (cross-modal joint reasoning to improve diagnostic confidence).

[0117] Data flow: The component identification results, spatial positioning information, and status diagnosis information are output to the topology parsing layer as the basis for topology relationship construction.

[0118] 4. Topology resolution layer:

[0119] Core function: Based on component-level information, construct the spatial topology of 10kV overhead lines and provide a structured line model.

[0120] Component modules: Component node construction (electrical components are treated as nodes, and assigned category, coordinates, and state attributes); Connection relationship reasoning (inferring node connections by combining geometric proximity and learning models); Spatial topology graph generation (constructing a topology graph G=(V,E)).

[0121] Data flow: The spatial topology map G is output to the application decision layer as the core input for autonomous inspection path planning.

[0122] 5. Application Decision-Making Level:

[0123] Core function: By combining the topology map G with the real-time inspection status, the system completes path planning, flight control, and diagnostic result output, thereby achieving automated inspection.

[0124] Components: Autonomous inspection path planning (planning the optimal path based on the topology map); flight control and mission scheduling (generating control commands and optimizing energy consumption); diagnostic report generation (integrating results and outputting a standardized report).

[0125] Data flow: Receive topology map G and real-time status feedback from the UAV platform to form a closed loop of "decision-execution-feedback", and finally output inspection and diagnosis report and task execution results.

[0126] Final application: Based on the topology map G, the system automatically plans the optimal inspection path covering all fault points, avoiding obstacles such as trees.

[0127] The technical solution of the present invention includes the following steps:

[0128] Step 1: Multimodal UAV Inspection Data Collection and Acquisition

[0129] The drone is equipped with a multimodal sensor system, including:

[0130] Optical and thermal imaging sensing unit: Includes a high-definition visible light camera, an infrared thermal imaging camera, and an acoustic imager, used to capture visible light and thermal feature images of lines, towers, insulators, transformers, and circuit breakers, providing a data foundation for feature recognition and extraction.

[0131] Acoustic and ultrasonic sensing unit: includes an acoustic imager (for detecting abnormal sound sources such as corona discharge) and an ultrasonic sensor to assist in the detection of nearby obstacles and the location of non-visible faults.

[0132] Spatial Perception and Positioning Unit: This unit includes a lidar unit, a GPS receiver, and an inertial measurement unit (IMU). The lidar unit is used for high-precision 3D environmental scanning and ranging; the GPS unit provides the UAV's global latitude and longitude coordinates; and the IMU unit is used to measure the UAV's three-axis acceleration and three-axis angular velocity in real time, providing high-frequency attitude data.

[0133] Step 2: Multimodal Feature Encoding and Hierarchical Extraction:

[0134] 2.1 Visible and Infrared Mode Feature Extraction:

[0135] Visible light images captured by drones and infrared thermal imaging images The input is first fed into a feature extraction network for encoding. The feature extraction network is preferably a convolutional neural network or a visual transformer network, including but not limited to ResNet, EfficientNet, or SwinTransformer. To simultaneously process power components of different scales, a feature pyramid structure is introduced, outputting multi-scale feature maps at different levels of the backbone network. Among them, low-level feature maps are used to represent high-resolution detail information; high-level feature maps are used to represent semantic information with a large receptive field. For infrared thermal imaging data, in addition to general visual features, the following physical features are further extracted: the highest temperature of the target area. Average temperature and temperature variance Area of ​​hot spot Shape characteristics (such as eccentricity and extensibility) and distribution characteristics. These infrared characteristics are used to characterize the thermal state of the components and provide a basis for subsequent anomaly diagnosis.

[0136] As described above, the visual imaging data is input into a feature extraction model to obtain the multi-scale visual features. The feature extraction model includes a backbone network and a feature pyramid structure network. The backbone network includes any one of the following: a convolutional neural network, a visual transformer network. The feature pyramid structure network is used to determine the multi-scale visual features based on the hierarchical features output by different levels of the backbone network. The backbone network includes at least a first level and a second level. The first level is used to extract detail information features, and the second level is used to extract semantic information features. The resolution corresponding to the first level is higher than the resolution corresponding to the second level.

[0137] 2.2. Laser point cloud modal feature extraction:

[0138] The three-dimensional point cloud data collected by lidar is used to characterize the spatial structure information of the line and its surrounding environment.

[0139] The point cloud data can be used for feature extraction using point convolutional networks, such as PointNet++ or its variants, to extract point-by-point features by learning the local neighborhood geometric relationships of points. This process can be formalized as follows: .in, It is a point The neighborhood, It is a mapping function (usually a multilayer perceptron). These are aggregate functions (such as maximum pooling, average pooling). This represents the network layer number. `i` represents the index of the currently processed center point. Specifically, `pi` is a target point in the point cloud, and the formula calculates the k-th layer feature `fpi(k)` for that point. Each point is used as a center point for feature extraction sequentially. `j` represents the index of neighboring points within the neighborhood of center point `pi`. `N(i)` is the set of all neighboring points within a given radius (or K nearest neighbors) centered at `pi`. `j` iterates through each point in this neighborhood. For a fixed center point `i`, each neighboring point `j` within its neighborhood contributes to the calculation of the center point's features. The formula aggregates the output of the mapping function for all neighboring points `j` (e.g., maximum or average pooling) to obtain the updated features of the center point. By stacking multiple layers, conductor suspension morphology, tower spatial configuration, and safety distance information can be extracted.

[0140] As described above, for each target point in the spatial depth point cloud data, a neighborhood point set centered on the target point is determined; based on the relative positional relationship between each neighboring point in the neighborhood point set and the target point, local geometric features of the neighborhood point set are extracted through a mapping function; the local geometric features are aggregated through an aggregation function to obtain the target point features corresponding to the target point; based on the target point features corresponding to each target point, global geometric features corresponding to the spatial depth point cloud data are obtained, wherein the global geometric features are used to characterize at least one of the following: the suspension shape of the conductor, the spatial configuration of the tower, and the safe distance between the conductor and surrounding objects.

[0141] 2.3 Acoustic Modal Feature Extraction:

[0142] Audio signals collected by acoustic sensors First, the acoustic spectral feature map is generated by transforming the time domain to the time-frequency domain using a short-time Fourier transform. .

[0143] Where, t: continuous time variable (unit: seconds), representing the sampling time of the original audio signal. Integration is performed over the entire time axis. S(t): original time-domain audio signal. Acquired by acoustic sensors (such as microphone arrays, acoustic imagers), reflecting the change of air pressure over time. For example, partial discharge produces high-frequency noise, while corona discharge produces a specific hissing sound. τ: center time position of the window function (i.e., time shift parameter). The short-time Fourier transform divides a long-time signal into multiple short-time segments by sliding a window function along the time axis. Each τ corresponds to the position of a segment, thus obtaining the spectral information of the signal changing over time. ω: angular frequency (unit: radians / second), related to frequency f by ω = 2πf. Each ω in the spectrum corresponds to a frequency channel, used to analyze the energy distribution of the signal at different frequencies. ω(t-τ): window function, which is non-zero only near t≈τ, used to truncate a local segment of the signal. The width of the window function determines the trade-off between time resolution and frequency resolution. e-jωt: complex exponential basis function (Fourier kernel). Used to perform Fourier analysis on the captured signal segment, decomposing it into different frequency components. Here, j is the imaginary unit (j² = -1). S(τ,ω): Acoustic spectral feature map. It is a two-dimensional array, with time τ on the horizontal axis and frequency ω on the vertical axis. Each pixel value represents the energy at that frequency at that moment. This map can be used as input to subsequent neural networks (such as MobileNet) to extract acoustic feature vectors faudio reflecting partial discharge, corona discharge, mechanical anomalies, etc.

[0144] Subsequently, the lightweight neural network MobileNet was used to analyze the spectral features. Encoding is performed to extract acoustic feature vectors that can characterize partial discharge, corona discharge, or mechanical anomalies. .

[0145] Similar to the above, Fourier transform operation is performed on the acoustic signal data to convert the acoustic signal data from the time domain to the time-frequency domain, obtaining acoustic time-frequency domain features; the acoustic time-frequency domain features are input into a convolutional neural network for encoding to obtain acoustic spectrum features, wherein the acoustic spectrum features are used to characterize at least one of the following: partial discharge features, corona discharge features, and mechanical anomaly features.

[0146] 2.4 Spatiotemporal attitude sequence feature extraction:

[0147] During autonomous inspection by unmanned aerial vehicles (UAVs), the attitude stability and trajectory smoothness of the flight platform directly affect the quality of sensor data acquisition and the accuracy of subsequent target identification and positioning. This invention comprehensively utilizes a GPS receiver and an inertial measurement unit to construct a high-frequency, high-precision spatiotemporal attitude sequence.

[0148] Suppose the UAV is in discrete time step The collected data can be represented as: .

[0149] in, These are longitude, latitude, and altitude, respectively.

[0150] These are roll angle, pitch angle, and yaw angle, which describe the attitude of the UAV.

[0151] The acceleration is a three-axis acceleration, reflecting the vibration and maneuvering state of the UAV;

[0152] It represents the three-axis angular velocity, reflecting the rotational rate of the drone.

[0153] The aforementioned multi-source heterogeneous data collectively constitute a high-dimensional time series signal. Where D=12 is the feature dimension. A stacked long short-term memory (LSTM) network is used to model the attitude sequence: the data is modeled to extract the UAV's flight attitude stability and trajectory change features. ,in In hidden state, The hidden layer dimension is determined by taking the hidden state at the last time step. Average pooling is performed at all time steps. Obtain the global pose feature vector The features are used to assist in localization and abnormal state judgment.

[0154] Based on the attitude inertial sequence data and position coordinate data mentioned above, a multi-dimensional temporal signal is constructed. This multi-dimensional temporal signal includes the UAV's three-dimensional position coordinate vectors, three-axis attitude angle vectors, three-axis acceleration vectors, and three-axis angular velocity vectors at multiple time steps. The vectors corresponding to multiple time steps are sequentially input into a stacked long short-term memory network to obtain hidden state vectors corresponding to each time step. These hidden state vectors represent the temporal feature information of the corresponding time step and all previous time steps. Average pooling is then performed on the hidden state vectors corresponding to multiple time steps to obtain a global attitude feature vector.

[0155] Step 3: Intelligent Recognition and Deconstruction Based on Deep Learning:

[0156] 1. Object detection and instance segmentation:

[0157] After obtaining multi-scale features, target detection networks (such as Faster R-CNN, YOLO) are used to locate and classify power components in visible light and infrared images. The identified objects include conductors, towers, insulators, clamps, vibration dampers, transformers, and circuit breakers. The detection results can be represented as follows: ,in For bounding box coordinates, As a category, The confidence level is used. For components requiring precise contour information, an instance segmentation network (such as Mask R-CNN) is further employed to output the corresponding pixel-level mask. To support precise measurement tasks such as length, angle, and deformation.

[0158] 2. Status recognition and anomaly diagnosis:

[0159] After component identification, a multi-task learning framework is constructed to analyze the component status, including: a defect classification task to determine whether a component has a specific defect; and a risk parameter regression task to calculate parameters such as conductor sag, safety clearance, and temperature gradient. The loss function of this multi-task learning framework can be defined as: .in, For classifying losses, To regress the loss, To divide the loss. , To balance the weighting coefficients of each task, the aforementioned tasks share a feature encoding layer, and joint optimization improves the accuracy of state recognition.

[0160] As described above, the preliminary identification information and multi-scale feature sets of multiple power component instances are input into the shared feature encoding model to obtain the component state information corresponding to each of the multiple power component instances. The shared feature encoding model includes an objective loss function, which is composed of a weighted sum of the classification loss of the defect classification task, the regression loss of the risk parameter regression task, and the segmentation loss of the instance segmentation task. The classification loss is used to measure the difference between the predicted category and the actual defect category of the defect classification task. The regression loss is used to measure the difference between the predicted risk parameter and the actual risk parameter of the risk parameter regression task. The segmentation loss is used to measure the difference between the predicted contour mask and the actual contour mask of the instance segmentation task.

[0161] 3. Multimodal evidence fusion diagnosis:

[0162] For complex fault scenarios, the system will integrate multimodal features from visible light, infrared, acoustics, and point clouds, and perform joint reasoning in a unified feature space to output the final anomaly diagnosis result, thereby improving the diagnostic confidence and robustness.

[0163] Step 4: Topological relation resolution:

[0164] After completing component instance-level identification and location, the line topology relationship parsing step is performed, including:

[0165] 1. Treat each identified power component as a node in a graph structure. Node attributes include component category, spatial coordinates, and status information;

[0166] 2. Based on the geometric proximity between components (e.g., Euclidean distance less than a threshold) (Or spatial orientation consistency) and learning-based relational reasoning models, inferring the connection relationships between nodes, i.e., edges.

[0167] 3. Construct a spatial topology diagram G=(V,E) to describe the connection relationships of 10kV overhead lines.

[0168] The topology map G serves as an important input for the UAV's autonomous inspection and decision-making, and is used to plan the autonomous flight path along the route.

[0169] As described above, multiple power component instances in the component instance information are respectively identified as nodes in the graph structure; based on the geometric proximity between nodes, the connection relationship between each node in the graph structure is determined, and the connection relationship is used as the edge in the graph structure; based on the nodes and edges, a line space topology graph is constructed.

[0170] The above optional implementation methods can achieve at least the following beneficial effects:

[0171] (1) Achieving unified modeling and hierarchical feature extraction of multimodal inspection data: This invention comprehensively utilizes various heterogeneous sensor data such as visible light, infrared, laser point cloud, acoustic and attitude data, and achieves unified expression of information of different scales and different physical attributes through a hierarchical feature encoding mechanism, effectively overcoming the problem of insufficient information of single modality perception.

[0172] (2) Improve the accuracy and robustness of identifying power components and their hidden defects: By introducing an intelligent identification mechanism that combines multi-scale feature extraction, target detection and instance segmentation, this invention can simultaneously take into account large-scale targets such as conductors and small-scale components such as insulators and fittings, thereby improving the identification accuracy in complex backgrounds and long-distance scenarios.

[0173] (3) Support for refined state identification and risk assessment through multi-task learning: Based on component identification, this invention constructs a multi-task learning framework to simultaneously realize defect type discrimination and key risk parameter regression, enabling the system to not only determine "whether it is abnormal", but also to quantify the degree of abnormality and enhance the practicality of inspection results.

[0174] (4) It has good scalability and engineering application value: The method and system of the present invention adopt a modular design, which can be flexibly adapted to different types of sensor configurations and inspection scenarios, and is easy to deploy in existing UAV inspection systems, and has good engineering promotion value.

[0175] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0177] Example 2

[0178] According to embodiments of the present invention, an apparatus for implementing the above-described method for determining component instance information is also provided. Figure 3 This is a structural block diagram of a device for determining component instance information according to an embodiment of the present invention, such as... Figure 3 As shown, the device includes: an acquisition module 302, a first determination module 304, a second determination module 306, a third determination module 308, and a fourth determination module 310. The device will be described in detail below.

[0179] The acquisition module 302 is used to acquire multimodal sensor data, which is collected by the sensor system mounted on the UAV. The multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. The first determination module 304, connected to the acquisition module 302, is used to determine a multi-scale feature set based on the multimodal sensor data. The multi-scale feature set includes multi-scale visual features, global geometric features, acoustic spectrum features, and a global attitude feature vector. The second determination module 306, connected to the first determination module 304, is used to perform power component target detection and instance segmentation operations based on the multi-scale visual features, obtaining multiple power component instances respectively. The corresponding preliminary identification information includes: the bounding box coordinates of the corresponding power component instance, the component category, and the contour mask. The component category includes: conductor and tower. The third determination module 308, connected to the second determination module 306, is used to determine the component status information corresponding to the multiple power component instances based on the preliminary identification information and multi-scale feature set of the multiple power component instances. The corresponding component status information includes at least one of the following: defect status information and risk parameter status information. The fourth determination module 310, connected to the third determination module 308, is used to perform an instance information fusion operation based on the preliminary identification information and component status information of the multiple power component instances to obtain component instance information.

[0180] It should be noted here that the above-mentioned acquisition module 302, first determination module 304, second determination module 306, third determination module 308 and fourth determination module 310 correspond to steps S102 to S110 in the method for determining the implementation component instance information. The multiple modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0181] Example 3

[0182] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement a method for determining component instance information of any of the above.

[0183] Example 4

[0184] According to another aspect of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method for determining component instance information of any of the above-mentioned embodiments.

[0185] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0186] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0188] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0190] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0191] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for determining component instance information, characterized in that, include: Acquire multimodal sensor data, wherein the multimodal sensor data is collected by the sensor system carried by the UAV, and the multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; Based on the multimodal sensor data, a multi-scale feature set is determined, wherein the multi-scale feature set includes multi-scale visual features, global geometric features, acoustic spectral features, and global pose feature vector. Based on the multi-scale visual features, power component target detection and instance segmentation operations are performed to obtain preliminary identification information corresponding to multiple power component instances. The corresponding preliminary identification information includes: the bounding box coordinates of the corresponding power component instance, the component category, and the contour mask. The component category includes: conductor and tower. Based on the preliminary identification information of the plurality of power component instances and the multi-scale feature set, component status information corresponding to the plurality of power component instances is determined, wherein the corresponding component status information includes at least one of the following: defect status information, risk parameter status information; Based on the preliminary identification information and component status information of the multiple power component instances, an instance information fusion operation is performed to obtain component instance information.

2. The method according to claim 1, characterized in that, Based on the preliminary identification information of the plurality of power component instances and the multi-scale feature set, the component status information corresponding to the plurality of power component instances is determined, including: The preliminary identification information of the multiple power component instances and the multi-scale feature set are input into a shared feature encoding model to obtain component state information corresponding to the multiple power component instances. The shared feature encoding model includes a target loss function, which is composed of a weighted sum of the classification loss of the defect classification task, the regression loss of the risk parameter regression task, and the segmentation loss of the instance segmentation task. The classification loss is used to measure the difference between the predicted category and the true defect category of the defect classification task. The regression loss is used to measure the difference between the predicted risk parameter and the true risk parameter of the risk parameter regression task. The segmentation loss is used to measure the difference between the predicted contour mask and the true contour mask of the instance segmentation task.

3. The method according to claim 1, characterized in that, Based on the multimodal sensor data, a multi-scale feature set is determined, including: The visual imaging data is input into a feature extraction model to obtain the multi-scale visual features. The feature extraction model includes a backbone network and a feature pyramid structure network. The backbone network includes any one of the following: a convolutional neural network and a visual transformer network. The feature pyramid structure network is used to determine the multi-scale visual features based on the hierarchical features output by different levels of the backbone network. The backbone network includes at least a first level and a second level. The first level is used to extract detail information features, and the second level is used to extract semantic information features. The resolution corresponding to the first level is higher than the resolution corresponding to the second level.

4. The method according to claim 1, characterized in that, Based on the multimodal sensor data, a multi-scale feature set is determined, including: For each target point in the spatial depth point cloud data, determine a set of neighborhood points centered on the target point; Based on the relative positional relationship between each neighboring point in the neighborhood point set and the target point, the local geometric features of the neighborhood point set are extracted through a mapping function; The local geometric features are aggregated using an aggregation function to obtain the target point features corresponding to the target point; Based on the target point features corresponding to each target point, global geometric features corresponding to the spatial depth point cloud data are obtained, wherein the global geometric features are used to characterize at least one of the following: the suspension shape of the conductor, the spatial configuration of the tower, and the safe distance between the conductor and surrounding objects.

5. The method according to claim 1, characterized in that, Based on the multimodal sensor data, a multi-scale feature set is determined, including: Perform a Fourier transform operation on the acoustic signal data to convert the acoustic signal data from a time-domain signal to a time-frequency domain, thereby obtaining acoustic time-frequency domain features; The acoustic time-frequency domain features are input into a convolutional neural network for encoding to obtain the acoustic spectrum features, wherein the acoustic spectrum features are used to characterize at least one of the following: partial discharge features, corona discharge features, and mechanical anomaly features.

6. The method according to claim 1, characterized in that, Based on the multimodal sensor data, a multi-scale feature set is determined, including: Based on the attitude inertial sequence data and the position coordinate data, a multidimensional time-series signal is constructed, wherein the multidimensional time-series signal includes the three-dimensional position coordinate vector, three-axis attitude angle vector, three-axis acceleration vector and three-axis angular velocity vector of the UAV at multiple time steps; The vectors corresponding to the multiple time steps are sequentially input into a stacked long short-term memory network to obtain the hidden state vectors corresponding to the multiple time steps respectively, wherein the hidden state vectors represent the temporal feature information of the corresponding time step and all previous time steps; Perform average pooling on the hidden state vectors corresponding to the multiple time steps to obtain the global pose feature vector, and then use the global pose feature vector.

7. The method according to any one of claims 1 to 6, characterized in that, Based on the preliminary identification information and component status information of the multiple power component instances, an instance information fusion operation is performed to obtain the component instance information, which then includes: The multiple power component instances in the component instance information are respectively identified as nodes in the graph structure; Based on the geometric proximity between nodes, the connection relationships between nodes in the graph structure are determined, and the connection relationships are used as edges in the graph structure. Based on the nodes and edges, construct a spatial topology graph of the route.

8. A device for determining component instance information, characterized in that, include: The acquisition module is used to acquire multimodal sensor data, wherein the multimodal sensor data is collected by the sensor system carried by the UAV, and the multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; The first determining module is used to determine a multi-scale feature set based on the multimodal sensor data, wherein the multi-scale feature set includes multi-scale visual features, global geometric features, acoustic spectrum features, and global attitude feature vector. The second determining module is used to perform power component target detection and instance segmentation operations based on the multi-scale visual features to obtain preliminary identification information corresponding to multiple power component instances respectively. The corresponding preliminary identification information includes: the bounding box coordinates of the corresponding power component instance, the component category and the contour mask. The component category includes: conductor and tower. The third determining module is used to determine the component status information corresponding to the multiple power component instances based on the preliminary identification information of the multiple power component instances and the multi-scale feature set, wherein the corresponding component status information includes at least one of the following: defect status information, risk parameter status information; The fourth determining module is used to perform an instance information fusion operation based on the preliminary identification information and component status information of the multiple power component instances respectively, to obtain component instance information.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for determining component instance information as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method for determining component instance information as described in any one of claims 1 to 7.

Images