Method, device and electronic equipment for performing unmanned aerial vehicle inspection task

Abstract

The application discloses an unmanned aerial vehicle inspection task execution method and device and electronic equipment. The method comprises the following steps: receiving and responding to a predetermined inspection task, acquiring multi-modal sensor data; performing cross-modal semantic alignment and feature fusion operation according to the multi-modal sensor data to obtain a fusion feature vector; determining component instance information according to the fusion feature vector; generating a first flight control instruction of the unmanned aerial vehicle according to the component instance information, and controlling the unmanned aerial vehicle to fly according to the first flight control instruction; and constructing a line space topology graph according to visual imaging data, spatial depth point cloud data, attitude inertia sequence data, position coordinate data and component instance information. The application solves the technical problems of low tracking flight accuracy of the unmanned aerial vehicle, lagging update of line topology information and missing connection relationship in the related art.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method, apparatus, and electronic device for performing unmanned aerial vehicle (UAV) inspection tasks. Background Technology

[0002] 10kV overhead lines, as a crucial component of the distribution network, operate in complex environments with large spans and diverse component types. They are constantly exposed to external conditions such as wind, rain, icing, and vegetation growth, making them prone to hazards such as insulator damage, foreign object snagging on conductors, wire clamp overheating, and insufficient safety distances. Traditional inspection methods primarily rely on manual on-site surveys or drone inspections based on single visible light images, which have significant limitations. Firstly, multimodal sensor data exhibits significant differences in data type and feature distribution. Current technologies typically process each modality of data independently, lacking a unified embedding representation and cross-modal semantic alignment mechanism. This makes it difficult to effectively integrate complementary information acquired by different sensors, creating information silos and resulting in poor perception reliability in complex environments such as nighttime and fog. Secondly, sensing and recognition, flight control, and topology construction... Figure 3 The various links are isolated from each other, failing to form a technological closed loop. The accuracy of the drone's autonomous tracking flight is insufficient, and the acquisition of line topology relationships still relies on manual surveying and data entry. This results in topology information updates lagging far behind the actual operating status, poor accuracy of connection relationships, and difficulty in supporting the digital operation and maintenance and intelligent scheduling of the power distribution network.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for performing unmanned aerial vehicle (UAV) inspection tasks, in order to at least solve the technical problems of low UAV tracking accuracy and delayed updates of line topology information and missing connection relationships in related technologies.

[0005] According to one aspect of the present invention, a method for executing a drone inspection task is provided, comprising: receiving and responding to a predetermined inspection task; acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on the drone, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; performing cross-modal semantic alignment and feature fusion operations based on the multimodal sensor data to obtain a fused representation vector; determining component instance information based on the fused representation vector, wherein the component instance information includes the three-dimensional spatial position of a power component instance, component category, and contour mask, the component category including: conductor, tower; generating a first flight control command for the drone based on the component instance information, and controlling the drone to fly according to the first flight control command, wherein the first flight control command is used to drive the drone to fly along the extension direction of the conductor; and constructing a line spatial topology map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information.

[0006] Optionally, based on the multimodal sensor data, cross-modal semantic alignment and feature fusion operations are performed to obtain a fused representation vector, including: performing multimodal encoding operations based on the multimodal sensor data to obtain multimodal embedding vectors, wherein the multimodal embedding vectors include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors; inputting the multimodal embedding vectors into a modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space, wherein the modality alignment model is trained on an initial alignment model using sample data and includes a cross-modal contrastive alignment loss function, which is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times; and performing feature fusion operations based on the aligned embedding vectors to obtain a fused representation vector.

[0007] Optionally, determining component instance information based on the fused representation vector includes: determining a multi-scale feature set based on the fused representation vector, wherein the multi-scale feature set includes multi-scale visual features and a global pose feature vector; performing power component target detection and instance segmentation operations based on the multi-scale visual features to obtain preliminary identification information corresponding to multiple power component instances, wherein the corresponding preliminary identification information includes: bounding box coordinates, component category, confidence score, and pixel-level segmentation mask of the corresponding power component instance; determining component state information corresponding to the multiple power component instances based on the preliminary identification information of the multiple power component instances and the multi-scale feature set, wherein the corresponding component state information includes at least one of the following: defect state information and risk parameter state information; and performing instance information fusion operations based on the preliminary identification information and component state information of the multiple power component instances to obtain component instance information.

[0008] Optionally, based on the component instance information, generating a first flight control command for the UAV and controlling the UAV to fly according to the first flight control command includes: fitting and generating a three-dimensional centerline spatial trajectory of the conductor based on the three-dimensional spatial position of the conductor instance in the component instance information, and using the three-dimensional centerline spatial trajectory of the conductor as a reference path; determining deviation data corresponding to the UAV based on the reference path and the current pose of the UAV, wherein the deviation data includes lateral deviation, altitude deviation, and heading angle deviation, the lateral deviation being the horizontal and vertical distance from the center of mass of the UAV to the nearest conductor reference point, the altitude ... The preset vertical following altitude difference between the drone and the guide wire, and the heading angle deviation is the angle difference between the tangent direction of the guide wire at the reference point and the yaw angle of the drone; the deviation data is input to the target composite controller to generate a first flight control command, and control the drone to fly according to the first flight control command, wherein the first flight control command is composed of a proportional control component, an integral control component, a derivative control component and a feedforward control component based on the curvature of the guide wire, and the first flight control command is used to control the drone to maintain a preset safe lateral distance and an upper following altitude from the guide wire, and to fly along the spatial extension direction of the guide wire.

[0009] Optionally, based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information, a line space topology map is constructed, including: determining a global three-dimensional point cloud map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, and the position coordinate data, wherein the global three-dimensional point cloud map is obtained based on the global UAV pose, the global UAV pose is obtained based on a pose optimization function and an initial three-dimensional point cloud map, the initial three-dimensional point cloud map is obtained based on an error correction function, the pose optimization function is used to eliminate the cumulative drift error of the UAV pose, and the error correction function is used to correct the measurement error of the sensor; obtaining device nodes representing energy-saving equipment and three-dimensional vector edges representing conductors based on the global three-dimensional point cloud map and the component category and contour mask of the power component instances in the component instance information; and performing a topology connection operation based on the device nodes and the three-dimensional vector edges to obtain the line space topology map.

[0010] Optionally, after constructing the line spatial topology map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information, the method further includes: integrating the multiple types of data in the line spatial topology map into structured topology data, wherein the multiple types of data include node layer data, topology connection table, and inspection data layer, the node layer data includes the spatial location and attribute information of the equipment, the topology connection table is used to describe the connection relationship between nodes and edges, and the inspection data layer includes the attached defect status information, risk parameter status information, and inter-deep point cloud data; converting the structured topology data into a target data format; and sending the line spatial topology map in the target data format to the power grid management system so that the power grid management system updates the line topology data based on the line spatial topology map.

[0011] According to one aspect of the present invention, an execution device for a drone inspection task is provided, comprising: a receiving module, configured to receive and respond to a predetermined inspection task by acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on the drone and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; a first determining module, configured to perform cross-modal semantic alignment and feature fusion operations based on the multimodal sensor data to obtain a fused representation vector; a second determining module, configured to determine component instance information based on the fused representation vector, wherein the component instance information includes the three-dimensional spatial position of a power component instance, component category, and contour mask, the component category including: conductor, tower; a generating module, configured to generate a first flight control command for the drone based on the component instance information and control the drone to fly according to the first flight control command, wherein the first flight control command is used to drive the drone to fly along the extension direction of the conductor; and a constructing module, configured to construct a line spatial topology map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information.

[0012] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method for executing a drone inspection task as described in any of the preceding claims.

[0013] According to one aspect of the present invention, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the execution method of the unmanned aerial vehicle inspection task described in any of the preceding claims.

[0014] According to one aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0015] In this embodiment of the invention, a predetermined inspection task is received and responded to, and multimodal sensor data is acquired. This multimodal sensor data is collected by a sensor system mounted on a UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Based on the multimodal sensor data, cross-modal semantic alignment and feature fusion operations are performed to obtain a fused representation vector. Based on the fused representation vector, component instance information is determined, including the three-dimensional spatial location of the power component instance, the component category, and a contour mask. The component category includes: conductors and towers. Based on the component instance information, a first flight control command for the UAV is generated, and the UAV is controlled to fly according to the first flight control command. The first flight control command is used to drive the UAV to fly along the direction of the conductor extension. Based on the visual imaging data, spatial depth point cloud data, attitude inertial sequence data, position coordinate data, and component instance information, a line spatial topology map is constructed. By integrating multimodal perception fusion, autonomous tracking control, and topology mapping, this method achieves the goal of unifying environmental perception, flight control, and topology mapping into a closed-loop autonomous inspection technology. This is achieved by performing cross-modal semantic alignment and feature fusion on multimodal sensor data to obtain a fused representation vector, determining component instance information based on the fused representation vector, generating the first flight control command based on the component instance information to drive the UAV to fly along the conductor, and constructing a spatial topology map of the line based on multimodal sensor data and component instance information. This results in improving the accuracy of UAV autonomous tracking flight and the efficiency of line topology information updates, thereby solving the technical problems of low UAV tracking flight accuracy and delayed line topology information updates and missing connection relationships in related technologies. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of a method for executing a drone inspection task according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of the 10kV line spatial topology map construction system based on multimodal sensing and autonomous tracking, provided by an optional embodiment of the present invention.

[0019] Figure 3 This is a structural block diagram of the execution device for a drone inspection task according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example 1

[0023] According to an embodiment of the present invention, an embodiment of a method for executing a drone inspection task is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] Figure 1 This is a flowchart of a method for executing a drone inspection task according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0025] Step S102: Receive and respond to the predetermined inspection task, and acquire multimodal sensor data. The multimodal sensor data is collected by the sensor system carried by the UAV, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data.

[0026] Among them, the scheduled inspection task refers to the work instruction set in advance by the operation and maintenance personnel or the back-end system for the automated inspection of a specific 10kV overhead line section. It can trigger the drone to autonomously execute the entire process of inspection without the need for real-time manual control. For example, it can perform a full line inspection of a 10kV line from tower #001 to tower #050.

[0027] Among them, multimodal sensor data refers to the heterogeneous data set collected simultaneously by five different types of sensors carried by the UAV in the same inspection mission. It can provide power line inspection with multi-dimensional complementary panoramic perception information from appearance and acoustic features to three-dimensional spatial structure and motion posture, overcoming the limitation of insufficient perception capability of a single sensor in complex environments such as night, fog, and backlight.

[0028] Among them, visual imaging data refers to image data collected by high-definition visible light cameras and infrared thermal imaging cameras, which can clearly present the appearance, color, markings and temperature distribution information of power equipment.

[0029] Acoustic signal data refers to audio signals collected by acoustic imagers and ultrasonic sensors, which can capture abnormal acoustic features such as partial discharge, corona discharge, and mechanical loosening.

[0030] Among them, spatial depth point cloud data refers to a set of three-dimensional spatial points generated by lidar by emitting laser beams and receiving reflected echoes. It can accurately measure the distance between UAVs and power lines, towers, and vegetation, providing centimeter-level precision spatial information for flight control and three-dimensional mapping.

[0031] Among them, attitude inertial sequence data refers to the time series of three-axis acceleration and three-axis angular velocity of the UAV output in real time by the inertial measurement unit, which can provide prior information on the UAV's body motion.

[0032] Among them, location coordinate data refers to the global latitude, longitude and elevation coordinates of the UAV obtained by the GPS receiver, which can provide absolute geographical reference for the UAV.

[0033] In this step, the system receives and responds to the predetermined inspection task, acquiring multimodal sensor data collected by the sensor system onboard the UAV. This multimodal sensor data includes visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Through this step, the system obtains comprehensive and multi-type basic data required for the inspection, providing a complete data source for subsequent data processing, component identification, flight control, and topology map construction.

[0034] Step S104: Based on the multimodal sensor data, perform cross-modal semantic alignment and feature fusion operations to obtain a fused representation vector;

[0035] Cross-modal semantic alignment refers to the operation of semantic matching of data from different modalities, which can unify the semantic standards of data from different modalities, such as semantic matching of visual data and acoustic data.

[0036] Feature fusion refers to the operation of integrating features of multimodal data, which can aggregate effective features of multimodal data, such as features from visual and acoustic data.

[0037] Among them, the fusion representation vector refers to the vector generated after the fusion of multimodal data features, which can represent the comprehensive features of multimodal data, such as the vector after integrating features of multiple types of data.

[0038] In this step, cross-modal semantic alignment and feature fusion operations are performed based on multimodal sensor data to obtain a fused representation vector. This step achieves semantic unification and feature integration of different types of multimodal sensor data, eliminating differences between different modalities and generating a vector that comprehensively represents the data features, providing accurate feature basis for subsequent component instance information determination.

[0039] Step S106: Based on the fused representation vector, determine the component instance information, which includes the three-dimensional spatial location of the power component instance, the component category, and the contour mask. The component category includes: conductor and tower.

[0040] Among them, component instance information refers to the relevant attribute information of power component instances, which can clearly identify the core characteristics of power components, such as the attribute information of conductors and towers.

[0041] Among them, "electric component instance" refers to the actual individual electric component in the line, such as individual conductors or towers in the line.

[0042] Among them, three-dimensional spatial location refers to the three-dimensional spatial coordinate information of the power component instance, which can determine the spatial orientation of the power component, such as the three-dimensional spatial coordinates of the pole.

[0043] Among them, component category refers to the classification of power component instances, which can distinguish different power components, such as conductors and towers.

[0044] Among them, the contour mask refers to the contour mask of the power component instance, which can define the contour range of the power component, such as the contour mask of the conductor.

[0045] Among them, conductors refer to power transmission components, which are the core components of power lines, such as the conductors of 10kV overhead lines.

[0046] Among them, poles and towers refer to power line support components used to support conductors, such as the poles and towers of 10kV overhead lines.

[0047] In this step, based on the fused representation vector, the 3D spatial location, component category, and contour mask information of the power component instance are determined. The component category includes conductors and towers. Through this step, the core information of the power component is accurately identified based on the fused feature vector, and the location, type, and contour of conductors and towers are clearly defined, providing a precise target reference for subsequent UAV flight control.

[0048] Step S108: Based on the component instance information, generate the first flight control command for the UAV and control the UAV to fly according to the first flight control command, wherein the first flight control command is used to drive the UAV to fly along the extension direction of the wire;

[0049] The first flight control command refers to the control command that drives the drone to fly, which can control the drone's flight trajectory, such as the command that drives the drone to fly along a guide wire.

[0050] In this step, based on the component instance information, the first flight control command for the UAV is generated, and the UAV is controlled to fly along the direction of the guide wire according to the first flight control command. Through this step, flight commands matching the route direction are generated based on the identified component instance information, enabling the UAV to fly autonomously along the guide wire without human intervention in the flight trajectory, ensuring that the inspection path conforms to the actual route of the line.

[0051] Step S110: Based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, position coordinate data, and component instance information, a spatial topology map of the line is constructed.

[0052] Among them, the line spatial topology map refers to the topology map that represents the spatial structure of the line, which can show the spatial layout and connection relationship of the line, such as the spatial topology map of a 10kV overhead line.

[0053] In this step, a spatial topology map of the railway line is constructed based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, position coordinate data, and component instance information. This step integrates multiple types of basic data and component instance information to complete the topology map construction, intuitively presenting the spatial structure of the railway line and providing visualized topology data support for railway line operation and maintenance management.

[0054] Through the above steps S102-S110, the system receives and responds to the predetermined inspection task, acquiring multimodal sensor data. This multimodal sensor data is collected by the sensor system mounted on the UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Based on the multimodal sensor data, cross-modal semantic alignment and feature fusion operations are performed to obtain a fused representation vector. Based on the fused representation vector, component instance information is determined, including the three-dimensional spatial location of the power component instance, component category, and contour mask. Component categories include: conductors and towers. Based on the component instance information, a first flight control command for the UAV is generated, and the UAV is controlled to fly according to the first flight control command. This first flight control command drives the UAV to fly along the direction of the conductor extension. Based on the visual imaging data, spatial depth point cloud data, attitude inertial sequence data, position coordinate data, and component instance information, a line spatial topology map is constructed. By integrating multimodal perception fusion, autonomous tracking control, and topology mapping, this method achieves the goal of unifying environmental perception, flight control, and topology mapping into a closed-loop autonomous inspection technology. This is achieved by performing cross-modal semantic alignment and feature fusion on multimodal sensor data to obtain a fused representation vector, determining component instance information based on the fused representation vector, generating the first flight control command based on the component instance information to drive the UAV to fly along the conductor, and constructing a spatial topology map of the line based on multimodal sensor data and component instance information. This results in improving the accuracy of UAV autonomous tracking flight and the efficiency of line topology information updates, thereby solving the technical problems of low UAV tracking flight accuracy and delayed line topology information updates and missing connection relationships in related technologies.

[0055] As an optional embodiment, based on multimodal sensor data, cross-modal semantic alignment and feature fusion operations are performed to obtain a fused representation vector, including: performing multimodal encoding operations based on multimodal sensor data to obtain multimodal embedding vectors, wherein the multimodal embedding vectors include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors; inputting the multimodal embedding vectors into a modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space, wherein the modality alignment model is trained on an initial alignment model using sample data, including a cross-modal contrastive alignment loss function, which is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times; and performing feature fusion operations based on the aligned embedding vectors to obtain a fused representation vector.

[0056] Multimodal coding refers to the operation of encoding multimodal sensor data, which can convert different modal data into embedding vectors in a unified format, such as encoding visual imaging data to obtain visual embedding vectors.

[0057] Among them, multimodal embedding vectors refer to vectors generated after encoding multimodal data, which can represent the feature information of each modality of data, such as visual imaging embedding vectors and acoustic embedding vectors.

[0058] Among them, the visual imaging embedding vector refers to the vector after the visual imaging data is encoded, which can characterize the features of the visual imaging data, such as the vector after the line image is encoded.

[0059] Among them, the acoustic embedding vector refers to the vector after acoustic signal data is encoded, which can characterize the features of acoustic signal data, such as the vector after line acoustic signal is encoded.

[0060] Among them, the point cloud embedding vector refers to the vector after spatial depth point cloud data is encoded, which can represent the characteristics of spatial depth point cloud data, such as the vector after line point cloud data is encoded.

[0061] Among them, the inertial embedding vector refers to the vector after the attitude inertial sequence data is encoded, which can characterize the features of the attitude inertial sequence data, such as the vector after the attitude data of a UAV is encoded.

[0062] Among them, the position embedding vector refers to the vector after the position coordinate data is encoded, which can represent the characteristics of the position coordinate data, such as the vector after the position coordinates of a UAV are encoded.

[0063] Among them, the modality alignment model refers to the model used to achieve the alignment of multimodal embedding vectors, which can map different modality vectors to a unified space, such as a well-trained cross-modality alignment model.

[0064] Among them, semantic co-referential embedding space refers to a unified semantic embedding space that enables semantic alignment of vectors of different modalities, such as the semantic embedding space for multimodal data sharing.

[0065] Among them, the aligned embedding vector refers to the vector that has completed semantic alignment in a unified space, which can eliminate semantic differences between modalities, such as the vectors after alignment of each modality at the same time.

[0066] Among them, sample data pairs refer to paired data used to train the model, which can provide data support for model training, such as multimodal sample data pairs collected at the same time.

[0067] The initial alignment model refers to the untrained base alignment model, which is the initial carrier for model training, such as a cross-modal alignment model that has not been trained with samples.

[0068] Among them, the cross-modal contrastive alignment loss function refers to the loss function used for model training, which can optimize the modal alignment effect, such as the loss function used to maximize the modal similarity at the same time.

[0069] In this embodiment, multimodal encoding is first performed on the multimodal sensor data to obtain five types of multimodal embedding vectors. These vectors are then input into a modal alignment model trained from sample data and including a cross-modal contrastive alignment loss function to obtain aligned embedding vectors in the semantic coreference embedding space. Finally, feature fusion is performed based on the aligned embedding vectors to obtain fused representation vectors.

[0070] This approach first transforms heterogeneous multimodal data into unified-format embedding vectors through multimodal encoding, addressing the issue of inconsistent data formats across different modalities and the inability to directly fuse them. Then, it constrains model training through a cross-modal contrastive alignment loss function, maximizing the similarity of embedding vectors from different modalities at the same acquisition time and minimizing the similarity at different acquisition times. This achieves precise cross-modal semantic alignment, completely eliminating semantic barriers between modalities. Finally, it generates fused representation vectors through feature fusion, fully aggregating effective features from each modality, eliminating redundant information, and significantly improving the completeness, robustness, and representational ability of the features. This provides a solid feature foundation for the accurate determination of subsequent component instance information.

[0071] Optionally, based on the aligned embedding vector, a feature fusion operation is performed to obtain a fused representation vector, including: when the target task is a fine-grained perception task, acquiring multiple feature sequences, and performing interactive fusion between the multiple feature sequences through a cross-modal attention mechanism to obtain a fused feature sequence as the fused representation vector. The multiple feature sequences include visual feature sequences, point cloud feature sequences, and acoustic feature sequences. The visual feature sequences include spatial grid feature sequences obtained after feature extraction from visual imaging data; the point cloud feature sequences include local geometric feature sequences of key points in spatial depth point cloud data; and the acoustic feature sequences include acoustic signal data obtained after time-frequency transformation. The time-frequency block feature sequence, the fine-grained perception task includes at least one of the following: wire tracking task, flight obstacle avoidance task and defect detection and localization task; and / or, when the target task is a semantic reasoning task, multiple global embedding vectors are obtained, and the multiple global embedding vectors are weighted and summed through a gated multimodal fusion strategy to obtain a fused embedding vector as a fused representation vector, wherein the multiple global embedding vectors include global embedding vectors corresponding to visual imaging embedding vector, acoustic embedding vector, point cloud embedding vector, inertial embedding vector and position embedding vector respectively, wherein the semantic reasoning task includes at least one of the following: equipment status diagnosis task and flight path planning task.

[0072] Optionally, multiple feature sequences are acquired, and a cross-modal attention mechanism is used to perform interactive fusion between the multiple feature sequences to obtain a fused feature sequence as a fused representation vector. This includes: acquiring multiple feature sequences; determining attention weights corresponding to the multiple feature sequences respectively, wherein the corresponding attention weights represent the degree of correlation between the target feature position in the corresponding feature sequence and the predetermined feature position in other feature sequences; and performing interactive fusion operations based on the attention weights corresponding to the multiple feature sequences through a cross-modal attention mechanism to obtain a fused feature sequence, wherein the interactive fusion operations include weighted fusion of feature information at each feature position in the multiple feature sequences.

[0073] Optionally, multiple global embedding vectors are obtained, and a weighted sum of these vectors is performed using a gated multimodal fusion strategy to obtain a fused embedding vector as the fused representation vector. This includes: obtaining multiple global embedding vectors; performing vector concatenation on the multiple global embedding vectors to obtain a joint embedding vector; inputting the joint embedding vector into a gated network, performing a linear transformation on the joint embedding vector using the learning parameter matrix in the gated network, and then processing it with a normalized exponential function to obtain the gate weights corresponding to the multiple global embedding vectors; and obtaining the fused embedding vector based on the multiple global embedding vectors and the gate weights corresponding to them.

[0074] Optionally, based on the multimodal sensor data, a multimodal encoding operation is performed to obtain a multimodal embedding vector, including: encoding the visual imaging data using a deep separable convolutional neural network to obtain a visual imaging embedding vector; converting the acoustic signal data into a Mel spectrogram, and then encoding the Mel spectrogram using a convolutional neural network with progressively reduced channel counts to obtain an acoustic embedding vector; downsampling the spatial depth point cloud data to obtain downsampled point cloud data, and then encoding the downsampled point cloud data using a multilayer perceptron-based point cloud processing network to obtain a point cloud embedding vector; encoding the attitude inertial sequence data using a target neural network to obtain an inertial embedding vector, wherein the target neural network includes at least one of the following: a gated recurrent unit network, a dilated convolutional temporal network; and encoding the position coordinate data using a fully connected network to obtain a position embedding vector.

[0075] Optionally, before inputting the multimodal embedding vectors into the modality alignment model to obtain the aligned embedding vectors in the semantic coreference embedding space, the method further includes: acquiring sample data and a cross-modal contrastive alignment loss function, wherein the sample data includes positive sample data and negative sample data, the positive sample data includes any combination of sample embedding vectors of two different modalities at the same acquisition time, and the negative sample data includes any combination of sample embedding vectors of two different modalities at different acquisition times; training the initial alignment model, and using the cross-modal contrastive alignment loss function to maximize the similarity between positive sample data and minimize the similarity between negative sample data, until the target condition is reached to obtain the modality alignment model, wherein the target condition includes at least one of the following: reaching a predetermined number of iterations, and the value of the cross-modal contrastive alignment loss function being lower than a preset loss threshold.

[0076] Optionally, after inputting the multimodal embedding vectors into the modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space, the method further includes: obtaining the abnormal embedding vector of the target modality, wherein the target modality is the modality in the multimodal embedding vectors that detects anomalies, and the abnormal embedding vector is the feature vector in the target modality corresponding to the abnormal event; determining the similarity between the abnormal embedding vector and the embedding vectors of other modalities in the semantic coreference embedding space; determining the associated embedding vector that matches the abnormal embedding vector from the embedding vectors corresponding to other modalities based on the similarity; and outputting fault association data corresponding to other modalities based on the data acquisition time and spatial location corresponding to the associated embedding vector.

[0077] As an optional embodiment, determining component instance information based on the fused representation vector includes: determining a multi-scale feature set based on the fused representation vector, wherein the multi-scale feature set includes multi-scale visual features and a global pose feature vector; performing power component target detection and instance segmentation operations based on the multi-scale visual features to obtain preliminary identification information corresponding to multiple power component instances, wherein the corresponding preliminary identification information includes: bounding box coordinates, component category, confidence score, and pixel-level segmentation mask of the corresponding power component instance; determining component state information corresponding to multiple power component instances based on the preliminary identification information and the multi-scale feature set, wherein the corresponding component state information includes at least one of the following: defect state information and risk parameter state information; and performing instance information fusion operations based on the preliminary identification information and component state information of multiple power component instances to obtain component instance information.

[0078] Among them, the multi-scale feature set refers to the combination of features at different scales, which can comprehensively represent the multi-dimensional features of the line, such as the set of multi-scale visual features and global geometric features.

[0079] Among them, multi-scale visual features refer to visual features of different scales that can be adapted to the recognition of power components of different sizes, such as line visual features of different resolutions.

[0080] Global geometric features refer to the overall geometric features of the line, which can characterize the spatial geometric structure of the line, such as the overall geometric features of towers and conductors.

[0081] Among them, acoustic spectrum characteristics refer to the spectral characteristics of acoustic signals, which can characterize the acoustic anomalies of a line, such as the acoustic spectrum characteristics of line discharge.

[0082] Among them, the global attitude feature vector refers to the feature vector of the UAV's global attitude, which can characterize the UAV's flight attitude features, such as the vector of the UAV's overall flight attitude.

[0083] Among them, power component target detection refers to the operation of identifying the location and category of power components, which can locate power components, such as detecting the location of conductors and towers.

[0084] Among them, instance segmentation operation refers to the operation of pixel-level segmentation of power components, which can accurately divide the component outline, such as segmenting the pixel area of ​​the conductor.

[0085] Preliminary identification information refers to the basic identification information of power components, which can preliminarily determine the component attributes, such as the component's bounding box and category information.

[0086] Among them, the bounding box coordinates refer to the bounding box position coordinates of the power component, which can locate the approximate range of the component, such as the bounding box coordinates of the tower.

[0087] The confidence score refers to the credibility score of the component identification result, which can measure the accuracy of identification, such as the confidence score of component identification.

[0088] Among them, pixel-level segmentation mask refers to the pixel-level contour mask of a component, which can accurately define the boundary of the component, such as the pixel-level segmentation mask of a wire.

[0089] Among them, component status information refers to the operating status information of power components, which can reflect the health status of the components, such as component defects and risk parameter information.

[0090] Among them, defect status information refers to defect-related information of power components, which can clearly identify the defect status of the components, such as defect information of insulator spontaneous explosion.

[0091] Among them, risk parameter status information refers to the risk parameter information of power components, which can reflect the operational risks of the components, such as the risk parameter of conductor sag.

[0092] Among them, the instance information fusion operation refers to the operation of integrating preliminary identification information and status information, which can generate complete component instance information, such as the operation of merging component location, category, and status.

[0093] In this embodiment, a multi-scale feature set including multi-scale visual, global geometric, acoustic spectrum, and global pose features is first determined based on the fused representation vector. Then, power component target detection and instance segmentation are performed based on the multi-scale visual features to obtain preliminary identification information. Next, the component state information is determined by combining the preliminary identification information with the multi-scale feature set. Finally, instance information fusion is performed to obtain component instance information.

[0094] This approach employs a multi-scale feature set to cover line features of different scales and dimensions, adapting to the identification needs of power components of varying sizes and avoiding missed or false detections. Through target detection and instance segmentation, it simultaneously achieves the localization, classification, and pixel-level contour division of power components, obtaining high-precision preliminary identification information. Combining multi-modal features further mines the component's defect, risk, and other state information, achieving an upgrade from component identification to state diagnosis. Finally, through information fusion, the basic identification information and state information are integrated to generate complete component instance information containing location, category, contour, and state, providing comprehensive, accurate, and multi-dimensional data support for subsequent UAV flight control and line topology construction.

[0095] Optionally, based on the preliminary identification information and multi-scale feature sets of multiple power component instances respectively, the component state information corresponding to each of the multiple power component instances is determined, including: inputting the preliminary identification information and multi-scale feature sets of multiple power component instances respectively into a shared feature encoding model to obtain the component state information corresponding to each of the multiple power component instances. The shared feature encoding model includes an objective loss function, which is composed of a weighted sum of the classification loss of the defect classification task, the regression loss of the risk parameter regression task, and the segmentation loss of the instance segmentation task. The classification loss is used to measure the difference between the predicted category and the true defect category of the defect classification task, the regression loss is used to measure the difference between the predicted risk parameter and the true risk parameter of the risk parameter regression task, and the segmentation loss is used to measure the difference between the predicted contour mask and the true contour mask of the instance segmentation task.

[0096] Optionally, based on multimodal sensor data, a multi-scale feature set is determined, including: inputting visual imaging data into a feature extraction model to obtain multi-scale visual features. The feature extraction model includes a backbone network and a feature pyramid structure network. The backbone network includes any one of the following: a convolutional neural network, a visual transformer network. The feature pyramid structure network is used to determine multi-scale visual features based on the hierarchical features output by different levels of the backbone network. The backbone network includes at least a first level and a second level. The first level is used to extract detail information features, and the second level is used to extract semantic information features. The resolution corresponding to the first level is higher than the resolution corresponding to the second level.

[0097] Optionally, based on multimodal sensor data, a multi-scale feature set is determined, including: for each target point in the spatial depth point cloud data, determining a neighborhood point set centered on the target point; extracting local geometric features of the neighborhood point set through a mapping function based on the relative positional relationship between each neighborhood point and the target point; aggregating the local geometric features through an aggregation function to obtain the target point features corresponding to the target point; and obtaining global geometric features corresponding to the spatial depth point cloud data based on the target point features corresponding to each target point, wherein the global geometric features are used to characterize at least one of the following: the suspension shape of the conductor, the spatial configuration of the tower, and the safe distance between the conductor and surrounding objects.

[0098] Optionally, based on multimodal sensor data, a multi-scale feature set is determined, including: performing a Fourier transform operation on the acoustic signal data to convert the acoustic signal data from a time-domain signal to a time-frequency domain to obtain acoustic time-frequency domain features; inputting the acoustic time-frequency domain features into a convolutional neural network for encoding to obtain acoustic spectral features, wherein the acoustic spectral features are used to characterize at least one of the following: partial discharge features, corona discharge features, and mechanical anomaly features.

[0099] Optionally, based on multimodal sensor data, a multi-scale feature set is determined, including: constructing a multi-dimensional temporal signal based on attitude inertial sequence data and position coordinate data, wherein the multi-dimensional temporal signal includes the three-dimensional position coordinate vector, three-axis attitude angle vector, three-axis acceleration vector, and three-axis angular velocity vector of the UAV at multiple time steps; sequentially inputting the vectors corresponding to multiple time steps into a stacked long short-term memory network to obtain hidden state vectors corresponding to multiple time steps, wherein the hidden state vectors represent the temporal feature information of the corresponding time step and all previous time steps; performing an average pooling operation on the hidden state vectors corresponding to multiple time steps to obtain a global attitude feature vector, and then...

[0100] As an optional embodiment, based on component instance information, a first flight control command for the UAV is generated, and the UAV is controlled to fly according to the first flight control command. This includes: fitting and generating a three-dimensional centerline spatial trajectory of the conductor based on the three-dimensional spatial position of the conductor instance in the component instance information, and using the three-dimensional centerline spatial trajectory of the conductor as a reference path; determining the deviation data corresponding to the UAV based on the reference path and the current pose of the UAV, wherein the deviation data includes lateral deviation, altitude deviation, and heading angle deviation, wherein the lateral deviation is the horizontal and vertical distance from the center of mass of the UAV to the nearest conductor reference point, the altitude deviation is the preset vertical following altitude difference between the UAV and the conductor, and the heading angle deviation is the angle difference between the tangent direction of the conductor at the reference point and the yaw angle of the UAV; inputting the deviation data to the target composite controller to generate the first flight control command, and controlling the UAV to fly according to the first flight control command, wherein the first flight control command consists of a proportional control component, an integral control component, a derivative control component, and a feedforward control component based on the curvature of the conductor, and the first flight control command is used to control the UAV to maintain a preset safe lateral distance and an upward following altitude from the conductor, and to fly along the spatial extension direction of the conductor.

[0101] Among them, the three-dimensional centerline spatial trajectory of the conductor refers to the three-dimensional spatial trajectory of the conductor center generated by fitting, which can characterize the extension direction of the conductor, such as the three-dimensional spatial curve trajectory of the conductor center.

[0102] The reference path refers to the reference path for the drone's flight, which can provide directional guidance for the drone's flight, such as the three-dimensional centerline spatial trajectory of a guide.

[0103] Among them, the current pose of the drone refers to the current position and attitude information of the drone, which can determine the real-time status of the drone, such as the drone's real-time position and flight attitude.

[0104] Among them, deviation data refers to the deviation information between the UAV and the reference path, which can reflect the flight deviation, such as lateral, altitude, and heading angle deviations.

[0105] Among them, lateral deviation refers to the horizontal and vertical distance from the center of mass of the UAV to the reference point of the guide wire, which can measure the degree of horizontal deviation of the UAV, such as the horizontal distance deviation between the UAV and the guide wire.

[0106] Among them, altitude deviation refers to the difference between the preset vertical following altitude of the drone and the guide wire, which can measure the degree of altitude deviation of the drone, such as the vertical height difference between the drone and the guide wire.

[0107] Among them, the heading angle deviation, which is the angle difference between the tangent direction of the guide line and the yaw angle of the UAV, can measure the degree of heading deviation of the UAV, such as the angle difference between the direction of the guide line and the flight direction of the UAV.

[0108] Among them, the target composite controller refers to the controller that generates flight control commands and can output precise flight control commands, such as a composite controller that includes proportional, integral, derivative and feedforward functions.

[0109] The proportional control component refers to the proportional adjustment part of the control command, which can quickly respond to changes in deviation, such as the proportional control part in flight commands.

[0110] The integral control component refers to the integral adjustment part of the control command, which can eliminate static deviations, such as the integral control part in flight commands.

[0111] Among them, the differential control component refers to the differential adjustment part of the control command, which can suppress sudden deviations, such as the differential control part in flight commands.

[0112] Among them, the feedforward control component refers to the feedforward adjustment part based on the curvature of the conductor, which can predict changes in the direction of the line in advance, such as the feedforward control part generated based on the curvature of the conductor.

[0113] Among them, the safe lateral distance refers to the preset horizontal safe distance between the drone and the guide wire, which can ensure flight safety, such as the horizontal safe distance maintained between the drone and the guide wire.

[0114] The "following height" refers to the preset vertical following height between the drone and the guide wire, which ensures the inspection view. For example, the drone is set to follow the guide wire at a preset height above the guide wire.

[0115] In this embodiment, the three-dimensional centerline spatial trajectory of the conductor is first fitted based on the three-dimensional spatial position of the conductor instance as a reference path. Then, the lateral, altitude, and heading angle deviation data are determined in combination with the current pose of the UAV. The deviation data is input into the target composite controller to generate a first flight control command composed of proportional, integral, derivative, and feedforward control components. This command controls the UAV to maintain a preset safe lateral distance and an upward following altitude while flying along the direction of the conductor extension.

[0116] This method uses the three-dimensional centerline spatial trajectory of the conductor as a reference path, ensuring that the drone's flight trajectory perfectly matches the actual route of the line, thus guaranteeing the accuracy of the inspection path. By quantitatively calculating three types of deviation data—lateral, altitude, and heading angle—the deviation of the drone from the reference path is accurately located, providing a precise basis for flight control adjustments. A composite controller incorporating proportional, integral, derivative, and feedforward components is employed. The proportional component quickly corrects real-time deviations, the integral component eliminates static cumulative deviations, the derivative component suppresses abrupt deviation changes, and the feedforward component adapts to changes in conductor curvature in advance, achieving smooth, stable, and precise flight control. At the same time, the drone is strictly controlled to maintain a preset safe lateral distance and overhead following altitude, eliminating the safety risk of collisions between the drone and the line while ensuring the best perspective and effect for inspection data acquisition.

[0117] Optionally, after controlling the UAV to fly according to the first flight control command, the process further includes: determining whether the distance between the UAV and the target tower instance is less than a preset safe distance threshold based on component instance information, and obtaining a first determination result; if the first determination result indicates that the distance is less than the preset safe distance threshold, triggering a tower processing mode; in the tower processing mode, determining the coordinates of the tower center point of the target tower instance, and constructing a local occupancy grid map of the target tower instance based on spatial depth point cloud data; using a target trajectory algorithm, taking the tower center point coordinates as the trajectory reference point and the target variable as the optimization objective, generating a tower flight path, wherein the target variable includes trajectory length, trajectory execution time, and the repulsive potential field between the trajectory point and the obstacle, and the obstacle is represented by the occupancy grid in the local occupancy grid map; determining a second flight control command based on the tower flight path, and controlling the UAV to fly according to the tower flight path.

[0118] Optionally, after controlling the UAV to fly according to the first flight control command, the process further includes: determining a second determination result based on component instance information to see if a line branch point has been identified; triggering a branch decision mode if the second determination result indicates that a line branch point has been identified; in the branch decision mode, determining a target branch line to be tracked from multiple candidate branch lines, wherein the method of determining the target branch line includes: determining the target branch line according to a preset tracking rule, wherein the preset tracking rule includes one of the following: selecting the candidate branch line with the largest cross-section, selecting the candidate branch line with the smallest deviation from the current heading angle of the UAV, selecting the candidate branch line that matches the preset mission target; and / or, inputting the conductor feature data of multiple candidate branch lines into a decision model to determine the target branch line, wherein the decision model is a model trained based on historical inspection data; determining a third flight control command based on the target branch line, and controlling the UAV to fly according to the third flight control command.

[0119] Optionally, after controlling the UAV to fly according to the first flight control command, the method further includes: determining the target obstacle on the flight path; determining the target correction path corresponding to the target obstacle; determining the correction flight control command according to the target correction path, and controlling the UAV to fly according to the correction flight control command, so as to control the UAV to bypass the target obstacle.

[0120] Optionally, determining the target correction path corresponding to the target obstacle includes: determining the target correction path for the static obstacle when the target obstacle includes a static obstacle, and / or determining the obstacle prediction trajectory corresponding to the dynamic obstacle when the target obstacle includes a dynamic obstacle; determining the minimum predicted distance and collision time between the UAV and the dynamic obstacle based on the obstacle prediction trajectory; determining the collision risk probability based on the minimum predicted distance and collision time; and generating a target correction path around the dynamic obstacle using a target path planning algorithm when the collision risk probability exceeds a preset risk threshold. The target path planning algorithm includes a target artificial potential field function, which includes a gravitational potential field term, a static obstacle repulsion potential field term, a dynamic obstacle repulsion potential field term, and a path-keeping term. The gravitational potential field term guides the UAV to move along the original path, the static obstacle repulsion potential field term maintains a safe distance between the UAV and the static obstacle, the dynamic obstacle repulsion potential field term enables the UAV to avoid the dynamic obstacle based on the relative speed and collision time, and the path-keeping term constrains the deviation of the correction path from the original path.

[0121] Optionally, after determining the third flight control command based on the target branch line and controlling the UAV to fly according to the third flight control command, the method further includes: generating traceable point information corresponding to the branch line point, wherein the traceable point information includes the branch point coordinate data of the branch line point; storing the traceable point information into the traceable point stack so that after the mission is interrupted, the traceable point information can be retrieved from the traceable point stack and the mission continuation operation can be performed based on the traceable point information.

[0122] Optionally, after controlling the UAV to fly according to the first flight control command, the method further includes: when a mission interruption occurs, performing positioning based on the UAV's position coordinate data and attitude inertial sequence data to obtain interruption positioning data; controlling the UAV to fly to the mission interruption area based on the interruption positioning data; acquiring the current environmental image corresponding to the mission interruption area, matching the current environmental image with the image in the storage area to obtain target positioning data; determining the target backtracking point from multiple backtracking points included in the backtracking point stack based on the target positioning data; and controlling the UAV to fly to the target backtracking point.

[0123] As an optional embodiment, a line space topology map is constructed based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, position coordinate data, and component instance information. This includes: determining a global 3D point cloud map based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. The global 3D point cloud map is obtained based on the global UAV pose, which is obtained based on a pose optimization function and an initial 3D point cloud map. The initial 3D point cloud map is obtained based on an error correction function. The pose optimization function is used to eliminate the cumulative drift error of the UAV pose, and the error correction function is used to correct the measurement error of the sensor. Based on the global 3D point cloud map and the component category and contour mask of the power component instance in the component instance information, device nodes representing energy-saving equipment and 3D vector edges representing conductors are obtained. Based on the device nodes and 3D vector edges, a topology connection operation is performed to obtain the line space topology map.

[0124] Among them, the global 3D point cloud map refers to the 3D point cloud map of the entire route area, which can completely represent the 3D spatial structure of the route, such as the 3D point cloud map of the entire route corridor.

[0125] Among them, global UAV pose refers to the position and attitude information of the UAV across the entire domain, which can determine the global spatial state of the UAV, such as the pose information of the UAV during inspection of the entire domain.

[0126] Among them, the pose optimization function refers to the function that optimizes the pose of the UAV and can eliminate the cumulative pose drift error, such as the function used to correct the pose drift of the UAV.

[0127] The initial 3D point cloud map refers to the preliminary generated 3D point cloud map, which is the basis for building the global map, such as the initial point cloud map without error correction.

[0128] Among them, the error correction function refers to the function that corrects the measurement error of the sensor, which can improve the accuracy of the data, such as the function that corrects the sensor acquisition error.

[0129] Among them, cumulative drift error refers to the offset error generated by the cumulative position of the UAV, which will reduce the positioning accuracy, such as the position drift error accumulated during the flight of the UAV.

[0130] Measurement error refers to the error in the data collected by the sensor, which affects the accuracy of the data. For example, the error in the sensor's position and attitude data.

[0131] Among them, equipment nodes refer to topology nodes that represent power equipment and can represent the position of power equipment in the topology graph, such as the topology node corresponding to a pole or tower.

[0132] Among them, the three-dimensional vector edge refers to the three-dimensional vector line segment that represents the conductor and can represent the spatial direction of the conductor, such as the three-dimensional vector edge corresponding to the conductor.

[0133] Among them, the topology connection operation refers to the operation of connecting device nodes with three-dimensional vector edges, which can construct line topology relationships, such as the operation of connecting tower nodes with conductor vector edges.

[0134] In this embodiment, a global 3D point cloud map is first determined based on multiple types of data (the global UAV pose is optimized by a pose optimization function to eliminate cumulative drift error, and the initial 3D point cloud map is corrected by an error correction function to correct sensor measurement error). Then, the device nodes and 3D vector edges are obtained by combining component instance information. Finally, a topology connection operation is performed to obtain a line space topology map.

[0135] This approach utilizes an error correction function to correct the native measurement errors of sensors, while a pose optimization function eliminates the cumulative pose drift error generated during UAV flight. This fundamentally improves the accuracy of the global 3D point cloud map and UAV positioning, solving the error problem in 3D modeling and spatial positioning. Power equipment such as poles and towers are abstracted as equipment nodes, and conductors are abstracted as 3D vector edges, simplifying the topological representation of complex physical lines and reducing the complexity of topology construction. Through topological relationship connection operations, the spatial connection logic of equipment and conductors is accurately restored. The generated line spatial topology map has both centimeter-level geometric accuracy and correct topological relationships, providing a high-precision spatial data foundation for digital line management.

[0136] Optionally, based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data, a global 3D point cloud map and a global UAV pose are determined, including: determining an initial 3D point cloud map and a corrected UAV pose based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; and performing a global pose optimization operation based on the initial 3D point cloud map and the corrected UAV pose to determine the global 3D point cloud map and the global UAV pose, wherein the global pose optimization operation is used to eliminate accumulated drift errors.

[0137] Optionally, based on the initial 3D point cloud map and the corrected UAV pose, a global pose optimization operation is performed to determine the global 3D point cloud map and the global UAV pose. This includes: comparing the similarity between the visual imaging data and historical keyframes to obtain a comparison result, where the historical keyframes are visual imaging frames collected and stored by the UAV in the explored area; if the comparison result indicates that the similarity is higher than a preset similarity threshold, a closed loop is identified, where a closed loop indicates that the current spatial position of the UAV is the same as the historical spatial position corresponding to the historical keyframe; if a closed loop is identified, a global pose optimization operation is performed based on the initial 3D point cloud map and the corrected UAV pose to determine the global 3D point cloud map and the global UAV pose.

[0138] Optionally, based on the initial 3D point cloud map and the corrected UAV pose, a global pose optimization operation is performed to determine the global 3D point cloud map and the global UAV pose. This includes: constructing a pose graph using the UAV poses corresponding to multiple time steps in the corrected UAV pose as pose nodes and the relative pose constraints between multiple pose nodes as edges; invoking a pose optimization function, wherein the pose optimization function aims to minimize the weighted sum of squared errors of the relative pose constraints between pose nodes; obtaining the global UAV pose based on the poses corresponding to multiple pose nodes in the pose graph and the pose optimization function; and updating the initial 3D point cloud map based on the global UAV pose to obtain the global 3D point cloud map.

[0139] Optionally, based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data, an initial 3D point cloud map and a corrected UAV pose are determined, including: constructing an initial state vector based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data, wherein the initial state vector includes the initial UAV pose state quantities corresponding to multiple time steps and the initial 3D coordinates of multiple map points, where the multiple map points represent points constituting the environmental surface in the 3D point cloud map; retrieving an error correction function, wherein the error correction function includes multiple error terms and corresponding weight coefficients for each of the multiple error terms, the multiple error terms including visual error terms, lidar error terms, inertial measurement unit error terms, and positioning error terms, and the weight coefficients corresponding to each of the multiple error terms are negatively correlated with the measurement uncertainty of the corresponding sensor; obtaining a target state vector based on the error correction function and the initial state vector; determining the corrected UAV pose based on the UAV pose state quantities corresponding to multiple time steps in the target state vector, and determining the initial 3D point cloud map based on the target 3D coordinates of the multiple map points in the target state vector.

[0140] Optionally, based on the global 3D point cloud map and the component category and contour mask of the power component instance in the component instance information, the device nodes representing the energy-saving equipment and the 3D vector edges representing the conductors are obtained, including: performing point cloud clustering operations based on spatial distance and geometric features on multiple map points in the global 3D point cloud map to obtain multiple device point cloud clusters, wherein a device point cloud cluster represents a set of map points belonging to the same device instance; determining the device type corresponding to each of the multiple device point cloud clusters; performing a parametric model fitting operation on the multiple device point cloud clusters to obtain the device pose information of multiple device instances, wherein the device pose information includes 3D spatial coordinates and device posture, and the parametric model fitting operation is to fit the device point cloud clusters into a parametric 3D model corresponding to the device type to determine the 3D spatial coordinates and device posture of the device instance; determining the multiple device instances as nodes representing the energy-saving equipment, and assigning the device attribute information of the multiple device instances to the corresponding nodes, wherein the device attribute information includes device type and device pose information.

[0141] Optionally, based on the global 3D point cloud map and the component category and contour mask of the power component instance in the component instance information, the node attribute information representing the energy-saving device and the 3D vector edge representing the conductor are obtained, including: determining the conductor point set based on the contour mask of the conductor instance in the component instance information and the global 3D point cloud map, wherein the conductor point set is the set of map points belonging to the conductor in the global 3D point cloud map; performing a 3D space curve fitting operation on the conductor point set to obtain the continuous 3D vector line corresponding to the conductor, wherein the continuous 3D vector line represents the continuous trajectory of the conductor in 3D space, and the 3D space curve fitting operation includes at least one of the following: catenary equation, polynomial fitting algorithm; and determining the 3D vector lines corresponding to multiple conductors as the 3D vector edges representing the conductor.

[0142] As an optional embodiment, after constructing the line spatial topology map based on visual imaging data, spatial depth point cloud data, attitude inertial sequence data, position coordinate data, and component instance information, the method further includes: integrating the various types of data in the line spatial topology map into structured topology data, wherein the various types of data include node layer data, topology connection table, and inspection data layer. The node layer data includes the spatial location and attribute information of the equipment, the topology connection table is used to describe the connection relationship between nodes and edges, and the inspection data layer includes the attached defect status information, risk parameter status information, and inter-node depth point cloud data; converting the structured topology data into a target data format; and sending the line spatial topology map in the target data format to the power grid management system so that the power grid management system can update the line topology data according to the line spatial topology map.

[0143] Structured topology data refers to standardized topology data that has been integrated, which can unify the data format. For example, structured data that integrates node, connection, and inspection data.

[0144] Among them, node layer data refers to the relevant data of device nodes in the topology diagram, which can represent the basic information of the device, such as the spatial location and attribute data of the device.

[0145] Among them, attribute information refers to the attribute-related information of power equipment, which can clearly identify the equipment attributes, such as the equipment model and category attribute information.

[0146] The inspection data layer refers to the inspection-related data mounted on the topology graph, which can carry inspection result information, such as inspection data of defects, risks, and point clouds.

[0147] The target data format refers to a standardized data format that conforms to standards and can adapt to system interface requirements, such as the data format of the power industry standards.

[0148] Among them, the power grid management system refers to the system platform used to manage the power grid, which can realize the management of line topology data, such as the power grid operation and maintenance management system of a power company.

[0149] Among them, line topology data refers to line topology-related data in the power grid management system, which can support power grid operation and maintenance management, such as line spatial topology data in the power grid system.

[0150] In this embodiment, after constructing the line spatial topology map, the node layer data, topology connection table, and inspection data layer are integrated into structured topology data, converted into the target data format, and sent to the power grid management system for updating the line topology data.

[0151] This method integrates scattered, multi-type data from the topology map into structured topology data, achieving data standardization and systematic organization, improving data readability, callability, and management efficiency; it converts the data into a standardized target data format, perfectly adapting to the data interface specifications of the power grid management system, achieving seamless data connection and transmission; and it synchronizes accurate line spatial topology data to the power grid management system in real time, completing the dynamic update of line topology data, completely solving the core problems of lagging updates and inconsistencies with actual lines in traditional power grid topology data, providing real-time, accurate, and reliable topology data support for power grid operation and maintenance scheduling, fault location, power restoration, and other businesses.

[0152] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0153] In related technologies, traditional distribution network topology identification mainly relies on manual on-site surveys and comparison with drawings. The core problem is that topology information updates lag significantly behind actual operating conditions. Especially after line switching, load transfer, or network reconfiguration, existing Geographic Information Systems (GIS) often fail to automatically synchronize topology relationships, leading to inconsistencies between graphical and physical connections. This static and isolated management approach has significant drawbacks: First, manual maintenance is prone to subjective errors and is inefficient, causing the reliability of the system topology to continuously deteriorate with frequent operations. Second, topology errors directly cause advanced application algorithms such as power flow calculation, fault location, and power restoration to fail, severely impacting the accuracy of operation and maintenance and the reliability of power supply. Furthermore, existing systems lack real-time data fusion mechanisms and cannot correct topology deviations through dynamic verification with systems such as SCADA and electricity consumption information collection, creating information silos. Therefore, there is an urgent need for a technical solution that can adaptively track network structure changes and achieve dynamic topology verification and automatic error correction to improve the overall perception and operation and maintenance management capabilities of the distribution network.

[0154] In view of this, an optional embodiment of the present invention provides a method for performing UAV inspection tasks, which can also be called a method for UAVs to automatically identify and track the route of 10kV overhead lines through multiple sensors. This method can achieve high-precision, fully autonomous line tracking and map construction. Figure 2 This is a flowchart of the 10kV line spatial topology map construction system based on multimodal sensing and autonomous tracking, provided by an optional embodiment of the present invention. Figure 2 As shown, the system includes: Multimodal data synchronous acquisition: At the system's starting point, multiple sensors on the UAV (high-definition camera, infrared, lidar, acoustic, IMU / GPS) synchronously acquire comprehensive data of the line corridor. Perception and fusion center: Feature extraction and intelligent recognition: Identifying equipment such as conductors, towers, and insulators, and their status from raw data. Multimodal information fusion and unified embedding: Aligning and fusing data from different sources in a unified semantic space to form a comprehensive environmental understanding. Decision and execution loop: Autonomous tracking and decision control: Generating decisions such as tracking, obstacle avoidance, and branch selection based on fused information. Flight control commands and UAV pose and status: Decisions are translated into specific actions; the UAV executes flight and provides real-time feedback on its pose, forming a rapid closed loop of "perception-decision-action". Map construction and output: SLAM and 3D environment reconstruction: Constructing an accurate 3D point cloud map using pose and fused features. Topological relationship reasoning and vectorization: Extracting equipment (nodes) and conductors (edges) from the 3D map and automatically connecting them to generate a structured topology map. Final application: The generated 10kV line spatial topology map is directly output to the GIS system and digital operation and maintenance platform for intelligent management and operation and maintenance of the power grid.

[0155] The technical solution of the present invention is as follows:

[0156] Step 1: Sensor System Configuration and Data Acquisition

[0157] The drone is equipped with a multimodal sensor system, including:

[0158] Optical and thermal imaging sensing unit: Includes a high-definition visible light camera, an infrared thermal imaging camera, and an acoustic imager, used to capture visible light and thermal feature images of lines, towers, insulators, transformers, and circuit breakers, providing a data foundation for feature recognition and extraction.

[0159] Acoustic and ultrasonic sensing unit: includes an acoustic imager (for detecting abnormal sound sources such as corona discharge) and an ultrasonic sensor to assist in the detection of nearby obstacles and the location of non-visible faults.

[0160] Spatial Perception and Positioning Unit: This unit includes a lidar unit, a GPS receiver, and an inertial measurement unit (IMU). The lidar unit is used for high-precision 3D environmental scanning and ranging; the GPS unit provides the UAV's global latitude and longitude coordinates; and the IMU unit is used to measure the UAV's three-axis acceleration and three-axis angular velocity in real time, providing high-frequency attitude data.

[0161] Step 2: Line identification and autonomous tracking based on multimodal information fusion:

[0162] 2.1 Lightweight Multimodal Embedding Model:

[0163] In resource-constrained scenarios such as drones, embedded devices, and edge computing, this invention constructs a lightweight multimodal embedding model specifically designed for edge computing to achieve deeper levels of perception and reasoning.

[0164] Specifically, the system aligns and fuses the six modal data collected:

[0165] Visual image: from a high-definition camera.

[0166] Thermal imaging: from an infrared camera.

[0167] The aforementioned visual images and thermal images are the same as the aforementioned visual imaging data.

[0168] Acoustic signals (same as the acoustic signal data above): from acoustic imagers and ultrasonic testing instruments (which can be regarded as a special audio stream).

[0169] Spatial depth / point cloud (same as the spatial depth point cloud data above): from LiDAR.

[0170] Attitude inertial sequence (same as the attitude inertial sequence data above): from IMU.

[0171] Location coordinates (same as the location coordinate data above): from GPS.

[0172] (1) Cross-modal semantic alignment based on contrastive learning:

[0173] To achieve semantic consistency and interchangeability of data from different modalities, a contrastive learning framework based on noise contrastive estimation is adopted.

[0174] a) Objective function and training paradigm:

[0175] Its core principle is to maximize the mutual information between positive sample pairs within the same scene context, while minimizing the similarity between irrelevant negative sample pairs. The Information Noise Contrast Estimation (InfoNCE) loss function is used as the loss function.

[0176]

[0177] Where vi and ti represent the embedding vectors of any two modalities (such as image-infrared, image-audio, audio-point cloud) from the same time (same scene), forming a positive sample pair.

[0178] tj represents the modal embeddings of other scenarios in the batch, which are used as negative samples.

[0179] s( ) is a similarity function (usually cosine similarity).

[0180] τ is a temperature hyperparameter used to control the sharpness of the probability distribution.

[0181] N is the batch size.

[0182] b) Specific implementation of alignment:

[0183] During training, a batch containing synchronized data from all sensors is constructed. For each data sample, the embedding vectors of any two modalities are mutually positive samples. For example:

[0184] High-resolution images of the conductor at the same time, infrared thermal images of the conductor, recorded ambient sounds, lidar point clouds, and IMU attitude data are all brought together in the embedded space.

[0185] The traverse image is then pushed away from unrelated ambient sound or point cloud data from another moment.

[0186] In this way, the model can learn the intrinsic relationships between conductor heating, corona discharge sound, and specific geometric structures without explicit category labels, forming a semantic common-reference space. In this space, different modal data describing the same electrical equipment or state will have highly similar embedding vectors.

[0187] Similar to the above, multimodal encoding is performed based on multimodal sensor data to obtain multimodal embedding vectors. These multimodal embedding vectors include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors. The multimodal embedding vectors are then input into a modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space. This model is trained from sample data using an initial alignment model and includes a cross-modal contrastive alignment loss function. This loss function maximizes the similarity between different modal embedding vectors at the same acquisition time and minimizes the similarity between different modal embedding vectors at different acquisition times. Based on the aligned embedding vectors, a feature fusion operation is performed to obtain a fused representation vector.

[0188] (2) Hierarchical feature fusion mechanism:

[0189] After obtaining the aligned embedding vectors, a hierarchical fusion mechanism is used to generate the final scene representation to adapt to downstream tasks of varying complexity.

[0190] a) Early interaction and integration:

[0191] For tasks requiring fine spatial awareness (such as obstacle avoidance and wire tracking), a lightweight cross-modal attention mechanism is employed to enable early interaction:

[0192] F fusion =Transformer_Block(E v E l E a );

[0193] Among them, E v E l E a These represent the embedding sequences for vision, LiDAR, and audio, respectively. This mechanism allows a point in point cloud data to focus on its corresponding pixel region in an image, or allows abnormal sounds to focus on hot spots in thermal imaging, achieving fine-grained fusion at the pixel / point level.

[0194] b) Late-stage semantic fusion:

[0195] For high-level semantic tasks (such as device status diagnosis and path planning), the global embedding vectors extracted from each modality by the encoder are directly fused. Specifically, a gated multimodal fusion strategy is adopted:

[0196]

[0197] in:

[0198] e Represents the global embedding vector for each modality. This is the embedding vector corresponding to the visible light image. This is the embedding vector corresponding to the infrared thermal imaging image. The embedding vector corresponding to the acoustic signal data. For learnable parameters, This is the weighted global embedding vector.

[0199] || indicates vector concatenation.

[0200] σ is the Sigmoid activation function.

[0201] g is a learned gating vector used to adaptively weight the importance of each modality. For example, at night or on foggy days, the weight of the visual modality is reduced, while the weights of the infrared and lidar modalities are automatically increased.

[0202] (3) Zero-sample transfer and cross-modal retrieval:

[0203] The alignment and fusion methods described above endow the system with powerful zero-shot inference capabilities. Once the model establishes robust cross-modal associations in the embedding space, it can achieve inter-modal transfer without specialized training.

[0204] Text-guided retrieval: Maintenance personnel input the text query "10kV tension pole," and the text encoder maps the query to a unified embedding space. The system then retrieves the visual, infrared, or audio embeddings most similar to the query vector within this space and locates the target.

[0205] Audio-to-image fault localization: When the acoustic sensor detects an audio pattern of "corona discharge", even if the model is not directly trained using "corona-image" pairing data, it can automatically retrieve and highlight the most likely normally operating non-fully insulated equipment, such as circuit breakers, in high-definition or infrared images by finding the nearest neighbor in the shared embedding space.

[0206] Cross-modal semantic alignment is achieved through contrastive learning, and robust scene representations are generated through a hierarchical fusion mechanism. This not only solves the problem of unified processing of heterogeneous data, but also endows the system with general cross-modal understanding and zero-shot transfer capabilities similar to ImageBind. Furthermore, a carefully designed lightweight network and training strategy ensure its feasibility for deployment on UAV edge computing platforms. This lays the core technical foundation for building a fully autonomous and highly intelligent automatic tracking and topology mapping system for 10kV overhead lines.

[0207] 2.2 Feature Extraction and Intelligent Recognition:

[0208] This stage is the core of the entire system's perception. Its task is to extract discriminative low-level and high-level features from multimodal raw data and use advanced intelligent algorithms to achieve accurate identification and understanding of 10kV overhead line components and their status. This process is a hierarchical processing pipeline that progresses from coarse to fine and from local to global.

[0209] (1) Multimodal feature encoding and hierarchical extraction:

[0210] a) Visual and infrared modal feature extraction:

[0211] Backbone Network and Feature Pyramid: For high-definition visible light and infrared thermal imaging video streams, a backbone network (such as ResNet, EfficientNet, or SwinTransformer) based on deep convolutional neural networks (CNNs) or visual transformers (ViTs) is used for feature encoding. To simultaneously handle targets of different scales (such as a wire crossing the screen and an insulator in the distance), a feature pyramid network is introduced to extract multi-scale feature maps at different levels of the backbone network. Lower-level feature maps have high resolution and are rich in detailed information, which is beneficial for accurate localization; higher-level feature maps have a large receptive field and contain rich semantic information, which is beneficial for target classification.

[0212] Encoding the physical meaning of infrared features: For infrared thermal imaging data, in addition to general visual features, additional temperature statistical features (such as regional maximum temperature, average temperature, and temperature variance) and hot spot morphological features are extracted. These features have a direct physical correlation with equipment failure modes such as overload, poor contact, and internal defects, providing crucial information for subsequent condition diagnosis.

[0213] b) Laser point cloud modal feature extraction:

[0214] 3D Geometric Structure Analysis: The disordered and sparse 3D point cloud data generated by LiDAR is processed through a dedicated architecture based on PointNet++ or Dynamic Graph Convolutional Network (DGCNN). These networks can operate directly on the point cloud data, learn the local neighborhood geometric structure features of each point, and effectively capture key 3D information such as the suspension curve of the conductor, the spatial configuration of the tower, and the safe distance from the surrounding vegetation.

[0215] Voxelization and 3D Convolution: As an alternative, point clouds can be voxelized (3D pixelated) into a regular mesh, and then a 3D convolutional neural network can be used for feature extraction. This method can efficiently learn the 3D spatial occupancy and geometric context of a scene, which is crucial for determining the real-time relative position of the drone and the route.

[0216] c) Acoustic modal feature extraction:

[0217] Time-frequency analysis and spectral feature extraction: The time-domain audio signal acquired by the acoustic imager is first converted into a time-frequency graph (such as a Mel spectrogram). Subsequently, lightweight CNNs or temporal convolutional networks (TCNs) are used to extract features from the spectrogram. These features can effectively capture the unique "fingerprints" of sounds generated by partial discharge, corona discharge, or mechanical loosening in the frequency domain, such as energy spikes or harmonic patterns in specific frequency bands.

[0218] d) Spatiotemporal attitude sequence feature extraction:

[0219] Sequence Modeling: IMU and GPS generate a series of timestamp-aligned sequence data. Long Short-Term Memory (LSTM) networks or one-dimensional temporal convolutional networks (1D-TCN) are used to model the UAV's own motion dynamics characteristics. These characteristics include flight attitude stability and trajectory smoothness, which can not only be used to assist in positioning but also serve as indirect indicators of abnormal states (such as high-frequency body jitter when encountering wind disturbances).

[0220] (2) Intelligent recognition and deconstruction based on deep learning:

[0221] a) Object detection and instance segmentation:

[0222] After obtaining powerful multi-scale features, an advanced detection and segmentation head network is used for accurate identification:

[0223] Component detection: Based on single-stage or Transformer-based detectors such as YOLOv7 or DETR, key power components, such as conductors, towers, insulators, tension clamps, vibration dampers, transformers, and circuit breakers, are located and classified in real time from visible and infrared images. The confidence score and class probability of the bounding box serve as the initial perception results for the component.

[0224] Pixel-level instance segmentation: For applications requiring precise contours, such as conductors and insulators, instance segmentation models like MaskR-CNN or SOLOv2 are used to output a binary mask for each target instance in the image. This is crucial for fine-grained measurement tasks such as calculating the actual length of conductors and determining the tilt angle of insulator strings.

[0225] b) Status recognition and anomaly diagnosis:

[0226] After identifying "what it is," the system further diagnoses "what its state is." This is a fine-grained classification and regression problem.

[0227] Multi-task learning framework: Design a multi-task learning head that shares the aforementioned extracted visual features.

[0228] Task 1: Defect Classification: Determine if a component has a specific defect, such as insulator spontaneous explosion, foreign object hanging from the conductor, or tower tilting. This is a classification task.

[0229] Task 2: Risk Regression: Regression calculations are performed on key parameters, such as the sag of the conductor, the distance between the conductor and trees, and the temperature gradient of the infrared hotspot. This is a regression task.

[0230] Multimodal evidence fusion diagnosis: For complex faults, such as "overheating of wire connectors", the system will fuse multimodal features from visible light images (identifying wire clamps), infrared images (detecting abnormal high temperatures), and acoustic data (detecting possible corona hissing sounds) and perform joint reasoning in a unified embedding space, which greatly improves the confidence and robustness of the diagnosis.

[0231] c) Topological relation resolution:

[0232] After completing instance-level identification and location of all components, the system enters the topology relationship resolution stage.

[0233] Graph structure construction: Each identified power component (tower, insulator, etc.) is regarded as a node in the graph, and its attributes include category, spatial coordinates (from GPS and visual SLAM), and status information.

[0234] Spatial and logical connection reasoning: Based on geometric proximity (such as a conductor connecting two adjacent towers) and a deep learning relational network, edges between nodes are inferred, thereby constructing a scene graph describing the line connection relationships. This scene graph serves as the direct data foundation for the subsequent automatic generation of 10kV line spatial topology maps.

[0235] The feature extraction and intelligent recognition module, through a complex algorithm system integrating multi-scale coding, multi-modal fusion, multi-task learning and graph structure reasoning, realizes end-to-end intelligent deconstruction from raw sensor data to power system components and their topological relationships, providing accurate, robust and semantically rich environmental perception information for the autonomous tracking and decision-making of UAVs.

[0236] Based on the aforementioned fusion representation vector, a multi-scale feature set is determined, which includes multi-scale visual features. Power component target detection and instance segmentation operations are performed to obtain preliminary identification information corresponding to multiple power component instances. This preliminary identification information includes: the bounding box coordinates of the corresponding power component instance, component category, confidence score, and pixel-level segmentation mask. Component categories include: conductors and towers. Based on the preliminary identification information and the multi-scale feature set of multiple power component instances, component state information corresponding to each power component instance is determined. This component state information includes at least one of the following: defect state information and risk parameter state information. Based on the preliminary identification information and component state information of multiple power component instances, an instance information fusion operation is performed to obtain component instance information.

[0237] 2.3 Autonomous Tracking and Decision Control:

[0238] This stage is the core of the system's execution, and its task is to transform the multimodal environment understanding output by the front-end perception module into precise, safe, and adaptive flight control commands. This process constructs a hierarchical control closed-loop system, realizing full-stack autonomous navigation from global path search to local dynamic obstacle avoidance.

[0239] (1) Tracking strategy based on hybrid behavior model:

[0240] To address the complexity of 10kV overhead line structures (such as branches, turns, and elevation changes), a hybrid behavior model was designed to enable UAVs to smoothly switch between different scenarios.

[0241] Mainline Tracking Mode: This is the basic flight mode for the UAV. The system uses the three-dimensional spatial vector of the guide line obtained by the front-end sensing as the reference path. The Flight Controller (FCU) adopts a proportional-integral-derivative and feedforward composite control algorithm. By calculating the lateral deviation, altitude deviation, and heading angle deviation of the guide line relative to the UAV, it generates attitude control commands to ensure that the UAV always maintains a preset safe lateral distance and overhead following altitude from the guide line.

[0242] Tower Node Processing Mode: When an approaching tower is detected, the system triggers the node processing routine. The UAV will temporarily use the tower's center as a virtual waypoint and execute a smooth circumnavigation or passage trajectory. This process deeply integrates LiDAR point cloud data for real-time motion planning, ensuring an absolutely safe distance from the tower and its auxiliary equipment (such as crossarms and guy wires).

[0243] Branching Decisions and Path Selection: Upon identifying a branching point (such as a T-junction or branching pole), the system enters a decision-making state. Its decision-making logic is based on a model combining a rule-based expert system and lightweight reinforcement learning.

[0244] Rule base: Prioritize following power grid topology conventions, such as "follow the conductor with a thicker cross-section", "continue along the original direction of travel", or "select a specific branch according to the preset task objective".

[0245] Learner: By learning from historical inspection data, the model can identify more likely trunk line features.

[0246] After making a decision, the system binds the high-precision GNSS coordinates, laser SLAM pose, and visual features of the starting point (branching point) of the selected branch and stores them in the stack, recording them as a traceable point.

[0247] (2) Multimodal perception-driven active obstacle avoidance and safety assurance:

[0248] The corridor environment contains many unstructured obstacles (such as trees, crossings, and temporary construction equipment), requiring drones to have dynamic obstacle avoidance capabilities.

[0249] Dynamic obstacle trajectory prediction and reactive obstacle avoidance: The system performs real-time clustering of laser point cloud and visual data to identify potential obstacles. For static obstacles, an improved A / B algorithm is used. Alternatively, a fast random tree algorithm can be used for local path replanning; for dynamic obstacles (such as birds), linear Kalman filtering or nonlinear particle filtering is used to predict their trajectory and calculate the collision risk. Once the risk exceeds a threshold, the UAV will initiate reactive obstacle avoidance, generating a collision-free path with the smallest deviation from the original tracking path based on the artificial potential field method or vector field histogram method.

[0250] Multi-sensor redundancy safety monitoring: High-frequency attitude data of the aircraft provided by the IMU and low-frequency global pose data provided by visual-laser SLAM are deeply fused through complementary filters or Kalman filters to form an attitude-position estimation system. When the system detects GPS signal loss, visual tracking failure, or distance from the guide wire exceeding the safety tolerance, it immediately triggers a safety fault protection mode, such as hovering, reversing along the original path, or ascending to a safe altitude, and waits for remote commands or autonomously attempts to resume tracking.

[0251] (3) Autonomous repositioning and task continuation based on memory maps:

[0252] When the drone automatically identifies a 10kV line branch point or encounters mission interruption issues due to obstacle avoidance, signal interruption, or power alarm, it can automatically achieve a high level of mission-level autonomy.

[0253] Construction of the topological memory map: During flight, the system not only records the original point cloud map, but also simultaneously constructs a sparse topological semantic map. This map is stored in a graph structure, with nodes representing poles and branch points, edges representing tracked line segments, and includes multimodal sensing feature descriptors for key locations.

[0254] Global localization and relocalization: When the mission is interrupted and needs to be resumed, the UAV first uses GNSS and IMU for coarse localization. After reaching the approximate area, it acquires the current environment image through the activated visual sensor, and compares it with keyframes in the topological memory map through image retrieval and feature matching technology to achieve visual relocalization with centimeter-level accuracy.

[0255] Task backtracking and continuation: After successful relocation, the UAV autonomously navigates to the last recorded backtrackable point (such as a branch point). Subsequently, the system pops the context information of that point from the stack, reinitializes the tracker, and seamlessly switches to "mainline tracking mode" or "branch decision mode" to continue executing the unfinished inspection task until the entire route topology traversal is completed.

[0256] The "Autonomous Tracking and Decision Control" module of this system utilizes a hybrid behavior model, an active obstacle avoidance mechanism, and topological memory. Figure 3 The core technology enables the construction of an autonomous aerial agent capable of understanding power grid topology semantics, coping with complex corridor environments, and recovering from outages. This not only ensures the integrity and continuity of the 10kV overhead line automatic tracking mission but also greatly enhances the robustness and practical value of the entire system in real-world, complex scenarios.

[0257] Based on the 3D spatial position of the conductor instance in the component instance information mentioned above, a 3D centerline spatial trajectory of the conductor is fitted and generated, and the 3D centerline spatial trajectory of the conductor is used as a reference path. Based on the reference path and the current pose of the UAV, the deviation data corresponding to the UAV is determined. The deviation data includes lateral deviation, altitude deviation, and heading angle deviation. The lateral deviation is the horizontal and vertical distance from the UAV's centroid to the nearest conductor reference point. The altitude deviation is the preset vertical following altitude difference between the UAV and the conductor. The heading angle deviation is the angle difference between the tangent direction of the conductor at the reference point and the yaw angle of the UAV. The deviation data is input to the target composite controller to generate a first flight control command, and the UAV is controlled to fly according to the first flight control command. The first flight control command consists of a proportional control component, an integral control component, a derivative control component, and a feedforward control component based on the conductor curvature. The first flight control command is used to control the UAV to maintain a preset safe lateral distance and an upward following altitude from the conductor, and to fly along the spatial extension direction of the conductor.

[0258] Step 3: Construction of 10kV line spatial topology:

[0259] This stage represents the final output of the system's data processing. Its core task is to transform the massive, multimodal, spatiotemporal sequence data collected in real time by the UAV during autonomous tracking into a precise, complete, and semantically rich digital route model. This model not only represents the geometric shape of the route but also profoundly describes the connection relationships and functional attributes between its components, thereby constructing a dynamic and computable three-dimensional spatial topology map.

[0260] 3.1 SLAM with Multi-Source Data Fusion and High-Precision Map Construction:

[0261] To address the challenge of accurate positioning and mapping in corridor environments where GPS signals may be unavailable, the system employs a tightly coupled multi-sensor fusion SLAM framework.

[0262] Vision-Inertial-LiDAR Tightly Coupled SLAM: This system uses visual feature points and line segment features (especially suitable for conductors) extracted by the front-end perception module, along with LiDAR point cloud features, as front-end association information. Combined with prior knowledge of the aircraft's motion provided by the IMU, joint state estimation is performed through graph optimization or filtering algorithms. This system can calculate the six-DOF pose of the UAV at any given time in real-time and simultaneously generate a 3D point cloud map with centimeter-level accuracy. This map clearly depicts the terrain, vegetation, and 3D outlines of all power facilities along the power corridor.

[0263] Global Pose Graph Optimization and Loop Closure Detection: To eliminate odometry drift errors accumulated during SLAM, the system introduces a loop closure detection mechanism. When the UAV revisits an explored area, loop closures are identified by comparing the similarity between the current visual scene and historical keyframes. Subsequently, pose graph optimization is initiated, treating all pose nodes (UAV positions) and observation constraints (including IMU pre-integration, visual reprojection, laser matching, and GPS absolute position) as an optimization problem for global adjustment, thereby obtaining a globally consistent high-precision map and trajectory.

[0264] 3.2 Semantic Extraction and Vector Deconstruction of Topological Elements:

[0265] Based on the high-precision point cloud map and UAV trajectory, the system extracts key semantic information and transforms the dense point cloud into structured vector elements.

[0266] Example extraction and location of towers and key equipment:

[0267] Point cloud clustering and classification: Apply point cloud clustering algorithms (such as DBSCAN) based on Euclidean distance and features to the global point cloud to separate independent device point cloud clusters.

[0268] Model matching and parametric fitting: Utilizing a pre-built parametric 3D template library of equipment such as poles and transformers, each point cloud cluster is identified and model-fitted using an iterative nearest-point algorithm or a deep learning point cloud segmentation network. The final output includes the type, 3D spatial coordinates (latitude, longitude, and elevation), and attitude (e.g., pole tilt) of each device, which are then used as nodes in the topology graph.

[0269] 3D Reconstruction and Vectorization of Conductors and Ground Wires:

[0270] Trajectory extraction of conductor point cloud: Combining visual recognition results, the point sets belonging to conductors and ground wires are segmented from the original point cloud.

[0271] Catenary model fitting: Based on physical knowledge, the catenary equation or polynomial fitting algorithm is used to fit a discrete conductor point cloud in three-dimensional space. This process outputs a continuous, smooth three-dimensional vector line for each conductor / ground wire and calculates key parameters such as the height of the lowest sag point and the amount of sag. These vector lines will serve as edges connecting nodes in the topology graph.

[0272] 3.3 Topological Relationship Reasoning and Structured Map Generation:

[0273] This step is crucial for giving the map "topological" properties, organizing discrete nodes and edges into a graph structure that describes the connections between them.

[0274] Spatial Relationships and Connection Logic Reasoning: Based on geometric proximity and electrical connection rules, the system automatically infers the connection relationships between nodes and edges. For example, a fitted conductor vector line must have both ends spatially connected to two tower nodes, forming an "edge". For branch points (such as T-junctions), the system will identify multiple conductors converging at the same node, thereby automatically generating a branch topology.

[0275] Attribute association and semantic enrichment: All attributes output by the front-end intelligent recognition module (such as device ID, model, identified defect type, infrared temperature measurement data, and acoustic diagnostic results) are associated with corresponding nodes and edges in the topology graph. This upgrades the topology graph from a simple geometric network into a semantic map carrying rich operational information.

[0276] Based on the aforementioned visual imaging data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data, a global 3D point cloud map is determined. This global 3D point cloud map is obtained from the global UAV pose, which in turn is derived from a pose optimization function and an initial 3D point cloud map. The initial 3D point cloud map is obtained from an error correction function. The pose optimization function is used to eliminate the cumulative drift error of the UAV pose, and the error correction function is used to correct the measurement error of the sensors. Based on the global 3D point cloud map and the component category and contour mask of the power component instances in the component instance information, device nodes representing energy-saving equipment and 3D vector edges representing conductors are obtained. Based on the device nodes and 3D vector edges, a topology connection operation is performed to obtain the line space topology map.

[0277] 3.4 Standardized Data Output and GIS System Integration:

[0278] Finally, the system converts the constructed structured topology data into a format that conforms to power industry standards (such as Common Information Model / CIM or GeoJSON), enabling seamless integration with existing power grid management systems.

[0279] Output: The generated 10kV line spatial topology map contains the following core elements:

[0280] Node layer: A table showing the spatial location and attributes of equipment such as poles, transformers, and switches.

[0281] Edge layer: The three-dimensional spatial orientation and electrical parameters of conductors and ground wires.

[0282] Topology connection table: explicitly describes the connection relationships between nodes and edges.

[0283] Inspection data layer: Real-time inspection data such as defects, temperature measurement, and point cloud data.

[0284] The spatial topology map construction module, through a progressive technical route of "multi-source fusion SLAM (localization and geometry construction) → semantic extraction and vectorization (objectification) → topological relationship reasoning (structuring)," successfully transforms the raw perception data from UAVs into a digital twin model of power infrastructure that combines geometric accuracy, topological correctness, and semantic richness. This model fundamentally solves the problems of lagging data updates and inaccurate topology in traditional GIS systems, providing a unique and reliable data foundation for the digital operation and maintenance, automated scheduling, and intelligent analysis of distribution networks.

[0285] The above-mentioned method integrates multiple types of data in the line spatial topology map into structured topology data. The multiple types of data include node layer data, topology connection table, and inspection data layer. Node layer data includes the spatial location and attribute information of equipment. The topology connection table is used to describe the connection relationship between nodes and edges. The inspection data layer includes the attached defect status information, risk parameter status information, and inter-depth point cloud data. The structured topology data is then converted into a target data format. The line spatial topology map in the target data format is sent to the power grid management system so that the power grid management system can update the line topology data based on the line spatial topology map.

[0286] The above optional implementation methods can achieve at least the following beneficial effects:

[0287] (1) Full-process autonomy and high intelligence: This invention realizes fully autonomous unmanned operation from route identification and flight tracking to map construction. By integrating multimodal perception and artificial intelligence algorithms, the UAV has human-like environmental understanding and decision-making capabilities, and can automatically cope with complex scenarios such as branches and towers, completely getting rid of the dependence on preset routes and manual operation, greatly improving inspection efficiency and reducing labor costs and safety risks.

[0288] (2) Strong robustness and high reliability of the sensing system: By introducing multi-source heterogeneous sensors such as visible light, infrared, lidar, and acoustic sensors, and utilizing advanced cross-modal embedding fusion technology, the system constructs a redundant and complementary sensing system. This system can effectively overcome the sensing limitations of a single sensor in harsh environments such as backlight, nighttime, fog, and complex backgrounds, ensuring the continuity and accuracy of line and status identification and tracking, and significantly improving the system's environmental adaptability and reliability.

[0289] (3) Accuracy and semantic richness of the topology map: The 10kV line spatial topology map constructed in this invention not only contains three-dimensional geometric information of lines and equipment with centimeter-level precision, but also embeds equipment attributes, connection relationships, and status information such as defects and thermal hazards found during inspections through semantic recognition and topological relationship reasoning. It generates a "computable" digital twin model that truly reflects the physical power grid, fundamentally solving the core pain points of inaccurate data and lagging updates in traditional GIS systems.

[0290] (4) Advanced cross-modal zero-shot transfer capability: By adopting the unified embedding space concept similar to ImageBind, the system achieves powerful cross-modal semantic understanding and zero-shot transfer capability. For example, it can retrieve and locate the corresponding visual target based solely on text description or abnormal sound. This capability greatly enhances the system's flexibility and scalability, enabling it to adapt to new inspection tasks and fault types in the future without retraining all models, thus possessing the potential for continuous learning.

[0291] (5) End-to-end closed loop and high practicality of the system: This invention forms a complete technical closed loop of "perception-decision-control-mapping". The specially designed autonomous return and task continuation mechanism ensures automatic recovery after encountering interference or task interruption, guaranteeing the integrity and success rate of the inspection task. The whole solution fully considers engineering practice, has a complete technical chain, and has extremely high practical value and industrial application prospects.

[0292] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0293] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0294] Example 2

[0295] According to an embodiment of the present invention, an apparatus for implementing the above-described method for performing unmanned aerial vehicle (UAV) inspection tasks is also provided. Figure 3 This is a structural block diagram of the execution device for a drone inspection task according to an embodiment of the present invention, such as... Figure 3 As shown, the device includes: a receiving module 302, a first determining module 304, a second determining module 306, a generating module 308, and a constructing module 310. The device will be described in detail below.

[0296] A receiving module 302 is used to receive and respond to a predetermined inspection task, acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on a UAV, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; a first determining module 304, connected to the receiving module 302, is used to perform cross-modal semantic alignment and feature fusion operations based on the multimodal sensor data to obtain a fused representation vector; a second determining module 306, connected to the first determining module 304, is used to determine component instance information based on the fused representation vector, wherein the component instance information includes power components. The three-dimensional spatial location, component category, and contour mask of the instance are provided. The component categories include: conductors and towers. A generation module 308, connected to the second determination module 306, is used to generate a first flight control command for the UAV based on the component instance information, and control the UAV to fly according to the first flight control command, wherein the first flight control command is used to drive the UAV to fly along the extension direction of the conductor. A construction module 310, connected to the generation module 308, is used to construct a line space topology map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information.

[0297] It should be noted that the receiving module 302, the first determining module 304, the second determining module 306, the generating module 308, and the building module 310 mentioned above correspond to steps S102 to S110 in the execution method of implementing the UAV inspection task. The multiple modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0298] Example 3

[0299] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the method for executing the unmanned aerial vehicle inspection task described above.

[0300] Example 4

[0301] According to another aspect of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the execution method of any of the above-described drone inspection tasks.

[0302] Example 5

[0303] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of any method.

[0304] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0305] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0306] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0307] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0308] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0309] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0310] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for executing a drone inspection task, characterized in that, include: Receive and respond to a predetermined inspection task, and acquire multimodal sensor data, wherein the multimodal sensor data is collected by the sensor system carried by the UAV, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; Based on the multimodal sensor data, cross-modal semantic alignment and feature fusion operations are performed to obtain a fused representation vector; Based on the fused representation vector, component instance information is determined, wherein the component instance information includes the three-dimensional spatial location of the power component instance, the component category, and the contour mask, and the component category includes: conductor, tower; Based on the component instance information, a first flight control command for the UAV is generated, and the UAV is controlled to fly according to the first flight control command, wherein the first flight control command is used to drive the UAV to fly along the extension direction of the wire; Based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information, a line spatial topology map is constructed.

2. The method according to claim 1, characterized in that, Based on the multimodal sensor data, cross-modal semantic alignment and feature fusion operations are performed to obtain a fused representation vector, including: Based on the multimodal sensor data, a multimodal encoding operation is performed to obtain a multimodal embedding vector, wherein the multimodal embedding vector includes a visual imaging embedding vector, an acoustic embedding vector, a point cloud embedding vector, an inertial embedding vector, and a position embedding vector. The multimodal embedding vectors are input into the modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space. The modality alignment model is trained on the initial alignment model using sample data and includes a cross-modal contrastive alignment loss function. The cross-modal contrastive alignment loss function is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times. Based on the alignment embedding vector, a feature fusion operation is performed to obtain a fused representation vector.

3. The method according to claim 1, characterized in that, Based on the fused representation vector, component instance information is determined, including: Based on the fused representation vector, a multi-scale feature set is determined, wherein the multi-scale feature set includes multi-scale visual features; Based on the multi-scale visual features, target detection and instance segmentation of power components are performed to obtain preliminary identification information corresponding to multiple power component instances. The corresponding preliminary identification information includes: bounding box coordinates, component category, confidence score, and pixel-level segmentation mask of the corresponding power component instance. Based on the preliminary identification information of the plurality of power component instances and the multi-scale feature set, component status information corresponding to the plurality of power component instances is determined, wherein the corresponding component status information includes at least one of the following: defect status information, risk parameter status information; Based on the preliminary identification information and component status information of the multiple power component instances, an instance information fusion operation is performed to obtain component instance information.

4. The method according to claim 1, characterized in that, Based on the component instance information, a first flight control command is generated for the UAV, and the UAV is controlled to fly according to the first flight control command, including: Based on the three-dimensional spatial position of the conductor instance in the component instance information, a three-dimensional centerline spatial trajectory of the conductor is fitted and generated, and the three-dimensional centerline spatial trajectory of the conductor is used as a reference path; Based on the reference path and the current pose of the UAV, the deviation data corresponding to the UAV is determined. The deviation data includes lateral deviation, altitude deviation and heading angle deviation. The lateral deviation is the horizontal and vertical distance from the center of mass of the UAV to the nearest guide reference point. The altitude deviation is the preset vertical following altitude difference between the UAV and the guide. The heading angle deviation is the angle difference between the tangent direction of the guide at the reference point and the yaw angle of the UAV. The deviation data is input to the target composite controller to generate a first flight control command, and the UAV is controlled to fly according to the first flight control command. The first flight control command consists of a proportional control component, an integral control component, a derivative control component, and a feedforward control component based on the curvature of the guide wire. The first flight control command is used to control the UAV to maintain a preset safe lateral distance and an upward following altitude from the guide wire, and to fly along the spatial extension direction of the guide wire.

5. The method according to claim 1, characterized in that, Based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information, a line spatial topology map is constructed, including: Based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, and the position coordinate data, a global 3D point cloud map is determined. The global 3D point cloud map is obtained based on the global UAV pose, which is obtained based on the pose optimization function and the initial 3D point cloud map. The initial 3D point cloud map is obtained based on the error correction function. The pose optimization function is used to eliminate the cumulative drift error of the UAV pose, and the error correction function is used to correct the measurement error of the sensor. Based on the global 3D point cloud map and the component category and contour mask of the power component instance in the component instance information, the device node representing the energy-saving device and the 3D vector edge representing the conductor are obtained. Based on the device nodes and the three-dimensional vector edges, a topology connection operation is performed to obtain a line space topology map.

6. The method according to any one of claims 1 to 5, characterized in that, After constructing the line spatial topology map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information, the method further includes: The various types of data in the line space topology map are integrated into structured topology data, which includes node layer data, topology connection table and inspection data layer. The node layer data includes the spatial location and attribute information of the equipment. The topology connection table is used to describe the connection relationship between nodes and edges. The inspection data layer includes the attached defect status information, risk parameter status information and inter-depth point cloud data. Convert the structured topology data into the target data format; The line spatial topology map in the target data format is sent to the power grid management system so that the power grid management system can update the line topology data based on the line spatial topology map.

7. An execution device for unmanned aerial vehicle (UAV) inspection tasks, characterized in that, include: The receiving module is used to receive and respond to a predetermined inspection task and acquire multimodal sensor data, wherein the multimodal sensor data is collected by the sensor system carried by the UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. The first determining module is used to perform cross-modal semantic alignment and feature fusion operations based on the multimodal sensor data to obtain a fused representation vector; The second determining module is used to determine component instance information based on the fused representation vector, wherein the component instance information includes the three-dimensional spatial location of the power component instance, the component category, and the contour mask, and the component category includes: conductor and tower; The generation module is used to generate a first flight control command for the UAV based on the component instance information, and control the UAV to fly according to the first flight control command, wherein the first flight control command is used to drive the UAV to fly along the extension direction of the wire; The construction module is used to construct a line spatial topology map based on the visual imaging data, the spatial depth point cloud data, the attitude inertial sequence data, the position coordinate data, and the component instance information.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for executing the UAV inspection task as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the execution method of the UAV inspection task as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images