A Multi-Agent Large-Model Collaborative Approach for UAV and Robot Group Power Grid Inspection
By constructing an environmental semantic map and classifying facility access methods, aerial and ground inspection tasks are generated, and sub-task dependencies are established, enabling multi-agent collaborative decision-making. This solves the problems of incomplete information collection and duplicate inspections in power grid inspections, and improves inspection efficiency and result accuracy.
Patent Information
- Application Number
- CN202611451251.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-18
- Publication Date
- 2026-11-17
AI Technical Summary
In existing power grid inspection technologies, the lack of task-level coordination during multi-agent collaborative operations leads to incomplete information collection, repeated inspections, and extended inspection cycles. Furthermore, the lack of consideration for the different access methods of different parts of the facilities results in missed or repeated inspections.
Construct an environmental semantic map, identify facility access methods and classify them into high-altitude, ground and hybrid access facilities, generate aerial and ground inspection tasks, establish sub-task dependencies, generate joint execution sequences through multi-agent collaborative decision-making, adjust paths and timing under state synchronization instructions, and exchange detection information for fusion.
It improved the coverage and accuracy of inspections, reduced repetitive work, shortened the inspection cycle, enhanced adaptability and robustness to complex environments, and improved the completeness and interpretability of inspection results.
Smart Images

Figure CN123113931A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of power grid inspection technology, and in particular to a method for power grid group inspection using a multi-agent large-scale model collaboration between drones and robots. Background Technology
[0002] Power grid inspections have long relied on human experience and stand-alone automated equipment. Drones typically inspect overhead transmission lines, towers, and other high-altitude targets along pre-set routes, while ground robots inspect equipment and cable tunnels within substations according to planned paths. As the scope of operation and maintenance expands, some organizations have begun to use multi-drone or multi-ground robot teams for inspections. However, the team members are of limited types, and task allocation is still pre-determined by maintenance personnel based on equipment ledgers. The various intelligent agents rarely collaborate at the task level during execution, and the collected data is stored separately, only to be manually summarized and analyzed after the inspection is completed.
[0003] Current practices do not link facility spatial configuration with task allocation. Inspection areas are typically divided into blocks based on geographical boundaries and assigned to single-type equipment, without considering the different access methods required for different parts of facilities such as poles and substations. For facilities with both high-altitude and ground-level inspection points, a single device can only acquire information from a local perspective, easily leading to missed or duplicate inspections. If data loss is found after the task is completed, on-site re-shooting must be arranged, extending the inspection cycle.
[0004] When multiple types of equipment work together, there is also a lack of process synchronization and task correlation. Each piece of equipment advances in the same area according to its own timeline, adjusting its path when encountering obstacles, without other equipment being able to promptly learn of this and adjust its own plans. Data collected by different equipment around the same object lacks unified identification and correspondence, and during later fusion, it can only rely on manual experience for time and space alignment, making it difficult to support accurate and overall status assessment. Low collaboration efficiency also directly affects the timeliness of reporting inspection results. Summary of the Invention
[0005] This invention provides a method for inspecting power grids using a large-scale collaborative model of multiple agents, involving drones and robots, which can solve the problems in the prior art.
[0006] A first aspect of this invention provides a method for inspecting power grids using a multi-agent large-model collaborative approach involving drones and robots, comprising:
[0007] Collect geographical environment data and facility distribution data of the power grid inspection area, and construct an environmental semantic map containing obstacle information and access constraints based on the geographical environment data;
[0008] Based on the environmental semantic map and facility distribution data, the access method of each facility is identified, and the facilities are classified into high-altitude access facilities, ground access facilities, and mixed access facilities.
[0009] For high-altitude access facilities and ground access facilities, aerial inspection tasks and ground inspection tasks are generated respectively. Based on the spatial structure and inspection requirements of hybrid access facilities, aerial inspection sub-tasks and ground inspection sub-tasks are generated, and the dependencies and data transmission interfaces between sub-tasks are established.
[0010] Multi-agent collaborative decision-making is carried out for each inspection task and sub-task. The sub-task dependency is used as a timing constraint and state synchronization instructions are inserted at the sub-task execution node to generate a joint execution sequence of UAV agent and ground robot agent.
[0011] Each intelligent agent executes tasks according to the joint execution sequence. At the state synchronization instruction trigger node, it exchanges position information and progress information and adjusts the subsequent execution path and timing accordingly. It also exchanges local detection information through the data transmission interface to form the inspection result.
[0012] Collect geographical environment data and facility distribution data of the power grid inspection area, and construct an environmental semantic map containing obstacle information and access constraints based on the geographical environment data, including:
[0013] Collect geographical environment data and facility distribution data of the power grid inspection area, perform spatial interpolation to complete the geographical environment data, generate a continuous three-dimensional spatial field expression, and extract the gradient vector field from the three-dimensional spatial field expression;
[0014] Divergence and curl are calculated on the gradient vector field. The source points of spatial density change are identified based on the divergence value, and the rotation center of the spatial structure is identified based on the curl value. The source points of density change and the rotation center are determined as feature anchor points of the obstacle. Region growth is performed using the feature anchor points as seed points. Growth stops when the magnitude of the gradient vector at the growth boundary is lower than the convergence threshold. The spatial region after growth is determined as a complete obstacle entity and its boundary envelope is extracted.
[0015] Based on the facility distribution data, calculate the spatial occlusion relationship between obstacle entities and facilities, determine the type of approach path restriction caused by the occlusion relationship, and generate corresponding detour passage constraints or height adjustment passage constraints for different restriction types.
[0016] The obstacle boundary envelope, spatial occlusion relationship, and generated passage constraints are semantically associated and encoded to construct an environmental semantic map containing obstacle information and passage constraints.
[0017] Based on the environmental semantic map and facility distribution data, the access method of each facility is identified, and the facilities are classified into high-altitude access facilities, ground access facilities, and mixed access facilities, including:
[0018] Extract the distribution of access constraints around each facility from the environmental semantic map, and map the access constraints to a spherical coordinate system centered on each facility according to their spatial location. The radial direction represents the distance to the facility, the polar angle represents the vertical angle, and the azimuth angle represents the horizontal angle.
[0019] In the spherical coordinate system, a vertical approach sector and a horizontal approach ring are defined based on the polar angle range. The vertical approach sector covers the airspace directly above the facility, and the horizontal approach ring covers the ground area around the facility. The traffic constraint density within the vertical approach sector and the horizontal approach ring is calculated respectively.
[0020] The minimum approach space size of each facility is read from the facility distribution data. The minimum approach space size is converted into a radial distance threshold in a spherical coordinate system. It is determined whether there is a passage constraint within the radial distance threshold range of the vertical approach sector and the air approach support is marked. It is determined whether there is a passage constraint within the radial distance threshold range of the horizontal approach ring and the ground approach support is marked.
[0021] Based on the marking results, facilities are classified as follows: those marked only with aerial access support are classified as high-altitude access facilities; those marked only with ground access support are classified as ground access facilities; and those marked with both types of access support are classified as hybrid access facilities.
[0022] Based on the spatial structure and inspection requirements of hybrid access facilities, aerial inspection sub-tasks and ground inspection sub-tasks are generated, and the dependencies and data transfer interfaces between the sub-tasks are established, including:
[0023] The detection requirement description of hybrid proximity facilities is extracted from the facility distribution data, parsed into a pairing relationship between detection targets and detection accuracy, and an observable envelope surface that meets the detection accuracy requirements is constructed for each detection target.
[0024] Calculate the spatial intersection between the observable envelope and the access constraints in the environmental semantic map. Based on the spatial intersection, divide the observable envelope into an air-accessible observation area and a ground-accessible observation area. For detection targets whose observable envelope crosses the air-accessible observation area and the ground-accessible observation area, set a collaborative observation reference point on the boundary line between the air-accessible observation area and the ground-accessible observation area.
[0025] For detection targets that include collaborative observation reference points, a dependency relationship is established between corresponding aerial inspection sub-tasks and ground inspection sub-tasks. A data transmission interface is established at the collaborative observation reference point. When the aerial inspection sub-task reaches the observation position corresponding to the collaborative observation reference point, it transmits local feature data. The ground inspection sub-task determines the observation reference reference based on the local feature data and then performs complementary observations.
[0026] Multi-agent collaborative decision-making is performed on each inspection task and sub-task. Sub-task dependencies are used as timing constraints, and state synchronization instructions are inserted at sub-task execution nodes to generate a joint execution sequence of UAV agents and ground robot agents, including:
[0027] Multi-agent collaborative decision-making is carried out for each inspection task and sub-task. The coordinates of the execution area of each inspection task and sub-task are extracted. The moving distance and time cost of the UAV agent and the ground robot agent to reach each execution area are calculated. A spatiotemporal reachability evaluation matrix is constructed. Based on the spatiotemporal reachability evaluation matrix, the inspection tasks and sub-tasks are assigned to the corresponding agents. A task execution node sequence containing the agent identifier and the planned execution time is generated.
[0028] The subtask dependencies are analyzed to extract the data flow path between subtasks. The cross-agent connection relationship between the predecessor subtask and the successor subtask belonging to different execution agents is identified in the data flow path. The end time of the task execution node corresponding to the predecessor subtask is increased by the data transmission delay and used as the start time of the task execution node corresponding to the successor subtask. This forms a time constraint and updates the task execution node sequence.
[0029] In the updated task execution node sequence, locate the successor task execution node corresponding to the cross-agent connection relationship, insert a state synchronization instruction into the successor task execution node, and trigger the predecessor execution agent to transmit the position coordinates and detection results to the successor execution agent; group the task execution node sequence according to the execution agent identifier and sort it according to the start time to form a joint execution sequence.
[0030] Based on the spatiotemporal reachability evaluation matrix, inspection tasks and subtasks are assigned to corresponding agents, generating a sequence of task execution nodes containing the agent identifier and the planned execution time, including:
[0031] Extract the numerical variance and range of each row of the executing agent in the spatiotemporal reachability assessment matrix, and calculate the degree of adaptability difference of the executing agent to different inspection tasks and sub-tasks;
[0032] Select the agents with the greatest degree of adaptability difference for task allocation. Assign the inspection tasks and sub-tasks identified by the column with the smallest value in the corresponding row to the selected agents. Remove the columns corresponding to the assigned inspection tasks and sub-tasks from the spatiotemporal reachability evaluation matrix. Recalculate the degree of adaptability difference for the remaining agents and continue to allocate tasks until all inspection tasks and sub-tasks are allocated.
[0033] The coordinates of the execution areas corresponding to the inspection tasks and subtasks assigned to each execution agent are counted. The movement path of the execution agent to traverse all execution areas is planned. The execution order of the inspection tasks and subtasks is determined according to the access order of the execution areas in the movement path.
[0034] For each inspection task and subtask in the execution sequence, a task execution node is generated. The task execution node contains the assigned execution agent identifier and the planned execution time calculated based on the cumulative movement time of the preceding position in the movement path. The task execution nodes are arranged in the execution sequence to form a task execution node sequence.
[0035] Each agent executes tasks according to the joint execution sequence, exchanges position and progress information at the state synchronization instruction trigger node, and adjusts subsequent execution paths and timing accordingly. Local detection information is exchanged and fused through the data transmission interface to form inspection results, including:
[0036] When each agent reaches the state synchronization instruction trigger node, it compares its current position coordinates and task execution progress with the expected position coordinates and expected progress recorded in the joint execution sequence, and generates a deviation report containing position deviation and progress deviation.
[0037] Each agent exchanges deviation reports through the data transmission interface, identifies agents whose deviations exceed a preset range as delayed agents, and extracts the unfinished task identifiers of the delayed agents.
[0038] According to the unfinished task identifier, query the task dependency relationship in the joint execution sequence. If the task to be executed depends on the unfinished task of the delayed agent, the planned execution time is pushed back. Based on the pushed planned execution time, the movement path to the next execution area is replanned, and the subsequent execution process is adjusted according to the replanned movement path.
[0039] Each intelligent agent collects detection data of the target object in the execution area as local detection information. After adding the collection location coordinates to the local detection information, it sends it to the fusion node through the data transmission interface. The fusion node performs spatial clustering of the local detection information according to the collection location coordinates and merges the local detection information in the same spatial cluster to form the inspection result.
[0040] A second aspect of this invention provides a multi-agent large-model collaborative UAV and robot power grid group inspection system, comprising:
[0041] The map building unit is used to collect geographical environment data and facility distribution data of the power grid inspection area, and to build an environmental semantic map containing obstacle information and passage constraints based on the geographical environment data.
[0042] The facility classification unit is used to identify the access method of each facility based on the environmental semantic map and facility distribution data, and classify the facilities into high-altitude access facilities, ground access facilities and mixed access facilities.
[0043] The task generation unit is used to generate aerial inspection tasks and ground inspection tasks for high-altitude access facilities and ground access facilities, respectively. Based on the spatial structure and inspection requirements of hybrid access facilities, it generates aerial inspection sub-tasks and ground inspection sub-tasks, and establishes the dependencies and data transmission interfaces between sub-tasks.
[0044] The collaborative decision-making unit is used to make multi-agent collaborative decisions on each inspection task and sub-task, taking the sub-task dependency relationship as a timing constraint and inserting state synchronization instructions at the sub-task execution node to generate a joint execution sequence of UAV agent and ground robot agent.
[0045] The execution fusion unit is used by each intelligent agent to execute tasks according to the joint execution sequence, exchange position information and progress information at the state synchronization instruction trigger node and adjust the subsequent execution path and timing accordingly, and exchange local detection information through the data transmission interface to form inspection results.
[0046] A third aspect of the present invention provides an electronic device, comprising:
[0047] processor;
[0048] Memory used to store processor-executable instructions;
[0049] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.
[0050] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.
[0051] This method constructs an environmental semantic map based on the geographical environment and facility distribution of the inspection area, enabling heterogeneous intelligent agents to understand obstacles and passage constraints before task planning, significantly reducing the risk of collisions and detours during inspection. It further categorizes facilities according to their access methods, matching differentiated inspection modes to high-altitude, ground-based, and mixed-access facilities, avoiding the inadequacy of a single platform for complex facilities. For mixed facilities, it breaks down aerial and ground-based sub-tasks, establishing dependencies and data interfaces to ensure the spatial and logical integrity of multi-angle detection data, overcoming blind spots in single-view detection and improving defect identification coverage.
[0052] Subtask dependencies are transformed into temporal constraints. After state synchronization instructions are embedded in execution nodes, UAVs and ground robots can advance at a collaborative pace, reducing repetitive inspections or waiting gaps caused by uncoordinated actions. The joint execution sequence pre-coordinates the behavior of multiple agents, improving the utilization of airspace and ground resources and shortening the overall inspection time. When state synchronization is triggered, position and progress information are exchanged, and each agent can instantly correct its local path and work sequence, effectively responding to dynamic obstacles or sudden conditions and enhancing the adaptability and robustness of the formation in complex power grid environments.
[0053] The data transmission interface supports semantic-level fusion of local detection results. Images, point clouds, or sensor data collected by different agents can be mutually verified within a unified framework, generating more complete defect descriptions and localization results than a single platform. Redundant data is filtered synchronously during the fusion process, reducing the workload of manual review and compensating for information gaps in weak textures and occluded areas. The accuracy and interpretability of inspection results are improved simultaneously, supporting subsequent maintenance decisions. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating the multi-agent large-model collaborative UAV and robot power grid group inspection method according to an embodiment of the present invention;
[0055] Figure 2 A flowchart illustrating the method for constructing an environmental semantic map according to an embodiment of the present invention;
[0056] Figure 3 A bar chart comparing facility access constraint density with classification thresholds;
[0057] Figure 4 A bar chart comparing the divergence values of characteristic anchor points with the curl modulus. Detailed Implementation
[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0059] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0060] Figure 1This is a flowchart illustrating the multi-agent large-model collaborative UAV and robot power grid group inspection method according to an embodiment of the present invention. The present invention provides a multi-agent large-model collaborative UAV and robot power grid group inspection method comprising:
[0061] Collect geographical environment data and facility distribution data of the power grid inspection area, and construct an environmental semantic map containing obstacle information and access constraints based on the geographical environment data;
[0062] Based on the environmental semantic map and facility distribution data, the access method of each facility is identified, and the facilities are classified into high-altitude access facilities, ground access facilities, and mixed access facilities.
[0063] For high-altitude access facilities and ground access facilities, aerial inspection tasks and ground inspection tasks are generated respectively. Based on the spatial structure and inspection requirements of hybrid access facilities, aerial inspection sub-tasks and ground inspection sub-tasks are generated, and the dependencies and data transmission interfaces between sub-tasks are established.
[0064] Multi-agent collaborative decision-making is carried out for each inspection task and sub-task. The sub-task dependency is used as a timing constraint and state synchronization instructions are inserted at the sub-task execution node to generate a joint execution sequence of UAV agent and ground robot agent.
[0065] Each intelligent agent executes tasks according to the joint execution sequence. At the state synchronization instruction trigger node, it exchanges position information and progress information and adjusts the subsequent execution path and timing accordingly. It also exchanges local detection information through the data transmission interface to form the inspection result.
[0066] Figure 2 This is a flowchart illustrating the method for constructing an environmental semantic map according to an embodiment of the present invention. It involves collecting geographical environmental data and facility distribution data of the power grid inspection area, and constructing an environmental semantic map containing obstacle information and access constraints based on the geographical environmental data, including:
[0067] Collect geographical environment data and facility distribution data of the power grid inspection area, perform spatial interpolation to complete the geographical environment data, generate a continuous three-dimensional spatial field expression, and extract the gradient vector field from the three-dimensional spatial field expression;
[0068] Divergence and curl are calculated on the gradient vector field. The source points of spatial density change are identified based on the divergence value, and the rotation center of the spatial structure is identified based on the curl value. The source points of density change and the rotation center are determined as the feature anchor points of the obstacle.
[0069] Region growth is performed using the feature anchor points as seed points. Growth stops when the gradient vector magnitude at the growth boundary is lower than the convergence threshold. The spatial region after growth is determined as a complete obstacle entity and its boundary envelope is extracted.
[0070] Based on the facility distribution data, calculate the spatial occlusion relationship between obstacle entities and facilities, determine the type of approach path restriction caused by the occlusion relationship, and generate corresponding detour passage constraints or height adjustment passage constraints for different restriction types.
[0071] The obstacle boundary envelope, spatial occlusion relationship, and generated passage constraints are semantically associated and encoded to construct an environmental semantic map containing obstacle information and passage constraints.
[0072] When collecting geographic environment data and facility distribution data for the power grid inspection area, the geographic environment data sources include point cloud scanning data from lidar carried by UAVs, multispectral camera image data, and 3D scanning data collected by ground mobile survey vehicles. Facility distribution data comes from the coordinates and attribute information of transmission lines, towers, substations, and other facilities already registered in the power grid geographic information system. Due to spatial resolution differences and local occlusion in multi-source sensor acquisition, the original geographic environment data often contains sampling gaps in space. Therefore, spatial interpolation is performed to complete the geographic environment data. The spatial interpolation method uses Kriging interpolation to estimate the missing areas based on the spatial correlation of known sampling points, completing the discrete 3D point cloud and elevation data into a continuously distributed 3D spatial field representation throughout the inspection area. This 3D spatial field representation is stored in a voxel grid format, with each voxel recording its spatial density value, which reflects the probability of the presence of a physical structure at that location.
[0073] When extracting the gradient vector field from the three-dimensional spatial field representation, along the path at each voxel position... , , The partial derivatives of the density value are calculated along each of the three coordinate axes. The combination of the partial derivatives in the three directions forms the gradient vector at that voxel location. The gradient vector points in the direction of the fastest increase in density, and its magnitude is... It reflects the drastic degree of density change at that location. The gradient vectors of all voxel locations within the entire inspection area constitute a complete gradient vector field. This vector field exhibits a large magnitude at obstacle boundaries and a magnitude close to zero in open areas. Therefore, the gradient vector field itself carries geometric information about the spatial structure of the obstacle.
[0074] Divergence and curl are calculated on the gradient vector field; these two operations characterize the spatial features of obstacles from different perspectives. Divergence The divergence value describes the flux change of the gradient vector within the neighborhood of a point. When the divergence value at a voxel location is a large positive value, it indicates that the location is a source point for the outward radiation of spatial density, corresponding to the internal core region of an obstacle entity in physical terms. When the divergence value is a large negative absolute value, it indicates that the location is a sink point for density convergence, corresponding to the transition region inside the obstacle boundary. Curl The curl magnitude describes the tendency of the gradient vector field to circulate within the neighborhood of a point. Larger gradients indicate a significant rotational structure in the gradient vector, corresponding to locations where the obstacle's geometry bends or turns, such as the corners of a tower's crossarm or the edges of a building. Sources of spatial density changes are identified based on divergence values, and centers of spatial structural rotation are identified based on curl values. These density change sources and rotation centers are collectively defined as the obstacle's feature anchor points. This set of feature anchor points covers the core locations and morphological transition points of the obstacle's geometry. For example... Figure 3 As shown in the figure, this diagram illustrates the divergence values (blue bars) and curl magnitudes (orange bars) of five feature anchor points (AP-01 to AP-05) identified from the gradient vector field through divergence and curl calculations. Larger positive divergence values indicate that the location is a source point of outward spatial density radiation, corresponding to the core region inside the obstacle; larger negative absolute divergence values indicate that the location is a sink point of density convergence, corresponding to the transition region inside the obstacle boundary. Locations with larger curl magnitudes indicate a significant rotational structure in the gradient vector, corresponding to locations where the obstacle's geometry bends or turns. In the figure, AP-01 has a divergence value of +4.82 and a curl modulus of only 0.31, indicating that this point is mainly a density source point, corresponding to the core of the tower body; AP-02 has a divergence value of -1.23 and a curl modulus of 3.91, indicating that this point has a significant rotational structure, corresponding to the corner of the tower crossarm; AP-03 has a divergence value of +3.67 and a curl modulus of 0.45, corresponding to the source point characteristics of the power station enclosure wall; AP-04 has a divergence value of -0.87 and a curl modulus of 2.54, corresponding to the rotation center of the distribution box edge; AP-05 has a divergence value of +5.12 and a curl modulus of 0.28, corresponding to the density source point at the center of the tree canopy.
[0075] Region growing is performed using feature anchor points as seed points. The growth process starts from each seed point and expands towards neighboring voxels along the gradient vector direction, incorporating voxels with density values higher than a set threshold into the current obstacle region. During growth, the magnitude of the gradient vector at the growth boundary is continuously monitored. When the magnitude of the gradient vector at the growth boundary falls below the convergence threshold, it indicates that the boundary has reached the end of the transition zone between the obstacle entity and free space, and the density change has stabilized; growth in that direction is then stopped. After region growing is completed for all seed points, the grown spatial region is defined as the complete obstacle entity, and the boundary envelope of the entity's outer surface is extracted. The boundary envelope uses a combination of convex hull algorithm and surface mesh reconstruction. For obstacles with regular shapes (such as rectangular buildings), an axis-aligned bounding box is used as the envelope; for obstacles with complex shapes (such as towers and trees), a fine triangular mesh is used as the envelope to ensure the accuracy of collision detection during subsequent path planning.
[0076] When calculating the spatial occlusion relationship between obstacle entities and facilities by combining facility distribution data, for each facility location, rays are emitted from multiple azimuth and pitch angles, and the number of times the rays intersect with the obstacle envelope before reaching the facility is counted. If a ray in a certain direction intersects with the obstacle envelope, the approach path corresponding to that direction is occluded by the obstacle. Based on the azimuth and pitch angle distribution of the occluded rays, the type of approach path restriction caused by the occlusion relationship is determined. When the occlusion is mainly concentrated in a certain angle range in the horizontal direction, it is determined to be a horizontal detour restriction, and a detour constraint is generated accordingly. The detour constraint specifies the obstacle area that the UAV or ground robot must bypass and the recommended detour direction when approaching the facility. When the occlusion is mainly concentrated in the lower pitch angle direction, it is determined to be a low-altitude restriction, and an altitude adjustment constraint is generated accordingly. The altitude adjustment constraint specifies the minimum flight altitude that needs to be increased when approaching the facility. This ensures that the drone can pass over the obstructed area from above the obstacle to reach the facility detection position; when obstruction exists in multiple directions at the same time, it is judged as a compound restriction, and both detour passage constraints and height adjustment passage constraints are generated. The two types of constraints together limit the feasible approach path space.
[0077] When semantically associating and encoding obstacle boundary envelopes, spatial occlusion relationships, and generated access constraints, each obstacle entity and each facility is treated as a node. The occlusion relationship is used as a directed edge connecting obstacle nodes and facility nodes affected by its occlusion, and corresponding access constraint attributes are attached to these directed edges. Obstacle nodes store geometric descriptions of their boundary envelopes, volume, height range, and other geometric attributes; facility nodes store attributes such as their type, coordinates, and detection surface direction; access constraint attributes store constraint type (detour or height adjustment) and constraint parameters (detour direction vector or minimum height). The nodes, edges, and attributes are structured and stored according to a unified semantic encoding format to construct an environmental semantic map containing obstacle information and passage constraints. This semantic map is organized in the form of a graph database, supporting queries for facilities to find their associated obstacles and passage constraints, as well as queries for all obstacle entities within a spatial range, providing structured environmental knowledge support for path planning in subsequent multi-agent collaborative decision-making. During the inspection task, if the sensors detect new obstacles that do not match the semantic map in real time, the semantic map can be locally updated, and the newly detected obstacle entities and their corresponding passage constraint increments can be written into the graph database to ensure the consistency between the environmental semantic map and the actual inspection environment.
[0078] Based on the environmental semantic map and facility distribution data, the access method of each facility is identified, and the facilities are classified into high-altitude access facilities, ground access facilities, and mixed access facilities, including:
[0079] Extract the distribution of access constraints around each facility from the environmental semantic map, and map the access constraints to a spherical coordinate system centered on each facility according to their spatial location. The radial direction represents the distance to the facility, the polar angle represents the vertical angle, and the azimuth angle represents the horizontal angle.
[0080] In the spherical coordinate system, a vertical approach sector and a horizontal approach ring are defined based on the polar angle range. The vertical approach sector covers the airspace directly above the facility, and the horizontal approach ring covers the ground area around the facility. The traffic constraint density within the vertical approach sector and the horizontal approach ring is calculated respectively.
[0081] The minimum approach space size of each facility is read from the facility distribution data. The minimum approach space size is converted into a radial distance threshold in a spherical coordinate system. It is determined whether there is a passage constraint within the radial distance threshold range of the vertical approach sector and the air approach support is marked. It is determined whether there is a passage constraint within the radial distance threshold range of the horizontal approach ring and the ground approach support is marked.
[0082] Based on the marking results, facilities are classified as follows: those marked only with aerial access support are classified as high-altitude access facilities; those marked only with ground access support are classified as ground access facilities; and those marked with both types of access support are classified as hybrid access facilities.
[0083] When extracting the distribution of access constraints around facilities from an environmental semantic map, it is necessary to traverse the spatial neighborhood of each facility on the map and read the marked obstacle boundaries, no-fly zone markers, impassable ground areas, and high-voltage protection barriers within that neighborhood. These access constraints are stored in three-dimensional Cartesian coordinates in the original map. To facilitate the analysis of the accessibility direction of a particular facility, they need to be remapped to a spherical coordinate system centered on that facility. In the spherical coordinate system, radial coordinates... Indicates the spatial distance between the constraint point and the center of the facility, polar angle. This indicates the angle of deflection of the point relative to the vertically upward direction of the facility center; azimuth. This represents the rotation angle of the point relative to the reference direction in the horizontal plane. After the coordinate transformation, each traffic constraint point is represented by... The data is recorded in triplets within a spherical coordinate space centered on the facility, providing the basis for subsequent sector and ring statistics.
[0084] After establishing the spherical coordinate system, the space is functionally divided according to the polar angle range. The vertically approaching sector is defined as the polar angle. satisfy The cone-shaped airspace region, in which The vertical sector half-apex angle threshold is determined based on the maneuver envelope angles of the UAV's vertical descent and ascent, typically ranging from 30° to 60°, covering the flyable airspace directly above and near the facility. The horizontal approach ring is defined as the polar angle. satisfy The annular region, in which The starting polar angle of the horizontal ring zone. The downward-extending tolerance angle is used to cover the approach path of ground robots on slightly sloping terrain, encompassing the passable areas of the ground and near-ground layers surrounding the facility. For vertical approach sectors, the number and volume percentage of passage constraint points falling within these sectors are statistically analyzed to obtain the vertical approach sector passage constraint density. For the horizontally approaching ring zone, the traffic constraint density of the horizontally approaching ring zone is obtained statistically in the same way. The traffic constraint density is calculated as the ratio of the number of constraint voxels in the region to the total number of voxels in the region. The value ranges from 0 to 1. The larger the value, the more obstructed the approach path in that direction.
[0085] When retrieving the minimum approach space dimension for each facility from facility distribution data, this dimension describes the minimum safe operating distance required for inspection equipment to approach the facility and perform inspection actions. For drones, the minimum approach space dimension typically includes the sum of the rotor deployment radius, obstacle avoidance safety margin, and the imaging or sensor working distance; for ground robots, this dimension includes half the vehicle width plus the robotic arm extension distance and safety clearance. The above minimum approach space dimension is then converted to a radial distance threshold in spherical coordinates. This means that the inspection equipment must be within a certain distance of the facility center. The detection operation can be completed within a certain range. Based on this, the radial distance within the vertically approaching sector is determined. Does the subspace contain traffic constraint points? If the traffic constraint density in the subspace is lower than the preset air-to-access threshold, then the subspace contains traffic constraint points. If the drone can approach the facility vertically, the facility is marked as having aerial access support; if the passage constraint density exceeds If the vertical approach path is blocked, then aerial approach support is not marked. Similarly, the radial distance within the horizontal approach ring is determined. Does the subspace contain traffic constraint points? If the traffic constraint density in the subspace is lower than the preset ground threshold and close to the usable threshold, then the subspace is considered a traffic constraint point. If the ground robot can approach the facility horizontally, the facility is marked as having ground access support; if the passage constraint density exceeds If so, ground proximity support is not marked. Figure 4 As shown in the figure, this diagram illustrates the vertical access sector traffic constraint density statistically obtained in spherical coordinates for six typical facilities (TWR-07, SUB-03, MIX-12, TWR-15, SUB-02, and MIX-05). (Blue bar) and horizontal near-ring zone traffic constraint density (Green bars), with red dashed lines indicating an airborne approach to the usable threshold of 0.2 and orange dashed lines indicating a ground-based approach to the usable threshold of 0.25. The graph clearly shows that TWR-07... =0.08 is lower than the air threshold, while =0.42, which is higher than the ground threshold, indicates that the vertical approach path to the facility is unobstructed, but the horizontal approach path is obstructed; therefore, it is classified as a high-altitude access facility; SUB-03 =0.35 is higher than the air threshold. A value of 0.10, below the ground threshold, indicates obstruction in the vertical direction but unobstructed access in the ground direction, classifying it as a ground-accessible facility; MIX-12's... =0.06、 A value of 0.15 is below the corresponding threshold, indicating that access conditions are met both in the air and on the ground, classifying it as a mixed access facility. The same logic applies to other facilities; the facility access method can be automatically classified by comparing the density value with the threshold.
[0086] In real-world power grid scenarios, transmission towers are typically erected on ridges or in open fields, with ample airspace directly above them and virtually no obstacles within the vertical access sector. However, the surrounding terrain often presents constraints such as shrubs, ditches, or slopes, resulting in a high density of passage constraints within the horizontal access loop. Such facilities will only be marked as aerial access support, thus classifying them as high-altitude access facilities, and will be assigned only to drones for inspection. Similarly, ground-based switchgear, low-voltage distribution boxes, and other equipment within substations are installed in enclosed buildings or walled courtyards. The airspace directly above them is obstructed by building rooftops, and there are significant structural constraints within the vertical access sector. However, the ground access is spacious, resulting in a low density of passage constraints within the horizontal access loop. Such facilities will only be marked as ground-based access support, thus classifying them as ground-based access facilities, and will be assigned only to ground robots for inspection. For composite facilities that integrate poles and ground-based auxiliary equipment, such as pole-mounted transformers with ground control boxes, the open airspace above them supports drone access, and the ground-side roads also provide conditions for robot passage. Both types of access support are marked, and the facility will be classified as a hybrid access facility. Subsequently, aerial inspection sub-tasks and ground inspection sub-tasks will be generated simultaneously, and a collaborative dependency relationship will be established.
[0087] Once the classification results are determined, the classification label for each facility, along with its spherical coordinate system parameters and radial distance threshold, is... Traffic constraint density in each sector and This information is also written into the facility attribute table for use in subsequent task generation phases. For hybrid access facilities, the available approach azimuth angle ranges for both the vertical access sector and the horizontal access ring also need to be recorded. This allows for the planning of optimal approach directions for both the UAV and the ground robot when generating sub-tasks, preventing the two types of agents from occupying mutually interfering spatial areas simultaneously. The entire classification process is executed for all facilities within the inspection area, ultimately forming a complete list of facility access methods, which serves as the input for multi-agent task allocation and joint execution sequence generation.
[0088] Based on the spatial structure and inspection requirements of hybrid access facilities, aerial inspection sub-tasks and ground inspection sub-tasks are generated, and the dependencies and data transfer interfaces between the sub-tasks are established, including:
[0089] The detection requirement description of hybrid proximity facilities is extracted from the facility distribution data, parsed into a pairing relationship between detection targets and detection accuracy, and an observable envelope surface that meets the detection accuracy requirements is constructed for each detection target.
[0090] Calculate the spatial intersection between the observable envelope and the access constraints in the environmental semantic map. Based on the spatial intersection, divide the observable envelope into an air-accessible observation area and a ground-accessible observation area. For detection targets whose observable envelope crosses the air-accessible observation area and the ground-accessible observation area, set a collaborative observation reference point on the boundary line between the air-accessible observation area and the ground-accessible observation area.
[0091] For detection targets that include collaborative observation reference points, a dependency relationship is established between corresponding aerial inspection sub-tasks and ground inspection sub-tasks. A data transmission interface is established at the collaborative observation reference point. When the aerial inspection sub-task reaches the observation position corresponding to the collaborative observation reference point, it transmits local feature data. The ground inspection sub-task determines the observation reference reference based on the local feature data and then performs complementary observations.
[0092] When extracting the inspection requirements description for hybrid proximity facilities from facility distribution data, the original description text needs to be structured and parsed, transforming it into a pairing relationship between several inspection targets and inspection accuracies. Each inspection target corresponds to a specific physical component or area, such as an insulator string, conductor connection point, tower weld, or transformer substation shell, while the inspection accuracy is quantified as the required spatial resolution or minimum identifiable defect size. Based on this, an observable envelope is constructed for each inspection target. The observable envelope is a set of three-dimensional surfaces surrounding the inspection target, where any point satisfies the following condition: the line of sight from that point can cover all critical surfaces of the corresponding inspection target with a resolution no less than the required inspection accuracy. When constructing the observable envelope, the sensor's focal length parameters, pixel resolution, and target surface normal vector distribution are comprehensively considered to ensure that the projection coverage of the target by each sampling point on the envelope meets the inspection accuracy requirements. For inspection targets with complex shapes, the observable envelope usually presents as an irregular closed surface, requiring approximation through a discretized set of sampling points.
[0093] Calculating the spatial intersection of the observable envelope and the access constraints in the environmental semantic map involves mapping discrete sampling points on the envelope to a voxel grid in the environmental semantic map one by one, and querying the access constraint type for each voxel location. Access constraint types include air-accessible areas, ground-accessible areas, and obstacle areas that are impassable in both. The set of all sampling points on the observable envelope located within air-accessible areas is defined as the air-accessible observation area, and the set of sampling points located within ground-accessible areas is defined as the ground-accessible observation area. For a given target, if its observable envelope simultaneously includes both air-accessible and ground-accessible observation areas, then the target is an object requiring air-ground collaborative observation. When setting up collaborative observation reference points on the boundary between the air-accessible observation area and the ground-accessible observation area, sampling is performed along the boundary at certain intervals. Points meeting the following conditions are selected as collaborative observation reference points: the point is located within the neighborhood of both the air-accessible observation area boundary and the ground-accessible observation area boundary, and the coverage of the target from the line of sight originating from the point reasonably overlaps with the coverage of adjacent air-accessible sampling points and adjacent ground-accessible sampling points to ensure sufficient feature correspondence during data transmission. Let the coordinates of the collaborative observation reference point be... Its corresponding voxel coordinates in the environmental semantic map are: This is used for precise positioning during subsequent path planning.
[0094] For targets containing collaborative observation reference points, a dependency relationship is established between corresponding airborne and ground-based inspection subtasks. The core semantics of this dependency relationship are: the ground-based inspection subtask must not enter the complementary observation phase of the corresponding target until the airborne inspection subtask completes its observation at the collaborative observation reference point and transmits local feature data. This dependency relationship is recorded as directed edges in the subtask dependency graph, with the starting point being the arrival event of the collaborative observation reference point for the airborne inspection subtask and the ending point being the initiation event of the complementary observation for the ground-based inspection subtask. At the collaborative observation reference point... The established data transmission interface defines the format and content of the transmitted data: the aerial inspection subtask occurs when the drone arrives... When the corresponding observation position is reached, local feature data corresponding to the overlapping part of the detected target in the current frame image or point cloud data is extracted. The local feature data includes 3D point cloud fragments of the visible surface of the detected target, texture feature descriptors, and the estimated pose of the target in the global coordinate system. ,in for The homogeneous transformation matrix describes the rotation and translation relationship between the local coordinate system of the detected target and the global coordinate system.
[0095] After receiving local feature data, the ground inspection subtask estimates the pose using the 3D point cloud fragments and the target. As an observation reference benchmark, complementary observations are performed. The specific process of complementary observation is as follows: the ground robot, based on... The system identifies surface areas of the target within the accessible ground observation zone that are not yet covered by the aerial viewpoint. It then plans ground observation paths capable of covering these areas and continuously performs online matching using received 3D point cloud fragments during execution to correct target localization errors caused by ground viewpoint obstruction or changes in lighting. Complementary observation data collected by the ground robot and local feature data transmitted from the aerial inspection subtask are fused through a data transmission interface. The fusion strategy is based on the target pose. After transforming both to the same coordinate system, point cloud stitching and feature fusion are performed to ultimately form a complete detection result covering all key surfaces of the target.
[0096] In actual power grid inspection scenarios, a typical example of hybrid accessibility facilities is the transmission tower: the tower top crossarms and insulator strings are usually only accessible to drones at a sufficiently close distance for high-precision photography, falling within the aerial observation zone; the tower base, grounding down conductors, and low-level markers are located within the ground observation zone, suitable for close-range inspection by ground robots; while certain structural components in the middle section of the tower have an observable envelope extending into both areas, requiring data connection through collaborative observation reference points. In this case, when the drone descends to the corresponding height in the middle section of the tower, it completes the collection and transmission of local feature data at the collaborative observation reference point. The ground robot then uses this data as a reference to perform complementary observations of the lower part of the tower in the area near the tower base, ensuring that the overall inspection coverage of the tower meets the requirements.
[0097] For mixed proximity facilities with a large number of detection targets, the observable envelopes of each target may spatially overlap, causing the collaborative observation reference points of different targets to be physically close to each other. In this case, for targets with a location distance less than a preset merging threshold... Multiple collaborative observation reference points are merged to reduce the number of times UAVs and ground robots need to stop. The merged collaborative observation reference points undertake the task of transmitting local feature data of multiple detection targets. The data transmission interface is correspondingly expanded to a multi-target parallel transmission format. The ground inspection sub-task parses the data one by one according to the target number and establishes the observation reference reference for each detection target before performing complementary observations. This merging mechanism effectively reduces the frequency of collaborative interactions and improves the overall inspection efficiency while ensuring the integrity of the detection.
[0098] Multi-agent collaborative decision-making is performed on each inspection task and sub-task. Sub-task dependencies are used as timing constraints, and state synchronization instructions are inserted at sub-task execution nodes to generate a joint execution sequence of UAV agents and ground robot agents, including:
[0099] Multi-agent collaborative decision-making is carried out for each inspection task and sub-task. The coordinates of the execution area of each inspection task and sub-task are extracted. The moving distance and time cost of the UAV agent and the ground robot agent to reach each execution area are calculated. A spatiotemporal reachability evaluation matrix is constructed. Based on the spatiotemporal reachability evaluation matrix, the inspection tasks and sub-tasks are assigned to the corresponding agents. A task execution node sequence containing the agent identifier and the planned execution time is generated.
[0100] The subtask dependencies are analyzed to extract the data flow path between subtasks. The cross-agent connection relationship between the predecessor subtask and the successor subtask belonging to different execution agents is identified in the data flow path. The end time of the task execution node corresponding to the predecessor subtask is increased by the data transmission delay and used as the start time of the task execution node corresponding to the successor subtask. This forms a time constraint and updates the task execution node sequence.
[0101] In the updated sequence of task execution nodes, locate the successor task execution node corresponding to the cross-agent connection relationship, insert a state synchronization instruction into the successor task execution node, and trigger the predecessor execution agent to transmit the position coordinates and detection results to the successor execution agent.
[0102] The task execution node sequence is grouped according to the execution agent identifier and sorted according to the start time to form a joint execution sequence.
[0103] When making multi-agent collaborative decisions for various inspection tasks and sub-tasks, it is necessary to extract the execution area coordinates corresponding to each inspection task and sub-task. The execution area coordinates are derived from the spatial location information of various facilities in the environmental semantic map. For aerial inspection tasks, the execution area coordinates are the three-dimensional coordinates of the hovering observation point or the start and end points of the flight path that the UAV agent needs to reach; for ground inspection tasks, the execution area coordinates are the two-dimensional coordinates of the inspection station that the ground robot agent needs to reach. After extracting the execution area coordinates, the required travel distance and time cost for both the UAV agent and the ground robot agent to reach each execution area from their current position or the end position of the previous task are calculated. The travel time cost of the UAV agent is calculated based on its maximum cruising speed and flight path length, while also adding the additional path increment caused by obstacle detours; the travel time cost of the ground robot agent is calculated based on its ground travel speed and the reachable path length of the road network, and the impact of terrain constraints on the path needs to be considered.
[0104] When constructing the spatiotemporal reachability assessment matrix, the agent ID is used as the row index and the task ID as the column index. Matrix elements record the comprehensive cost value for the corresponding agent to reach the corresponding task execution region. The comprehensive cost value is obtained by a weighted sum of the travel time cost and the execution time cost, where the execution time cost reflects the detection operation time required to complete the task. Assume the agent set contains... There are [number] intelligent agents, and the task set contains [number] items. For each task, the spatiotemporal reachability evaluation matrix is... The dimension is Matrix elements Indicates the first The agent executes the first... The overall cost of each task, of which , Based on the spatiotemporal reachability assessment matrix, an allocation strategy minimizing total cost is adopted. Aerial inspection tasks are prioritized for allocation to UAV agents, and ground inspection tasks are prioritized for allocation to ground robot agents. For aerial and ground inspection sub-tasks derived from hybrid accessibility facilities, agents of the corresponding type are matched according to the task type. After allocation, a task execution node containing the agent identifier and planned execution time is generated for each task, and all task execution nodes are organized in chronological order to form an initial task execution node sequence.
[0105] After obtaining the initial task execution node sequence, it is necessary to parse the subtask dependencies to extract the data flow paths between subtasks. The data flow path describes the logical link through which detection data generated by the preceding subtask is passed to the subsequent subtask. For aerial and ground inspection subtasks of hybrid access facilities, a typical dependency exists: image data of the facility's top or high-altitude areas acquired by the aerial inspection subtask needs to be passed to the ground robot agent before the ground inspection subtask is executed, so that the ground robot agent can adjust the ground inspection focus area based on the existing aerial detection results. In the data flow path, the cases where the preceding and subsequent subtasks belong to different executing agents are identified, and such relationships are marked as cross-agent connections. For each cross-agent connection, the planned end time of the corresponding task execution node of the preceding subtask is taken, and the data transmission delay is added to this time. The resulting time is taken as the earliest allowed start time of the corresponding task execution node of the subsequent subtask. The data transmission delay is estimated based on the amount of data to be transmitted and the actual bandwidth of the communication link between the agents. If the agent is in a communication dead zone, additional delays introduced by relay forwarding are also required. The above timing constraints are written into the task execution node sequence. The planned start times of subsequent task execution nodes affected by the constraints are updated accordingly. Downstream task execution nodes that experience cascading delays due to time updates are recursively adjusted until the entire sequence achieves timing consistency, forming the updated task execution node sequence.
[0106] In the updated task execution node sequence, the positions of all subsequent task execution nodes corresponding to cross-agent connections are located, and state synchronization instructions are inserted at these nodes. The state synchronization instructions include trigger conditions, the identifier of the data sender agent, the identifier of the data receiver agent, and the data type and format specifications to be transmitted. The trigger condition is set to activate immediately after the completion signal of the predecessor task execution node arrives at the receiver. The data type covers the real-time position coordinates of the predecessor agent when completing the predecessor sub-task and the local detection results generated by that sub-task. The position coordinates are transmitted in three-dimensional coordinates in the global coordinate system, and the detection results are transmitted in structured data packets containing a detection timestamp, detection object identifier, detection index value, and confidence score. After inserting the state synchronization instructions, each cross-agent successor node in the task execution node sequence has explicit data reception waiting logic, ensuring that the successor agent has obtained all the necessary information transmitted by the predecessor agent before starting the successor sub-task.
[0107] After the state synchronization instructions are inserted, the complete sequence of task execution nodes is grouped according to the agent identifiers, with each group corresponding to all the nodes to be executed for one agent. Within each group, the task execution nodes are sorted from earliest to latest according to their planned start times, forming an ordered execution list for that agent. The ordered execution lists of all agents are then aggregated to form the final joint execution sequence. In the joint execution sequence, the execution list of the UAV agent includes all its aerial inspection task nodes and aerial inspection sub-task nodes, as well as the corresponding state synchronization instruction nodes; the execution list of the ground robot agent includes all its ground inspection task nodes and ground inspection sub-task nodes, as well as the corresponding state synchronization instruction nodes. The joint execution sequence fully records the action plan of each agent throughout the entire inspection process, including the start time, end time, execution area coordinates, required sensor configuration, and interaction times with other agents for each task node. This provides a complete scheduling basis for subsequent sequential execution of tasks by each agent and dynamic adjustment of paths and timings during operation.
[0108] Based on the spatiotemporal reachability evaluation matrix, inspection tasks and subtasks are assigned to corresponding agents, generating a sequence of task execution nodes containing the agent identifier and the planned execution time, including:
[0109] Extract the numerical variance and range of each row of the executing agent in the spatiotemporal reachability assessment matrix, and calculate the degree of adaptability difference of the executing agent to different inspection tasks and sub-tasks;
[0110] Select the agents with the greatest degree of adaptability difference for task allocation. Assign the inspection tasks and sub-tasks identified by the column with the smallest value in the corresponding row to the selected agents. Remove the columns corresponding to the assigned inspection tasks and sub-tasks from the spatiotemporal reachability evaluation matrix. Recalculate the degree of adaptability difference for the remaining agents and continue to allocate tasks until all inspection tasks and sub-tasks are allocated.
[0111] The coordinates of the execution areas corresponding to the inspection tasks and subtasks assigned to each execution agent are counted. The movement path of the execution agent to traverse all execution areas is planned. The execution order of the inspection tasks and subtasks is determined according to the access order of the execution areas in the movement path.
[0112] For each inspection task and subtask in the execution sequence, a task execution node is generated. The task execution node contains the assigned execution agent identifier and the planned execution time calculated based on the cumulative movement time of the preceding position in the movement path. The task execution nodes are arranged in the execution sequence to form a task execution node sequence.
[0113] After constructing the spatiotemporal reachability assessment matrix, the cost information contained in the matrix needs to be transformed into specific task allocation schemes. For each row in the matrix (corresponding to an executing agent), the variance and range of all values in that row are extracted as indicators to measure the degree of adaptability difference. Variance reflects the dispersion of the agent's cost assessment of different tasks, while range directly reflects the cost span between the best and worst tasks. If the overall cost difference of an agent for different tasks is significant, i.e., both variance and range are large, it indicates that the agent has a strong preference for tasks, with a clear distinction between "good at tasks" and "bad at tasks"; if both variance and range are small, it indicates that the agent's execution cost for each task is relatively balanced, and arbitrary allocation will not cause significant efficiency loss.
[0114] The degree of adaptive variability can be calculated by normalizing the variance and range, then weighting and fusing them to obtain a comprehensive variability score for each agent. ,in To implement the agent's number index. Specifically, let the number index be... The variance of the row values corresponding to each executing agent is The range is The normalized overall difference score is ,in and These are the normalized values of variance and range across all agents, respectively. and The weighting coefficients are satisfied. The greater the degree of adaptive difference in the executing agent, the greater the impact of its allocation decision on the overall efficiency. Therefore, prioritizing task allocation to such agents can effectively prevent high-preference agents from being forced to perform extremely costly tasks due to improper allocation order.
[0115] The task allocation execution logic follows a greedy iterative strategy. In each iteration, a comprehensive difference score is selected from the set of agents currently without assigned tasks. The highest-performing agent finds the column with the smallest value in the corresponding row of the spatiotemporal reachability evaluation matrix and assigns the inspection task or subtask identified by that column to that agent. After one assignment, the column corresponding to the assigned task is removed from the matrix, and the agent that has completed the assignment is marked from the candidate set of the current round (if a strategy of assigning one task per agent per round is adopted) or retained (if a single agent is allowed to undertake multiple tasks). After removing the column, the variance and range of each row are recalculated for the remaining executing agents, the comprehensive difference score is updated, and the next round of iteration begins, until all inspection tasks and subtasks have been assigned. This dynamic recalculation method ensures that each round of assignment decisions is based on the latest state of the current task set, avoiding the problem of static scores becoming invalid after changes in the task set.
[0116] After task allocation, the coordinates of the execution areas corresponding to all inspection tasks and subtasks undertaken by each agent are statistically analyzed. These execution area coordinates originate from the spatial location information generated for each facility or subtask during the task definition phase, including the 3D aerial approach point coordinates for UAV agents and the 2D ground approach point coordinates for ground robot agents. For UAV agents, the set of execution area coordinates constitutes a discrete set of points in 3D space, requiring the planning of a movement path traversing all execution areas while considering flight altitude constraints and obstacle avoidance. For ground robot agents, the set of execution area coordinates constitutes a discrete set of points in a 2D plane, requiring the planning of a ground movement path under traffic constraints. Path planning can employ a graph search-based method, using the actual traversable distance or estimated travel time between execution areas as edge weights. The traversal order is obtained by approximating the traveling salesman problem, thereby determining the execution order of each inspection task and subtask.
[0117] After path planning is completed, a timeline is established for the task sequence of each agent based on the order in which the execution areas are visited along the movement path. Using the agent's departure time as a baseline, the movement time is accumulated segment by segment along the path to obtain the estimated arrival time for each execution area. The calculation of movement time comprehensively considers the segment distance, the agent's rated speed, and the environmental drag coefficient; for UAV agents, the time consumed by ascent and descent also needs to be added. Let the... The executing agent visits the path along the first... The cumulative movement time for each execution region is Then the planned execution time of the task corresponding to that execution region. From the departure reference time and The result of superposition is that If there are nodes in the path where the preceding task needs to wait for the cooperating agent to complete state synchronization, a waiting time buffer is introduced at that node, so that the planned execution time of subsequent tasks is postponed accordingly, in order to ensure that the subtask dependencies are satisfied in the time dimension.
[0118] The generation of task execution nodes is based on the aforementioned time estimation results. Each task execution node contains three basic elements: first, the assigned execution agent identifier, used to clarify the responsible party in the joint execution sequence; second, the planned execution time. The first part is used to coordinate the concurrent behavior of multiple agents in the temporal dimension; the second part is the descriptive information of the task itself, including the target facility identifier, the approach method type, and the detection content requirements. All task execution nodes of the same agent are arranged in the order of execution to form the task execution node sequence of that agent; merging the task execution node sequences of all agents constitutes a complete joint task execution node sequence.
[0119] Regarding the handling of subtask dependencies, when there is a data transfer dependency between the aerial inspection subtask and the ground inspection subtask of a hybrid proximity facility, the task execution nodes of the UAV agent responsible for the aerial subtask and the ground robot agent responsible for the ground subtask need to be aligned and verified in terms of timing. Specifically, if the ground subtask depends on the detection results of the aerial subtask, the planned execution time of the corresponding node of the ground subtask must be later than the sum of the planned execution time of the aerial subtask node and the estimated detection time. Otherwise, the path planning or departure time of the ground robot agent needs to be adjusted until the timing constraints are met. After dependency verification and timing alignment, each node in the task execution node sequence is executable, providing a complete scheduling basis for subsequent agents to execute tasks in sequence, exchange information at the state synchronization command trigger node, and dynamically adjust the path.
[0120] Each agent executes tasks according to the joint execution sequence, exchanges position and progress information at the state synchronization instruction trigger node, and adjusts subsequent execution paths and timing accordingly. Local detection information is exchanged and fused through the data transmission interface to form inspection results, including:
[0121] When each agent reaches the state synchronization instruction trigger node, it compares its current position coordinates and task execution progress with the expected position coordinates and expected progress recorded in the joint execution sequence, and generates a deviation report containing position deviation and progress deviation.
[0122] Each agent exchanges deviation reports through the data transmission interface, identifies agents whose deviations exceed a preset range as delayed agents, and extracts the unfinished task identifiers of the delayed agents.
[0123] According to the unfinished task identifier, query the task dependency relationship in the joint execution sequence. If the task to be executed depends on the unfinished task of the delayed agent, the planned execution time is pushed back. Based on the pushed planned execution time, the movement path to the next execution area is replanned, and the subsequent execution process is adjusted according to the replanned movement path.
[0124] Each intelligent agent collects detection data of the target object in the execution area as local detection information. After adding the collection location coordinates to the local detection information, it sends it to the fusion node through the data transmission interface. The fusion node performs spatial clustering of the local detection information according to the collection location coordinates and merges the local detection information in the same spatial cluster to form the inspection result.
[0125] Upon reaching the state synchronization command trigger node, each agent extracts its current position coordinates and task execution progress, comparing them item by item with the pre-recorded expected position coordinates and expected progress in the joint execution sequence. Position deviation is calculated by the Euclidean distance between the current actual position and the expected position, while progress deviation is calculated by the difference between the number of completed tasks and the planned number of tasks. These two types of deviations together constitute a deviation report, which also includes the trigger node's identifier and timestamp, enabling the fusion node to accurately pinpoint the spatiotemporal location of the deviation during subsequent processing. The generation of the deviation report is automatic, without human intervention, and is completed by each agent upon reaching the trigger node, ensuring the real-time nature and consistency of the deviation information.
[0126] After each agent generates a deviation report, it broadcasts the report to all participating agents in the group via a data transmission interface. Upon receiving the deviation report, the position and progress deviations of each agent are compared with preset deviation tolerance ranges. Agents whose position deviations exceed the position tolerance threshold or whose progress deviations exceed the progress tolerance threshold are identified as delayed agents. For individuals identified as delayed agents, a list of their currently incomplete task identifiers is extracted and recorded for subsequent task dependency queries. The process of identifying delayed agents is re-executed at each trigger node, thus dynamically reflecting changes in the overall execution status of the group and preventing subsequent task collaboration failures due to early deviation accumulation.
[0127] After obtaining the incomplete task identifiers of delayed agents, each non-delayed agent queries task dependencies in the joint execution sequence based on these identifiers. If an agent's pending task has a dependency edge on an incomplete task of a delayed agent in the dependency graph, that agent needs to adjust the timing of its subsequent execution plan. Specifically, the planned execution time of the affected task is shifted backward, and the shift amount is determined based on the difference between the estimated completion time and the original planned completion time estimated by the delayed agent's progress deviation. Let the estimated additional time corresponding to the progress deviation of the delayed agent be... Then the planned execution time of the task to be executed under dependency constraints will be adjusted to the original planned execution time plus... With a buffer margin The sum, i.e., the adjusted planned execution time. satisfy ,in The originally scheduled execution time, The reserved collaborative buffer time is used to absorb any additional delays that may occur during the path replanning process.
[0128] Based on the rescheduled planned execution time, each affected agent replans its movement path from its current position to the next execution area. This replanning is based on the environmental semantic map, combined with the real-time location information of each agent, and employs a path search method under time window constraints to ensure that agents can reach the target execution area before the adjusted planned execution time. If the delay is too large, causing the path to no longer meet the time constraints after the rescheduled time, a task reallocation process is triggered, transferring the affected tasks to other agents with spatiotemporal reachability. After replanning, the execution sequence stored locally by each agent is updated, and subsequent execution proceeds according to the updated sequence, ensuring that the overall collaborative execution of the group does not suffer from long waiting times or task gaps due to local delays.
[0129] Within the execution area, each agent collects detection data from the target object. This data includes image data, temperature data, vibration data, and other detection items determined based on the facility type. The collected data is stored as local detection information in the agent's local cache. To ensure the accuracy of subsequent spatial fusion, each piece of local detection information is appended with the agent's position coordinates at the time of acquisition during storage. These coordinates are acquired in real-time by an airborne or vehicle-mounted positioning module, with accuracy meeting the resolution requirements of spatial clustering. After appending the position coordinates, the local detection information is sent to the fusion node in the form of structured data packets via a data transmission interface. The fusion node can be a specific agent within the group or an independently deployed ground station service process; its responsibility is to aggregate local detection information from all agents and perform fusion calculations.
[0130] After receiving local detection information from each agent, the fusion node performs spatial clustering using the collected location coordinates as clustering features. During clustering, the spatial distance between location coordinates is used as a similarity metric, and clusters with spatial distances less than a threshold cluster radius are selected. Local detection information is grouped into the same spatial cluster. The formation of spatial clusters reflects the overlapping observations of multiple agents on the same target object or the same spatial region. When fusing local detection information within the same spatial cluster, image data is generated by multi-view stitching or weighted overlay to produce a high-coverage target image. Numerical data (such as temperature and vibration amplitude) is calculated using a weighted average method. The weights are determined based on the acquisition distance and sensor confidence level of each piece of information. Detection information with smaller acquisition distances and higher sensor confidence levels has a greater weight in the fusion process, thereby suppressing the interference of long-distance or low-quality acquisition data on the fusion result.
[0131] After fusion, each spatial cluster generates a corresponding inspection result record. This record includes the spatial location of the target object, the fused detection value, a list of agent IDs covering the target, and the acquisition time of each agent. The inspection result records from all spatial clusters are aggregated to form a complete inspection result set. This set is stored in a structured format and can be accessed by downstream processing flows such as anomaly analysis and report generation. For isolated local detection information not covered by any spatial cluster, the fusion node retains it separately and marks it as a low-confidence record, prompting for subsequent manual review or triggering a supplementary inspection task. The entire fusion process is continuously executed in a streaming manner at the fusion node, without waiting for all agents to complete all tasks before unified processing, thereby reducing the latency of inspection result generation and supporting rapid response to anomalies.
[0132] A second aspect of this invention provides a multi-agent large-model collaborative UAV and robot power grid group inspection system, comprising:
[0133] The map building unit is used to collect geographical environment data and facility distribution data of the power grid inspection area, and to build an environmental semantic map containing obstacle information and passage constraints based on the geographical environment data.
[0134] The facility classification unit is used to identify the access method of each facility based on the environmental semantic map and facility distribution data, and classify the facilities into high-altitude access facilities, ground access facilities and mixed access facilities.
[0135] The task generation unit is used to generate aerial inspection tasks and ground inspection tasks for high-altitude access facilities and ground access facilities, respectively. Based on the spatial structure and inspection requirements of hybrid access facilities, it generates aerial inspection sub-tasks and ground inspection sub-tasks, and establishes the dependencies and data transmission interfaces between sub-tasks.
[0136] The collaborative decision-making unit is used to make multi-agent collaborative decisions on each inspection task and sub-task, taking the sub-task dependency relationship as a timing constraint and inserting state synchronization instructions at the sub-task execution node to generate a joint execution sequence of UAV agent and ground robot agent.
[0137] The execution fusion unit is used by each intelligent agent to execute tasks according to the joint execution sequence, exchange position information and progress information at the state synchronization instruction trigger node and adjust the subsequent execution path and timing accordingly, and exchange local detection information through the data transmission interface to form inspection results.
[0138] A third aspect of the present invention provides an electronic device, comprising:
[0139] processor;
[0140] Memory used to store processor-executable instructions;
[0141] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.
[0142] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.
[0143] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.
[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for multi-agent large-scale model collaborative inspection of power grids using drones and robots, characterized in that, include: Collect geographical environment data and facility distribution data of the power grid inspection area, and construct an environmental semantic map containing obstacle information and access constraints based on the geographical environment data; Based on the environmental semantic map and facility distribution data, the access method of each facility is identified, and the facilities are classified into high-altitude access facilities, ground access facilities, and mixed access facilities. For high-altitude access facilities and ground access facilities, aerial inspection tasks and ground inspection tasks are generated respectively. Based on the spatial structure and inspection requirements of hybrid access facilities, aerial inspection sub-tasks and ground inspection sub-tasks are generated, and the dependencies and data transmission interfaces between sub-tasks are established. Multi-agent collaborative decision-making is carried out for each inspection task and sub-task. The sub-task dependency is used as a timing constraint and state synchronization instructions are inserted at the sub-task execution node to generate a joint execution sequence of UAV agent and ground robot agent. Each intelligent agent executes tasks according to the joint execution sequence. At the state synchronization instruction trigger node, it exchanges position information and progress information and adjusts the subsequent execution path and timing accordingly. It also exchanges local detection information through the data transmission interface to form the inspection result.
2. The method according to claim 1, characterized in that, Collect geographical environment data and facility distribution data of the power grid inspection area, and construct an environmental semantic map containing obstacle information and access constraints based on the geographical environment data, including: Collect geographical environment data and facility distribution data of the power grid inspection area, perform spatial interpolation to complete the geographical environment data, generate a continuous three-dimensional spatial field expression, and extract the gradient vector field from the three-dimensional spatial field expression; Divergence and curl are calculated on the gradient vector field. The source points of spatial density change are identified based on the divergence value, and the rotation center of the spatial structure is identified based on the curl value. The source points of density change and the rotation center are determined as feature anchor points of the obstacle. Region growth is performed using the feature anchor points as seed points. Growth stops when the magnitude of the gradient vector at the growth boundary is lower than the convergence threshold. The spatial region after growth is determined as a complete obstacle entity and its boundary envelope is extracted. Based on the facility distribution data, calculate the spatial occlusion relationship between obstacle entities and facilities, determine the type of approach path restriction caused by the occlusion relationship, and generate corresponding detour passage constraints or height adjustment passage constraints for different restriction types. The obstacle boundary envelope, spatial occlusion relationship, and generated passage constraints are semantically associated and encoded to construct an environmental semantic map containing obstacle information and passage constraints.
3. The method according to claim 1, characterized in that, Based on the environmental semantic map and facility distribution data, the access method of each facility is identified, and the facilities are classified into high-altitude access facilities, ground access facilities, and mixed access facilities, including: Extract the distribution of access constraints around each facility from the environmental semantic map, and map the access constraints to a spherical coordinate system centered on each facility according to their spatial location. The radial direction represents the distance to the facility, the polar angle represents the vertical angle, and the azimuth angle represents the horizontal angle. In the spherical coordinate system, a vertical approach sector and a horizontal approach ring are defined based on the polar angle range. The vertical approach sector covers the airspace directly above the facility, and the horizontal approach ring covers the ground area around the facility. The traffic constraint density within the vertical approach sector and the horizontal approach ring is calculated respectively. The minimum approach space size of each facility is read from the facility distribution data. The minimum approach space size is converted into a radial distance threshold in a spherical coordinate system. It is determined whether there is a passage constraint within the radial distance threshold range of the vertical approach sector and the air approach support is marked. It is determined whether there is a passage constraint within the radial distance threshold range of the horizontal approach ring and the ground approach support is marked. Based on the marking results, facilities are classified as follows: those marked only with aerial access support are classified as high-altitude access facilities; those marked only with ground access support are classified as ground access facilities; and those marked with both types of access support are classified as hybrid access facilities.
4. The method according to claim 1, characterized in that, Based on the spatial structure and inspection requirements of hybrid access facilities, aerial inspection sub-tasks and ground inspection sub-tasks are generated, and the dependencies and data transfer interfaces between the sub-tasks are established, including: The detection requirement description of hybrid proximity facilities is extracted from the facility distribution data, parsed into a pairing relationship between detection targets and detection accuracy, and an observable envelope surface that meets the detection accuracy requirements is constructed for each detection target. Calculate the spatial intersection between the observable envelope and the access constraints in the environmental semantic map. Based on the spatial intersection, divide the observable envelope into an air-accessible observation area and a ground-accessible observation area. For detection targets whose observable envelope crosses the air-accessible observation area and the ground-accessible observation area, set a collaborative observation reference point on the boundary line between the air-accessible observation area and the ground-accessible observation area. For detection targets that include collaborative observation reference points, a dependency relationship is established between corresponding aerial inspection sub-tasks and ground inspection sub-tasks. A data transmission interface is established at the collaborative observation reference point. When the aerial inspection sub-task reaches the observation position corresponding to the collaborative observation reference point, it transmits local feature data. The ground inspection sub-task determines the observation reference reference based on the local feature data and then performs complementary observations.
5. The method according to claim 1, characterized in that, Multi-agent collaborative decision-making is performed on each inspection task and sub-task. Sub-task dependencies are used as timing constraints, and state synchronization instructions are inserted at sub-task execution nodes to generate a joint execution sequence of UAV agents and ground robot agents, including: Multi-agent collaborative decision-making is carried out for each inspection task and sub-task. The coordinates of the execution area of each inspection task and sub-task are extracted. The moving distance and time cost of the UAV agent and the ground robot agent to reach each execution area are calculated. A spatiotemporal reachability evaluation matrix is constructed. Based on the spatiotemporal reachability evaluation matrix, the inspection tasks and sub-tasks are assigned to the corresponding agents. A task execution node sequence containing the agent identifier and the planned execution time is generated. The subtask dependencies are analyzed to extract the data flow path between subtasks. The cross-agent connection relationship between the predecessor subtask and the successor subtask belonging to different execution agents is identified in the data flow path. The end time of the task execution node corresponding to the predecessor subtask is increased by the data transmission delay and used as the start time of the task execution node corresponding to the successor subtask. This forms a time constraint and updates the task execution node sequence. In the updated task execution node sequence, locate the successor task execution node corresponding to the cross-agent connection relationship, insert a state synchronization instruction into the successor task execution node, and trigger the predecessor execution agent to transmit the position coordinates and detection results to the successor execution agent; group the task execution node sequence according to the execution agent identifier and sort it according to the start time to form a joint execution sequence.
6. The method according to claim 5, characterized in that, Based on the spatiotemporal reachability evaluation matrix, inspection tasks and subtasks are assigned to corresponding agents, generating a sequence of task execution nodes containing the agent identifier and the planned execution time, including: Extract the numerical variance and range of each row of the executing agent in the spatiotemporal reachability assessment matrix, and calculate the degree of adaptability difference of the executing agent to different inspection tasks and sub-tasks; Select the agents with the greatest degree of adaptability difference for task allocation. Assign the inspection tasks and sub-tasks identified by the column with the smallest value in the corresponding row to the selected agents. Remove the columns corresponding to the assigned inspection tasks and sub-tasks from the spatiotemporal reachability evaluation matrix. Recalculate the degree of adaptability difference for the remaining agents and continue to allocate tasks until all inspection tasks and sub-tasks are allocated. The coordinates of the execution areas corresponding to the inspection tasks and subtasks assigned to each execution agent are counted. The movement path of the execution agent to traverse all execution areas is planned. The execution order of the inspection tasks and subtasks is determined according to the access order of the execution areas in the movement path. For each inspection task and subtask in the execution sequence, a task execution node is generated. The task execution node contains the assigned execution agent identifier and the planned execution time calculated based on the cumulative movement time of the preceding position in the movement path. The task execution nodes are arranged in the execution sequence to form a task execution node sequence.
7. The method according to claim 1, characterized in that, Each agent executes tasks according to the joint execution sequence, exchanges position and progress information at the state synchronization instruction trigger node, and adjusts subsequent execution paths and timing accordingly. Local detection information is exchanged and fused through the data transmission interface to form inspection results, including: When each agent reaches the state synchronization instruction trigger node, it compares its current position coordinates and task execution progress with the expected position coordinates and expected progress recorded in the joint execution sequence, and generates a deviation report containing position deviation and progress deviation. Each agent exchanges deviation reports through the data transmission interface, identifies agents whose deviations exceed a preset range as delayed agents, and extracts the unfinished task identifiers of the delayed agents. According to the unfinished task identifier, query the task dependency relationship in the joint execution sequence. If the task to be executed depends on the unfinished task of the delayed agent, the planned execution time is pushed back. Based on the pushed planned execution time, the movement path to the next execution area is replanned, and the subsequent execution process is adjusted according to the replanned movement path. Each intelligent agent collects detection data of the target object in the execution area as local detection information. After adding the collection location coordinates to the local detection information, it sends it to the fusion node through the data transmission interface. The fusion node performs spatial clustering of the local detection information according to the collection location coordinates and merges the local detection information in the same spatial cluster to form the inspection result.
8. A multi-agent large-model collaborative UAV and robot power grid group inspection system, used to implement the method as described in any one of claims 1-7, characterized in that, include: The map building unit is used to collect geographical environment data and facility distribution data of the power grid inspection area, and to build an environmental semantic map containing obstacle information and passage constraints based on the geographical environment data. The facility classification unit is used to identify the access method of each facility based on the environmental semantic map and facility distribution data, and classify the facilities into high-altitude access facilities, ground access facilities and mixed access facilities. The task generation unit is used to generate aerial inspection tasks and ground inspection tasks for high-altitude access facilities and ground access facilities, respectively. Based on the spatial structure and inspection requirements of hybrid access facilities, it generates aerial inspection sub-tasks and ground inspection sub-tasks, and establishes the dependencies and data transmission interfaces between sub-tasks. The collaborative decision-making unit is used to make multi-agent collaborative decisions on each inspection task and sub-task, taking the sub-task dependency relationship as a timing constraint and inserting state synchronization instructions at the sub-task execution node to generate a joint execution sequence of UAV agent and ground robot agent. The execution fusion unit is used by each intelligent agent to execute tasks according to the joint execution sequence, exchange position information and progress information at the state synchronization instruction trigger node and adjust the subsequent execution path and timing accordingly, and exchange local detection information through the data transmission interface to form inspection results.
9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.
