Power station heterogeneous intelligent agent cooperative cruise system and method based on digital twinning

Abstract

This invention discloses a collaborative patrol system and method for heterogeneous intelligent agents in substations based on digital twins, belonging to the field of intelligent substation inspection technology. The system includes a point cloud database module, a visual inspection and spatial mapping module, a spatial positioning and registration module, a heterogeneous trajectory planning module, and a digital twin simulation module connected sequentially. This invention achieves accurate mapping from two-dimensional detection results to three-dimensional point clouds through ray tracing and inverse projection. It overcomes the shortcomings of traditional point cloud registration, which is prone to getting trapped in local optima, by employing a multi-hypothesis concurrent FPFH+ICP registration algorithm. Differentiated trajectory planning strategies are customized for UAVs and robotic sensors respectively, and a closed-loop architecture from planning to simulation and then to data feedback is constructed based on UE5. This invention significantly improves the positioning accuracy and inspection automation level of substation equipment, effectively reduces operational safety risks, and can be widely applied to unmanned collaborative inspection scenarios in substations.

Description

Technical Field

[0001] This invention relates to the fields of intelligent power plant inspection, digital twins, 3D point cloud processing, computer vision and robot trajectory planning, and particularly to the field of system technology for unmanned autonomous inspection operations applicable to power scenarios such as substations and converter stations. Specifically, it relates to a collaborative patrol system and method for heterogeneous intelligent agents in power plants based on digital twins. Background Technology

[0002] With the deepening of the intelligent transformation of power systems, substation equipment inspection has gradually evolved from the traditional manual point-to-point inspection mode to robotic autonomous inspection. Current inspection solutions in the industry mainly suffer from the following technical shortcomings: First, most existing inspection solutions rely on pure two-dimensional visual recognition, which can only acquire the bounding box of the target within the image plane and cannot directly output the three-dimensional spatial coordinates and orientation information of the equipment, making it difficult to support the pose requirements of high-precision autonomous inspection; some solutions using depth cameras are greatly affected by ambient lighting and equipment texture, resulting in poor stability in outdoor substation scenarios. Second, some localization solutions using 3D point cloud registration generally employ the Iterative Closest Point (ICP) algorithm with a single initial pose, which is extremely sensitive to the initial pose. In complex scenarios with dense substation equipment and numerous point cloud noise, it is prone to getting trapped in local optima, and the registration success rate and positioning accuracy are difficult to meet engineering requirements. Furthermore, current drone and ground robot inspection systems are mostly independent, each planning its own flight path and executing its own tasks, making collaborative air-ground operations impossible. Flight path planning also relies heavily on manual point-by-point annotation, resulting in low efficiency, poor consistency, and a high dependence on operator experience. Finally, existing flight path planning is often directly deployed after completion, lacking a pre-implementation digital twin simulation verification step. This prevents the early detection of collision risks, perspective deviations, and other issues, posing safety hazards such as drone crashes and equipment collisions. Additionally, operational data cannot be used to optimize the recognition model, hindering the system's ability to iterate autonomously. Based on these existing technical problems, a more compatible inspection and patrol system is urgently needed. Summary of the Invention

[0003] To address the shortcomings mentioned in the background technology, this application provides a digital twin-based heterogeneous intelligent agent collaborative patrol system and method for power plants. This system solves one of the technical problems in existing power plant inspections: low 3D positioning accuracy of equipment, easy getting stuck in local optima during point cloud registration, inability of air-to-ground robots to coordinate planning, high safety risks during actual flight, and inability of the system to self-optimize. This system achieves automation, high precision, and high safety throughout the entire power plant inspection process.

[0004] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0005] This invention provides a power plant heterogeneous intelligent agent collaborative cruise system based on digital twins, comprising a point cloud database module, a visual detection and spatial mapping module, a spatial positioning and registration module, a heterogeneous trajectory planning module, and a digital twin simulation module that are connected in sequence.

[0006] The point cloud database module is used to construct a 3D point cloud map of the power plant with color information, complete the interactive cropping, posture standardization and marker point configuration of single device point clouds, and output a standardized device point cloud model and marker point data containing position and normal vector information.

[0007] The visual detection and spatial mapping module is used to automatically render the 3D map from multiple perspectives to generate image samples, identify power station equipment through the target detection model, and map the 2D detection ROI area into a 3D spatial ray set through ray tracing and inverse projection to obtain the coarse positioning point cloud area of ​​the equipment.

[0008] The spatial positioning and registration module is used to perform random sampling and consistent clustering of the detection frustums of similar devices to eliminate erroneous detection results and extract the precise ROI range. It also calculates the precise three-dimensional pose of the device through a multi-hypothesis concurrent point cloud registration algorithm and converts the local coordinates into general geographic coordinates.

[0009] The heterogeneous trajectory planning module includes a UAV trajectory planning unit and a mechanical dog trajectory planning unit, which generate executable inspection trajectories with gimbal action commands based on the motion characteristics of airborne and land-based inspection vehicles, respectively.

[0010] The digital twin simulation module loads inspection tracks on the UE5 platform to perform safety distance verification and simulated inspection, and supports feeding back the simulated and rendered device images to the visual inspection and spatial mapping module to form a self-optimizing closed loop of the system.

[0011] Correspondingly, the present invention also provides a method for cooperative navigation of heterogeneous intelligent agents in power plants based on digital twins, comprising the following steps:

[0012] S1. Construct a 3D point cloud map of the power plant with color information, complete the equipment point cloud clipping, attitude standardization and marker point configuration, and output a standardized equipment point cloud model and marker point data containing position and normal vectors;

[0013] S2. Automatically render the 3D map from multiple perspectives to generate images, complete device detection through the target detection model, and map the 2D detected ROI area into a 3D spatial ray set through ray tracing and inverse projection to obtain the device coarse positioning point cloud area.

[0014] S3. Perform RANSAC clustering on the detection frustums of similar devices to extract the precise ROI range, solve the precise 3D pose of the device through a multi-hypothesis concurrent point cloud registration algorithm, and convert it into general geographic coordinates;

[0015] S4. Generate inspection tracks with action commands for both drones and robotic dogs. The drone tracks support 3D obstacle avoidance and multi-drone relay scheduling, while the robotic dog tracks support traversable area mapping and real-time gimbal attitude calculation.

[0016] S5. Import the planned flight path into the UE5 digital twin environment for safety distance verification and simulated inspection. The simulated and rendered equipment images can be fed back to the target detection model to achieve system self-optimization closed loop.

[0017] Beneficial effects:

[0018] 1. This invention uses RANSAC clustering of view cones to initially screen effective detection results, and then combines this with a multi-hypothesis concurrent fast point feature histogram ICP registration scheme, effectively overcoming the shortcomings of traditional single-point cloud registration, which is prone to getting trapped in local optima. Real-world test data shows that in real power plant scenarios, the registration success rate for devices such as surge arresters, circuit breakers, and voltage transformers can reach 100%, and the mean square error of positioning can be controlled within 10cm, far superior to traditional single initial attitude registration schemes.

[0019] 2. It adopts ray tracing and inverse projection to achieve accurate mapping from 2D detection results to 3D point cloud. It does not rely on additional depth sensors. It can quickly extract the coarse positioning point cloud area of ​​the device through geometric calculations. It has low computational load, high efficiency, and is compatible with various point cloud map formats.

[0020] 3. Customized trajectory planning strategies were developed for the motion characteristics of both drones and robotic dogs. The drone side implemented K-tree 3D obstacle avoidance, multi-drone relay scheduling, and graded speed control, while the robotic dog side implemented high-precision mapping of passable areas and real-time gimbal calculations, forming a unified air-ground inspection planning system that significantly improves inspection efficiency and coverage.

[0021] 4. Based on UE5, a pre-simulation verification stage is built, which can complete the flight path safety verification and shooting effect preview before the actual flight, avoiding collision safety risks from the source; at the same time, the simulation rendering data can feed back to the incremental training of the target detection model, realize the self-iterative improvement of system performance, and form a complete closed loop of data acquisition, flight path planning, simulation verification and data feedback. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall system architecture and entire process of the present invention;

[0024] Figure 2 A schematic diagram illustrating the automatic alignment principle of 2D principal component analysis for device point cloud models [The right image shows the model before automatic rotation, and the left image shows the model after automatic rotation; the blue axis is the z-axis, the red axis is the x-axis, and the green axis is the y-axis].

[0025] Figure 3 A schematic diagram of automatic estimation and fine-tuning of the normal vector for the marked point [the top is before fine-tuning, the bottom is after fine-tuning, white represents the marked point, and yellow line represents the initial normal vector];

[0026] Figure 4 This is a schematic diagram of the view frustum centerline, shortest distance, and RANSAC clustering in one embodiment;

[0027] Figure 5 This is a schematic diagram of the overall results of clustering and localization of the ROI view frustum frame read into the Shangbanqiao Power Station.

[0028] Figure 6 Model diagram of the circuit breaker before registration for Shangbanqiao Power Station (marked points are white, and the marked points face yellow).

[0029] Figure 7 Model diagram of the circuit breaker after registration for Shangbanqiao Power Station (marked points are in blue, and the direction of the marked points is yellow).

[0030] Figure 8 Model diagram of the circuit breaker before registration for Shangbanqiao Power Station (marked points are white, and the marked points face yellow).

[0031] Figure 9 Model diagram of the circuit breaker after registration for Shangbanqiao Power Station (marked points are in blue, and the direction of the marked points is yellow).

[0032] Figure 10 A schematic diagram of the overall results of clustering and localization of the ROI view frustum frame read into the Laoxiongjing Power Station.

[0033] Figure 11 Model diagram of the circuit breaker before registration for Laoxiongjing Power Station (marked points are white, and the marked points face yellow).

[0034] Figure 12 Model diagram of the circuit breaker after registration for Laoxiongjing Power Station (marked points are blue, and the marked points face yellow). Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0037] Example 1:

[0038] This embodiment provides a complete power plant heterogeneous intelligent agent cooperative patrol system, the overall architecture of which is as follows: Figure 1 As shown, the system comprises a point cloud database construction module, a visual inspection and spatial mapping module, a spatial positioning and registration module, a heterogeneous trajectory planning module, and a digital twin simulation module, which are sequentially connected through a data interface. The functions of each module are as follows:

[0039] Point Cloud Database Module: This module provides the system with basic maps and standard equipment models. It first integrates raw point cloud data collected by LiDAR and high-precision RTK-GPS, then uses Gaussian sputtering rendering to generate a 3D power station point cloud map with realistic color information. The map uses the UTM projection coordinate system, with units in meters, and is stored in LCC format. Based on this, the module supports interactive equipment clipping and denoising, performs decentralized and attitude normalization processing on single-equipment point clouds, and generates standard equipment models with uniform attitudes. It also supports configuring inspection markers on the equipment models, automatically calculating the surface normal vectors and orientations of the markers, and finally outputting PLY format equipment point cloud files and JSON format marker data files for subsequent registration and trajectory planning.

[0040] Visual Inspection and Spatial Mapping Module: This module realizes the inverse mapping from 2D visual inspection to 3D space, completing the coarse positioning of equipment. First, based on the power station's working range and camera parameters, the module automatically calculates multi-view rendering points covering the entire area, generating power station image samples in batches based on a 3D point cloud map. After annotation, the samples are used to train the target detection model. The trained model is encapsulated as a dynamic link library, capable of real-time identification of five core power station equipment types: main transformers, disconnect switches, voltage transformers, surge arresters, and circuit breakers. For the 2D ROI bounding boxes output by the detection, the module uses a ray tracing inverse projection algorithm to convert screen pixels into 3D spatial ray beams. The corresponding 3D point cloud is selected based on the distance from the point to the ray, obtaining the coarse positioning point cloud region for the equipment, thus narrowing the calculation range for subsequent high-precision registration.

[0041] Spatial Positioning and Registration Module: This module is the core positioning unit of the system, responsible for calculating the precise 3D pose of the device. First, the module groups all detection frustums of similar devices and uses the RANSAC clustering algorithm to spatially cluster the centerlines of the detection frustums, eliminating invalid frustums with incorrect detections. This calculates the approximate center position of the device and generates a dynamic bounding box (ROI) that adapts to the device size, cropping the local device point cloud. Subsequently, the module employs a multi-hypothesis concurrent point cloud registration algorithm to generate multiple initial rotational attitudes. It then performs coarse FPFH feature registration and fine ICP registration in parallel using multi-threaded execution, selecting the globally optimal result with the highest coverage and smallest error as the precise pose of the device. Finally, it uses a Gaussian projection inverse algorithm to convert the local UTM coordinates to WGS84 latitude and longitude coordinates, simultaneously calculating the global position and orientation of the marker points, and outputting this result to the trajectory planning module.

[0042] Heterogeneous Flight Path Planning Module: This module includes a UAV flight path planning unit and a robotic dog flight path planning unit, generating executable flight paths for airborne and land-based inspection vehicles, respectively. The UAV flight path planning unit uses KDTree point cloud indexing for 3D obstacle avoidance and automatically calculates safe flight altitudes for each segment; it combines the position and normal vector of equipment marker points to calculate the optimal shooting position for the UAV; for long-endurance missions, it automatically implements multi-UAV relay scheduling and deduplication of repetitive points; it also supports hierarchical speed control and automatic injection of action commands, ultimately exporting multi-format standard flight path files. The robotic dog flight path planning unit generates a high-precision top-down view through orthographic projection rendering and supports interactive drawing of passable areas; when the robotic dog performs tasks, it achieves autonomous positioning through real-time point cloud registration and dynamically calculates the gimbal angle and focal length to ensure the lens is always pointed at the target equipment.

[0043] Digital Twin Simulation Module: This module builds a digital twin simulation environment based on the UE5 platform for safety verification and system self-optimization before track deployment. After loading the planned drone and robot dog track, the module automatically verifies the safe distance between each waypoint and surrounding obstacles, identifies risk levels using color, and supports manual correction. It also provides first-person and third-person dual-view simulated inspection, allowing real-time preview of captured images and robot movement. Verified tracks can be exported to control formats compatible with multiple robot brands. The module also supports selecting device ROIs to render new image samples, feeding them back to the target detection model for incremental training. Triggering mechanism and selection strategy: The system triggers the data feedback process weekly, statistically analyzing the device detection confidence of all rendered images in the simulation verification phase within that week. Only difficult samples with confidence scores below 0.60 or whose detection results do not match the device categories labeled at waypoints are selected. After automatic labeling, based on known device categories at waypoints and manual sampling (e.g., a sampling rate of 5%), these samples are added to the training set for model fine-tuning. Each feedback iteration adds no fewer than 200 and no more than 2000 new samples to prevent catastrophic forgetting. This mechanism enables targeted performance improvements to the model under complex lighting and viewing angles, achieving self-iteration of system performance and forming a complete self-optimization loop.

[0044] Example 2:

[0045] This embodiment provides a method for cooperative navigation of heterogeneous intelligent agents in power plants based on digital twins. The entire process consists of five steps, which correspond one-to-one with the system modules in Embodiment 1, and are combined with the appendix. Figure 1 As shown, the specific process of the cruise method provided in this embodiment is as follows:

[0046] Step S1: Standardize and build a database of equipment models. First, use LiDAR with high-precision GPS to scan the entire power station and generate a colored 3D point cloud map through Gaussian sputtering rendering. Interactively crop the point cloud data of a single device from the point cloud map to remove surface noise and irrelevant debris. Perform decentralization and attitude normalization on the device point cloud to ensure that the device center is located at the origin of the coordinate system and the main direction is aligned with the global X-axis. Select marker points at key locations on the device, automatically calculate the surface normal vector and orientation of the marker points, and save the standard equipment model and marker point data after completion.

[0047] Step S2: Target Detection and 2D-3D Inverse Mapping; Delineate the working area of ​​the power station, automatically calculate multi-view rendering viewpoints based on the camera's field of view and working distance, and batch render and generate power station images; Construct a training dataset and train the target detection model to identify five types of power station equipment; Deploy the trained model to the inference stage, perform real-time detection on the rendered images, and output the 2D ROI bounding box of the equipment; Convert the 2D ROI region into a 3D spatial ray beam using a ray tracing inverse projection algorithm, filter out the corresponding 3D point cloud, and obtain the coarse localization point cloud region of the equipment.

[0048] Step S3: High-precision spatial positioning and coordinate transformation; group all detection results by equipment category and extract the center line of the view frustum corresponding to each detection result; use the RANSAC clustering algorithm to spatially cluster the center lines of the view frustum, remove erroneous detection results, calculate the approximate center of the equipment, and generate a dynamic ROI bounding box; use a multi-hypothesis concurrent point cloud registration algorithm to register the field point cloud within the ROI with the standard equipment model, and calculate the precise three-dimensional position and orientation of the equipment; convert the local UTM coordinates to WGS84 latitude and longitude coordinates through Gaussian projection inverse solution, and simultaneously calculate the global pose information of the marker points.

[0049] Step S4: Heterogeneous carrier trajectory planning; For UAVs: Construct a KDTree index based on the full-field point cloud and calculate the safe flight altitude for each segment; Combine the marker point position and normal vector to calculate the optimal shooting position; If the total mission duration exceeds the endurance of a single UAV, automatically split the trajectory to achieve multi-UAV relay; Inject graded speed and shooting action commands into the trajectory and export standard format trajectory files.

[0050] For robotic dogs: a high-precision top view is generated through orthographic rendering, and passable areas are interactively drawn and inspection points are mapped; when the robotic dog performs tasks, it achieves autonomous positioning through real-time point cloud registration and dynamically calculates gimbal parameters to align with the target device.

[0051] Step S5: Digital twin simulation verification and closed-loop optimization; import the planned trajectory into the UE5 digital twin environment, automatically verify the safe distance, and identify risk waypoints through visualization; conduct simulated inspections in first-person and third-person modes to preview the shooting effect and trajectory rationality; after verification, export the executable robot control file; at the same time, use the simulation environment to render and generate new training samples to feed back into the incremental training of the target detection model, thereby achieving system self-optimization.

[0052] In this embodiment, regarding the implementation of simulation verification and data feedback, the functions of the digital twin simulation module are refined and optimized as follows: The planned flight paths of the UAV and the robotic dog are imported into the UE5 digital twin system. The system automatically calculates the spatial distance between each waypoint and surrounding equipment, buildings, and towers, and uses red and blue colors to intuitively indicate whether the waypoints meet the safety distance requirements: red indicates insufficient distance and a risk of collision, while blue indicates compliance with safety distance requirements. A ranging tool is also provided to assist users in manually correcting waypoint positions in three-dimensional space, enabling flight path safety verification and visualization. Simultaneously, the system supports two simulated inspection modes:

[0053] First-person perspective: The main screen displays the real-time perspective of the robot's camera, allowing you to preview the footage taken during its movement and verify the shooting angle, composition, and target clarity.

[0054] Third-person perspective: The scene presents a 3D model of the robot moving along its path. Users can freely navigate the global view with the mouse and keyboard to observe the overall path and the robot's movement.

[0055] For verified flight paths, the format conversion module can decouple and export them into proprietary control formats compatible with different brands of drones and robotic dogs, achieving one-stop planning and multi-device compatibility. The simulation system supports selecting device ROI regions to render and generate new image samples, which, after annotation, can be used for incremental training of the target detection model, continuously improving the model's recognition accuracy under complex angles and lighting conditions. This forms a complete closed loop of "data acquisition, recognition and localization, flight path planning, simulation verification, and data feedback," enabling the system's performance to self-iterate and optimize.

[0056] Example 3:

[0057] This embodiment refines and optimizes the pose normalization and marker configuration in the point cloud database construction process. The specific implementation method is as follows:

[0058] 3.1 Attitude Normalization Algorithm Based on Two-Dimensional Principal Component Analysis; To eliminate attitude differences caused by different scanning batches and different initial angles, the cropped device point cloud is subjected to attitude normalization processing, so that the device center is located at the origin of the coordinate system and the principal direction is towards the positive direction of the global X-axis. The specific steps are as follows:

[0059] (1) Decentralized processing; assume that the clipped point cloud contains N three-dimensional points. Calculate the average coordinates of all points as the centroid c of the point cloud: Subtract the centroid coordinates from the coordinates of all points to translate the center of the point cloud to the origin of the coordinate system. ;

[0060] (2) Two-dimensional principal component analysis to extract principal directions; since the power station equipment is placed vertically, orientation normalization is only required in the XY horizontal plane, keeping the Z-axis direction unchanged. Project each three-dimensional point onto the XY plane to obtain two-dimensional points. Calculate the mean vector μ of a two-dimensional point set: Construct a two-dimensional covariance matrix C to characterize the dispersion and correlation of the point cloud in the X and Y directions:

[0061] ;

[0062] Eigenvalue decomposition of the covariance matrix C yields two eigenvalues. , and its corresponding eigenvectors , The eigenvector corresponding to the largest eigenvalue is selected as the principal direction of the horizontal plane of the point cloud. :

[0063] , ;

[0064] (3) Rotate and align to the global X-axis; set the principal direction vector. The angle θ between the principal direction and the global X-axis is calculated using the arctangent function: Take the rotation angle Generate a 3D rotation matrix around the Z-axis, and align the principal direction with the positive global X-axis:

[0065] ;

[0066] To avoid additional translation of the point cloud during rotation, the rotation center is used as the rotation center, employing a transformation method of "translate to the origin, rotate, and translate back to the original position": ;in, This is the translation transformation matrix.

[0067] (4) Orientation ambiguity elimination and manual fine-tuning: Since principal component analysis can only determine non-directional principal axes, and For the same geometric axis, automatic alignment may result in a 180° orientation ambiguity. The system provides two manual fine-tuning methods: one is to click the endpoint of the coordinate axis to achieve discrete rotations of 90° each time, and the other is to hold down the corresponding axis key and drag the mouse to achieve continuous fine-tuning, with a sensitivity of 0.5° / pixel; at the same time, the X-axis pitch angle is limited to the range of [-89.43°, +89.43°] to avoid extreme tilting postures. Optionally, the rotation step size can be configured within the range of 5°-90° according to usage requirements.

[0068] 3.2 Automatic Estimation and Fine-tuning Method for Marker Point Normal Vectors

[0069] Marker points are used to define the key shooting positions and suggested observation directions for drones or robotic sensors. Each marker point contains three-dimensional coordinates and a unit normal vector. The specific generation steps are as follows:

[0070] (1) Automatic estimation of local normal vectors; after the user clicks to select a marker point on the device point cloud, the system searches the surrounding point cloud in a spherical neighborhood with radius r=0.10m centered on that point; when the number of points in the neighborhood is not less than 3, the three-dimensional covariance matrix of the neighborhood point set is calculated. :

[0071] ;

[0072] in, Let j be the coordinates of the j-th point in the neighborhood. The coordinates of the neighborhood point set are given. The eigenvalues ​​of the 3D covariance matrix are decomposed, and the eigenvector corresponding to the smallest eigenvalue is taken as the local surface normal vector. This principle is based on the geometric property that "the variance of the point distribution in the vertical direction within a local plane is minimized".

[0073] (2) Normal vector orientation constraint; To avoid the normal vector randomly pointing into the device, the system calculates the line of sight from the current camera viewpoint to the marker point. Determine the orientation of the normal vector by the dot product:

[0074] ;

[0075] like Then, invert the normal vector to ensure that the normal vector always points towards the side of the observation viewpoint:

[0076] ;in, The physical meaning is the local surface normal vector (unit vector) at the marked point, which is obtained by decomposing the three-dimensional covariance matrix of the point cloud in the neighborhood of the marked point, and represents the vertical direction of that point on the device surface; This represents the unit vector of the line of sight, pointing from the camera's viewpoint to the marker point, and represents the direction of the current observation angle. This represents the dot product of the normal vector and the line-of-sight vector. Result > 0: The angle between the two is < 90°, and the normal vector is pointing towards the camera side; Result < 0: The angle between the two is > 90°, and the normal vector is pointing away from the camera and towards the inside of the device. Invert the normal vector, that is, the vector with the completely opposite direction.

[0077] (3) Manual fine-tuning of the normal vector

[0078] When the automatically estimated normal vector does not match the actual inspection requirements, users can fine-tune the normal vector of the selected marker point, supporting one-click reversal of orientation, or rotation around the X / Y / Z axes in ±5° increments. Fine-tuning is achieved using the axis angle rotation formula: , Where a is the unit axis of rotation and φ is the rotation angle. Let be the antisymmetric cross product matrix corresponding to vector 'a'. Finally, the normalized device point cloud is saved in PLY format, and the marker data is saved in JSON format, including the marker name, applicable carrier, 3D coordinates x / y / z, and unit normal vectors nx / ny / nz, with the coordinate unit being meters. Optionally, the neighborhood radius of the marker can be adjusted within the range of 0.08-0.12m according to the point cloud density.

[0079] Example 4:

[0080] This embodiment proposes an optimized technical solution based on embodiments 1-2. This embodiment mainly refines and optimizes the visual detection and spatial mapping process, and elaborates on multi-view rendering, target detection model and ray tracing inverse projection algorithm. Specifically, it includes a full-coverage multi-view rendering automatic frustum calculation method, target detection model training and deployment method and ray tracing-based inverse projection point cloud screening algorithm implementation.

[0081] Specifically, the automatic calculation method for view frustum in full-coverage multi-view rendering includes the following steps:

[0082] To ensure that all equipment within the power plant is rendered and captured, the system automatically calculates rendering viewpoints covering the entire working area. The specific steps are as follows:

[0083] The first step is to manually delineate the working area of ​​the power station map and determine the boundary dimensions in the X, Y, and Z directions. , , Exclude surrounding trees, houses, and other irrelevant areas.

[0084] The second step is to calculate the field of view size at the working distance based on the camera's intrinsic parameters. Let the camera's horizontal field of view angle be... , If the working distance is set to D, then the width W and height H of the field of view at the working distance are respectively: ; The third step is to calculate the number of viewpoints and their coordinates required in each direction. Taking the X-direction as an example, the required number of viewpoints... Similarly, we can obtain the Y direction The Z direction is The total number of viewpoints is .

[0085] It should be noted that the multi-view rendering coverage calculation in this embodiment is based on the assumption of orthographic projection of the camera's optical axis perpendicularly downwards along the negative Z-axis. In this case, the field of view size at the working distance D is constant, and an equidistant grid arrangement can be used. If an oblique view is used in actual applications, the view frustum needs to be projected onto the ground according to the camera's extrinsic parameters, and the coverage union is obtained through polygon Boolean operations to ensure full coverage. Those skilled in the art can, based on the equidistant grid principle disclosed in this solution, combine perspective projection transformation to achieve full viewpoint coverage calculation under an oblique viewpoint; this is an equivalent transformation of this invention. The spatial coordinates of each viewpoint are arranged at equal intervals according to the field of view size, and the X-coordinate of the i-th X-direction viewpoint is... The same applies to the Y and Z directions; the detection cones arranged in this way can achieve omnidirectional coverage of the working area, ensuring that all devices are rendered and captured.

[0086] The training and deployment process of the object detection model is based on the aforementioned multi-viewpoint approach. Power plant images from corresponding viewpoints are batch-rendered on the LCC point cloud map as the training dataset for the object detection model. The CVAT annotation tool is used to annotate five types of power plant equipment in the images, including: main transformers, disconnectors, voltage transformers, surge arresters, and circuit breakers. The YOLOv9 algorithm is used to train the object detection model. After training, the model is encapsulated as a C++ dynamic link library (DLL) adapted for both CPU and GPU, which can be directly integrated into the main program for real-time inference. Optionally, the object detection algorithm can also be replaced with other versions of the YOLO series, Faster R-CNN, SSD, and other mainstream object detection algorithms; all of these are equivalent alternatives to this invention.

[0087] The following section details the inverse projection point cloud filtering algorithm based on ray tracing. For the 2D ROI bounding box output by the model detection, the screen area is mapped to a 3D spatial point cloud set through inverse projection using ray tracing. The specific principles and steps are as follows:

[0088] (1) Ray construction: For each pixel within the ROI region, a spatial ray is constructed using the principle of camera inverse projection: , t>0; where O is the position of the camera in world space, D is the unit direction vector from the camera to the pixel, and t is the ray depth parameter.

[0089] (2) Point-to-ray distance calculation: Since the point cloud is a discrete set of points and does not have a patch structure, traditional ray intersection calculation is not possible. Therefore, the "shortest distance from point to ray" is used as the basis for hit determination. The formula for the shortest distance from a spatial point P to a ray is: This formula is derived from the geometric meaning of the cross product of vectors, and the result is equal to the height of the parallelogram formed by the point and the ray, that is, the perpendicular distance from the point to the ray.

[0090] (3) Hit determination and point cloud filtering: Set a distance threshold. If the shortest distance d from a 3D point to a ray is less than the threshold, the point is determined to be hit by the ray and belongs to the 3D region corresponding to the ROI; if it is greater than the threshold, it is determined to be an irrelevant point. The distance threshold is usually 0.5-1 times the average spacing of the point cloud or the size of the rendered point. If the threshold is too small, it will lead to missed detection of the point cloud; if it is too large, it will introduce too many irrelevant background points. Perform the above determination on all rays in the ROI, collect all hit 3D points and remove duplicates, and finally obtain the device coarse positioning point cloud region corresponding to the detection target. In order to improve the computational efficiency, it is not necessary to generate rays for every pixel in the ROI. A uniform interval sampling method can be used to significantly reduce the amount of computation while ensuring accuracy. This method is implemented only through geometric calculation and does not rely on the UE5 real-time ray tracing pipeline. It has low computational overhead and flexible deployment.

[0091] Example 5:

[0092] This embodiment, based on Embodiment 2, elaborates on RANSAC frustum clustering, the multi-hypothesis concurrent registration algorithm, and the coordinate transformation method. This embodiment refines and optimizes the spatial localization and registration process, and is the core algorithm embodiment of this invention. The specific implementation method is as follows:

[0093] First, we describe a device ROI extraction method based on RANSAC clustering of the view frustum centerline. This method is used to cluster the spatial range of the real device from multiple detection view frustums, while removing invalid view frustums that are detected incorrectly, and generating an accurate ROI bounding box.

[0094] (1) Generation of the view frustum centerline: First, all two-dimensional detection results are grouped according to device category, and low-quality detection results with a confidence level below 0.50 are filtered out. Each detection ROI corresponds to a three-dimensional detection view frustum, defined by 8 vertices: 4 vertices on the near plane and 4 vertices on the far plane. See details. Figure 4 As shown, calculate the center of the near-end plane of the visual cone. Center of the far end plane : ; Using the near-end center as the ray origin, and normalizing the difference between the far-end center and the near-end center, we obtain the direction vector D of the view frustum centerline. The final parametric equation of the view frustum centerline is: , .

[0095] (2) Calculation of the shortest distance between two lines: For any two center lines of the view cone and Find the closest point on two straight lines in space. , The distance between two points is used as the consistency index of the two centerlines, and the midpoint M of the two points is used as the candidate intersection point.

[0096] , .

[0097] (3) RANSAC clustering algorithm process: The Random Sampling Consensus (RANSAC) algorithm is used for view frustum clustering. The specific steps are as follows: 1. Randomly select at least two view frustum centerlines to establish an initial intersection hypothesis; 2. Calculate the point-to-line distance from all other centerlines to the common point of the hypothesis. The distance formula is:

[0098] Where X is the candidate common point and L is the center line of the view cone to be determined. 3. Center lines whose distance to a point is less than a threshold are determined as interior points, and the number of interior points is counted. 4. Repeat the above random sampling process, iterating up to 5000 times, and retain the hypothesis with the most interior points as a device cluster. 5. For all center lines within the cluster, solve for the optimal common intersection point using the least squares method.

[0099] ; ; Where I is a 3×3 identity matrix. It is a 3×1 unit column vector (the direction vector of the center line of the view cone). It is a 3×1 column vector (coordinates of the center of the near-end plane); therefore, Let A be a 3×3 matrix and b be a 3×1 column vector. Solving the linear equation system A*X=b yields X, which is a 3×1 column vector representing the three-dimensional spatial coordinates (in meters) of the optimal common intersection point of the cluster. X is the initial estimate of the equipment center obtained from the solution.

[0100] (4) Dynamic bounding box ROI generation: After obtaining the cluster center, the midpoints of the intersection of the center lines of each pair within the cluster are statistically analyzed, and outliers that are too far from the mean are removed. Finally, the mean is taken as the center point of the ROI.

[0101] Based on the actual dimensions of the device model, a dynamic bounding box is generated proportionally: the XY plane dimensions are... ;in, These are the physical dimensions (in meters) of the standardized device model in step S1 along the X and Y axes, respectively, directly obtained from the bounding box side lengths after cropping the device point cloud. The Z-axis dimension is... This allows for the precise ROI point cloud of the device to be obtained through cropping. The thresholds used in the above process all employ an adaptive strategy, correlated with the device size, with specific parameters as follows:

[0102] Centerline consistency threshold: The default value is 2.0m when no device size is specified.

[0103] Candidate point to line threshold: Same as centerline consistency threshold;

[0104] Effective intersection distance threshold: The default value is 1.4m when no device dimensions are specified.

[0105] Outlier threshold at intersection point: Same as the effective intersection distance threshold. Through adaptive threshold design, this method can be adapted to power plant equipment of different sizes without the need for manual parameter tuning for each type.

[0106] Secondly, the following is a description of the multi-hypothesis concurrent FPFH+ICP high-precision point cloud registration algorithm: This algorithm is used to solve the precise three-dimensional pose (position + orientation) of the device from the ROI point cloud, which is one of the core innovations of this invention.

[0107] (1) Point cloud preprocessing and layering strategy: First, the point cloud of the ROI local map is translated to the vicinity of the coordinate origin to eliminate the loss of floating-point calculation accuracy caused by large coordinate values. Then, the point cloud is voxel downsampled and divided into two resolution levels; specifically, the coarse registration layer: voxel size 0.15m, used for fast feature matching and initial pose estimation; the fine registration layer: voxel size 0.05m, used for high-precision pose optimization.

[0108] (2) Construction of Discrete Rotation Hypothesis Set; To address the problem of traditional ICP easily getting trapped in local optima, this invention adopts a multi-hypothesis concurrent strategy to construct a discrete rotation hypothesis set in the Z-axis direction:

[0109] ;in, Discrete rotation assumption angle set, containing all rotation angles about the Z-axis used for the initial attitude; here, {} represents the standard notation for a mathematical set, and the brackets contain all the elements of the set; The first rotation assumption: The model is not rotated initially and directly participates in registration; Indicates the rotation step size. This represents the last assumed rotation angle. Since a 360° rotation around the Z-axis is completely equivalent to a 0° orientation, to avoid redundant calculations, we terminate at 360° minus one step; a step of 20° corresponds to 340°.

[0110] Rotation step size under default configuration A total of 18 initial rotation hypotheses were generated. Optionally, the rotation step size can be adjusted within the range of 10°-30° according to computing power and accuracy requirements: the smaller the step size, the more hypotheses are generated, and the higher the registration success rate, but the longer the computation time; the larger the step size, the faster the computation speed, but the success rate decreases. Experimental data shows that a step size of 20° can achieve the optimal balance between efficiency and accuracy with a success rate of 89.1%.

[0111] (3) Multi-threaded parallel registration process: The registration process for all rotation assumptions is executed in parallel using OpenMP multi-threading, with each thread independently completing the following steps:

[0112] 1. Rotate the point cloud of the standard equipment model to the corresponding initial angle; 2. Extract FPFH features from the coarse-resolution point cloud; 3. Perform coarse registration using the RANSAC algorithm to obtain the initial transformation matrix; the maximum number of RANSAC iterations is 100,000, the confidence level is 0.999, the corresponding point distance threshold is 0.60m, and side length consistency checks are enabled; 4. Based on the coarse registration results, perform fine registration optimization using the point-to-point ICP algorithm, with a maximum corresponding distance of 0.20m and a maximum number of iterations of 100, and output the final transformation matrix.

[0113] (4) Optimal Result Screening and Attitude Constraints: After all threads have completed their calculations, all registration results are sorted and screened: First sorting criterion: Registration coverage, i.e., the proportion of matched inliers to the total number of points in the model; Second sorting criterion: Inlier Mean Square Error (RMSE), when the Fitness difference between two results is less than 100%. When choosing a model, select the one with the smaller RMSE. Simultaneously apply a vertical attitude constraint: the angle between the local +Z axis of the transformed model and the global +Z axis of the map must not exceed 30° to prevent vertical devices from being incorrectly matched as tilted.

[0114] (5) Retry mechanism and success criteria: Since RANSAC sampling is random, the results may fluctuate in a single run. Therefore, a maximum of 4 retry mechanisms are set, including 1 initial attempt and 3 retry opportunities.

[0115] Success criteria: Fitness > 0.75 and RMSE < 0.20m. If both conditions are met, registration is considered successful, and subsequent retries are immediately terminated. If the success criteria are not met after 4 retries, the device registration is considered a failure. In this case, the system automatically performs hierarchical fault tolerance: ① The center of the device's coarse positioning point cloud region is used as its approximate location, and the normal vector follows the default orientation of similar devices, generating a backup waypoint with a 'low confidence' label; ② The backup waypoint is highlighted in red in the digital twin simulation module, prompting manual review; ③ The system continues to perform registration and track planning for the remaining devices, without interrupting the entire task due to a single device failure. After manual review, waypoints can be corrected and re-exported in the simulation environment with one click. Early termination condition: If a result shows Fitness ≤ 0.10 and RMSE ≥ 0.50m, it is considered obviously unusable, and the calculation for that round is terminated early. Optionally, the number of retries can be configured to 3-5 times depending on the scenario complexity.

[0116] Finally, the process of converting UTM to WGS84 high-precision coordinates is described. The device pose obtained through registration is in local UTM metric coordinates. To adapt to the navigation requirements of UAVs, it is converted to WGS84 latitude and longitude coordinates using a Gaussian projection inverse solution algorithm. The conversion process sequentially completes UTM pseudo-offset elimination, meridian arc length calculation, base point latitude inverse calculation, and curvature radius parameter calculation, ultimately solving for high-precision latitude, longitude, and altitude. The conversion accuracy can reach the centimeter level, fully meeting the navigation requirements of UAV close-range inspection.

[0117] The rotation matrix obtained from registration is simultaneously applied to the local normal vector of the marker point to obtain the orientation of the marker point in the global coordinate system, and the yaw and pitch angles are further calculated:

[0118] ; ;in, This is the unit normal vector of the marker point in the global coordinate system.

[0119] Example 6:

[0120] This embodiment, based on embodiments 2, 4, and 5 above, refines and optimizes the function of the UAV trajectory planning unit. Specific implementation methods include:

[0121] The KDTree-based 3D path corridor obstacle avoidance algorithm first constructs a KDTree spatial index based on the XY plane coordinates of the entire point cloud to improve the efficiency of nearest neighbor point retrieval. The specific obstacle avoidance height calculation steps are as follows:

[0122] Extend the straight path between two adjacent waypoints by 12 meters on each side to construct a three-dimensional cuboid flight path corridor. Use KDTree to quickly retrieve all point clouds within the corridor area and calculate the maximum Z-coordinate of these points as the maximum obstacle height for this flight segment. Add an 8-meter safety margin to the maximum obstacle height to obtain the safe flight altitude for this segment. If the safe flight altitude exceeds the UAV's maximum ceiling or is higher than the top of nearby towers, the system determines that the straight flight segment is impassable and automatically compresses the corridor width to 8 meters and recalculates. If it is still impassable after compression, KDTree A detour path search is initiated to bypass areas with dense obstacles. The default 12-meter safe corridor width is a conservative initial value and can be adjusted by the user in the configuration file within the range of 2-20 meters according to the density of power station equipment. Ensure that the corridor width is not lower than the legally permissible takeoff safety altitude.

[0123] Automatic Optimal Shooting Position Calculation: For each equipment marker point to be shot, the system automatically calculates the optimal shooting position for the drone, taking into account both shooting angle and collision safety: Horizontal direction: Move back 5 meters horizontally in the opposite direction of the marker point's normal vector to ensure a safe shooting distance from the equipment; Vertical direction: Add a relative inspection height of 3.5 meters to the height of the equipment marker point to obtain a good overhead shooting angle; At the same time, based on the relative position of the shooting position point and the marker point, the system automatically calculates the gimbal's yaw and pitch angles to ensure that the lens is aimed at the target.

[0124] Long-endurance multi-drone relay scheduling algorithm: For scenarios with dense inspection points in large substations and a total duration exceeding the endurance of a single drone, automatic multi-drone relay scheduling is implemented. The steps are as follows: Accumulate the flight time of all segments and the hovering time at each inspection point to calculate the total time of the entire mission; if the total time exceeds the set endurance threshold for a single drone, the total time is divided equally according to the number of available drones to obtain the mission time quota for each drone; starting from the starting point, the time at each inspection point is accumulated one by one, and the waypoint with the accumulated time closest to the quota threshold is found as the optimal relay point; intelligent deduplication is performed when splitting the flight path: the first drone completes the entire process to the relay point and completes the corresponding shooting; subsequent relay drones fly directly to the relay point after takeoff, skipping the shooting action at that point, and start the process from the next inspection point to avoid repeated shooting.

[0125] Graded speed control and automatic motion injection: The system divides the single-device inspection process into four flight phases and implements a graded speed strategy to balance efficiency and shooting stability.

[0126] Approach phase: Flight speed 8m / s, rapidly approaching the target equipment;

[0127] Pre-positioning phase: Flight speed 3m / s, decelerate to enter the shooting position;

[0128] During the inspection and shooting phase: the flight speed is 0.5m / s, almost hovering, to ensure stable image quality.

[0129] Departure phase: Flight speed 3m / s, departing from the current equipment and heading to the next target.

[0130] Simultaneously, a strict distinction is made between the climb rate of 3 m / s and the descent rate of 2.5 m / s to ensure a smooth and safe takeoff and landing process. Automatic action commands are injected at key nodes in the flight path, including: hovering and waiting actions upon reaching waypoints, taking photos upon reaching inspection points, starting and stopping video recording before and after entering the inspection area, and pre-adjusting the gimbal attitude, achieving fully unmanned autonomous execution.

[0131] And multi-format standard track export: The planned track can be exported in multiple standard formats:

[0132] CSV format: Contains detailed coordinates, speed, and movement information for each waypoint, making it easy to view and edit;

[0133] JSON format: Task attribute configuration conforming to DJI Pilot2 specifications;

[0134] KML / KMZ format: can be imported into 3D map software such as Google Earth for intuitive preview and review;

[0135] WPML format: DJI's proprietary engineering track format, encapsulating all waypoints, speeds, actions, and safety policies, which can be directly uploaded to the remote controller for one-click execution.

[0136] Example 7:

[0137] This embodiment optimizes upon embodiments 1 and 2, primarily focusing on refining the functionality of the mechanical dog's trajectory planning unit. It details the technologies for constructing passable areas, autonomous positioning, and gimbal-based computation. The specific implementation is as follows: Traditional sparse point cloud-based passable area construction suffers from severe detail loss and poor accuracy in planar areas. This invention utilizes a 3D Gaussian sputtering (3DGS) high-precision map, generating a high-precision top-view through orthographic projection rendering. The specific method is as follows:

[0138] By setting the rendering viewpoint to infinity on the Z-axis and employing orthographic transformation, all light rays are projected perpendicularly along the negative Z-axis, generating a top-down view without perspective distortion. This top-down view fully preserves ground details, equipment outlines, and path information, with each pixel corresponding to a real geographic coordinate.

[0139] The pixel coordinate system of this top-view view is mapped one-to-one with the UTM coordinate system of the global point cloud map through a homography matrix. Let the ground sampling interval of the top-view view be r (unit: meters / pixel, default value is 0.05m / pixel), and the UTM coordinates of the southwest corner of the map be (…). Then, the actual geographic coordinates corresponding to any pixel (u,v) in the top view are (u,v). Calculate using the following formula: ; Where u and v are the pixel indices (non-negative integers) in the row and column directions of the image, and r is adaptively determined based on the point cloud density and rendering resolution. The global pose of the robot dog obtained through real-time point cloud registration is also based on the same UTM coordinate system, which allows interactively drawn 2D path points to be directly converted into navigation target points, ensuring the consistency between the planning coordinate system and the execution coordinate system.

[0140] Users can interactively draw the navigable area of ​​the robotic dog on a high-precision top-down view and map 3D inspection points into 2D waypoints to generate the robotic dog's basic cruising path. When performing tasks, the robotic dog scans the surrounding environment in real time using its onboard LiDAR, generating real-time point cloud data. The system matches and registers the real-time point cloud with a pre-imported global point cloud map, and uses the ICP algorithm to calculate the robotic dog's real-time position and attitude in the global coordinate system, achieving unmarked autonomous navigation and positioning, and completing autonomous positioning based on real-time point cloud registration.

[0141] Then, based on the real-time pose of the robotic dog and the 3D coordinates of the target marker point, combined with the pinhole camera's physical model, the control parameters of the camera gimbal are dynamically calculated: 1. The relative coordinates of the target point in the robotic dog's body coordinate system are calculated; 2. The yaw and pitch angles of the gimbal are calculated using the arctangent function to ensure the lens optical axis is aligned with the target; 3. Based on the target distance and device size, the camera focal length / zoom ratio is automatically calculated to ensure the target device is at an appropriate size in the frame. During the robotic dog's movement, the gimbal parameters are updated in real time to ensure the target device remains centered in the frame, achieving dynamic gimbal attitude calculation.

[0142] Example 8:

[0143] See appendix Figures 5-12 As shown, this embodiment is a verification embodiment of the effect of the present invention. To verify the technical effect of the present invention, field tests were conducted at the 110kV Shangbanqiao Substation and the 220kV Laoxiongjing Substation. Some core test data are as follows:

[0144] main transformer 2 0.88862 9.800 80.878 50 lightning arrester 4 0.99703 5.280 1.428 100 breaker 7 0.99170 5.903 8.798 100 disconnect switch 16 0.86918 9.155 3603.4 88.8 Voltage transformer 4 0.98309 7.391 0.477 100

[0145] Table 1 shows the equipment registration results at Laoxiongjing.

[0146] Comparative tests were conducted on three rotation step sizes of 10°, 20°, and 30° under the same hardware configuration. The results are shown in Table 2 below.

[0147] 10° 33 8.726 0.93014 7.564 81.5 20° 33 4.228 0.92565 7.821 89.1 30° 33 3.236 0.91363 7.774 83.7

[0148] Table 2 shows the comparison test of rotation step length.

[0149] Test results show that a 20° rotation step size achieves the optimal balance between registration success rate, positioning accuracy, and computational efficiency, verifying the effectiveness of the multi-hypothesis concurrent registration strategy and the rationality of parameter selection in this invention. Compared with the traditional ICP registration scheme with a single initial attitude, this scheme improves the registration success rate by more than 30% and reduces the positioning error by about 40%, demonstrating significant technical advantages.

[0150] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A digital twin-based heterogeneous intelligent agent cooperative patrol system for power plants, characterized in that: It includes a point cloud database module, a visual inspection and spatial mapping module, a spatial positioning and registration module, a heterogeneous trajectory planning module, and a digital twin simulation module, which are connected sequentially. The point cloud database module is used to construct a 3D point cloud map of the power plant with color information, complete the interactive cropping, posture standardization and marker point configuration of single device point cloud, and output a standardized device point cloud model and marker point data containing position and normal vector information; The visual detection and spatial mapping module is used to automatically render the three-dimensional map from multiple perspectives to generate image samples, identify power station equipment through the target detection model, and map the two-dimensional detection ROI area into a three-dimensional spatial ray set through ray tracing and inverse projection to obtain the coarse positioning point cloud area of ​​the equipment. The spatial positioning and registration module is used to perform random sampling and consistent clustering of the detection frustums of similar devices to eliminate erroneous detection results and extract the precise ROI range. It also calculates the precise three-dimensional pose of the device through a multi-hypothesis concurrent point cloud registration algorithm, while converting local coordinates into general geographic coordinates. The heterogeneous trajectory planning module includes a UAV trajectory planning unit and a mechanical dog trajectory planning unit, which generate executable inspection trajectories with gimbal action commands based on the motion characteristics of airborne and land-based inspection vehicles, respectively. The digital twin simulation module loads inspection tracks on the UE5 platform to perform safety distance verification and simulated inspection, and supports feeding back the simulated and rendered device images to the visual inspection and spatial mapping module to form a self-optimizing closed loop of the system.

2. The power plant heterogeneous intelligent agent cooperative patrol system based on digital twin as described in claim 1, characterized in that, The point cloud database construction module is specifically used for: Gaussian sputtering rendering is performed on the power station cloud data collected by lidar and high-precision GPS to generate a 3D power station map with color information. Decentralized processing is achieved by interactively cropping and extracting point clouds from a single device, removing noise, and calculating the centroid of the point cloud. Two-dimensional principal component analysis is used to extract the principal direction of the point cloud in the XY horizontal plane. The principal direction is then aligned to the global X-axis by rotating around the Z-axis to complete the device attitude normalization. The surface normal vector is estimated based on the 3D covariance matrix of the local point cloud in the neighborhood of the marker point. The orientation of the normal vector is constrained by the camera's line of sight, and incremental fine-tuning around the axis is supported. Finally, the device point cloud in PLY format and the marker point file in JSON format are output.

3. The power plant heterogeneous intelligent agent cooperative patrol system based on digital twin as described in claim 1, characterized in that, The visual detection and spatial mapping module is specifically used for: automatically calculating the position and number of multi-view rendering viewpoints based on preset field of view, working distance and map range, and batch generating rendered images as training samples; annotating samples through an annotation platform and training a target detection model based on YOLOv9, encapsulating it into a C++ dynamic link library adapted to CPU and GPU, used to identify five types of power station equipment: main transformers, disconnect switches, voltage transformers, surge arresters and circuit breakers; for the ROI area obtained by two-dimensional detection, constructing corresponding ray beams through camera inverse projection, calculating the shortest spatial distance from each point in the three-dimensional point cloud to the ray, and filtering point clouds with distances less than a set threshold as equipment coarse positioning point cloud areas.

4. The power plant heterogeneous intelligent agent cooperative patrol system based on digital twin as described in claim 1, characterized in that, The spatial positioning and registration module is specifically used for: extracting the center lines of the view cones corresponding to all detection results of similar devices, calculating the shortest spatial distance between any two center lines, eliminating erroneous view cones through a random sampling consensus clustering algorithm, and generating a dynamic bounding box ROI that is adaptive to the device size; generating a discrete set of rotation hypotheses from 0° to 360°, performing fast point feature histogram feature coarse registration and iterative nearest point fine registration in parallel using multiple threads, selecting the result with the highest coverage and the lowest mean square error as the device's precise pose; converting the local UTM projection coordinates into WGS84 latitude and longitude coordinates through a Gaussian projection inverse algorithm, and outputting the device's precise position, orientation, and global pose information of the marked points.

5. The power plant heterogeneous intelligent agent cooperative patrol system based on digital twin as described in claim 1, characterized in that, The UAV trajectory planning unit is specifically used for: constructing a three-dimensional safety corridor for the trajectory based on a K-dimensional tree point cloud index; detecting the maximum height of obstacles in the corridor and superimposing a safety margin; automatically calculating the safe flight altitude for each segment; horizontally retracting a preset shooting distance along the opposite direction of the normal vector of the equipment marker point and superimposing a vertical inspection height to calculate the optimal shooting position coordinates of the UAV; for long-duration missions where the total time exceeds the endurance of a single UAV, automatically splitting the trajectory and determining the relay point to achieve multi-UAV relay scheduling and deduplication of repeated shooting points; injecting graded flight speed and gimbal action commands into key nodes of the trajectory; and exporting standard trajectory files in multiple formats such as WPML, KMZ, CSV, and JSON.

6. The power plant heterogeneous intelligent agent cooperative patrol system based on digital twin as described in claim 1, characterized in that, The robotic dog's trajectory planning unit is specifically used for: generating a high-precision top view by orthographically projecting and rendering a 3D point cloud map; interactively drawing passable areas and mapping 2D inspection points based on the top view; when the robotic dog performs a task, it scans the point cloud in real time using its own sensors and registers it with a preset map to achieve autonomous positioning in the global coordinate system; and dynamically calculating the yaw angle, pitch angle, and focal length parameters of the camera gimbal based on the robotic dog's real-time pose and the coordinates of the target device, combined with the pinhole camera model, to ensure that the lens is aligned with the target device.

7. A method for cooperative navigation of heterogeneous intelligent agents in power plants based on digital twins, characterized in that, Includes the following steps: S1. Construct a 3D point cloud map of the power plant with color information, complete the equipment point cloud clipping, attitude standardization and marker point configuration, and output a standardized equipment point cloud model and marker point data containing position and normal vectors; S2. Automatically render the 3D map from multiple perspectives to generate images, complete device detection through the target detection model, and map the 2D detected ROI area into a 3D spatial ray set through ray tracing and inverse projection to obtain the device coarse positioning point cloud area. S3. Randomly sample and uniformly cluster the detection cones of similar devices to extract the precise ROI range. Solve the precise 3D pose of the device through a multi-hypothesis concurrent point cloud registration algorithm and convert it into general geographic coordinates. S4. Generate inspection tracks with action commands for both drones and robotic dogs. The drone tracks support 3D obstacle avoidance and multi-drone relay scheduling, while the robotic dog tracks support traversable area mapping and real-time gimbal attitude calculation. S5. Import the planned flight path into the UE5 digital twin environment for safety distance verification and simulated inspection. The simulated and rendered equipment images can be fed back to the target detection model to achieve system self-optimization closed loop.

8. The method for cooperative navigation of heterogeneous intelligent agents in power plants based on digital twins according to claim 7, characterized in that, In step S3, the specific process of the multi-hypothesis concurrent point cloud registration is as follows: generate a discrete rotation hypothesis sequence of 0°-360° with a step size of 10°-30°, perform fast point feature histogram feature matching and random sampling coarse registration in parallel through multi-threaded execution, and then complete fine registration through point-to-point iterative nearest point algorithm; select the global optimal result with registration coverage and mean square error of interior points as evaluation indicators, while constraining the angle between the vertical axis of the model and the global vertical axis to not exceed 30°; configure a retry mechanism of up to 3-5 times, if a single registration does not reach the preset success threshold, re-execute random sampling coarse sampling until the registration is successful or the retry limit is reached.

9. The method for cooperative navigation of heterogeneous intelligent agents in power plants based on digital twins according to claim 7, characterized in that, In step S4, the UAV trajectory planning also includes graded speed and automatic motion injection control: the single device inspection flight process is divided into the approach stage, pre-positioning stage, inspection and shooting stage and departure stage, corresponding to preset flight speeds of 8m / s, 3m / s, 0.5m / s and 3m / s respectively; hovering waiting, fixed-point photography, video recording start and stop and gimbal attitude pre-adjustment commands are automatically injected at key waypoints of the trajectory to realize unmanned autonomous execution of the entire inspection process.

10. The method for cooperative navigation of heterogeneous intelligent agents in power plants based on digital twins according to claim 7, characterized in that, In step S1, the specific process for device attitude normalization and marker point configuration is as follows: the centroid coordinates of the cropped device point cloud are calculated, and the centering is achieved through coordinate translation; the principal direction of the horizontal plane of the point cloud is extracted through two-dimensional principal component analysis, and the principal direction is aligned to the global X-axis by rotating around the Z-axis, retaining the 90° discrete rotation and axis constraint fine-tuning functions, and eliminating the 180° orientation ambiguity caused by principal component analysis; the surface normal vector of the marker point is estimated by local principal component analysis with a neighborhood radius of 0.08-0.12m, and the orientation of the normal vector is constrained by the consistency of the camera's line of sight direction to ensure that the normal vector points to the observation side.

Images