Unmanned aerial vehicle cluster hydrological cooperative mapping method based on image semantics and language instructions

Abstract

The application discloses a kind of unmanned aerial vehicle cluster hydrology collaborative surveying and mapping methods based on image semantics and language instruction, belong to water conservancy and hydrology monitoring technical field.Method includes: constructing water area image and surveying and mapping task instruction expert database;With visual basic model, the structured description is output to the semantic segmentation of water area image, and with large language model as inference engine, task instruction is generated by matching expert database, and collaborative task allocation model is established;Under the distributed Gaussian splashing mapping framework, each unmanned aerial vehicle estimates pose through visual inertial odometry, shares key frame and Gaussian parameter between machines, and after loop detection and pose optimization, global water area three-dimensional scene is constructed by subgraph fusion.The application realizes the full-process automation of water area surveying and mapping from image understanding to three-dimensional reconstruction, significantly improves the surveying and mapping efficiency and reconstruction quality, and is suitable for digital twin river basin construction and low-altitude hydrology emergency monitoring scene.

Description

Technical Field

[0001] This invention relates to the field of water conservancy and hydrological monitoring technology, specifically to a method for collaborative hydrological mapping by unmanned aerial vehicle (UAV) swarms based on image semantics and language commands. Background Technology

[0002] In recent years, with the rapid development of the low-altitude economy and the in-depth advancement of water conservancy informatization, unmanned aerial vehicle (UAV) swarms have shown broad application prospects in the fields of water area monitoring and hydrological mapping. Traditional aerial and satellite remote sensing methods have inherent limitations in terms of timeliness, resolution, and flexibility, while ground-based manual measurement methods are inefficient and have limited coverage, making it difficult to meet the demand for high-precision, high-timeliness spatial data in digital twin watershed construction and hydrological emergency monitoring.

[0003] Unmanned aerial vehicle (UAV) swarms, with their advantages of mobility, cost control, and parallel operation, have become an important means of hydrological monitoring in water areas. However, the current application of UAV swarms in water mapping still faces the following prominent problems:

[0004] First, the planning of surveying and mapping tasks relies excessively on human experience. Task planning aspects such as formation selection, trajectory planning, and sensor configuration for drone swarms heavily depend on the individual experience of pilots and hydrographic surveying experts. The surveying plans developed by different operators vary significantly, and expert experience is difficult to systematically accumulate and reuse, resulting in inconsistent planning efficiency and quality for surveying and mapping tasks.

[0005] Second, there is a semantic gap between scene understanding of water area images and mission command generation. Traditional methods typically use water area images as a mapping reference rather than a basis for decision-making, with environmental information extracted manually from the images and then manually converted into flight mission commands. This process is not only time-consuming and labor-intensive, but also prone to information omissions or misjudgments due to human error, reducing the accuracy and reliability of mapping tasks.

[0006] Third, the efficiency of collaborative mapping and 3D reconstruction in UAV swarms urgently needs improvement. In existing technologies, mapping data collected by multiple UAVs typically needs to be transmitted back to a ground station for centralized processing. This involves large data volumes and high bandwidth requirements, making it difficult to achieve real-time or near-real-time 3D scene construction. Furthermore, the lack of effective information sharing and collaborative mapping mechanisms among the UAVs leads to data redundancy and limited reconstruction accuracy.

[0007] Reference 1, "A Method and System for Unmanned Aerial Vehicle (UAV) River and Lake Patrol Path Planning Based on BeiDou Navigation" (Patent CN121876995A), proposes a method for UAV river and lake patrol path planning based on BeiDou navigation. This method matches real-time UAV location information with preset key monitoring point information, collects environmental perception data and constructs a safe flight zone when adjustments are needed, calculates temporary monitoring points that meet data collection requirements, and constructs an optimized patrol path. This effectively solves the problems of inflexible route adjustments, low efficiency, and easy omission of important monitoring points in traditional patrols. However, this method mainly addresses the problem of dynamic trajectory adjustment and does not involve the full-process automation from image semantic understanding to multi-UAV collaborative task generation, nor does it achieve deep integration of cluster collaborative mapping and 3D reconstruction.

[0008] Reference 2, "A Heterogeneous Multi-Agent Cooperative SLAM Method and System for Air-Sea Collaboration" (Patent CN118500381A), proposes a heterogeneous multi-agent cooperative SLAM method for air-sea collaboration. Through the collaboration of UAVs and unmanned surface vessels (USVs), each USV platform independently estimates its initial pose and extracts keyframes and map points, sending them to a backend server. After loop closure detection and global optimization, co-localization factors are generated and sent back to each platform, achieving the second-stage pose estimation and local map update through factor graph optimization, thus improving the cooperative localization accuracy of multi-agents in large-scale scenarios. However, this method focuses on the underlying technologies of localization and mapping, relies on predefined collaborative strategies, cannot adaptively adjust the mapping scheme based on the semantic features of the water area images, and lacks a mechanism to directly integrate expert knowledge into the task generation process.

[0009] Reference 3, "Research on Automatic Extraction of Urban Water Bodies Based on Domestic High-Resolution Satellite Imagery" (Urban Surveying, 2024, Vol. 2, pp. 88-92), uses domestic high-resolution satellite (Jilin-1) imagery. Addressing the issues of mis-extraction and omission caused by building shadows and small water bodies in complex urban environments, it employs a deep learning model (U-Net) with both classification and pixel localization capabilities. By creating urban water body samples and iteratively training, it achieves automatic, rapid, and accurate extraction of water bodies within the Jiading urban area of ​​Shanghai, solving the problem of urban water body extraction to some extent. However, this method only focuses on the semantic interpretation of satellite remote sensing images and fails to effectively integrate the perception results with the autonomous task planning of UAV swarms, making it difficult to directly apply to actual water body surveying operations.

[0010] In summary, existing technologies have the following shortcomings: First, there is a lack of end-to-end automated connection between semantic understanding of water images and UAV swarm mission planning; second, there is a lack of systematic digital accumulation and reuse mechanisms for the implicit experience of pilots and hydrological experts; and third, the efficiency of collaborative mapping and 3D reconstruction of UAV swarms in the surveying process is not high, making it difficult to achieve real-time high-fidelity scene construction. Summary of the Invention

[0011] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and achieve full-process automation from user-captured water images to UAV swarm mapping task execution and 3D reconstruction through the collaborative work of structured perception driven by visual basic models, task generation constrained by expert knowledge, and distributed Gaussian splash mapping.

[0012] To solve the above problems, the present invention adopts the following technical solution:

[0013] This invention proposes a method for collaborative hydrological mapping using UAV swarms based on image semantics and language commands, comprising the following steps:

[0014] S1. Construct an expert database of water area images and surveying mission instructions, collect multimodal image samples of typical water area scenes, organize the operational experience of pilots and hydrological experts, and form a structured mission template database covering formation strategies, trajectory planning and sensor configuration.

[0015] S2. Establish a method for structured description and task conversion of aquatic environment based on the collaboration of a visual basic model and a large language model. Utilize the visual basic model with zero-shot generalization capability to perform semantic segmentation on the input aquatic image and output a structured aquatic environment description. Use the large language model as the inference engine to dynamically match the structured aquatic environment description with the expert database to generate UAV swarm task instructions. Establish a collaborative task allocation model with the goal of minimizing mapping blind spots and maximizing reconstruction accuracy, constrained by swarm communication bandwidth and single-unit power consumption. The optimization variables are the track points, formation, and scanning load of each UAV in the swarm.

[0016] S3. Under the framework of distributed Gaussian splashing collaborative mapping of UAV clusters, solve the collaborative mapping task. Each UAV estimates its own pose through visual inertial odometry, shares key frame features and local Gaussian parameters among the UAVs, and performs closed loop detection and relative pose calculation. Based on the distributed pose graph, optimize the unified multi-UAV coordinate system and construct a global three-dimensional water scene through sub-graph fusion.

[0017] Preferably, the specific steps for constructing the expert database of water area images and surveying task instructions in step S1 are as follows:

[0018] Collect multimodal image samples of typical hydrological scenarios, including rivers, lakes, reservoirs, wetlands, and estuaries and coastlines. The multimodal image samples include visible light images, multispectral images, and depth images.

[0019] The operational experience of drone pilots and hydrological experts was compiled, including formation selection criteria, flight path planning strategies, sensor parameter configuration schemes, and obstacle avoidance strategies for typical scenarios.

[0020] The multimodal image samples and operational experience are structured and encoded to form a structured task template library covering formation strategy, trajectory planning and sensor configuration;

[0021] The expert database is evaluated for quality and updated iteratively through reviews by domain experts.

[0022] Preferably, in step S2, the visual base model is a visual base model with zero-shot generalization capability, including SAM or CLIP, used for depth perception and semantic segmentation of the input water area image.

[0023] Preferably, in step S2, the structured aquatic environment description includes at least the water boundary coordinate sequence, obstacle location and type, water depth area classification, land feature category labeling, and scene confidence score, and the standard format is JSON or structured text format.

[0024] Preferably, in step S2, the structured environment description is dynamically matched with the expert database using a large language model as the inference engine. The specific steps are as follows:

[0025] The structured aquatic environment description is encoded into a query vector. Each task template in the expert database is encoded into a set of key vectors. , where N is the total number of task templates in the expert database.

[0026] Calculate the matching score between the query vector and each key vector. ,in Let i be the cosine similarity function, and i be the task template index. A higher score indicates a higher degree of matching between the current aquatic environment and the corresponding task template.

[0027] The task template with the highest matching score is selected as the basic scheme, and the large language model fills in and adapts the basic scheme with parameters according to the specific parameters in the structured aquatic environment description to generate executable UAV swarm task instructions.

[0028] Preferably, in step S2, the objective function of the collaborative task allocation model is as shown in formula (1):

[0029] (1);

[0030] In formula (1), To comprehensively optimize the objective function; The target for mapping coverage is defined as the ratio of the area of ​​the scanned region to the total area of ​​the target water body. The target for 3D reconstruction accuracy is defined as the inverse normalized value of the rendering error of the Gaussian splash model; This is a communication bandwidth penalty term used to constrain the load on inter-machine communication; , These are the weighting coefficients; Let be the coordinates of the three-dimensional track point of the i-th UAV at time t; Let i be the formation index of the i-th UAV at time t; The scanning load allocation ratio for the i-th UAV at time t represents the share of the total mapping task undertaken by that UAV.

[0031] The constraints of the collaborative task allocation model include:

[0032] Inter-machine communication bandwidth constraints:

[0033] (2);

[0034] Single-machine energy consumption constraints:

[0035] (3);

[0036] in, The total number of drone swarms participating in the surveying and mapping mission; The set of neighboring drone indices that maintain a communication link with the i-th drone; For a moment drones and Communication bandwidth between them The total available bandwidth of the cluster. To advance the power function, For sensor operating power, For communication power function, For the first Drone battery capacity, Let be the flight speed of the i-th drone at time t. Let t be the total bandwidth consumption for communication between the i-th UAV and all neighboring UAVs.

[0037] Preferably, the specific steps of the distributed Gaussian splashing collaborative mapping of the UAV cluster in step S3 are as follows:

[0038] S3.1 Set collaborative mapping parameters, including inter-machine communication frequency band and bandwidth, key frame sharing frequency, Gaussian subgraph fusion threshold, maximum number of iterations for distributed pose graph optimization, and loop closure detection matching threshold;

[0039] S3.2 Initialize the local Gaussian splash model of each UAV, and run the visual inertial odometry independently to estimate the local camera pose sequence;

[0040] S3.3 Each UAV extracts feature descriptors from the local keyframes and broadcasts them to neighboring UAVs via the inter-UAV communication link;

[0041] S3.4 Each UAV receives the feature descriptors of neighboring UAVs, matches them with local keyframes, and performs inter-UAV closed-loop detection.

[0042] S3.5 If an inter-machine closed loop is detected, calculate the relative pose transformation matrix. ,in Represent the rigid body transformation group in three-dimensional space, and add the closed-loop constraint to the local pose graph;

[0043] S3.6 Execute the distributed pose graph optimization algorithm, with each machine asynchronously updating its local pose graph to eliminate cumulative drift and achieve a unified coordinate system across multiple machines;

[0044] S3.7 Each machine sends the optimized Gaussian subgraph to the central fusion node or exchanges it via point-to-point method;

[0045] S3.8 The central fusion node or designated pilot drone receives all Gaussian subgraphs, performs subgraph fusion and deduplication of overlapping areas, and generates a global water area Gaussian splash 3D model.

[0046] S3.9 Determine whether the reconstruction accuracy meets the preset threshold. If it does, output the final model; otherwise, mark the under-reconstructed area, guide the UAV cluster to perform supplementary scanning, and jump to step S3.2.

[0047] Preferably, in step S3.6, the objective function of the distributed pose graph optimization algorithm is specifically as shown in formula (4):

[0048] (4);

[0049] In formula (4), For nodes and The pose to be optimized; For the edge The measurement transformation originates from the odometer constraint. or closed-loop constraints ; for arrive Logarithmic mapping; To measure the covariance matrix.

[0050] Preferably, in step S3.8, the sub-image fusion and overlapping region deduplication are specifically as follows:

[0051] All Gaussian elements in the Gaussian subgraphs are transformed to a unified world coordinate system through an optimized global pose transformation.

[0052] A spatial index is built for the transformed Gaussian set, and an octree or KD tree structure is used to accelerate the nearest neighbor search.

[0053] For Gaussian units that are spatially adjacent, if the spatial distance between two Gaussian units is less than a preset fusion threshold... Then perform the fusion operation: the center position of the fused Gaussian elements. The opacity after merging is calculated using an opacity-weighted average. Take the maximum of the two values, and the resulting covariance matrix is... Choose the one with the smaller covariance;

[0054] Output a global Gaussian splash 3D model without redundancy.

[0055] Preferably, the method further includes a continuous learning mechanism: after the mapping schemes generated by the system are reviewed by domain experts, the adopted schemes are entered into the positive sample library, and the corrected schemes are entered into the corrected sample library; the two types of data are used regularly for incremental training of the model, the positive sample library is used to strengthen the correct behavior of the model, and the corrected sample library is used to correct the erroneous tendencies of the model, so that the mapping capability of the system continues to improve with the increase of usage time.

[0056] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0057] 1. By using a structured perception driven by a visual foundation model, deep semantic understanding of water images is achieved, avoiding information loss in the feature extraction and semantic generation process of traditional CNN-LSTM methods, and significantly improving the accuracy and structure of water environment description;

[0058] 2. By using an expert knowledge-constrained task generation mechanism, the implicit experience of senior drone pilots and hydrological experts is transformed into reusable digital assets, effectively solving the problem of excessive reliance on human experience in traditional surveying and mapping task planning and lowering the threshold for surveying and mapping task planning.

[0059] 3. Through the distributed Gaussian splash collaborative mapping framework, real-time pose estimation, inter-drone information sharing, and global 3D scene construction of UAV swarms during flight were realized, which greatly improved the efficiency and reconstruction quality of water area mapping;

[0060] 4. Through an end-to-end process of "expert database construction - image understanding - task planning - cluster collaboration - real-time reconstruction", the entire process from user-captured images to the generation of 3D hydrological models is fully automated, which is especially suitable for digital twin watershed construction and low-altitude hydrological emergency monitoring scenarios. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the method flow involved in the present invention.

[0062] Figure 2 A schematic diagram of the expert database construction and structured perception process.

[0063] Figure 3 Flowchart for solving the collaborative task allocation model.

[0064] Figure 4 Flowchart for distributed Gaussian splash collaborative graph building. Detailed Implementation

[0065] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0066] This embodiment is a specific implementation of a UAV swarm hydrological collaborative mapping method based on image semantics and language commands, referencing... Figure 1 The details are as follows:

[0067] S1. Construct an expert database of water area images and surveying and mapping task instructions through multi-source surveys.

[0068] Multimodal image samples of typical aquatic scenarios are collected, including at least rivers, lakes, reservoirs, wetlands, and estuaries / coasts. These multimodal image samples include visible light images, multispectral images, and depth images. The operational experience of senior pilots and hydrological experts is compiled, including at least formation selection criteria, trajectory planning strategies, sensor parameter configuration schemes, and obstacle avoidance strategies for typical scenarios. The multimodal image samples and operational experience are structured and encoded to form a structured task template library covering formation strategies, trajectory planning, and sensor configuration. The expert library is evaluated and iteratively updated through domain expert review to provide knowledge constraints for subsequent semantic parsing and task generation.

[0069] S2: Establish a structured description and task transformation method for the aquatic environment based on the collaboration of a visual basic model and a large language model.

[0070] like Figure 2 As shown, firstly, a visual basic model with zero-shot generalization capability (preferably CLIP (Contrastive Language-Image Pre-Training) in this embodiment) is used to perform depth perception and semantic segmentation on the input water area image, extracting refined spatial structured information such as water area boundaries, obstacle distribution, water depth regions, and land feature categories, and directly outputting a structured water environment description in a standard format (preferably JSON or structured text format in this embodiment). The structured water environment description includes at least the water area boundary coordinate sequence, obstacle location and type, water depth region classification, land feature category labeling, and scene confidence score.

[0071] Secondly, using a large language model as the inference engine, the structured environment description is dynamically matched with the expert database constructed in S1. The structured aquatic environment description is encoded as a query vector q, and each task template in the expert database is encoded as a set of key vectors. ,in Encode the Nth task, where N is the total number of task templates in the expert database; calculate the matching score between the query vector and each key vector. ,in Let i be the cosine similarity function and i be the task template index. The task template with the highest matching score is selected as the basic scheme, and the large language model fills in and adapts the basic scheme according to the specific parameters in the structured environment description to generate an executable UAV swarm task instruction that includes formation type, trajectory strategy and load distribution.

[0072] Finally, a collaborative task allocation model is established with multiple objectives of minimizing mapping blind spots and maximizing reconstruction accuracy, constrained by cluster communication bandwidth and individual unit power consumption. The optimization variables are the waypoints, formation, and scanning load of each UAV within the cluster. Figure 3 As shown, the objective function of the collaborative task allocation model is specifically shown in formula (1):

[0073] (1);

[0074] In formula (1), To comprehensively optimize the objective function; The target for mapping coverage is defined as the ratio of the area of ​​the scanned region to the total area of ​​the target water body. The target for 3D reconstruction accuracy is defined as the inverse normalized value of the rendering error of the Gaussian splash model; This is a communication bandwidth penalty term used to constrain the load on inter-machine communication; , These are the weighting coefficients; Let be the coordinates of the three-dimensional track point of the i-th UAV at time t; Let be the formation index of the i-th UAV at time t; Let be the scanning load allocation ratio for the i-th UAV at time t, representing the share of the total mapping task undertaken by that UAV.

[0075] The constraints of the collaborative task allocation model include:

[0076] Inter-machine communication bandwidth constraints:

[0077] (2);

[0078] Single-machine energy consumption constraints:

[0079] (3);

[0080] in, The total number of drone swarms participating in the surveying and mapping mission; The set of neighboring drone indices that maintain a communication link with the i-th drone; For a moment drones and Communication bandwidth between them The total available bandwidth of the cluster. To advance the power function, For sensor operating power, For communication power function, For the first Drone battery capacity, Let be the flight speed of the i-th drone at time t. Let t be the total bandwidth consumption for communication between the i-th UAV and all neighboring UAVs.

[0081] S3. Solve the collaborative mapping task within the framework of distributed Gaussian splashing collaborative mapping of UAV clusters.

[0082] like Figure 4 As shown, the specific steps of the distributed Gaussian splashing collaborative mapping of the UAV cluster in step S3 are as follows:

[0083] S3.1: Set collaborative mapping parameters, including inter-machine communication frequency band and bandwidth, key frame sharing frequency, Gaussian subgraph fusion threshold, maximum number of iterations for distributed pose graph optimization, and loop closure detection matching threshold;

[0084] S3.2: Initialize the local Gaussian splash model for each UAV, and run the visual inertial odometry independently for each UAV to estimate the local camera pose sequence;

[0085] S3.3: Each UAV extracts feature descriptors from the local keyframes and broadcasts them to neighboring UAVs via the inter-UAV communication link;

[0086] S3.4: Each UAV receives the feature descriptors of neighboring UAVs, matches them with local keyframes, and performs inter-UAV loop closure detection.

[0087] S3.5: If an inter-machine closed loop is detected, calculate the relative pose transformation matrix. And add the closed-loop constraint to the local pose graph;

[0088] S3.6: Run the distributed pose graph optimization algorithm, with each machine asynchronously updating its local pose graph to eliminate accumulated drift and achieve a unified coordinate system across multiple machines. The objective function of the distributed pose graph optimization algorithm is shown in formula (4):

[0089] (4);

[0090] In formula (4), For nodes and The pose to be optimized; For the edge The measurement transformation originates from the odometer constraint. or closed-loop constraints ; for arrive Logarithmic mapping; To measure the covariance matrix.

[0091] S3.7: Each machine sends the optimized Gaussian subgraph to the central fusion node or exchanges it via point-to-point method;

[0092] S3.8: The central fusion node or designated lead drone receives all Gaussian subgraphs and performs subgraph fusion and deduplication of overlapping regions. Specifically: Gaussian elements in all Gaussian subgraphs are transformed to a unified world coordinate system using optimized global pose; a spatial index is established for the transformed Gaussian element set, using an octree or KD-tree structure to accelerate nearest neighbor search; for spatially adjacent Gaussian elements, if the spatial distance between two Gaussian elements is less than a preset fusion threshold... Then perform the fusion operation: the center position of the fused Gaussian elements. The opacity after merging is calculated using an opacity-weighted average. Take the maximum of the two values, and the resulting covariance matrix is... Select the model with the smaller covariance; output a global Gaussian splash 3D model without redundancy. The specific mathematical process is as follows:

[0093] Step 1: Coordinate system transformation:

[0094] Let the local Gaussian subgraph of the k-th UAV be . ,in Let Gaussian elements be the number of Gaussian elements in this subgraph, and each Gaussian element be represented as:

[0095] (5);

[0096] in: Let be the position vector of the Gaussian metacenter in the world coordinate system; The covariance matrix describes the spatial distribution shape of the Gaussian elements along the three axes; RGB color vector; Opacity.

[0097] The global pose transformation matrix obtained through distributed pose graph optimization in step S3.6 Transform all Gaussian elements to a unified world coordinate system:

[0098] (6);

[0099] (7);

[0100] in, and Transformation matrices The rotation and translation components, color vector, and opacity remain invariant during coordinate system transformations.

[0101] (8).

[0102] Step 2: Spatial Index Construction and Proximity Search

[0103] For the transformed set of all Gaussian elements Establish a spatial index structure, using an octree or KD-tree to divide the space into a recursive subspace hierarchy, and center the Gaussian elements. Using it as an index key, the time complexity of nearest neighbor search is reduced from... Reduce to ,in The total number of all Gaussian elements.

[0104] For each Gaussian element Search the spatial index for all Gaussian elements that satisfy the following conditions.

[0105] (9);

[0106] in This is the preset sub-image fusion threshold, in meters. The nearest neighbor set is

[0107] Step 3: Gaussian merging operation:

[0108] like Then for The Gaussian elements and all their nearest neighbors are fused to generate new Gaussian elements.

[0109] (10);

[0110] That is, the center position of the merged Gaussian elements according to their opacity Weighted average calculation. Let g be the opacity of the Gaussian element. Let g be the center position of the Gaussian element g.

[0111] (11);

[0112] The opacity after fusion is taken as the maximum value of the opacity among all Gaussian elements involved in the fusion, in order to preserve high-confidence information.

[0113] ,in (12);

[0114] The covariance matrix after fusion is taken from the Gaussian elements that participate in the fusion, which has the smallest determinant of the covariance matrix and corresponds to the Gaussian element with the most compact spatial distribution, in order to ensure the geometric accuracy of the fused model.

[0115] (13);

[0116] The blended color vectors are calculated using an opacity-weighted average to achieve a smooth color transition.

[0117] After merging, the original Gaussian elements are removed from the global set. Add the merged Gaussian elements .

[0118] Step 4: Global Render Verification

[0119] After fusion, it is rendered using a differentiable rasterization function. Verify the reconstruction quality of the global model. For a given query camera pose... The color of pixel u in the rendered image is calculated by mixing all Gaussian elements:

[0120] (14);

[0121] in:

[0122] (15);

[0123] (16);

[0124] In the formula, This is a camera projection function that projects 3D world coordinates onto image pixel coordinates. For camera rotation matrix; For projection function The Jacobian matrix at the projection point describes the local linear mapping relationship of the projection transformation as the spatial coordinates change.

[0125] The final output is a global Gaussian splash 3D model without redundancy. The model is based on Based on this, after the above fusion and deduplication operations, it is output in a structured point cloud file format (.ply or .splat).

[0126] S3.9: Determine whether the reconstruction accuracy meets the preset threshold. If it does, output the final model; otherwise, mark the under-reconstructed area, guide the UAV cluster to perform supplementary scanning, and jump to S3.2.

[0127] As a preferred embodiment of the present invention, the method further includes a continuous learning mechanism: after the mapping schemes generated by the system are reviewed by domain experts, the adopted schemes are entered into the positive sample library, and the corrected schemes are entered into the corrected sample library; the two types of data are used periodically for incremental training of the model, the positive sample library is used to strengthen the correct behavior of the model, and the corrected sample library is used to correct the erroneous tendencies of the model, so that the mapping capability of the system continues to improve with the increase of usage time.

[0128] This invention significantly improves the automation and intelligence level of water area mapping by deeply integrating structured perception driven by a visual basic model, task generation constrained by expert knowledge, and distributed Gaussian splash mapping. It is especially suitable for digital twin watershed construction and low-altitude hydrological emergency monitoring scenarios.

[0129] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. An unmanned aerial vehicle cluster hydrological cooperative surveying and mapping method based on image semantics and language instructions, characterized in that, Includes the following steps: S1. Construct an expert database of water area images and surveying mission instructions, collect multimodal image samples of typical water area scenes, organize the operational experience of pilots and hydrological experts, and form a structured mission template database covering formation strategies, trajectory planning and sensor configuration. S2. Establish a method for structured description and task conversion of aquatic environment based on the collaboration of a visual basic model and a large language model. Utilize the visual basic model with zero-shot generalization capability to perform semantic segmentation on the input aquatic image and output a structured aquatic environment description. Use the large language model as the inference engine to dynamically match the structured aquatic environment description with the expert database to generate UAV swarm task instructions. Establish a collaborative task allocation model with the goal of minimizing mapping blind spots and maximizing reconstruction accuracy, constrained by swarm communication bandwidth and single-unit power consumption. The optimization variables are the track points, formation, and scanning load of each UAV in the swarm. S3. Under the framework of distributed Gaussian splashing collaborative mapping of UAV clusters, solve the collaborative mapping task. Each UAV estimates its own pose through visual inertial odometry, shares key frame features and local Gaussian parameters among the UAVs, and performs closed loop detection and relative pose calculation. Based on the distributed pose graph, optimize the unified multi-UAV coordinate system and construct a global three-dimensional water scene through sub-graph fusion.

2. The method according to claim 1, characterized in that, The specific steps for constructing the expert database of water area images and surveying task instructions in step S1 are as follows: Collect multimodal image samples of typical hydrological scenarios, including rivers, lakes, reservoirs, wetlands, and estuaries and coastlines. The multimodal image samples include visible light images, multispectral images, and depth images. The operational experience of drone pilots and hydrological experts was compiled, including formation selection criteria, flight path planning strategies, sensor parameter configuration schemes, and obstacle avoidance strategies for typical scenarios. The multimodal image samples and operational experience are structured and encoded to form a structured task template library covering formation strategy, trajectory planning and sensor configuration; The expert database is evaluated for quality and updated iteratively through reviews by domain experts.

3. The method according to claim 1, characterized in that, In step S2, the visual base model is a visual base model with zero-shot generalization capability, including SAM or CLIP, used for depth perception and semantic segmentation of the input water area image.

4. The method according to claim 1, characterized in that, In step S2, the structured aquatic environment description includes at least the water boundary coordinate sequence, obstacle location and type, water depth area classification, land feature category labeling, and scene confidence score. The standard format is JSON or structured text format.

5. The method according to claim 1, characterized in that, In step S2, the structured environment description is dynamically matched with the expert database using a large language model as the inference engine. The specific steps are as follows: The structured aquatic environment description is encoded into a query vector. Each task template in the expert database is encoded into a set of key vectors. , where N is the total number of task templates in the expert database; Calculate the matching score between the query vector and each key vector. ,in Let i be the cosine similarity function, and i be the task template index. A higher score indicates a higher degree of matching between the current aquatic environment and the corresponding task template. The task template with the highest matching score is selected as the basic scheme, and the large language model fills in and adapts the basic scheme with parameters according to the specific parameters in the structured aquatic environment description to generate executable UAV swarm task instructions.

6. The method according to claim 1, characterized in that, In step S2, the objective function of the collaborative task allocation model is specifically shown in formula (1): （1）； In formula (1), To comprehensively optimize the objective function; The target for mapping coverage is defined as the ratio of the area of ​​the scanned region to the total area of ​​the target water body. The target for 3D reconstruction accuracy is defined as the inverse normalized value of the rendering error of the Gaussian splash model; This is a communication bandwidth penalty term used to constrain the load on inter-machine communication; , These are the weighting coefficients; Let be the coordinates of the three-dimensional track point of the i-th UAV at time t; Let be the formation index of the i-th UAV at time t; Let represent the scanning load allocation ratio for the i-th UAV at time t, and denot represent the total mapping task share undertaken by that UAV. The constraints of the collaborative task allocation model include: Inter-machine communication bandwidth constraints: （2）； Single-machine energy consumption constraints: （3）； in, The total number of drone swarms participating in the surveying and mapping mission; The set of neighboring drone indices that maintain a communication link with the i-th drone; For a moment drones and Communication bandwidth between them The total available bandwidth of the cluster. To advance the power function, For sensor operating power, For communication power function, For the first Drone battery capacity, Let be the flight speed of the i-th drone at time t. Let t be the total bandwidth consumption for communication between the i-th UAV and all neighboring UAVs.

7. The method according to claim 1, characterized in that, The specific steps of the distributed Gaussian splashing collaborative mapping of the UAV swarm described in step S3 are as follows: S3.1 Set collaborative mapping parameters, including inter-machine communication frequency band and bandwidth, key frame sharing frequency, Gaussian subgraph fusion threshold, maximum number of iterations for distributed pose graph optimization, and loop closure detection matching threshold; S3.2 Initialize the local Gaussian splash model of each UAV, and run the visual inertial odometry independently to estimate the local camera pose sequence; S3.3 Each UAV extracts feature descriptors from the local keyframes and broadcasts them to neighboring UAVs via the inter-UAV communication link; S3.4 Each UAV receives the feature descriptors of neighboring UAVs, matches them with local keyframes, and performs inter-UAV closed-loop detection. S3.5 If an inter-machine closed loop is detected, calculate the relative pose transformation matrix. ,in Represent the rigid body transformation group in three-dimensional space, and add the closed-loop constraint to the local pose graph; S3.6 Execute the distributed pose graph optimization algorithm, with each machine asynchronously updating its local pose graph to eliminate cumulative drift and achieve a unified coordinate system across multiple machines; S3.7 Each machine sends the optimized Gaussian subgraph to the central fusion node or exchanges it via point-to-point method; S3.8 The central fusion node or designated pilot drone receives all Gaussian subgraphs, performs subgraph fusion and deduplication of overlapping areas, and generates a global water area Gaussian splash 3D model. S3.9 Determine whether the reconstruction accuracy meets the preset threshold. If it does, output the final model; otherwise, mark the under-reconstructed area, guide the UAV cluster to perform supplementary scanning, and jump to step S3.

2.

8. The method according to claim 7, characterized in that, In step S3.6, the objective function of the distributed pose graph optimization algorithm is specifically shown in formula (4): （4）； In formula (4), For nodes and The pose to be optimized; For the edge The measurement transformation originates from the odometer constraint. or closed-loop constraints ; for arrive Logarithmic mapping; To measure the covariance matrix.

9. The method according to claim 7, characterized in that, In step S3.8, the sub-image fusion and overlapping region deduplication are specifically as follows: All Gaussian elements in the Gaussian subgraphs are transformed to a unified world coordinate system through an optimized global pose transformation. A spatial index is built for the transformed Gaussian set, and an octree or KD tree structure is used to accelerate the nearest neighbor search. For Gaussian units that are spatially adjacent, if the spatial distance between two Gaussian units is less than a preset fusion threshold... Then perform the fusion operation: the center position of the fused Gaussian elements. The opacity after merging is calculated using an opacity-weighted average. Take the maximum of the two values, and the resulting covariance matrix is... Choose the one with the smaller covariance; Output a global Gaussian splash 3D model without redundancy.

10. The method according to claim 1, characterized in that, The method also includes a continuous learning mechanism: after the mapping schemes generated by the system are reviewed by domain experts, the adopted schemes are entered into the positive sample library, and the corrected schemes are entered into the corrected sample library; the two types of data are used regularly for incremental training of the model, the positive sample library is used to strengthen the correct behavior of the model, and the corrected sample library is used to correct the erroneous tendencies of the model, so that the mapping capability of the system continues to improve with the increase of usage time.

Images