Multi-machine cooperative path planning method and system based on visual language large model

Abstract

The invention discloses a multi-machine collaborative path planning method and system based on a visual language large model, and the method specifically comprises the steps: firstly, carrying out the semantic information extraction of a received text instruction based on a large language model; positioning a flight target area in a satellite map by combining the extracted semantic information with the reasoning ability of a visual language model; a contour coefficient method and a clustering algorithm are adopted to carry out adaptive analysis on the target area, and intelligent task distribution of multiple unmanned aerial vehicles is realized; and finally, generating an initial feasible path covering a task area for each unmanned aerial vehicle based on the task environment understanding capability of the visual language model, performing joint optimization by taking the minimum flight path, time and energy consumption as targets, and outputting a collaborative flight path. According to the method, intelligent planning fusing the multi-modal understanding ability of the visual language large model and the multi-unmanned aerial vehicle collaborative optimization mechanism is realized, and the autonomous planning ability, collaborative execution efficiency and adaptive ability to a complex task environment of an unmanned aerial vehicle cluster are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent path planning technology for unmanned aerial vehicles (UAVs), and in particular to a multi-UAV collaborative path planning method and system based on a large visual language model. Background Technology

[0002] With the rapid development of drone technology and artificial intelligence, drones are widely used in disaster monitoring, environmental inspection, precision agriculture, logistics and transportation, and military reconnaissance, gradually becoming important equipment for achieving intelligent and autonomous airspace operations. Compared with single-drone operations, multi-drone collaborative path planning systems can significantly improve operational efficiency, coverage, and system robustness through task decomposition and information sharing, thus becoming a key research direction for intelligent unmanned systems. However, in large-scale and complex mission scenarios, existing manual-led planning methods still suffer from operational complexity and low efficiency, limiting the intelligent and autonomous development of drone systems.

[0003] Currently, multi-UAV collaborative path planning mainly employs rule-based optimization algorithms. For example, invention patent CN115115785 A discloses a multi-UAV collaborative 3D modeling system and method for search and rescue in mountainous and forest environments. This system uses a ground station control module to communicate with the UAVs, sending the initially generated coarse flight trajectory to the UAVs, controlling their flight, and receiving their status information. A real-time aerial image stitching module performs real-time stitching based on feature-matched images. A depth estimation module uses a deep learning-based monocular image depth estimation model to obtain a global depth map of the stitched image. The multi-UAV collaborative path planning module identifies high-altitude areas in the global scene as key areas, generates oblique photography trajectories for these key areas, and then uses an improved k-clustering algorithm to divide all trajectory points into categories with balanced workloads, allowing the UAVs to traverse all trajectory points in each category. A 3D reconstruction model of the search area is then built based on the images transmitted by the UAVs. However, such methods typically rely on manually designed mathematical models for solving problems, making them unsuitable for intelligent decision-making in complex environments. In recent years, with the rapid development of artificial intelligence technology, data-driven methods such as deep learning and reinforcement learning have been gradually introduced into the field of UAV path planning. Through model training, the mapping from environmental perception to decision output is achieved, which improves the intelligence level of path planning to a certain extent. For example, invention patent CN 120688710 A discloses a method, system, device, and storage medium for multi-UAV collaborative path planning in urban areas. By defining a task scenario in urban low-altitude airspace, it uses the MP GWO algorithm to generate initial paths for multi-UAV collaboration, sets differentiated penalty coefficients for different types of no-fly zones, embeds the K-means clustering mechanism into the GWO main loop, divides parallel subpopulations based on multi-objective fitness vectors, and each subpopulation independently executes α, β, and δ guided update strategies. It integrates a triple termination judgment mechanism of fixed iteration count, convergence threshold, and elite stagnation to output the current optimal planned path, achieving efficient path optimization in complex urban environments and is suitable for multi-UAV collaborative tasks in urban low-altitude airspace. However, this type of method relies on a large amount of high-quality data for model training, resulting in high data acquisition costs and limited model generalization ability. Furthermore, existing path planning methods are mostly based on single-modal input and cannot understand task semantic information and multimodal environmental features. With the rise of large language models, building a multimodal intelligent path planning framework that can understand and utilize semantic information has become a new research trend.

[0004] On the other hand, existing research mainly focuses on semantic understanding and decision optimization for single drones, while there is still significant room for improvement in multi-drone collaborative semantic understanding and joint planning. Therefore, there is an urgent need to construct an intelligent planning framework that integrates the multimodal understanding capabilities of a large visual language model with a multi-drone collaborative optimization mechanism. This framework would enable the system to perform task semantic parsing, target recognition, collaborative allocation, and joint path optimization in complex task environments, thereby significantly improving the autonomous planning capability and collaborative execution efficiency of drone swarms. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-drone collaborative path planning method and system that can improve the autonomous planning capability and collaborative execution efficiency of unmanned aerial vehicle (UAV) swarms.

[0006] The technical solution to achieve the purpose of this invention is: a multi-machine collaborative path planning method based on a large visual language model, comprising the following steps:

[0007] Step 1: The target extraction module receives natural language instructions, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information.

[0008] Step 2: Based on the target semantics extracted by the target extraction module, load satellite images of the relevant region, use the reasoning ability of the visual language model to locate the region where the target may appear, and mark the location of the target object in the processed image;

[0009] Step 3: Calculate the optimal number of drones to be scheduled using the silhouette coefficient method, and rationally divide and allocate the task area based on the K-means clustering algorithm;

[0010] Step 4: Combining the visual language model with the understanding and reasoning of the mission environment, the MST-DFS strategy is used to generate an initial feasible path covering the mission area for each UAV, and joint optimization is performed with the goal of minimizing the total flight path, time and energy consumption, and the optimal cooperative flight path is output.

[0011] Furthermore, the target extraction module described in step 1 receives natural language instruction input, automatically identifies semantics and extracts task-related targets using a large language model, and outputs key information, as follows:

[0012] Step 1.1: The target extraction module parses the user's natural language commands, further extracting key task targets and object types; the natural language commands are parsed and represented as:

[0013]

[0014] in, Provide input suggestions; For instruction units; The length of the input prompt varies with the complexity of the task;

[0015] Step 1.2: The domestically developed large language model, Tongyi Qianwen, is used as the backend inference engine. User input is received via a standardized API interface, and the model is guided to extract key information using pre-set prompt templates. Finally, the data structure is output in JSON format, i.e.:

[0016]

[0017] in, The list of targets to be identified, derived from the instructions of the natural language task, provides a clear target for the next stage of processing; A standardized string label representing a target; The length of the list.

[0018] Furthermore, in step 2, based on the target semantics extracted by the target extraction module, satellite images of the relevant region are loaded, the reasoning ability of the visual language model is used to locate the region where the target may appear, and the location of the target object is marked in the processed image, as follows:

[0019] Step 2.1: Use the target search module for visual understanding and task target localization. This module receives JSON format task instructions from the target extraction module, then loads high-resolution satellite images of the corresponding area and constructs text prompts.

[0020] Step 2.2: Use the Molmo-7B-O-bnb-4bit lightweight visual language model for multimodal reasoning, combining image content and text prompts to generate text output containing the location information of each target object:

[0021]

[0022] in, , They represent the first The vertical and horizontal coordinates of each target in the image coordinate system;

[0023] Step 2.3: Extract the initial position of the target object using a coordinate analysis function:

[0024]

[0025] in, It expresses the relative position of the target in the graph as a percentage;

[0026] Step 2.4: Use coordinate transformation functions to determine the relative position of the target in the graph. Mapped to latitude and longitude coordinates in real geographic space:

[0027]

[0029] Furthermore, step 3 involves calculating the optimal number of drones to be scheduled using the contour coefficient method, and then rationally dividing and allocating the task area based on the K-means clustering algorithm, as detailed below:

[0030] Step 3.1: Calculate the optimal number of drones to be scheduled, and set the number of drones to be scheduled as follows. Frame, among which The optimal number of drones is determined by calculating the optimal profile coefficient. :

[0031]

[0032] in, Represents the profile coefficient, when The closer the value is to 1, the smaller the intra-cluster distance and the larger the inter-cluster distance, resulting in better clustering performance and a higher target point. Correctly assigned to a cluster; Indicates the target point The average distance to other target points in the same cluster; Indicates the target point The average distance to all target points in each cluster is calculated, and the minimum value is taken.

[0033] Step 3.2: Determine the number of drones that need to be scheduled. After setting up, gather the target points. Divided into Each cluster, i.e. Each cluster corresponds to the operating area of ​​one drone. The objective is to minimize the sum of Euclidean distances from points within the cluster to the cluster center, as follows:

[0034] First, randomly select from the target points in the scene. Target points As the initial centroid of each cluster, the distance from all target points in the scene to the initial centroid is calculated, and the target points are assigned to the nearest cluster; then, the average coordinates of all target points in each cluster are taken. Update the centroids of each cluster; finally, reselect the initial centroids and repeat the update process until the centroid positions no longer change. Stop the loop and determine the best heart. The number of drones to be scheduled is obtained, and the spatial adaptive division of the target points in the task area is realized.

[0035] Furthermore, step 4 involves combining the visual language model with the understanding and reasoning of the task environment, using the MST-DFS strategy to generate an initial feasible path covering the task area for each UAV, and performing joint optimization with the objective of minimizing the total flight path, time, and energy consumption to output the optimal cooperative flight path, as detailed below:

[0036] Step 4.1: Using a strategy combining minimum spanning tree and depth-first search (MST-DFS), construct an initial feasible flight path covering all target points, as follows:

[0037] The set of target points obtained by each drone Let the edge weight be the Euclidean distance between any two points, i.e. ,in, Let the starting point of the drone's flight be... A one-way path that can cover all target points is ,in Let the coordinates of the path points be denoted as: Then the length of the UAV's flight path is expressed as:

[0038]

[0039] in, Representing a path The total flight length corresponding to the optimal path is expressed as:

[0040]

[0041] The initial path solution for multi-machine collaborative processing is constructed using the MST-DFS strategy. First, an MST is built. Starting from any target point, the strategy of "connecting visited and unvisited points with the shortest edge" is adopted to gradually expand and generate an acyclic connected graph that covers all target points and has the minimum total edge weight; this is the minimum spanning tree. The total edge weight of MST is defined as follows:

[0042]

[0043] Among them, minimum spanning tree The following formula can be used to solve for:

[0044]

[0045] Since any path covering all target points can form a spanning tree by adding one edge, therefore we have That is, the total weight of MST is less than the optimal path length;

[0046] Then, the constructed MST is traversed using DFS, starting from the starting point. Begin by exploring the tree's edges, prioritizing child nodes. Once a leaf node is reached, backtrack and continue exploring other branches, recording the initial order of node visits to form a complete visit sequence. This represents the drone's flight path; since each edge is traversed once during the DFS traversal, the total traversal length is... It is twice the weight of MST, that is:

[0047]

[0048] Finally, note the first-visit order of the DFS sequence. Construct the final drone flight path:

[0049]

[0050] Step 4.2: Employ the 2-opt local optimization algorithm to optimize the flight path by attempting to swap edges along the path, as detailed below:

[0051] First, for pairs that are not adjacent in the path... and , Then perform the "swap operation", which involves changing the path interval. Reverse, the new path is:

[0052]

[0053] The two edges that were deleted are , The two newly added edges are , When the path length increments The new path has a shorter total length, so the path is updated; Represented as:

[0054]

[0055] Iterate through non-adjacent pairs in the path until no better swap can be found, finally obtaining the flight path:

[0056]

[0058] A multi-machine collaborative path planning system based on a large visual language model is disclosed. This system implements the aforementioned multi-machine collaborative path planning method based on a large visual language model. The system includes a target extraction module, a visual language model inference module, and:

[0059] The target extraction module receives natural language commands, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information.

[0060] The visual language model inference module, based on the target semantics extracted by the target extraction module, loads satellite images of the relevant region, uses the inference ability of the visual language model to locate the region where the target may appear, and marks the location of the target object in the processed image;

[0061] The task area division module uses the contour coefficient method to calculate the optimal number of UAVs to be scheduled, and uses the K-means clustering algorithm to reasonably divide and allocate the task area.

[0062] The joint optimization module combines the understanding and reasoning of the mission environment by the visual language model, uses the MST-DFS strategy to generate an initial feasible path covering the mission area for each UAV, and performs joint optimization with the goal of minimizing the total flight path, time and energy consumption, and outputs the optimal cooperative flight path.

[0063] A mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multi-machine collaborative path planning method based on a large visual language model.

[0064] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multi-machine cooperative path planning method based on a large visual language model.

[0065] A computer device includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the multi-machine cooperative path planning method based on a large visual language model.

[0066] A computer program product includes computer instructions for causing a computer to execute the multi-machine collaborative path planning method based on a large visual language model.

[0067] Compared with the prior art, the present invention has the following significant advantages: (1) Based on the visual language big model and a variety of optimization algorithms, it realizes intelligent planning that integrates the multimodal understanding capability of the visual language big model with the multi-UAV collaborative optimization mechanism, and realizes the effective combination of the visual language big model and the multi-UAV collaborative UAV path planning, which significantly improves the autonomous planning capability and collaborative execution efficiency of the UAV cluster; (2) The intelligent planning method that integrates the multimodal understanding capability of the visual language big model with the multi-UAV collaborative optimization mechanism enables the system to realize task semantic parsing, target recognition, collaborative allocation and path joint optimization in complex task environments, which improves the adaptability to complex task environments. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating a multi-machine collaborative path planning method based on a large visual language model according to the present invention.

[0069] Figure 2 This is a schematic diagram of the target search results based on the visual language big model in this embodiment.

[0070] Figure 3 This is a schematic diagram of the multi-machine collaborative path planning results based on a large visual language model provided in this embodiment.

[0071] Figure 4 This is a comparison chart of the average flight length of manually planned and multi-aircraft collaboratively planned paths provided in the embodiment. Detailed Implementation

[0072] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, this invention provides a multi-machine cooperative path planning method based on a large visual language model, comprising the following steps:

[0074] Step 1: The target extraction module receives natural language instructions, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information.

[0075] Step 2: Based on the target semantics extracted by the target extraction module, load satellite images of the relevant region, use the reasoning ability of the visual language model to locate the region where the target may appear, and mark the location of the target object in the processed image;

[0076] Step 3: Calculate the optimal number of drones to be scheduled using the silhouette coefficient method, and rationally divide and allocate the task area based on the K-means clustering algorithm;

[0077] Step 4: Combining the visual language model with the understanding and reasoning of the mission environment, the MST-DFS strategy is used to generate an initial feasible path covering the mission area for each UAV, and joint optimization is performed with the goal of minimizing the total flight path, time and energy consumption, and the optimal cooperative flight path is output.

[0078] As a specific example, the target extraction module in step 1 receives natural language instructions, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information, as follows:

[0079] Step 1.1: The target extraction module parses the user's natural language commands to further extract key task targets and object types; the system parses and represents the natural language commands as follows:

[0080]

[0081] in, Provide input suggestions; For instruction units; The length of the input prompt varies with the complexity of the task;

[0082] Step 1.2: The domestically developed large language model, Tongyi Qianwen, is used as the backend inference engine. User input is received via a standardized API interface, and the model is guided to extract key information using pre-set prompt templates. Finally, the data structure is output in JSON format, i.e.:

[0083]

[0084] in, The list of targets to be identified, derived from the instructions of the natural language task, provides a clear target for the next stage of processing; A standardized string label representing a target; The length of the list.

[0085] As a specific example, step 2 involves loading satellite images of the relevant region based on the target semantics extracted by the target extraction module, utilizing the reasoning ability of the visual language model to locate the region where the target may appear, and marking the location of the target object in the processed image, as detailed below:

[0086] Step 2.1: Use the target search module for visual understanding and task target localization. This module receives JSON format task instructions from the target extraction module, then loads high-resolution satellite images of the corresponding area and constructs text prompts.

[0087] Step 2.2: Use the Molmo-7B-O-bnb-4bit lightweight visual language model for multimodal reasoning, combining image content and text prompts to generate text output containing the location information of each target object:

[0088]

[0089] in, , They represent the first The vertical and horizontal coordinates of each target in the image coordinate system;

[0090] Step 2.3: Extract the initial position of the target object using a coordinate analysis function:

[0091]

[0092] in, It expresses the relative position of the target in the graph as a percentage;

[0093] Step 2.4: Use coordinate transformation functions to determine the relative position of the target in the graph. Mapped to latitude and longitude coordinates in real geographic space:

[0094]

[0095] As a specific example, step 3 uses the silhouette coefficient method to calculate the optimal number of UAVs to be scheduled, and uses the K-means clustering algorithm to reasonably divide and allocate the task area, as detailed below:

[0096] Step 3.1: Calculate the optimal number of drones to be scheduled, and set the number of drones to be scheduled as follows. Frame, among which The optimal number of drones is determined by calculating the optimal profile coefficient. :

[0097]

[0098] in, Represents the profile coefficient, when The closer the value is to 1, the smaller the intra-cluster distance and the larger the inter-cluster distance, resulting in better clustering performance and a higher target point. It can be correctly assigned to a cluster; Indicates the target point The average distance to other target points in the same cluster; Indicates the target point The average distance to all target points in each cluster is calculated, and the minimum value is taken.

[0099] Step 3.2: Determine the number of drones that need to be scheduled. After the frame is set, the system will collect the target points. Divided into Each cluster, i.e. Each cluster corresponds to the operating area of ​​one drone. The goal is to minimize the sum of Euclidean distances from points within the cluster to the cluster center, ensuring a relatively balanced workload for each drone and a compact flight path, as detailed below:

[0100] First, randomly select from the target points in the scene. Target points As the initial centroid of each cluster, calculate the distance from all target points in the scene to the initial centroid and assign them to the nearest cluster; then take the average coordinates of all target points in each cluster. Update the centroids of each cluster; finally, repeat the above steps until the centroid positions no longer change. Stop the loop and determine the best heart. This allows for the determination of the number of drones to be scheduled and the adaptive spatial division of target points within the task area, ensuring the geographical compactness and load balance of each drone's operating range while avoiding manual specification. The subjectivity and blindness of value.

[0101] As a specific example, step 4 combines the understanding and reasoning of the task environment using a visual language model, uses the MST-DFS strategy to generate an initial feasible path covering the task area for each UAV, and performs joint optimization with the goal of minimizing the total flight path, time, and energy consumption, outputting the optimal cooperative flight path, as follows:

[0102] Step 4.1: Using a strategy combining minimum spanning tree and depth-first search (MST-DFS), construct an initial feasible flight path covering all target points, as follows:

[0103] The set of target points obtained by each drone Let the edge weight be the Euclidean distance between any two points, i.e. ,in, Let the starting point of the drone's flight be... A one-way path that can cover all target points is ,in Let the coordinates of the path points be denoted as , then the length of the UAV's flight path can be expressed as:

[0104]

[0105] in, Representing a path Corresponding to the total flight length, the total flight length of the optimal path can be expressed as:

[0106]

[0107] The initial path solution for multi-machine collaborative processing is constructed using the MST-DFS strategy. First, an MST is built. Starting from any target point, the strategy of "connecting visited and unvisited points with the shortest edge" is adopted to gradually expand and generate an acyclic connected graph that covers all target points and has the minimum total edge weight; this is the minimum spanning tree. The total edge weight of MST is defined as follows:

[0108]

[0109] Among them, minimum spanning tree The following formula can be used to solve for:

[0110]

[0111] Since any path covering all target points can form a spanning tree by adding just one edge, therefore... That is, the total weight of MST is less than the optimal path length;

[0112] Then, the constructed MST is traversed using DFS, starting from the starting point. Begin by exploring the tree's edges, prioritizing child nodes. Once a leaf node is reached, backtrack and continue exploring other branches, recording the initial order of node visits to form a complete visit sequence. This represents the drone's flight path; since each edge is traversed once during the DFS traversal, the total traversal length is... It is twice the weight of MST, that is:

[0113]

[0114] Finally, note the first-visit order of the DFS sequence. Construct the final drone flight path:

[0115]

[0116] Step 4.2: Employ the 2-opt local optimization algorithm to optimize the flight path by attempting to swap edges along the path, as detailed below:

[0117] First, iterate through the non-adjacent pairs in the path. and , Then perform the "swap operation", which involves changing the path interval. Reverse, the new path is:

[0118]

[0119] The two edges that were deleted are , The two newly added edges are , When the path length increments The new path has a shorter total length, so the path is updated; It can be represented as:

[0120]

[0121] Repeat the above process until no better swap can be found, and finally obtain the flight path:

[0122]

[0123] This invention also provides a multi-machine collaborative path planning system based on a large visual language model. This system is used to implement the aforementioned multi-machine collaborative path planning method based on a large visual language model. The system includes a target extraction module, a visual language model inference module, and:

[0124] The target extraction module receives natural language commands, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information.

[0125] The visual language model inference module, based on the target semantics extracted by the target extraction module, loads satellite images of the relevant region, uses the inference ability of the visual language model to locate the region where the target may appear, and marks the location of the target object in the processed image;

[0126] The task area division module uses the contour coefficient method to calculate the optimal number of UAVs to be scheduled, and uses the K-means clustering algorithm to reasonably divide and allocate the task area.

[0127] The joint optimization module combines the understanding and reasoning of the mission environment by the visual language model, uses the MST-DFS strategy to generate an initial feasible path covering the mission area for each UAV, and performs joint optimization with the goal of minimizing the total flight path, time and energy consumption, and outputs the optimal cooperative flight path.

[0128] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-machine collaborative path planning method based on a large visual language model.

[0129] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps in the multi-machine cooperative path planning method based on a large visual language model.

[0130] The present invention also provides a computer device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the multi-machine cooperative path planning method based on a large visual language model.

[0131] The present invention also provides a computer program product, including computer instructions, which are used to cause a computer to execute the multi-machine collaborative path planning method based on a large visual language model.

[0132] Example

[0133] This embodiment uses the standardized dataset UAV-VLPA-nano-30, specifically built for unmanned aerial vehicle (UAV) mission generation, to verify the multi-aircraft cooperative path planning method based on a large visual language model proposed in this invention. This dataset contains 30 high-resolution satellite images acquired from the open-source platform USGS EarthExplorer, with a resolution of approximately 1.5 meters per pixel and a single image coverage area of ​​approximately 760 square meters. 2It covers geographical environments such as cities, suburbs, and rural areas within the United States, encompassing semantic targets such as buildings, infrastructure, and natural landforms. It also includes geographic metadata to support the accurate mapping of semantic target points to the real geographic coordinate system, generating feasible trajectories.

[0134] To evaluate the advantages of this invention in autonomous planning, a multi-drone cooperative path planning experiment based on a large visual language model was conducted in 30 typical task scenarios in the dataset. The results were compared with those of a path manually planned by an experienced drone operator in Mission Planner. The starting point of each experiment was set at 10% of the width and height of the upper left corner of the image, and flight altitude was ignored. Figure 2 This is a schematic diagram of target search results based on a large visual language model. Figure 3 This is a schematic diagram of the multi-machine collaborative path planning results based on a large visual language model. Experimental results of path planning by manual planning and multi-machine collaborative planning are shown below. Figure 4 As shown, in terms of planning time, the UAVs-VLP system can complete a planning task that would take 35 minutes manually in just 5.98 minutes, improving efficiency by 5.85 times; in terms of path length, the average flight length per UAV in the UAVs-VLP system is reduced by 65.17% compared to manual planning. Experimental results demonstrate that the multi-aircraft collaborative path planning method based on a large visual language model provided by this invention has significant advantages in terms of planning speed and execution efficiency.

[0135] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-machine collaborative path planning method based on a large visual language model, characterized in that, Includes the following steps: Step 1: The target extraction module receives natural language instructions, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information. Step 2: Based on the target semantics extracted by the target extraction module, load satellite images of the relevant region, use the reasoning ability of the visual language model to locate the region where the target may appear, and mark the location of the target object in the processed image; Step 3: Calculate the optimal number of drones to be scheduled using the silhouette coefficient method, and rationally divide and allocate the task area based on the K-means clustering algorithm; Step 4: Combining the visual language model with the understanding and reasoning of the mission environment, the MST-DFS strategy is used to generate an initial feasible path covering the mission area for each UAV, and joint optimization is performed with the goal of minimizing the total flight path, time and energy consumption, and the optimal cooperative flight path is output.

2. The multi-machine collaborative path planning method based on a large visual language model according to claim 1, characterized in that, The target extraction module described in step 1 receives natural language instructions, automatically identifies semantics and extracts task-related targets using a large language model, and outputs key information, as follows: Step 1.1: The target extraction module parses the user's natural language commands, further extracting key task targets and object types; the natural language commands are parsed and represented as: ； in, Provide input suggestions; For instruction units; The length of the input prompt varies with the complexity of the task; Step 1.2: The domestically developed large language model, Tongyi Qianwen, is used as the backend inference engine. User input is received via a standardized API interface, and the model is guided to extract key information using pre-set prompt templates. Finally, the data structure is output in JSON format, i.e.: ； in, The list of targets to be identified, derived from the instructions of the natural language task, provides a clear target for the next stage of processing; A standardized string label representing a target; The length of the list.

3. The multi-machine collaborative path planning method based on a large visual language model according to claim 1, characterized in that, Step 2 involves extracting target semantics from the target extraction module, loading satellite images of the relevant region, utilizing the reasoning ability of the visual language model to locate the region where the target may appear, and marking the target object's location in the processed image, as detailed below: Step 2.1: Use the target search module for visual understanding and task target localization. This module receives JSON format task instructions from the target extraction module, then loads high-resolution satellite images of the corresponding area and constructs text prompts. Step 2.2: Use the Molmo-7B-O-bnb-4bit lightweight visual language model for multimodal reasoning, combining image content and text prompts to generate text output containing the location information of each target object: ； in, , They represent the first The vertical and horizontal coordinates of each target in the image coordinate system; Step 2.3: Extract the initial position of the target object using a coordinate analysis function: ； in, It expresses the relative position of the target in the graph as a percentage; Step 2.4: Use coordinate transformation functions to determine the relative position of the target in the graph. Mapped to latitude and longitude coordinates in real geographic space: 。 4. The multi-machine collaborative path planning method based on a large visual language model according to claim 1, characterized in that, Step 3 involves using the silhouette coefficient method to calculate the optimal number of drones for scheduling, and then using the K-means clustering algorithm to rationally divide and allocate the task area, as detailed below: Step 3.1: Calculate the optimal number of drones to be scheduled, and set the number of drones to be scheduled as follows. Frame, among which The optimal number of drones is determined by calculating the optimal profile coefficient. : ； in, Represents the profile coefficient, when The closer the value is to 1, the smaller the intra-cluster distance and the larger the inter-cluster distance, resulting in better clustering performance and a higher target point. Correctly assigned to a cluster; Indicates the target point The average distance to other target points in the same cluster; Indicates the target point The average distance to all target points in each cluster is calculated, and the minimum value is taken. Step 3.2: Determine the number of drones that need to be scheduled. After setting up, gather the target points. Divided into Each cluster, i.e. Each cluster corresponds to the operating area of ​​one drone. The objective is to minimize the sum of Euclidean distances from points within the cluster to the cluster center, as follows: First, randomly select from the target points in the scene. Target points As the initial centroid of each cluster, the distance from all target points in the scene to the initial centroid is calculated, and the target points are assigned to the nearest cluster; then, the average coordinates of all target points in each cluster are taken. Update the centroids of each cluster; finally, reselect the initial centroids and repeat the update process until the centroid positions no longer change. Stop the loop and determine the best heart. The number of drones to be scheduled is obtained, and the spatial adaptive division of the target points in the task area is realized.

5. The multi-machine collaborative path planning method based on a large visual language model according to claim 1, characterized in that, Step 4 involves combining visual language models to understand and reason about the task environment, using the MST-DFS strategy to generate initial feasible paths covering the task area for each UAV, and performing joint optimization with the goal of minimizing the total flight path, time, and energy consumption to output the optimal cooperative flight path, as detailed below: Step 4.1: Using a strategy combining minimum spanning tree and depth-first search (MST-DFS), construct an initial feasible flight path covering all target points, as follows: The set of target points obtained by each drone Let the edge weight be the Euclidean distance between any two points, i.e. ,in, Let the starting point of the drone's flight be... A one-way path that can cover all target points is ,in Let the coordinates of the path points be denoted as: Then the length of the UAV's flight path is expressed as: ； in, Representing a path The total flight length corresponding to the optimal path is expressed as: ； The initial path solution for multi-machine collaborative processing is constructed using the MST-DFS strategy. First, an MST is built. Starting from any target point, the strategy of "connecting visited and unvisited points with the shortest edge" is adopted to gradually expand and generate an acyclic connected graph that covers all target points and has the minimum total edge weight; this is the minimum spanning tree. The total edge weight of MST is defined as follows: ； Among them, minimum spanning tree The following formula can be used to solve for: ； Since any path covering all target points can form a spanning tree by adding one edge, therefore we have That is, the total weight of MST is less than the optimal path length; Then, the constructed MST is traversed using DFS, starting from the starting point. Begin by exploring the tree's edges, prioritizing child nodes. Once a leaf node is reached, backtrack and continue exploring other branches, recording the initial order of node visits to form a complete visit sequence. This represents the drone's flight path; since each edge is traversed once during the DFS traversal, the total traversal length is... It is twice the weight of MST, that is: ； Finally, note the first-visit order of the DFS sequence. Construct the final drone flight path: ； Step 4.2: Employ the 2-opt local optimization algorithm to optimize the flight path by attempting to swap edges along the path, as detailed below: First, for pairs that are not adjacent in the path... and , Then perform the "swap operation", which involves changing the path interval. Reverse, the new path is: ； The two edges that were deleted are , The two newly added edges are , When the path length increments The new path has a shorter total length, so the path is updated; Represented as: ； Iterate through non-adjacent pairs in the path until no better swap can be found, finally obtaining the flight path: 。 6. A multi-machine cooperative path planning system based on a large visual language model, characterized in that, This system is used to implement the multi-machine collaborative path planning method based on a large visual language model as described in any one of claims 1 to 5, the system comprising a target extraction module, a visual language model inference module, and: The target extraction module receives natural language commands, uses a large language model to automatically identify semantics and extract task-related targets, and outputs key information. The visual language model inference module, based on the target semantics extracted by the target extraction module, loads satellite images of the relevant region, uses the inference ability of the visual language model to locate the region where the target may appear, and marks the location of the target object in the processed image; The task area division module uses the contour coefficient method to calculate the optimal number of UAVs to be scheduled, and uses the K-means clustering algorithm to reasonably divide and allocate the task area. The joint optimization module combines the understanding and reasoning of the mission environment by the visual language model, uses the MST-DFS strategy to generate an initial feasible path covering the mission area for each UAV, and performs joint optimization with the goal of minimizing the total flight path, time and energy consumption, and outputs the optimal cooperative flight path.

7. A mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-machine collaborative path planning method based on a large visual language model as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-machine collaborative path planning method based on a large visual language model as described in any one of claims 1 to 5.

9. A computer device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the multi-machine collaborative path planning method based on a large visual language model as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, It includes computer instructions, which are used to cause the computer to execute the multi-machine collaborative path planning method based on a large visual language model as described in any one of claims 1 to 5.

Images