Multi-modal unmanned cluster distributed cooperative exploration planning method and device

Abstract

The application discloses a multi-modal unmanned cluster distributed cooperative exploration planning method and device. The method comprises the following steps: acquiring a local task grid, determining a boundary set of unknown areas in the local task grid; generating an air-ground view point attribute set according to the boundary set, the air-ground view point attribute set comprising pose attributes and modal attributes of each view point of each boundary in the boundary set; determining a first air-ground view point chain according to the air-ground view point attribute set, the first air-ground view point chain passing through a view point of each boundary and the sum of path costs of all adjacent view points thereof being minimum, the path cost comprising a penalty caused by switching modal attributes; and performing motion cost optimization on the first air-ground view point chain based on the air-ground view point attribute set, obtaining a second air-ground view point chain and generating an exploration trajectory of a multi-modal unmanned aerial vehicle. The application realizes distributed cooperative exploration planning of a multi-modal unmanned cluster in a complex unknown environment, and improves exploration efficiency and cooperative consistency.

Description

Technical Field

[0001] This invention relates to the field of unmanned swarm technology, and in particular to a multimodal unmanned swarm distributed collaborative exploration planning method, apparatus, readable storage medium, terminal equipment, and software product. Background Technology

[0002] Currently, existing collaborative exploration planning methods for unmanned swarms mainly target single-modality unmanned swarms. Examples include the RACER collaborative exploration planning method for UAV swarms and the MGGPlanner collaborative exploration planning method for unmanned vehicle swarms. These methods provide relatively mature solutions for area exploration and path planning in purely aerial or purely ground-based scenarios, respectively.

[0003] However, existing methods are designed only for a single mode (air or ground). For multimodal unmanned swarms, they cannot efficiently utilize the complementary characteristics of the ground and air modes to improve the exploration efficiency of unknown environments. Furthermore, since they only consider the motion characteristics of a single mode, they lack the planning capability for autonomous switching between the ground and air modes, making it difficult to meet the needs of multimodal UAVs for intelligent and continuous operation in complex and unknown environments. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a distributed collaborative exploration planning method and apparatus for multimodal unmanned swarms, aiming to realize distributed collaborative exploration planning of multimodal unmanned swarms in complex and unknown environments, thereby improving exploration efficiency and collaborative consistency.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a multimodal unmanned swarm distributed cooperative exploration planning method is provided, comprising: Obtain the local task grid and determine the boundary set of unknown regions in the local task grid; A ground-to-air viewpoint attribute set is generated based on the boundary set. The ground-to-air viewpoint attribute set includes the pose and modal attributes of each viewpoint of each boundary segment in the boundary set. The modal attributes include aerial viewpoints and ground viewpoints. A first ground-to-air viewpoint chain is determined based on the ground-to-air viewpoint attribute set to cover the unknown area. The first ground-to-air viewpoint chain passes through each of the boundaries and one viewpoint. The sum of the path costs of all adjacent viewpoints on the first ground-to-air viewpoint chain is minimized. When the modal attributes of adjacent viewpoints are different, the path cost includes the penalty for switching the modal attributes. Based on the set of ground-to-air viewpoint attributes, the motion cost of the first ground-to-air viewpoint chain is optimized to obtain the second ground-to-air viewpoint chain, and the exploration trajectory of the multimodal UAV is generated according to the second ground-to-air viewpoint chain.

[0006] Furthermore, the local task grid is an unexplored multi-level grid known to the multimodal UAV. The multimodal UAV stores the grid attributes of the unexplored multi-level grid locally, and the grid attributes include grid level and grid index. The path cost is the product of a penalty factor and a base cost; the base cost is related to the spatial distance or motion time between the two viewpoints; when the modal attributes of adjacent viewpoints are different, the penalty factor is a first value, otherwise it is a second value less than the first value.

[0007] Furthermore, a ground-to-air viewpoint attribute set is generated based on the boundary set, including: Sampling is performed in the known free space corresponding to each boundary in the boundary set to generate the pose attributes of the candidate viewpoints of each boundary, the pose attributes including position, height and exploration direction; Based on the position and exploration direction of the candidate viewpoint, calculate the field of view coverage of the candidate viewpoint over the unknown area of ​​the corresponding boundary; Candidate viewpoints with a field of view coverage greater than a preset coverage threshold are selected as viewpoints. The modal attribute of the viewpoint is determined based on the height of the viewpoint.

[0008] Furthermore, determining a first ground-to-air viewpoint chain to cover the unknown area based on the ground-to-air viewpoint attribute set includes: From the set of ground-air viewpoint attributes, select the viewpoint with the largest visual coverage of the unknown area for each boundary as the first target viewpoint; Using the first target viewpoint as the node to be visited, add virtual start point and virtual end point without imposing constraints on the return, construct the objective and constraints of the asymmetric traveling salesman problem. The objective includes visiting all the first target viewpoints, minimizing the sum of the path costs and minimizing the mode switching. The constraints include in-degree constraints, sub-loop elimination constraints, and virtual start point and end point constraints. Solving the asymmetric traveling salesman problem yields the first ground-to-space viewpoint chain.

[0009] Furthermore, based on the aforementioned ground-to-air viewpoint attribute set, motion cost optimization is performed on the first ground-to-air viewpoint chain to obtain a second ground-to-air viewpoint chain, including: Obtain the current viewpoint of the multimodal UAV. Starting from the current viewpoint, and based on the first ground-to-air viewpoint chain and the ground-to-air viewpoint attribute set, the first target viewpoint is optimized with the constraint of minimizing the total motion cost. A second target viewpoint is selected from the multiple viewpoints corresponding to each of the boundaries to obtain the second ground-to-air viewpoint chain composed of the second target viewpoint.

[0010] Furthermore, before acquiring the local task grid, interaction with other multimodal drones in the multimodal drone swarm is performed to update the local task grid, including: Based on the local task grid and interaction history, the interaction object is determined, a request message containing a task reassignment scheme is sent, and the local task grid is updated according to the response message.

[0011] According to a second aspect of the present invention, a multimodal unmanned swarm distributed cooperative exploration planning device is provided, comprising: The grid acquisition module is used to acquire the local task grid and determine the boundary set of unknown regions in the local task grid; The set generation module is used to generate a ground-air viewpoint attribute set based on the boundary set. The ground-air viewpoint attribute set includes the pose attributes and modal attributes of the viewpoints of each boundary segment in the boundary set. The modal attributes include aerial viewpoints and ground viewpoints. The path planning module is used to determine a first ground-air viewpoint chain to cover the unknown area based on the ground-air viewpoint attribute set. The first ground-air viewpoint chain passes through each of the boundaries and one viewpoint, and the sum of the path costs of all adjacent viewpoints on the first ground-air viewpoint chain is minimized. When the modal attributes of adjacent viewpoints are different, the path cost includes the penalty for switching the modal attributes. The path optimization module is used to optimize the motion cost of the first ground-to-air viewpoint chain based on the ground-to-air viewpoint attribute set to obtain a second ground-to-air viewpoint chain, and generate the exploration trajectory of the multimodal UAV based on the second ground-to-air viewpoint chain.

[0012] According to a third aspect of the present invention, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the steps of the multimodal unmanned swarm distributed cooperative exploration planning method as described in the first aspect of the present invention.

[0013] According to a fourth aspect of the present invention, a terminal device is provided, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the multimodal unmanned swarm distributed cooperative exploration planning method as described in the first aspect of the present invention.

[0014] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps in the multimodal unmanned cluster distributed cooperative exploration planning method as described in the first aspect of the present invention.

[0015] The embodiments of the present invention have at least one of the following advantages or beneficial effects: This invention addresses the multimodal characteristics of multimodal unmanned swarms by constructing a set of ground-air viewpoint attributes. It integrates both aerial and ground viewpoints into a unified planning framework, overcoming the limitations of single-modality (purely aerial or purely ground-based) planning. By incorporating a mode-switching penalty mechanism into the path cost, it enables multimodal UAVs to operate efficiently in a "ground-to-air chain" manner, effectively reducing unnecessary frequent mode switching. While fully utilizing the complementary characteristics of ground and air modes, it significantly improves exploration efficiency in complex and unknown environments.

[0016] In this embodiment of the invention, the ground-to-air viewpoint attribute set incorporates the modal attributes of the viewpoint, determines the modal attributes of the viewpoint based on the viewpoint height, and then integrates the ground-to-air modal attributes into the asymmetric traveling salesman problem and solves it. This enables multimodal UAVs to adaptively decide when and where to switch modes based on environmental information, achieving efficient and intelligent generation of ground and air trajectories, and meeting the needs of intelligent and continuous operation in complex and unknown environments.

[0017] The embodiments of the present invention adopt a two-stage strategy of "coarse planning + fine optimization". First, a better first ground-to-air viewpoint chain is obtained with the goal of minimizing the sum of path costs and the fewest mode switching. Then, motion cost optimization is performed to obtain a better second ground-to-air viewpoint chain. This phased optimization strategy effectively reduces computational complexity while ensuring the quality of path planning, and achieves synergistic optimization of exploration coverage and motion efficiency.

[0018] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0019] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0020] Figure 1 This is a schematic diagram of the main process of the multimodal unmanned cluster distributed collaborative exploration planning method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of each region in the local task grid of an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of generating candidate viewpoints according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating viewpoint optimization according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the main modules of the multimodal unmanned cluster distributed collaborative exploration and planning device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the composition principle of the terminal device in the embodiment of the present invention. Detailed Implementation

[0021] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0022] Example 1 Figure 1 This is a schematic diagram of the main flow of a multimodal unmanned swarm distributed cooperative exploration planning method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the multimodal unmanned swarm distributed cooperative exploration planning method of the present invention in this embodiment is used for any one multimodal unmanned swarm in a multimodal unmanned swarm, and includes the following steps S101 to S104.

[0023] Step S101: Obtain the local task grid and determine the boundary set of unknown regions in the local task grid; Step S102: Generate a ground-to-air viewpoint attribute set based on the boundary set. The ground-to-air viewpoint attribute set includes the pose attributes and modal attributes of the viewpoints of each boundary segment in the boundary set. The modal attributes include aerial viewpoints and ground viewpoints. Step S103: Determine a first ground-to-air viewpoint chain to cover the unknown area based on the ground-to-air viewpoint attribute set. The first ground-to-air viewpoint chain passes through each of the boundaries and one viewpoint. The sum of the path costs of all adjacent viewpoints on the first ground-to-air viewpoint chain is minimized. When the modal attributes of adjacent viewpoints are different, the path cost includes the penalty for switching the modal attributes. Step S104: Optimize the motion cost of the first ground-to-air viewpoint chain based on the ground-to-air viewpoint attribute set to obtain the second ground-to-air viewpoint chain, and generate the exploration trajectory of the multimodal UAV based on the second ground-to-air viewpoint chain.

[0024] Understandably, spatial grids are used to represent and manage exploration areas in two-dimensional or three-dimensional space during exploration planning. The task grid is the unexplored spatial grid of a multimodal drone swarm, while the local task grid is the unexplored spatial grid that each multimodal drone locally learns about within its exploration area and needs to explore. Specifically, in this embodiment and some embodiments of the present invention, the local task grid in step S101 is a multi-level grid. The multi-level grid uses a spatial hierarchical structure such as an octree or quadtree for discretized spatial representation and indexing. Each multimodal drone stores the task grids it perceives through its own sensors or learns from notifications from neighbors locally. During storage, the attribute data of the task grids, including grid level and grid index, is stored as a task list. Specifically, the multimodal UAV first stores its own highest-level task grid. When the proportion of known voxels in a task grid exceeds a first threshold and its grid level is not the finest, the task grid will be uniformly subdivided. After subdivision, grid indices are assigned to the task grids at the next level whose proportion of known voxels is less than the threshold, and the network indices and network levels corresponding to these next-level task grids are stored. If the proportion of unknown voxels in a task grid is less than the threshold and its grid level is the finest, the task grid is not stored, meaning that the local task grid has been explored.

[0025] It is understood that in this embodiment and some embodiments of the present invention, by performing task management based on the local task grid as described above, unexplored areas can be accurately characterized based on real-time perception information, ensuring the comprehensiveness and reliability of the exploration process.

[0026] Figure 2 This is a schematic diagram of various regions in the local task grid according to an embodiment of the present invention. Specifically, in this embodiment and some embodiments of the present invention, such as... Figure 2 As shown, in step S101, the boundary set of unknown areas in the local task grid is determined. Based on the local task grid, the multimodal UAV generates a corresponding navigation grid map. The grid map corresponding to the local task grid contains free areas, obstacle areas, and unknown areas. Relying on the sensors of the multimodal UAV swarm (the multimodal UAV's own sensors and the sensors of neighboring multimodal UAVs), the free area is a spatial area in the grid map that has been confirmed as obstacle-free and safe to pass through. Figure 2 In the center, white indicates that the voxel occupancy probability is below a set threshold. Obstacle areas are spatial regions in the grid map that have been identified by the sensor as containing obstacles and impassable. Figure 2 In black, areas whose voxel occupancy probability exceeds a set threshold are represented. Unknown areas are spatial regions in the raster map that have not yet been explored, or whose status cannot be determined due to insufficient exploration data. Figure 2The gray area is used to represent this. Furthermore, the boundary is the critical surface between the unknown region and the free region of the native task mesh, such as... Figure 2 The grids are shown in green, blue, yellow, and pink. Specifically, in this embodiment and some embodiments of the present invention, the grids representing boundaries, or free regions adjacent to unknown regions, can be identified through sliding window or connected component analysis. It is understood that the boundary set represents the latest boundary between free and unknown regions in the grid map. Updates to the local task grid or the receipt of new sensor data will update the grid map, and the boundaries will change accordingly, requiring the boundary set to be redefined according to set rules or periods.

[0027] Specifically, in this embodiment and some embodiments of the present invention, the boundary set in step S101 includes one or more boundary segments. The length of each boundary segment is suitable for one exploration by a multimodal UAV. Boundaries that are too large will be decomposed into multiple segments and then added to the boundary set. See also Figure 2 Each boundary segment will be assigned a unique identifier for differentiation.

[0028] Understandably, simply navigating a multimodal UAV to the midpoint of a boundary set will not maximize the exploration of the unknown areas within that boundary. To ensure the continuity of search navigation and the efficiency of exploration traversal, it is necessary to select appropriate viewpoints and exploration angles to ensure that the multimodal UAV can quickly complete its exploration task.

[0029] Specifically, in this embodiment and some embodiments of the present invention, step S102 includes steps S102a to S102d.

[0030] Step S102a: Sample the known free space corresponding to each boundary in the boundary set to generate the pose attributes of the candidate viewpoints of each boundary. The pose attributes include position, height and exploration direction.

[0031] Step S102b: Calculate the field coverage of the candidate viewpoint to the unknown area of ​​the corresponding boundary based on the position and exploration direction of the candidate viewpoint.

[0032] Step S102c: Select the candidate viewpoints whose field of view coverage is greater than a preset coverage threshold as the viewpoints.

[0033] Step S102d: Determine the modal attribute of the viewpoint based on its height. Specifically, if the height of the viewpoint is greater than a preset height threshold, the modal attribute of the viewpoint is an aerial viewpoint; otherwise, it is a ground viewpoint.

[0034] Understandably, the sensor field of view (FVR) is the spatial range that the sensors on a multimodal UAV can explore in its current position and attitude (which determines the sensor's exploration direction). It is typically determined by the sensor's horizontal field of view, vertical field of view, and effective ranging range. Within its sensor FVR, a multimodal UAV can calculate local environmental information from sensor data, including attributes such as obstacles, free space, unknown areas, and their locations.

[0035] Figure 3 This is a schematic diagram illustrating the principle of generating candidate viewpoints according to an embodiment of the present invention. More specifically, as... Figure 3 As shown, in this embodiment and some embodiments of the present invention, when a boundary Once created, the side of the boundary closest to the free region can be sampled, and the generated candidate viewpoints will be located... ,high and exploration direction And based on its sensor field of view, it covers the unknown areas outside the boundary. Figure 3 The yellow voxels represent the boundary, the light red represents the free region inside the boundary, and the green dots represent the generated candidate viewpoints. The candidate viewpoints are selected based on the azimuth sampling interval. Distance sampling interval and height sampling interval The free region is discretized and generated.

[0036] Specifically, in this embodiment and some embodiments of the present invention, since the multimodal UAV has ground and air modes of motion and exploration capabilities, the boundary set needs to include the modal attributes of each viewpoint to identify whether the task of that viewpoint should be completed on the ground or in the air. This attribute is determined by comprehensively considering the pose attributes of the candidate viewpoints.

[0037] Furthermore, in this embodiment and some embodiments of the present invention, in order to reduce the amount of data in the boundary set and ensure simplified calculations for subsequent viewpoint selection, for each viewpoint in the free space on the known side of the boundary... Exploring directions The direction selected is the direction in which the sensor's field of view achieves maximum coverage of the unknown area outside the boundary when the multimodal UAV is at this viewpoint. Only viewpoints with a field of view coverage exceeding a certain threshold and their exploration directions are retained.

[0038] More specifically, in this embodiment and some embodiments of the present invention, boundaries are defined. Indicates the first Segment boundaries, each segment boundary consists of a set of spatially adjacent boundary grids, plus a set of grids located on the unknown side. The grid set of all free regions within the boundaries is In the algorithm implementation, Used for sampling viewpoints, Used to calculate field of view coverage.

[0039] In step S102a, a set of candidate viewpoints is sampled from the free space of the known region:

[0040] in, Indicates the location viewpoint and boundary The shortest distance, It is the maximum sampling distance.

[0041] For each candidate viewpoint Calculate the exploration direction according to the following formula :

[0042] Among them, the coverage function , Is the sensor in position? Exploration direction Sensor field of view at that time Indicates intersection, Functions for calculating area or volume.

[0043] Candidate viewpoints that meet the coverage threshold are retained as viewpoints in the ground-to-air viewpoint attribute set. :

[0044] in, It is the coverage threshold.

[0045] Specifically, in this embodiment and some embodiments of the present invention, in step S102d, the viewpoint of the ground-to-air viewpoint attribute set... According to its height Determine modal attributes Specifically, the modal attribute decision function is defined as follows. calculate: ,in, This is the height threshold.

[0046] It is understandable that in this embodiment and some embodiments of the present invention, the final set of ground-to-air viewpoint attributes... for: .

[0047] Understandably, multimodal UAVs need to reach all boundaries, thus requiring dynamic planning of viewpoints across all boundaries. Each planning iteration must determine a path that traverses all boundaries while minimizing overall cost. In this embodiment and some embodiments of the present invention, the problem is modeled as an asymmetric traveling salesman problem without any constraints on the return journey. The optimal closed-loop path found under this state yields the optimal viewpoint chain for autonomous search by the multimodal UAV.

[0048] Specifically, in this embodiment and some embodiments of the present invention, since the multimodal UAV has the ability to move and explore on the ground and in the air, it is necessary to consider penalizing frequent ground / air mode switching in step S103. In the cost modeling of the non-traveling salesman problem, a penalty factor is set on the connection edge between the ground viewpoint and the air viewpoint to penalize the frequent switching of control modes, so as to explore more efficiently through the ground viewpoint chain-air viewpoint chain.

[0049] Specifically, in this embodiment and some embodiments of the present invention, step S103 includes steps S103a and S103c.

[0050] Step S103a: From the set of ground-to-air viewpoint attributes, select the viewpoint with the largest visual coverage of the unknown area for each boundary as the first target viewpoint; that is, from the set of ground-to-air viewpoint attributes... For each boundary, select a viewpoint with the largest field of view coverage, denoted as . , This represents the number of first target viewpoints, i.e., the number of boundaries.

[0051] Step S103b: Take the first target viewpoint as the node to be visited, add a virtual start point and a virtual end point without imposing constraints on the return, and construct the objective and constraints of the asymmetric traveling salesman problem. The objective includes passing through all the first target viewpoints, minimizing the sum of the path costs and minimizing the mode switching. The constraints include in-degree constraints, sub-loop elimination constraints, and virtual start-end point constraints.

[0052] Specifically, in this embodiment and some embodiments of the present invention, for the set Two primary target viewpoints Path cost As a penalty factor and basic cost The product of, i.e. Then, the cost matrix is ​​constructed first. Basic cost Related to the spatial distance or motion time between the two viewpoints, for example When the modal attributes of adjacent viewpoints are different, the penalty factor... The penalty factor is either a first value or a second value less than the first value. Specifically, in some embodiments of the present invention, when the modal attributes of adjacent viewpoints are different, the corresponding penalty factor is... The value is Otherwise, the value is , , In this embodiment, .

[0053] Understandably, the cost matrix above models the spatial distance cost, or communication cost, between multimodal viewpoints. When the penalty factor... The value is In this case, the algorithm tends to cluster viewpoints with the same modal attributes together for access, thereby reducing the number of modal switching and improving exploration efficiency, thus balancing motion efficiency and modal switching cost overall.

[0054] Specifically, in this embodiment and some embodiments of the present invention, the addition of a virtual starting point is used to enable the algorithm to obtain an open-loop path, and a virtual starting point is set. The path cost to all viewpoints is All viewpoints to the virtual endpoint The path cost is The path cost from the virtual endpoint to the virtual origin is ,in, This is a large number used to ensure that the relevant virtual path is always selected. Furthermore, in the cost matrix... Based on this, the extended cost matrix is ​​obtained. It is understandable that the virtual starting point and virtual ending point can be chosen arbitrarily; in this embodiment and some embodiments of the present invention, the origin (0,0) is selected.

[0055] Specifically, in this embodiment and some embodiments of the present invention, the objective of the asymmetric traveling salesman problem is represented as: , Indicates the first target viewpoint The value of the edge is 0, which means that the link contains the edge, and 1 means that it does not contain the edge.

[0056] Specifically, in this embodiment and some embodiments of the present invention, the in-degree constraint is expressed as follows: , , This means that each first target viewpoint has exactly one outgoing edge and one incoming edge.

[0057] Specifically, in this embodiment and some embodiments of the present invention, the elimination of constraints in sub-loops is represented as follows:

[0058]

[0059] in, As an auxiliary variable, it represents the first... The first objective perspective The sequence number in the link.

[0060] Specifically, in this embodiment and some embodiments of the present invention, the virtual start-end point constraint is represented as: Virtual Start Point The out-degree is 1 and the in-degree is 0; virtual endpoint The in-degree is 1 and the out-degree is 0.

[0061] Step S103c: Solve the asymmetric traveling salesman problem to obtain the first ground-to-air viewpoint chain. Specifically, in this embodiment and some embodiments of the present invention, the heuristic LKH algorithm is used to solve the problem. The LKH algorithm is based on the Lin-Kernighan idea, and improves the path solution by repeatedly cutting and reconnecting the connecting edges on the graph. The more edges exchanged, the better the quality of the solution.

[0062] Specifically, in this embodiment and some embodiments of the present invention, during the solution process, the link The sum of path costs is:

[0063] in, , These respectively represent the links The Middle The, the The first target viewpoint. If the link If the sum of path costs is minimized, then the virtual start point and virtual end point are removed from the link to obtain the first ground-to-air viewpoint chain.

[0064] Understandably, this embodiment and some embodiments of the present invention use a first ground-to-air viewpoint chain to enable multimodal UAVs to operate in a ground-to-air exploration chain mode as much as possible, reducing frequent mode switching and improving overall mission execution efficiency.

[0065] Specifically, in this embodiment and some embodiments of the present invention, step S104 optimizes the motion cost of the first ground-to-air viewpoint chain based on the ground-to-air viewpoint attribute set to obtain the second ground-to-air viewpoint chain, including steps S104a to S104b.

[0066] Step S104a: Obtain the current viewpoint of the multimodal UAV. Step S104b: Starting from the current viewpoint, based on the first ground-to-air viewpoint chain and the ground-to-air viewpoint attribute set, optimize the first target viewpoint with the constraint of minimizing the total motion cost, and select a second target viewpoint from the multiple viewpoints corresponding to each of the boundaries to obtain the second ground-to-air viewpoint chain composed of the second target viewpoint.

[0067] Figure 4 This is a schematic diagram illustrating viewpoint optimization according to an embodiment of the present invention. Specifically, as shown... Figure 4 As shown, in this embodiment and some embodiments of the present invention, in step S104b, a directed acyclic graph is constructed with the current state of the UAV as the starting point, the end point of the first ground-to-air viewpoint chain as the ending point, and the viewpoints in the aforementioned ground-to-air viewpoint attribute set as the path points. The nodes of the directed acyclic graph include the current state of the UAV, multiple viewpoints corresponding to each boundary, and the path points of the viewpoints. Figure 4 The graph shows three boundaries (yellow, blue, and green), and points near the boundaries that are the same color as the boundaries. The edges of the directed acyclic graph only connect viewpoints of adjacent boundaries. Considering motion constraints such as environmental constraints and time constraints, and with the goal of minimizing the total motion cost, an optimal path is determined in the directed acyclic graph using a graph search algorithm, which serves as the second ground-to-air viewpoint chain.

[0068] It is understood that, in this embodiment and some embodiments of the present invention, compared with the first ground-to-air viewpoint chain, the second ground-to-air viewpoint chain, while maintaining the macro-planning information of the first ground-to-air viewpoint chain, refines the selection of specific viewpoints in each boundary area, making the final generated exploration path more advantageous in terms of actual flight capability, smooth motion, and local obstacle avoidance, and can be more time-saving and reliable.

[0069] Understandably, in this embodiment and some embodiments of the present invention, factors such as energy consumption, speed, and safety are comprehensively considered to design a multi-optimization objective heuristic search function, employing dynamics A The algorithm and the B-spline algorithm can obtain the exploration trajectory of the multimodal UAV in actual flight based on the second ground-to-air viewpoint chain.

[0070] Specifically, in this embodiment and some embodiments of the present invention, dynamics A The algorithm needs to maintain two lists: an open list (OPEN) and a closed list (CLOSE). The OPEN list stores all explored nodes, and the dynamics A... The algorithm repeatedly selects the node with the smallest estimated value from the OPEN list as the next node to expand and adds it to the CLOSE list. After the search is complete, the dynamics A... The algorithm provides a feasible trajectory, which is further optimized using B-splines. Each node of the B-spline curve has the same time interval. The feasible trajectory is represented by a B-spline curve, which is determined by its order. , Control points and a node vector The decision, and the optimized curve, is the exploration trajectory.

[0071] Specifically, in this embodiment and some embodiments of the present invention, before step S101, step S100 is included, in which the multimodal drone interacts with other multimodal drones in the multimodal drone swarm to update the local task grid. It is understood that before interacting with neighboring multimodal drones, each multimodal drone maintains a local task grid and records the state of each local task grid, thus providing a clear and unambiguous physical basis for the task allocation of different multimodal drones.

[0072] Specifically, in this embodiment and some embodiments of the present invention, in step S100, the interaction object is determined based on the local task grid and the interaction history, a request message containing a task reassignment scheme is sent, and the local task grid is updated according to the response message, including steps S100a to S100e.

[0073] Step S100a: For each multimodal UAV, such as the first A multimodal drone It will periodically manage the current task grid set. and the timestamp of its last attempt to initiate an interaction .

[0074] Step S100b: The A multimodal drone Before initiating an interaction, it identifies all other devices currently within the communication range. A multimodal drone , denoted as set And first exclude those that have just happened in a very short time. Interactive drones, i.e. Multimodal drones, to avoid conflict, from ensemble The remaining multimodal drones are selected, and the one with the longest interval between its last successful interaction and the timestamp is chosen. This fair scheduling strategy, which involves interacting with neighbors and minimizing interaction time, ensures that all neighbors have an equal opportunity to negotiate.

[0075] Step S100c, based on two multimodal drones and Current task grid in charge and Given the current positions of the two multimodal UAVs, a capacity-constrained vehicle routing problem (CVRP) is constructed. The goal is to solve this problem by finding a way to repartition the tasks between the two multimodal UAVs, i.e., to form a new task mesh set. and This minimizes the total path length required for both to cover all their task grids, while balancing the workload allocated to each multimodal UAV, i.e., the "capacity constraint".

[0076] Step S100d, the first A multimodal drone This calculated optimal allocation scheme, task grid set and Send as a request to the multimodal drone .

[0077] Step S100d, Multimodal UAV Upon receiving the request, I will check again to see if I have done so within the very short time just now. Other interactions have already been conducted to avoid conflicts caused by simultaneous interactions with multiple drones due to communication delays; if the check passes, the multimodal drone... Reply to A multimodal drone Success, and two multimodal drones and All are based on the CVRP solution, i.e., the new task grid set. and Update their respective local task grids and record the timestamp of this successful interaction; if the check fails, the multimodal drone... If you have already interacted with someone, then reply with the [number]th [response]. A multimodal drone Failed. This interaction was ignored, and the task assignment remains unchanged.

[0078] Understandably, the embodiments of the present invention employ the aforementioned distributed task grid allocation. Since there is no central node, all decisions are made collaboratively between pairs of UAVs, improving the robustness of the multimodal UAV swarm to network instability and limited communication range. Furthermore, through continuous, local pairwise optimization, the task allocation of the entire swarm gradually tends towards a more optimal global state.

[0079] Example 2 Figure 5 This is a schematic diagram of the constituent modules of a multimodal unmanned swarm distributed cooperative exploration and planning device according to an embodiment of the present invention. Figure 5As shown, the collaborative spectrum sensing device based on a deep convolutional neural network in this embodiment of the present invention includes: The grid acquisition module is used to acquire the local task grid and determine the boundary set of unknown regions in the local task grid; The set generation module is used to generate a ground-air viewpoint attribute set based on the boundary set. The ground-air viewpoint attribute set includes the pose attributes and modal attributes of the viewpoints of each boundary segment in the boundary set. The modal attributes include aerial viewpoints and ground viewpoints. The path planning module is used to determine a first ground-air viewpoint chain to cover the unknown area based on the ground-air viewpoint attribute set. The first ground-air viewpoint chain passes through each of the boundaries and one viewpoint, and the sum of the path costs of all adjacent viewpoints on the first ground-air viewpoint chain is minimized. When the modal attributes of adjacent viewpoints are different, the path cost includes the penalty for switching the modal attributes. The path optimization module is used to optimize the motion cost of the first ground-to-air viewpoint chain based on the ground-to-air viewpoint attribute set to obtain a second ground-to-air viewpoint chain, and generate the exploration trajectory of the multimodal UAV based on the second ground-to-air viewpoint chain.

[0080] Example 3 like Figure 6 As shown, Embodiment 3 of the present invention provides a terminal device, including at least one processor and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps in the multimodal unmanned swarm distributed cooperative exploration planning method as described in the first aspect of the present invention.

[0081] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripherals, voltage regulators, and power management circuits, via interfaces, as is well known in the art. Interfaces provide a connection between the bus and the transceiver, such as communication interfaces or user interfaces. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0082] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0083] Example 4 Embodiment 4 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multimodal unmanned cluster distributed cooperative exploration planning method as described in the first aspect of the present invention.

[0084] Those skilled in the art will understand from the foregoing description that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic storage devices, and optical storage devices.

[0085] Example 5 Embodiment 5 of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the multimodal unmanned cluster distributed cooperative exploration planning method as described in the first aspect of the present invention.

[0086] Based on the detailed description of the embodiments of the present invention above, it can be more clearly understood that the present invention has the following advantages: This invention addresses the multimodal characteristics of multimodal unmanned swarms by constructing a set of ground-air viewpoint attributes. It integrates both aerial and ground viewpoints into a unified planning framework, overcoming the limitations of single-modality (purely aerial or purely ground-based) planning. By incorporating a mode-switching penalty mechanism into the path cost, it enables multimodal UAVs to operate efficiently in a "ground-to-air chain" manner, effectively reducing unnecessary frequent mode switching. While fully utilizing the complementary characteristics of ground and air modes, it significantly improves exploration efficiency in complex and unknown environments.

[0087] In this embodiment of the invention, the ground-to-air viewpoint attribute set incorporates the modal attributes of the viewpoint, determines the modal attributes of the viewpoint based on the viewpoint height, and then integrates the ground-to-air modal attributes into the asymmetric traveling salesman problem and solves it. This enables multimodal UAVs to adaptively decide when and where to switch modes based on environmental information, achieving efficient and intelligent generation of ground and air trajectories, and meeting the needs of intelligent and continuous operation in complex and unknown environments.

[0088] The embodiments of the present invention adopt a two-stage strategy of "coarse planning + fine optimization". First, a better first ground-to-air viewpoint chain is obtained with the goal of minimizing the sum of path costs and the fewest mode switching. Then, motion cost optimization is performed to obtain a better second ground-to-air viewpoint chain. This phased optimization strategy effectively reduces computational complexity while ensuring the quality of path planning, and achieves synergistic optimization of exploration coverage and motion efficiency.

[0089] This invention ensures complete coverage of unknown boundary areas by multimodal UAVs by sampling viewpoints within known free space within the boundary and filtering effective viewpoints based on exploration coverage. The embodiments of this invention employ a multi-level grid structure for task management, which can accurately characterize unexplored areas based on real-time sensing information, ensuring the comprehensiveness and reliability of the exploration process. Furthermore, the embodiments of this invention determine the interaction objects based on the local task grid and interaction history, and send a request message containing a task reassignment scheme, realizing the coordination of multimodal UAV distributed collaborative exploration planning, avoiding single point of failure and communication bottlenecks in centralized planning, and improving the robustness and scalability of the system.

[0090] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed cooperative exploration planning method for multimodal unmanned aerial vehicle (UAV) swarms, used for any one multimodal UAV in a multimodal UAV swarm, characterized in that, include: Obtain the local task grid and determine the boundary set of unknown regions in the local task grid; A ground-to-air viewpoint attribute set is generated based on the boundary set. The ground-to-air viewpoint attribute set includes the pose and modal attributes of each viewpoint of each boundary segment in the boundary set. The modal attributes include aerial viewpoints and ground viewpoints. A first ground-to-air viewpoint chain is determined based on the ground-to-air viewpoint attribute set to cover the unknown area. The first ground-to-air viewpoint chain passes through each of the boundaries and one viewpoint. The sum of the path costs of all adjacent viewpoints on the first ground-to-air viewpoint chain is minimized. When the modal attributes of adjacent viewpoints are different, the path cost includes the penalty for switching the modal attributes. Based on the set of ground-to-air viewpoint attributes, the motion cost of the first ground-to-air viewpoint chain is optimized to obtain the second ground-to-air viewpoint chain, and the exploration trajectory of the multimodal UAV is generated according to the second ground-to-air viewpoint chain.

2. The method according to claim 1, characterized in that, The local task grid is an unexplored multi-level grid known to the multimodal UAV. The multimodal UAV stores the grid attributes of the unexplored multi-level grid locally, and the grid attributes include grid level and grid index. The path cost is the product of a penalty factor and a base cost; the base cost is related to the spatial distance or motion time between the two viewpoints; when the modal attributes of adjacent viewpoints are different, the penalty factor is a first value, otherwise it is a second value less than the first value.

3. The method according to claim 1, characterized in that, A set of ground-to-air viewpoint attributes is generated based on the boundary set, including: Sampling is performed in the known free space corresponding to each boundary in the boundary set to generate the pose attributes of the candidate viewpoints of each boundary, the pose attributes including position, height and exploration direction; Based on the position and exploration direction of the candidate viewpoint, calculate the field of view coverage of the candidate viewpoint over the unknown area of ​​the corresponding boundary; Candidate viewpoints with a field of view coverage greater than a preset coverage threshold are selected as viewpoints. The modal attribute of the viewpoint is determined based on the height of the viewpoint.

4. The method according to claim 1, characterized in that, Determining a first ground-to-air viewpoint chain to cover the unknown area based on the ground-to-air viewpoint attribute set includes: From the set of ground-air viewpoint attributes, select the viewpoint with the largest visual coverage of the unknown area for each boundary as the first target viewpoint; Using the first target viewpoint as the node to be visited, add virtual start point and virtual end point without imposing constraints on the return, construct the objective and constraints of the asymmetric traveling salesman problem. The objective includes visiting all the first target viewpoints, minimizing the sum of the path costs and minimizing the mode switching. The constraints include in-degree constraints, sub-loop elimination constraints, and virtual start point and end point constraints. Solving the asymmetric traveling salesman problem yields the first ground-to-space viewpoint chain.

5. The method according to claim 4, characterized in that, Based on the aforementioned set of ground-to-air viewpoint attributes, motion cost optimization is performed on the first ground-to-air viewpoint chain to obtain a second ground-to-air viewpoint chain, including: Obtain the current viewpoint of the multimodal UAV. Starting from the current viewpoint, and based on the first ground-to-air viewpoint chain and the ground-to-air viewpoint attribute set, the first target viewpoint is optimized with the constraint of minimizing the total motion cost. A second target viewpoint is selected from the multiple viewpoints corresponding to each of the boundaries to obtain the second ground-to-air viewpoint chain composed of the second target viewpoint.

6. The method according to any one of claims 1-5, characterized in that, Before acquiring the local task grid, interacting with other multimodal drones in the multimodal drone swarm to update the local task grid includes: Based on the local task grid and interaction history, the interaction object is determined, a request message containing a task reassignment scheme is sent, and the local task grid is updated according to the response message.

7. A multimodal unmanned swarm distributed cooperative exploration and planning device, characterized in that, include: The grid acquisition module is used to acquire the local task grid and determine the boundary set of unknown regions in the local task grid; The set generation module is used to generate a ground-air viewpoint attribute set based on the boundary set. The ground-air viewpoint attribute set includes the pose attributes and modal attributes of the viewpoints of each boundary segment in the boundary set. The modal attributes include aerial viewpoints and ground viewpoints. The path planning module is used to determine a first ground-air viewpoint chain to cover the unknown area based on the ground-air viewpoint attribute set. The first ground-air viewpoint chain passes through each of the boundaries and one viewpoint, and the sum of the path costs of all adjacent viewpoints on the first ground-air viewpoint chain is minimized. When the modal attributes of adjacent viewpoints are different, the path cost includes the penalty for switching the modal attributes. The path optimization module is used to optimize the motion cost of the first ground-to-air viewpoint chain based on the ground-to-air viewpoint attribute set to obtain a second ground-to-air viewpoint chain, and generate the exploration trajectory of the multimodal UAV based on the second ground-to-air viewpoint chain.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the multimodal unmanned cluster distributed cooperative exploration planning method as described in any one of claims 1-6.

9. A terminal device, characterized in that, It includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein when the processor executes the program, it implements the multimodal unmanned swarm distributed cooperative exploration planning method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal unmanned cluster distributed collaborative exploration planning method as described in any one of claims 1-6.

Images