Air-ground cooperative autonomous exploration method and system in large-scale unknown environment

Abstract

The application discloses an air-ground cooperative autonomous exploration method and system in a large-scale unknown environment, and the method comprises the following steps: firstly, autonomous exploration of an aerial robot is performed through construction of an overview view and a coverage exploration strategy for the overview view; then, autonomous exploration of a ground robot is performed based on an attention mechanism; finally, the trajectory of the ground robot is refined according to a 2D ESDF map, and the air-ground cooperative autonomous exploration is completed. The embodiment of the application can quickly cover a high-yield area for efficient exploration, and can be widely applied to the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for air-ground collaborative autonomous exploration in large-scale unknown environments. Background Technology

[0002] During robot exploration, the robot continuously calculates the potential gain at each viewpoint based on the acquired point cloud, balancing factors such as boundary coverage, robot posture, and travel distance. It then sequentially visits a series of high-gain viewpoints to explore unknown areas. This is an effective strategy for UGVs with limited local information. However, due to the lack of global information, this can lead to repeated exploration of known areas or neglect of unexplored areas, severely impacting autonomous exploration efficiency, especially in large-scale environments. To address this issue, recent developments have focused on providing robots with prior information about the area to be explored, enabling them to formulate more informed global exploration strategies based on this prior knowledge. While these methods improve exploration efficiency, they require significant human and effort to provide accurate prior information. Incorrect prior information may hinder rather than aid the exploration process.

[0003] Furthermore, with the increasing prevalence of multi-robot systems, collaborative exploration through sharing and negotiating exploration areas is gaining attention. However, most research focuses on homogeneous robot swarms, while studies on heterogeneous collaboration between ground and aerial robots are limited. Considering the characteristics of heterogeneous robots, ground robots possess strong payload capacity and endurance, but lack the agility and maneuverability of aerial robots. Conversely, aerial robots are more agile and maneuverable, but are limited by their limited endurance. Utilizing the advantages of both ground and aerial robots to improve exploration efficiency is an area worthy of further exploration.

[0004] The main problems in related technologies are as follows:

[0005] 1. For UGVs with limited local information, when visiting a series of high-gain viewpoints in sequence to explore unknown areas, the lack of global information may lead to repeated exploration of known areas or neglect of unexplored areas, which seriously affects the efficiency of autonomous exploration, especially in large-scale environments.

[0006] 2. Most research focuses on homogeneous robot swarms, while research on heterogeneous collaboration between ground and aerial robots is limited. Summary of the Invention

[0007] The main objective of this invention is to propose an air-ground collaborative autonomous exploration method and system for large-scale unknown environments, which can quickly cover high-yield areas for efficient exploration.

[0008] To achieve the above objectives, one aspect of this invention proposes an air-ground cooperative autonomous exploration method for large-scale unknown environments, comprising the following steps:

[0009] Autonomous exploration by aerial robots is carried out by constructing a bird's-eye view and using a coverage exploration strategy for the bird's-eye view;

[0010] Autonomous exploration of ground robots based on attention mechanisms;

[0011] Based on the 2D ESDF map, the trajectory of the ground robot is refined to complete the autonomous exploration in an air-ground collaborative manner.

[0012] In some embodiments, the autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy based on the bird's-eye view includes the following steps:

[0013] Construct a bird's-eye view based on the projections of different objects on the ground;

[0014] Based on the bird's-eye view, an exploration strategy is constructed to cover the area to be explored, thereby enabling the aerial robot to explore autonomously.

[0015] In some embodiments, constructing a bird's-eye view based on the projections of different objects on the ground includes the following steps:

[0016] As the drone moves, it captures a series of images using its onboard camera. The semantic information in each image is used to generate a bird's-eye view by mapping different objects onto the ground.

[0017] Grounded-SAM and RAM are used to automatically label and segment object masks in images. For the i-th image, after automatic labeling and segmentation, a mask and a corresponding list of labels can be obtained.

[0018] Each pixel in the image is converted to camera coordinates and then mapped to its corresponding coordinates; the conversion process from pixel coordinates to camera coordinates is as follows:

[0019]

[0020] Where h is the flight altitude of the drone. f is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system; K is the intrinsic matrix of the camera, where f x and f y c represents focal length. x and c y It is the optical center; p c Represents the corresponding position in the camera coordinate system; x vRepresents the x-axis coordinate of pixel pv in the image; y v Represents the y-axis coordinate of pixel pv in the image;

[0021] The transformation expression from the camera coordinate system to the world coordinate system is:

[0022]

[0023] in, and Represents the rotation matrix and translation vector from the UAV's fuselage coordinate system to the world coordinate system; and Represents the rotation matrix and translation vector from the camera coordinate system to the UAV body coordinate system;

[0024] The location of object segments above the ground plane is determined by intersecting their continuous mappings; a topology is established between ground segments and other object segments to locate the same object segments in continuous images;

[0025] By combining the topology with UAV positioning information, the corresponding ground segments can be adjusted by stretching and shrinking.

[0026] In some embodiments, the step of constructing an exploration strategy based on the bird's-eye view to cover the area to be explored, thereby enabling the aerial robot to explore autonomously, includes the following steps:

[0027] Areas not covered by the drone's field of view are defined as unknown areas, and areas covered by the drone are defined as known areas.

[0028] Construct a level-restricted quadtree to decompose the bird's-eye view into a set of disjoint cells; during the decomposition process, any cell at level i is divided into four sub-cells at level i+1 to represent the exploration area; after subdivision, the center of the cell is replaced by the center of the sub-cell;

[0029] Uniformly sample a series of candidate waypoints along the boundary of the known area, excluding locations already occupied by obstacles;

[0030] Calculate the potential gain for each waypoint, and progressively add or remove candidate waypoints as you move;

[0031] For a quadtree with n cells, each cell... Represented as a set of data in Indicates the position of the cell center. Indicates the size of the cell. Indicates the state of the cell;

[0032] Evaluate a given candidate waypoint w based on the number of new cells observed from the waypoint and the distance to the waypoint. i Potential gain, potential benefit s(w i The formula for calculating ) is:

[0033]

[0034] Among them, the function It is an indicator function that takes the value 1 when the condition in parentheses is true, and 0 otherwise; T s,j T represents the Ts value of the j-th cell; T p,j Tp represents the value of cell j; T o,j This represents the To value of the j-th cell.

[0035] In some embodiments, the autonomous exploration of the ground robot based on the attention mechanism includes the following steps:

[0036] Based on an attention mechanism, a ground robot performs global exploration; specifically, it calculates the gradient from the global navigation point by checking whether the vector formed by two adjacent points is perpendicular to the gradient of a given point, with a tolerance of ∈. The global guide path is used to navigate all closed loops; the function `trilinearInterpolation()` uses trilinear interpolation to estimate the gradient, while the function `nearestPoints()` uses an incremental kd-tree to quickly search for the two closest points to a given point; multiple closed loop sub-paths are connected to generate a global path; the function `findTwoClosestPointsIndex()` determines the two closest points between two closed loop sub-loops; the TSP problem is extended between the two closest points on both sides of each closed loop sub-loop until the solution stabilizes.

[0037] Local exploration is carried out by a ground robot based on local exploration triggers and sparse route maps;

[0038] Generate frontier information and viewpoint information; specifically: evaluate the information gain of the viewpoint by calculating the number of boundaries directly observable from the viewpoint; when updating the information gain, only update the newly generated boundary distances less than d. max The old viewpoint; when the number of connected boundaries becomes zero, the viewpoint will be deleted, and the boundary will be marked as known only after being covered N times;

[0039] Based on the aforementioned cutting-edge information and viewpoint information, a local exploration path is constructed.

[0040] In some embodiments, the local exploration by a ground robot based on local exploration triggers and sparse route maps includes the following steps:

[0041] When the ground robot detects an unknown area not shown in the bird's-eye view, the local exploration trigger is activated;

[0042] The algorithm determines whether a region is within a ground segment by using ray casting. Specifically, it calculates the number of intersections between the rays cast from the region and the edge of the ground segment. If the number of intersections is odd, the region is inside the polygon; if it is outside, a local exploration trigger is activated.

[0043] During local exploration, a sparse route graph is used to store and query the 3D map. This sparse route graph is a graph structure composed of nodes and edges, represented as follows: node Randomly sampled in the free space of the exploration region, the topological connection between a node and its neighboring nodes is an edge ε;

[0044] Nodes generated near adjacent nodes, if the distance is less than the threshold d th Nodes that do not meet the distance requirements will be pruned, and only nodes that meet the distance requirements will be added to the sparse route graph.

[0045] In some embodiments, refining the trajectory of the ground robot based on a 2D ESDF map to complete air-ground cooperative autonomous exploration includes the following steps:

[0046] Construct a 2D ESDF map and generate locally feasible path searches; specifically, obstacles on the ground and in space, as well as dynamic obstacles, are mapped to the ESDF map with a corresponding value of -1; narrow passages that the ground robot cannot safely traverse and height differences between adjacent cells exceeding a threshold α are identified. th The area is also mapped to -1; considering the size of the UGV, the cell with a value of -1 in the ESDF map is inflated into its 8-neighborhood, and all other cells are marked as passable; the value of passable cells in the ESDF map is calculated step by step; when the path planned by the global and local explorers passes through the area with a value of -1 in the ESDF map, a hybrid state is used to search for a local feasible path in the ESDF map; a unified trajectory optimizer is applied to refine the feasible path and control commands are issued to the ground robot for execution;

[0047] Determine the polynomial representation of the exploration path;

[0048] Based on the polynomial representation of the exploration path, the trajectory is optimized according to the safety, dynamic feasibility, and smoothness of the final trajectory.

[0049] In some embodiments, optimizing the trajectory based on the safety, dynamic feasibility, and smoothness of the final trajectory according to the polynomial representation of the exploration path includes the following steps:

[0050] After representing the trajectory segment using a polynomial, based on the linear equations in the equation MC = b, and considering the safety, dynamic feasibility, and smoothness of the final trajectory, an unconstrained optimization problem is proposed. The time of each trajectory segment is discretized into k. i There are time intervals, among which The position of the trajectory at the j-th interval is represented by equation x. i (t) Calculation yields the following expression for the optimization problem:

[0051]

[0052] Among them, f s ,f d and f m These represent penalty terms for safety, dynamic feasibility, and smoothness, respectively, with corresponding coefficients λ. s , λ d and λ m , This is a time penalty used to minimize the total execution time of the trajectory;

[0053] Among them, the safety penalty is used to ensure that the UGV maintains a certain distance from the obstacle. min The minimum distance to avoid a collision, the expression for the safety penalty is:

[0054]

[0055] Where d(x) represents the distance from point x to the nearest obstacle; d min This represents the minimum distance the UGV maintains from the obstacle;

[0056] The dynamic feasibility penalty is used to ensure that the trajectory segment does not exceed the physical limitations of the ground robot. The expression for the dynamic feasibility penalty is:

[0057] f d =σ v f d,v +σ a f d,a ,

[0058] Where, σ v Represents the speed penalty parameter; f d,v Represents speed dynamics penalty; f d,a Represents acceleration dynamics penalty;

[0059] The expression for the smoothness penalty is:

[0060]

[0061] in, represents the first derivative of x; M represents the total number of segments the trajectory is divided into.

[0062] Another aspect of this invention provides an air-ground cooperative autonomous exploration system for large-scale unknown environments, comprising:

[0063] The first module is used to enable the aerial robot to conduct autonomous exploration by constructing a bird's-eye view and using a coverage exploration strategy for the bird's-eye view;

[0064] The second module is used for autonomous exploration of ground robots based on attention mechanisms;

[0065] The third module is used to refine the trajectory of the ground robot based on the 2D ESDF map, and complete the autonomous exploration of air and ground cooperation.

[0066] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0067] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0068] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0069] The embodiments of this invention include at least the following beneficial effects: This invention provides a method and system for autonomous exploration in a large-scale unknown environment through air-ground collaboration. The scheme first constructs a bird's-eye view and a coverage exploration strategy based on that view to enable autonomous exploration by an aerial robot; then, based on an attention mechanism, it enables autonomous exploration by a ground robot; finally, based on a 2DESDF map, the trajectory of the ground robot is refined to complete the air-ground collaborative autonomous exploration. The embodiments of this invention can quickly cover high-yield areas for efficient exploration. Attached Figure Description

[0070] Figure 1 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0071] Figure 2 This is a flowchart illustrating the implementation process in a specific scenario provided by the embodiments of the present invention;

[0072] Figure 3This is a schematic diagram of the bird's-eye view BEV generation process provided in an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram showing the position of an object segment above the ground plane provided in an embodiment of the present invention;

[0074] Figure 5 This is a schematic diagram of the division of the exploration area provided in an embodiment of the present invention;

[0075] Figure 6 This is a schematic diagram of global waypoints provided in an embodiment of the present invention;

[0076] Figure 7 This is a schematic diagram of the sparse route map provided in an embodiment of the present invention;

[0077] Figure 8 This is a schematic diagram of the viewpoint generation provided in an embodiment of the present invention;

[0078] Figure 9 This is a schematic diagram of the construction of the ESDF map provided in an embodiment of the present invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0080] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0081] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0083] Before providing a detailed description of the embodiments of the present invention, some related technologies involved in the embodiments of the present invention will be described first, as follows:

[0084] BEV: BEV stands for Bird's-Eye-View, a vivid expression used to describe the perspective gained from a high altitude, much like the view a bird sees when flying down. In mapmaking, a bird's-eye view is a special map representation that simulates the effect of viewing the Earth's surface vertically from directly above, clearly showing the spatial relationships and layout of geographical features such as terrain, buildings, and road networks.

[0085] Autonomous exploration: Autonomous robot exploration refers to a robot's ability to autonomously perceive its environment, plan its path, and execute actions without direct human guidance or control, using its built-in intelligent algorithms and sensor systems to explore and understand the unknown. This exploration can occur in various fields, including but not limited to scientific research, industrial applications, military uses, and daily life. In this process, the robot needs to possess a certain degree of intelligence and autonomy, capable of making adaptive decisions based on changes in the environment to complete tasks and achieve goals.

[0086] The air-ground collaborative autonomous exploration method and system for large-scale unknown environments provided in this invention relates to the field of computer technology. The air-ground collaborative autonomous exploration method for large-scale unknown environments provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, drone, smart robot, or vehicle-mounted terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the air-ground collaborative autonomous exploration method for large-scale unknown environments, but is not limited to the above forms.

[0087] This invention provides a method for air-ground cooperative autonomous exploration in large-scale unknown environments, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an air-ground collaborative autonomous exploration method for large-scale unknown environments applied to servers, provided as an embodiment of the present invention. The executing entity of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 1 The method may include the following steps:

[0088] Autonomous exploration by aerial robots is carried out by constructing a bird's-eye view and using a coverage exploration strategy for the bird's-eye view;

[0089] Autonomous exploration of ground robots based on attention mechanisms;

[0090] Based on the 2D ESDF map, the trajectory of the ground robot is refined to complete the autonomous exploration in an air-ground collaborative manner.

[0091] In some embodiments, the autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy based on the bird's-eye view includes the following steps:

[0092] Construct a bird's-eye view based on the projections of different objects on the ground;

[0093] Based on the bird's-eye view, an exploration strategy is constructed to cover the area to be explored, thereby enabling the aerial robot to explore autonomously.

[0094] In some embodiments, constructing a bird's-eye view based on the projections of different objects on the ground includes the following steps:

[0095] As the drone moves, it captures a series of images using its onboard camera. The semantic information in each image is used to generate a bird's-eye view by mapping different objects onto the ground.

[0096] Grounded-SAM and RAM are used to automatically label and segment object masks in images. For the i-th image, after automatic labeling and segmentation, a mask and a corresponding list of labels can be obtained.

[0097] Each pixel in the image is converted to camera coordinates and then mapped to its corresponding coordinates; the conversion process from pixel coordinates to camera coordinates is as follows:

[0098]

[0099] Where h is the flight altitude of the drone. f is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system; K is the intrinsic matrix of the camera, where fx and f y c represents focal length. x and c y It is the optical center; p c Represents the corresponding position in the camera coordinate system; x v Represents the x-axis coordinate of pixel pv in the image; y v Represents the y-axis coordinate of pixel pv in the image;

[0100] The transformation expression from the camera coordinate system to the world coordinate system is:

[0101]

[0102] in, and Represents the rotation matrix and translation vector from the UAV's fuselage coordinate system to the world coordinate system; and Represents the rotation matrix and translation vector from the camera coordinate system to the UAV body coordinate system;

[0103] The location of object segments above the ground plane is determined by intersecting their continuous mappings; a topology is established between ground segments and other object segments to locate the same object segments in continuous images;

[0104] By combining the topology with UAV positioning information, the corresponding ground segments can be adjusted by stretching and shrinking.

[0105] In some embodiments, the step of constructing an exploration strategy based on the bird's-eye view to cover the area to be explored, thereby enabling the aerial robot to explore autonomously, includes the following steps:

[0106] Areas not covered by the drone's field of view are defined as unknown areas, and areas covered by the drone are defined as known areas.

[0107] Construct a level-restricted quadtree to decompose the bird's-eye view into a set of disjoint cells; during the decomposition process, any cell at level i is divided into four sub-cells at level i+1 to represent the exploration area; after subdivision, the center of the cell is replaced by the center of the sub-cell;

[0108] Uniformly sample a series of candidate waypoints along the boundary of the known area, excluding locations already occupied by obstacles;

[0109] Calculate the potential gain for each waypoint, and progressively add or remove candidate waypoints as you move;

[0110] For a quadtree with n cells, each cell... Represented as a set of data in Indicates the position of the cell center. Indicates the size of the cell. Indicates the state of the cell;

[0111] Evaluate a given candidate waypoint w based on the number of new cells observed from the waypoint and the distance to the waypoint. i Potential gain, potential benefit s(w i The formula for calculating ) is:

[0112]

[0113] Among them, the function It is an indicator function that takes the value 1 when the condition in parentheses is true, and 0 otherwise; T s,j T represents the Ts value of the j-th cell; T p,j Tp represents the value of cell j; T o,j This represents the To value of the j-th cell.

[0114] In some embodiments, the autonomous exploration of the ground robot based on the attention mechanism includes the following steps:

[0115] Based on an attention mechanism, a ground robot performs global exploration; specifically, it calculates the gradient from the global navigation point by checking whether the vector formed by two adjacent points is perpendicular to the gradient of a given point, with a tolerance of ∈. The global guide path is used to navigate all closed loops; the function `trilinearInterpolation()` uses trilinear interpolation to estimate the gradient, while the function `nearestPoints()` uses an incremental kd-tree to quickly search for the two closest points to a given point; multiple closed loop sub-paths are connected to generate a global path; the function `findTwoClosestPointsIndex()` determines the two closest points between two closed loop sub-loops; the TSP problem is extended between the two closest points on both sides of each closed loop sub-loop until the solution stabilizes.

[0116] Local exploration is carried out by a ground robot based on local exploration triggers and sparse route maps;

[0117] Generate frontier information and viewpoint information; specifically: evaluate the information gain of the viewpoint by calculating the number of boundaries directly observable from the viewpoint; when updating the information gain, only update the newly generated boundary distances less than d. max The old viewpoint; when the number of connected boundaries becomes zero, the viewpoint will be deleted, and the boundary will be marked as known only after being covered N times;

[0118] Based on the aforementioned cutting-edge information and viewpoint information, a local exploration path is constructed.

[0119] In some embodiments, the local exploration by a ground robot based on local exploration triggers and sparse route maps includes the following steps:

[0120] When the ground robot detects an unknown area not shown in the bird's-eye view, the local exploration trigger is activated;

[0121] The algorithm determines whether a region is within a ground segment by using ray casting. Specifically, it calculates the number of intersections between the rays cast from the region and the edge of the ground segment. If the number of intersections is odd, the region is inside the polygon; if it is outside, a local exploration trigger is activated.

[0122] During local exploration, a sparse route graph is used to store and query the 3D map. This sparse route graph is a graph structure composed of nodes and edges, represented as follows: node Randomly sampled in the free space of the exploration region, the topological connection between a node and its neighboring nodes is an edge ε;

[0123] Nodes generated near adjacent nodes, if the distance is less than the threshold d th Nodes that do not meet the distance requirements will be pruned, and only nodes that meet the distance requirements will be added to the sparse route graph.

[0124] In some embodiments, refining the trajectory of the ground robot based on a 2D ESDF map to complete air-ground cooperative autonomous exploration includes the following steps:

[0125] Construct a 2D ESDF map and generate locally feasible path searches; specifically, obstacles on the ground and in space, as well as dynamic obstacles, are mapped to the ESDF map with a corresponding value of -1; narrow passages that the ground robot cannot safely traverse and height differences between adjacent cells exceeding a threshold α are identified. th The area is also mapped to -1; considering the size of the UGV, the cell with a value of -1 in the ESDF map is inflated into its 8-neighborhood, and all other cells are marked as passable; the value of passable cells in the ESDF map is calculated step by step; when the path planned by the global and local explorers passes through the area with a value of -1 in the ESDF map, a hybrid state is used to search for a local feasible path in the ESDF map; a unified trajectory optimizer is applied to refine the feasible path and control commands are issued to the ground robot for execution;

[0126] Determine the polynomial representation of the exploration path;

[0127] Based on the polynomial representation of the exploration path, the trajectory is optimized according to the safety, dynamic feasibility, and smoothness of the final trajectory.

[0128] In some embodiments, optimizing the trajectory based on the safety, dynamic feasibility, and smoothness of the final trajectory according to the polynomial representation of the exploration path includes the following steps:

[0129] After representing the trajectory segment using a polynomial, based on the linear equations in the equation MC = b, and considering the safety, dynamic feasibility, and smoothness of the final trajectory, an unconstrained optimization problem is proposed. The time of each trajectory segment is discretized into k. i There are time intervals, among which The position of the trajectory at the j-th interval is represented by equation x. i (t) Calculation yields the following expression for the optimization problem:

[0130]

[0131] Among them, f s ,f d and f m These represent penalty terms for safety, dynamic feasibility, and smoothness, respectively, with corresponding coefficients λ. s , λ d and λ m , This is a time penalty used to minimize the total execution time of the trajectory;

[0132] Among them, the safety penalty is used to ensure that the UGV maintains a certain distance from the obstacle. min The minimum distance to avoid a collision, the expression for the safety penalty is:

[0133]

[0134] Where d(x) represents the distance from point x to the nearest obstacle; d min This represents the minimum distance the UGV maintains from the obstacle;

[0135] The dynamic feasibility penalty is used to ensure that the trajectory segment does not exceed the physical limitations of the ground robot. The expression for the dynamic feasibility penalty is:

[0136] f d =σ v f d,v +σ a f d,a ,

[0137] Where, σ v Represents the speed penalty parameter; f d,v Represents speed dynamics penalty; f d,a Represents acceleration dynamics penalty;

[0138] The expression for the smoothness penalty is:

[0139]

[0140] in, represents the first derivative of x; M represents the total number of segments the trajectory is divided into.

[0141] The implementation process of the embodiments of the present invention in specific application scenarios will be described in detail below with reference to the accompanying drawings:

[0142] In view of the problems existing in the prior art, the purpose of this invention is to propose an efficient air-ground collaborative autonomous exploration framework.

[0143] This technology has the following characteristics:

[0144] 1. Drones rapidly explore high-yield areas to generate bird's-eye view BEVs, and ground robots (UGVs) utilize BEVs for efficient exploration.

[0145] 2. Based on the constructed BEV, a rapid coverage exploration strategy for drones is proposed, enabling drones to quickly cover high-yield areas.

[0146] 3. A layered UGV exploration strategy is introduced, in which the global explorer uses BEV and attention mechanisms to establish a global exploration path, allowing UGV to plan the globally optimal path and pay more attention to textured areas. Local explorers are triggered to explore areas not included in the BEV.

[0147] refer to Figure 2 The collaborative air-to-ground reconnaissance system proposed in this invention integrates the capabilities of UAVs and UGVs to improve exploration efficiency in large-scale unknown environments. UAVs equipped with onboard cameras capture images and use Grounded-Segment-Anything (GSM) and Random Object Recognition (RAM) models to segment and identify different objects and areas. These segmented masks are further mapped to create BEVs, providing a comprehensive overview of the area and highlighting key areas for further UGV exploration. Furthermore, the BEVs guide the rapid exploration of the UAVs by sampling viewpoints within known areas and calculating their potential benefits, thereby enabling rapid site coverage by the UAVs.

[0148] During UGV exploration, the shared BEV (Browser Exploration Vehicle) makes the UGV's exploration more informed. The UGV employs a hierarchical strategy: a high-level global explorer provides global navigation based on the BEV, while a low-level local explorer handles areas not covered by the BEV. The high-level global explorer finds the globally optimal guidance route under given conditions from the real-time BEV. To focus UGV exploration on areas rich in environmental features, rather than featureless flat ground, an attention mechanism is introduced to optimize the selection of global guidance routes. When encountering unknown areas not covered by the BEV, a local explorer is triggered, allowing the UGV to identify boundaries and generate local exploration paths to access high-gain viewpoints, ensuring comprehensive coverage of unknown areas.

[0149] While global planning and local exploration planning are highly efficient, they can involve sharp turns that require the UGV to slow down, impacting exploration efficiency. Furthermore, following a path may lead to collisions with dynamic obstacles. Therefore, a trajectory optimizer is used to further refine the trajectory. This process involves analyzing the traversability of the UGV in the 3D environment and mapping obstacles to a 2D Euclidean signed distance field (ESDF) that moves with the UGV. When encountering impassable obstacles, locally feasible paths are searched within the ESDF. Finally, considering factors such as safety, dynamic feasibility, and smoothness, a polynomial-based ballistic optimization problem is proposed to ensure the optimal and safe trajectory of the UGV.

[0150] The core steps of this invention include:

[0151] Step 1: Autonomous Exploration by Aerial Robots. This step mainly includes the construction process of the BEV (Balanced Exploration Vehicle) and proposes a rapid coverage exploration algorithm based on the BEV to facilitate UAVs covering as many high-yield exploration areas as possible in a short time. The detailed steps are as follows:

[0152] Step 1: Build BEV:

[0153] The BEV used in the proposed framework only provides coarse prior information to UAVs and UGVs, and is therefore defined as the projection of different objects onto the ground. It can be obtained through a direct and rapid mapping process, unlike other high-precision BEVs that require complex data processing and fusion. As the UAV moves, the onboard camera captures a series of images. Each image contains rich semantic information, which can be used to generate BEVs by mapping different objects onto the ground. Figure 3 The BEV generation process is demonstrated, where Grounded-SAM and RAM are used to automatically label and segment object masks in an image. For the i-th image, after automatic labeling and segmentation, a mask O = {O1, O2, ..., O...} can be obtained. n} and corresponding tags The list includes information on ground masks, which are of significant help to ground robots.

[0154] Without sacrificing generality, we assume the ground is a plane and that the UAV and UGV share the same world coordinate system, with the UAV flying at a fixed altitude. Each pixel p in the image... v (x v ,y v All of these can be converted to the camera coordinate system and then mapped to their corresponding coordinates p. w (x w ,y w The transformation process from pixel coordinates to camera coordinates is as follows:

[0155]

[0156] Where h is the flight altitude of the drone. It is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system. K is the intrinsic matrix of the camera, where f x and f y c represents focal length. x and c y It is the optical center. The transformation from the camera coordinate system to the world coordinate system is then defined as:

[0157]

[0158] in, and This represents the rotation matrix and translation vector from the UAV's fuselage coordinate system to the world coordinate system. Similarly, and This represents the rotation matrix and translation vector from the camera coordinate system to the UAV body coordinate system.

[0159] In the above mapping process, various object segments that do not belong to the ground plane are directly projected onto the ground plane, resulting in projection distortion in their positioning. To alleviate this problem, the positions of object segments above the ground plane are determined by continuous mappings that intersect them, such as... Figure 4 The black cell at the intersection of the red and blue cells is shown. Furthermore, due to the limited voxel resolution of the image, distant object segments may cause significant deviations after mapping. Therefore, a topology is established between ground segments and other object segments, allowing the location of the same object segments in consecutive images. By combining this topology with UAV positioning information, the corresponding ground segments can be adjusted by stretching and shrinking, thereby improving the accuracy of BEV (Balance of Vehicle Equivalent).

[0160] Step 2: Quickly explore the coverage:

[0161] In the proposed framework, the UAV provides the UGV with the BEV (Browser Energy Scope) of the area to be explored. To achieve autonomous and rapid coverage of high-yield areas, a BEV-based exploration strategy is proposed. Areas not covered by the UAV's Field of View (FoV) are defined as unknown areas, while covered areas are known areas, including free space and impassable obstacles. A level-restricted quadtree is used to decompose the BEV into a set of disjoint cells. Each cell in the quadtree can be further subdivided into four higher-level subcells, forming cells with different resolutions from coarse to fine. During the decomposition process, any cell at level i can be divided into four subcells at level i+1, thus allowing for flexible representation of the exploration area. Figure 5 This illustrates the division of the exploration area, where red dots represent the center of a cell. After further subdivision, the center of a cell will be replaced by the center of its sub-cells. Due to the limited number of levels, the highest-level cells covering more than half of the area are marked as known areas.

[0162] To achieve autonomous exploration, a series of candidate waypoints are uniformly sampled along the boundary of the known region, excluding positions already occupied by obstacles, and denoted as follows: like Figure 5 The black dots in the diagram indicate this. The UAV calculates the potential gain for each waypoint, progressively adding or removing candidate waypoints as it moves. For a quadtree with n cells... Each cell It can be represented as a group in Indicates the position of the cell center. Indicates the size of the cell. This indicates the state of the cell (1 for known state, 0 for unknown state). Given candidate waypoints. The potential gain is assessed based on the number of new cells observable from the waypoint and the distance to the waypoint. Potential gain s(w) i It can be calculated as follows:

[0163]

[0164] Among them, the function This is an indicator function that takes the value 1 when the condition in parentheses is true, and 0 otherwise. The drone selects the option with the highest gain. As the next target location, the drone's exploration process is constantly being updated. And move to the candidate waypoint with the greatest potential gain. When the potential gain of any waypoint is below a threshold s th At that time, that is

[0165] The process of constructing a BEV relies on the assumption that the ground is planar. When mapping object segments to a quadtree, due to projection distortion, some obstacles whose actual height exceeds the ground (such as trees and buildings) may occupy a large area in the quadtree. Therefore, in practice, the cell size occupied by mapped object segments is limited. Although this method introduces some errors into the quadtree, our extensive testing shows that it does not significantly affect the UAV's exploration strategy. Furthermore, this embodiment of the invention does not consider obstacle avoidance during UAV flight, as this is handled automatically by the UAV's low-level controller.

[0166] Step Two: Autonomous Exploration by the Ground Robot. To achieve efficient exploration, this invention designs a hierarchical framework for the UGV, including a global resource manager and a local resource manager. The global explorer uses BEVs shared from the UAV to create global guide waypoints, thereby generating high-level exploration routes that avoid redundant exploration. When an unknown area not represented in the BEV is detected, the local explorer is activated. Furthermore, this embodiment of the invention also introduces an attention mechanism that focuses the UGV's exploration on areas requiring fine reconstruction, rather than open or featureless areas. Detailed steps are as follows:

[0167] Step 1: Global Exploration

[0168] As the UGV gradually receives real-time BEVs, its exploration becomes more intelligent, relying on prior information about the area to be explored. Although this information may be somewhat coarse at the edges of the BEV due to segmentation errors introduced by Grounded-SAM with RAM when processing images with limited resolution, it still provides significant assistance to the UGV's exploration. In exploration tasks, the areas that need to be reconstructed are regions with rich textures, rather than vast ground areas or large open spaces such as plazas. Notably, feature-rich areas are often located at the edges of BEV ground segments. Therefore, an attention mechanism is introduced to focus the UGV's attention on the areas requiring detailed reconstruction.

[0169] Each point p in the BEV ground segment is assigned an attention value, which is higher for points closer to the ground segment boundary. This means that the attention value of each point p is inversely proportional to the Euclidean distance d(p) from that point to the nearest boundary of the ground segment. Therefore, the attention value of a point p is calculated as follows:

[0170]

[0171] Here, d(p) can be calculated using an incremental method. Global waypoints With the same attention value ψ th Uniform sampling at the location, such as Figure 6The black dots are shown in the image. To facilitate quick exploration, all guide waypoints are displayed. By connecting them end-to-end, a global tour guide is constructed. Although this itinerary can be formulated as the Traveling Salesman Problem (TSP) and solved using algorithms such as x-opt and Lin-Kern ighan, the computational cost increases with the size of the TSP.

[0172] This invention does not directly solve for the TSP, but first utilizes the attention value and the internal connectivity of the ground segment to sequentially connect the waypoints in each ground segment along a direction perpendicular to the attention gradient. Generate a closed-loop global guided sub-tour. Then, establish a local TSP between the two nearest points of each pair of guided sub-tours to create a connection. The TSP gradually extends outward from the two nearest points until the solution stabilizes, ultimately forming the final global tour, as shown below. Figure 6 The red closed-loop navigation is shown in the image.

[0173] The detailed algorithm is shown in Algorithm 1. Lines 1-15 calculate the gradient from the global point to the navigation point by checking whether the vector formed by two adjacent points is perpendicular to the gradient of the given point, with tolerance ∈. The algorithm performs a global guide path tour of all closed loop sub-travels. The function `trilinearInterpolation()` uses trilinear interpolation to estimate the gradient, while the function `nearestPoints()` utilizes an incremental kd-tree to quickly search for the two closest points to a given point. Rows 16-25 connect multiple closed loop sub-travels to generate a global path. The function `findTwoClosestPointsIndex()` determines the two closest points between two closed loop sub-travels. Finally, the algorithm extends the TSP problem between the two closest points on either side of each closed loop sub-travel until a solution stabilizes. The TSP problem is solved using the Lin-Kern ighan heuristic to ensure computational efficiency.

[0174] The global navigation will be continuously updated as the UGV moves and receives updated BEVs. Covered guide waypoints will be removed from the global guidance path and the guidance will be reconnected accordingly. Additionally, due to obstacles in the UAV FoV, there may be multiple disconnected grounding points in the BEV, such as... Figure 6 The two yellow ground segments are shown in the diagram. If the UGV detects a connection between these segments, it applies the same strategy described in lines 16-27 of Algorithm 1 to regenerate a global navigation spanning multiple ground segments.

[0175]

[0176]

[0177] Step 2: Local Exploration

[0178] 1) Local exploration trigger

[0179] When the UGV detects an unknown area not represented in the BEV, a local exploration trigger is activated. A ray casting algorithm is used to determine if an area is within a ground segment. This is done by calculating the number of intersections between the rays cast from the area and the edge of the ground segment. If the number of intersections is odd, the area is inside the polygon; if it is outside, the local explorer is triggered.

[0180] 2) Sparse route map

[0181] During local exploration, UGVs frequently need to enter already explored areas and find efficient exploration paths within traversable regions. Therefore, a sparse route graph is used instead of directly storing and querying the 3D map to minimize memory and computational resource consumption. A sparse route graph is a graph structure consisting of nodes and edges, represented as... like Figure 7 As shown by the blue line in the diagram. Node Nodes are randomly sampled in the free space of the exploration region, and the topological connection between a node and its neighboring nodes is an edge ε. To ensure the sparsity of the route graph, nodes generated near neighboring nodes are considered valid if the distance is less than a threshold d. th Nodes that do not meet the distance requirements will be pruned, and only nodes that meet the distance requirements will be added to the sparse route graph.

[0182] 3) Frontier generation and viewpoint generation

[0183] Local explorers, based on FEP (Free-Edge Program), enable UGVs to rapidly cover boundaries, thus exploring unknown territories. Boundaries are defined as the intersection of unknown and known areas and are gradually updated as the UGVs move. To cover the boundaries, the UGVs move to designated viewpoints, which are uniformly sampled within an area centered on the UGV with a radius of d. v Smaller than LiDAR d max The maximum sensing distance. For example... Figure 8 As shown. At position p i The generated viewpoint is marked in blue, while at position p i+1 The newly generated viewpoints are marked in orange.

[0184] For efficient exploration, local exploration paths with high single-exploration gains should be planned to guide the UGV to continuously visit a series of viewpoints. However, as the scale of the explored scene increases, the computational cost of calculating viewpoint information gain also increases. To reduce this cost, two strategies are adopted. First, similar to FAEL, viewpoint information gain is evaluated by calculating the number of boundaries directly observable from the viewpoint, rather than calculating volumetric information gain. Boundaries are connected to the nearest visible viewpoint, such as... Figure 8 As shown by the purple dashed line, unconnected viewpoints will be deleted. Secondly, when updating the information gain, only viewpoints with a newly generated boundary distance less than d are updated. max The old viewpoints are deleted when the number of connected boundaries becomes zero. Furthermore, because the error in LiDAR measurements increases with the sensing distance, boundaries are only marked as known after N coverages.

[0185] 4) Local exploration path

[0186] Constructing local exploration paths can be viewed as a Traveling Science (TSP) problem, aiming to minimize total travel distance and turns while maximizing exploration gains. Given a set in This is the current location of the UGV. The problem of optimizing a local exploration path for the viewpoint to be visited is defined as follows:

[0187]

[0188] Where G(V) i P(V) represents information gain. i The function l(V0, V) represents the penalty term, where c1 and c2 are weighting coefficients. The penalty term consists of two parts: the first part is the distance traveled as a penalty, which aims to minimize the total travel length. i Calculate the distance from the position V0 of the UGV to the viewpoint V. i The cumulative distance, where V i-1 and V i The distance between them is searched using A* in a sparse route map. The second part is the turning penalty, designed to reduce the frequency of turns by the UGV and select the cycle with the fewest turns among multiple turns of the same length. The function g(V) i-1 V i The definition of a turn between two tourist routes is as follows:

[0189]

[0190] equation The optimization problem is solved using an improved 2-opt search. Once the local explorer has completed exploring regions that do not exist in the BEV, it returns to the global explorer to continue exploring.

[0191] While the paths planned by global and local explorers are efficient and safe, they may include sharp turns that require the UGV to slow down for tracking, such as... Figure 7As shown by the red line in the diagram. Furthermore, the presence of both static and dynamic obstacles makes directly following the path impractical. Therefore, further refinement and optimization are needed to generate a safe, dynamically feasible, and smooth trajectory.

[0192] Step 3: Trajectory Refinement for the Ground Robot. In this step, we address the need to improve and optimize global and local exploration path planning to ensure the generation of a safe, dynamically feasible, and smooth trajectory for the UGV. Furthermore, to handle static and dynamic obstacles that may affect UGV exploration, a 2D ESDF map is introduced for local feasible path search. Finally, a trajectory optimization problem is formulated based on polynomial piecewise operations, considering multiple penalty terms.

[0193] Step 1: 2D ESDF map and local feasible path search.

[0194] Although the UGV's movement is guided by prior information received from the BEV, it needs to avoid collisions with obstacles, especially moving vehicles and pedestrians, during actual exploration. Therefore, local path replanning is necessary. Directly storing and querying 3D maps of large-scale scenes consumes significant memory and time, and typical 2D maps cannot contain spatial information, leading to collisions with obstacles such as low bushes. To address this issue, a fixed-size 2D ESDF map is maintained and moved along with the UGV. It is generated after analyzing traversability in 3D space, used to search for locally feasible paths when some paths become impassable, and to optimize the trajectory.

[0195] The construction of the ESDF map involves two steps. First, obstacles on the ground and in space, as well as dynamic obstacles, are mapped onto the ESDF map with a corresponding value of -1. Additionally, narrow passages that UGVs cannot safely traverse and height differences between adjacent cells exceeding a threshold α are also considered. th The region is also mapped to -1, such as Figure 9 The purple cells in the diagram are shown. Considering the significant size of the UGV, cells with a value of -1 in the ESDF map are inflated to their 8-neighborhood to ensure safety, while all other cells are marked as passable. Subsequently, the values ​​of passable cells in the ESDF map are calculated incrementally.

[0196] When the paths planned by the global and local explorers traverse areas with a value of -1 in the ESDF map, a hybrid state A* is used to search for locally feasible paths in the ESDF map. Subsequently, a unified trajectory optimizer is applied to refine the feasible paths, and control commands are issued to the UGV for execution.

[0197] Step 2: Trajectory Representation.

[0198] The m-dimensional path of the global and local explorer planning can be represented in a unified form as M segments of trajectories, and each segment is a polynomial of order N = 2s - 1, s > 1. The ith segment of the trajectory can be expressed as:

[0199]

[0200] where β(t) = (1, t,... t N ) T is the basis function of the polynomial trajectory, represents the coefficient matrix of the polynomial. Each segment corresponds to a specific time T = {T1, T2,..., T M}. To ensure that all segments except the first and the last are connected at the start and end points, and the trajectory passes through the viewpoints in the planned path, the intermediate conditions Here, f i represents the number of specified intermediate states that must be reached at the intersection of two adjacent line segments, The intermediate conditions in the form of the coefficient matrix can be obtained from the following:

[0201]

[0202] where E i , F i ∈R 2s×2s :

[0203]

[0204] In the intermediate condition constraint (E i F i ), when f i = 0, it ensures that the adjacent segments are connected and satisfy the differentiable and continuous of order N. When 0 < f i < N, the intermediate conditions also guarantee that the connection points of the adjacent segments must pass through the specified states.

[0205] In addition, the initial state X0 ∈ R s×m and the initial boundary condition F0 = (β(0),..., β s-1 (0)) T ∈R s×2s are introduced to correspond to the current motion state of the UGV, ensuring the continuity and smoothness between the newly optimized trajectory and the ongoing trajectory. The terminal state X M ∈R s×m and the terminal boundary condition E i = (β(T M ),..., β (s-1) (T M )) T ∈R s×2sUsed to constrain the motion state required for the UGV to reach the end of the trajectory. Combined with equation (E i F i The following system of linear equations can be constructed:

[0206] MC = b

[0207] Among them here M∈R 2Ms×2Ms and b∈R 2Ms×2m Defined as:

[0208]

[0209] Step 3: Trajectory optimization.

[0210] After representing the trajectory segments using polynomials, an unconstrained optimization problem is proposed based on the linear equations in the equation MC = b, considering the safety, dynamic feasibility, and smoothness of the final trajectory. The time of each trajectory segment is discretized into k. i There are 10 time intervals, where 10, 20, 30, 40, 50, The position of the trajectory at the j-th interval is represented by equation x. i (t) is calculated. The optimization problem is defined as follows:

[0211]

[0212] Where, f s ,f d and f m These represent penalty terms for safety, dynamic feasibility, and smoothness, respectively, with corresponding coefficients λ. s , λ d and λ m . The time penalty aims to minimize the total execution time of the trajectory.

[0213] 1) Safety Penalty: Safety penalties ensure that the UGV maintains a safe distance from obstacles. min The minimum distance to avoid collision is defined as follows:

[0214]

[0215] Where d(x) represents the distance from point x to the nearest obstacle, which is calculated during the construction of the local ESDF map and does not require additional calculation. Γ(x1,x2) is a piecewise function defined as:

[0216]

[0217] 2) Dynamic feasibility penalty: The dynamic feasibility penalty ensures that the trajectory segment does not exceed the physical limitations of the UGV, such as the maximum speed v. maxMaximum acceleration a max and maximum angular velocity These punishments can be defined as follows:

[0218]

[0219] The final dynamic feasibility penalty is:

[0220] f d =σ v f d,v +σ a f d,a .

[0221] 3) Smoothness Penalty: A smoothness penalty is used to make the UGV's movement smoother and reduce unnecessary speed adjustments. This penalty is defined as:

[0222]

[0223] Solving equations After optimizing the problem, a transient optimal trajectory considering safety, smoothness, and dynamic feasibility is obtained. This trajectory is then sent to the lower-level controller of the UGV for trajectory tracking, thereby promoting efficient exploration.

[0224] In summary, this invention proposes a Collaborative Autonomous Exploration Framework for Air-Ground Robots (CAGE), leveraging the complementary strengths of Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) to improve the exploration efficiency and detailed reconstruction of UGVs in large-scale unknown environments. To our knowledge, this is the first framework to use UAVs to construct Bird's-Eye Views (BEVs) to assist UGV exploration. In this framework, UAVs equipped with onboard cameras utilize Grounded-Segmented Anything (GSAM) and Recognize Anything Model (RAM) to segment images and construct BEVs, identifying key areas for ground exploration. To rapidly cover high-yield areas, a quadtree-based exploration strategy is proposed to guide the movement of the UAVs. LiDAR-equipped UGVs use a hierarchical exploration mechanism for detailed probing based on real-time BEVs shared by the UAVs. To enhance reconstructed details, an attention mechanism is proposed to guide the UGVs to focus more on areas requiring detailed mapping rather than spacious and featureless areas. The effectiveness of the CAGE system is validated through practical experiments, demonstrating a significant improvement in exploration efficiency compared to state-of-the-art methods.

[0225] This invention has the following characteristics:

[0226] 1. Drones rapidly detect high-yield areas to generate BEVs, and ground robots utilize BEVs for efficient exploration.

[0227] 2. A rapid coverage exploration strategy for drones based on building BEV (Browser Active Vehicle) enables drones to quickly cover high-yield areas.

[0228] 3. A layered UGV exploration strategy is introduced, in which the global explorer uses BEV and attention mechanisms to establish a global exploration path, allowing UGV to plan the globally optimal path and pay more attention to textured areas. Local explorers are triggered to explore areas not included in the BEV.

[0229] Compared with the prior art, the present invention has the following advantages:

[0230] 1. High efficiency: Reduces autonomous exploration time and shortens the total distance traveled during autonomous exploration.

[0231] 2. High precision: The introduction of the attention mechanism brings higher point cloud density and better detail preservation.

[0232] 3. High mobility: The drone has high mobility and a wide field of view camera, which can quickly explore key areas and provide BEV priors for autonomous vehicle exploration.

[0233] 4. Long endurance: Autonomous vehicles have longer endurance and LiDAR equipment, which can perform detailed reconstruction of the area under the guidance of the proposed attention mechanism.

[0234] Another aspect of this invention provides an air-ground cooperative autonomous exploration system for large-scale unknown environments, comprising:

[0235] The first module is used to enable the aerial robot to conduct autonomous exploration by constructing a bird's-eye view and using a coverage exploration strategy for the bird's-eye view;

[0236] The second module is used for autonomous exploration of ground robots based on attention mechanisms;

[0237] The third module is used to refine the trajectory of the ground robot based on the 2D ESDF map, and complete the autonomous exploration of air and ground cooperation.

[0238] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0239] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned air-ground collaborative autonomous exploration method in a large-scale unknown environment. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0240] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0241] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned air-ground collaborative autonomous exploration method in a large-scale unknown environment.

[0242] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0243] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0244] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0245] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0246] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0247] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0248] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0249] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0250] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0251] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0252] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0253] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0254] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0255] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method for air-ground collaborative autonomous exploration in large-scale unknown environments, characterized in that, Includes the following steps: Autonomous exploration by aerial robots is carried out by constructing a bird's-eye view and using a coverage exploration strategy for the bird's-eye view; Autonomous exploration of ground robots based on attention mechanisms; Based on the 2D ESDF map, the trajectory of the ground robot is refined to complete the autonomous exploration in an air-ground collaborative manner; The autonomous exploration of the aerial robot by constructing a bird's-eye view and employing a coverage exploration strategy based on that bird's-eye view includes the following steps: Construct a bird's-eye view based on the projections of different objects on the ground; Based on the bird's-eye view, an exploration strategy based on the bird's-eye view is constructed to complete the coverage of the area to be explored, thereby realizing the autonomous exploration of the aerial robot; The process of constructing an exploration strategy based on the bird's-eye view to cover the area to be explored, thereby enabling the aerial robot to explore autonomously, includes the following steps: Areas not covered by the drone's field of view are defined as unknown areas, and areas covered by the drone are defined as known areas. Construct a level-restricted quadtree to decompose the bird's-eye view into a set of disjoint cells; during the decomposition process, any cell at level i is divided into four sub-cells at level i+1 to represent the exploration area; after subdivision, the center of the cell is replaced by the center of the sub-cell; Uniformly sample a series of candidate waypoints along the boundary of the known area, excluding locations already occupied by obstacles; Calculate the potential gain for each waypoint, and progressively add or remove candidate waypoints as you move; For a quadtree with n cells, each cell... Represented as a set of data in Indicates the position of the cell center. Indicates the size of the cell. Indicates the state of the cell; Evaluate a given candidate waypoint w based on the number of new cells observed from the waypoint and the distance to the waypoint. i Potential gain, potential benefit s(w i The formula for calculating ) is: Among them, the function It is an indicator function that takes the value 1 when the condition in parentheses is true, and 0 otherwise; T s,j T represents the Ts value of the j-th cell; T p,j Tp represents the value of cell j; T o,j This represents the To value of the j-th cell; The autonomous exploration of ground robots based on attention mechanisms includes the following steps: Based on an attention mechanism, a ground robot performs global exploration; specifically, it calculates the gradient from the global navigation point by checking whether the vector formed by two adjacent points is perpendicular to the gradient of a given point, with a tolerance of ∈. The global guide path is used to navigate all closed loops; the function `trilinearInterpolation()` uses trilinear interpolation to estimate the gradient, while the function `nearestPoints()` uses an incremental kd-tree to quickly search for the two closest points to a given point; multiple closed loop sub-paths are connected to generate a global path; the function `findTwoClosestPointsIndex()` determines the two closest points between two closed loop sub-loops; the TSP problem is extended between the two closest points on both sides of each closed loop sub-loop until the solution stabilizes. Local exploration is carried out by a ground robot based on local exploration triggers and sparse route maps; Generate frontier information and viewpoint information; specifically: evaluate the information gain of the viewpoint by calculating the number of boundaries directly observable from the viewpoint; when updating the information gain, only update the newly generated boundary distances less than d. max The old viewpoint; when the number of connected boundaries becomes zero, the viewpoint will be deleted, and the boundary will be marked as known only after being covered N times; Based on the aforementioned cutting-edge information and viewpoint information, a local exploration path is constructed.

2. The air-ground collaborative autonomous exploration method in a large-scale unknown environment according to claim 1, characterized in that, The process of constructing a bird's-eye view based on the projections of different objects on the ground includes the following steps: As the drone moves, it captures a series of images using its onboard camera. The semantic information in each image is used to generate a bird's-eye view by mapping different objects onto the ground. Grounded-SAM and RAM are used to automatically label and segment object masks in images. For the i-th image, after automatic labeling and segmentation, the mask and the corresponding label list are obtained. Each pixel in the image is converted to camera coordinates and then mapped to its corresponding coordinates; the conversion process from pixel coordinates to camera coordinates is as follows: Where h is the flight altitude of the drone. f is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system; K is the intrinsic matrix of the camera, where f x and f y c represents focal length. x and c y It is the optical center; p c Represents the corresponding position in the camera coordinate system; x v Represents the x-axis coordinate of pixel pv in the image; y v Represents the y-axis coordinate of pixel pv in the image; The transformation expression from the camera coordinate system to the world coordinate system is: in, and Represents the rotation matrix and translation vector from the UAV's fuselage coordinate system to the world coordinate system; and Represents the rotation matrix and translation vector from the camera coordinate system to the UAV body coordinate system; The location of object segments above the ground plane is determined by intersecting their continuous mappings; a topology is established between ground segments and other object segments to locate the same object segments in continuous images; By combining the topology with UAV positioning information, the corresponding ground segments can be adjusted by stretching and shrinking.

3. The air-ground collaborative autonomous exploration method in a large-scale unknown environment according to claim 1, characterized in that, The method of local exploration by a ground robot based on local exploration triggers and sparse route maps includes the following steps: When the ground robot detects an unknown area not shown in the bird's-eye view, the local exploration trigger is activated; The algorithm determines whether a region is within a ground segment by using ray casting. Specifically, it calculates the number of intersections between the rays cast from the region and the edge of the ground segment. If the number of intersections is odd, the region is inside the polygon; if it is outside, a local exploration trigger is activated. During local exploration, a sparse route graph is used to store and query the 3D map. This sparse route graph is a graph structure composed of nodes and edges, represented as follows: node Randomly sampled in the free space of the exploration region, the topological connection between a node and its neighboring nodes is an edge ε; Nodes generated near adjacent nodes, if the distance is less than the threshold d th Nodes that do not meet the distance requirements will be pruned, and only nodes that meet the distance requirements will be added to the sparse route graph.

4. The air-ground collaborative autonomous exploration method in a large-scale unknown environment according to claim 1, characterized in that, The process of refining the trajectory of the ground robot based on the 2D ESDF map to complete the air-ground cooperative autonomous exploration includes the following steps: Construct a 2D ESDF map and generate locally feasible path searches; specifically, obstacles on the ground and in space, as well as dynamic obstacles, are mapped to the ESDF map with a corresponding value of -1; narrow passages that the ground robot cannot safely traverse and height differences between adjacent cells exceeding a threshold α are identified. th The area is also mapped to -1; considering the size of the UGV, the cell with a value of -1 in the ESDF map is inflated into its 8-neighborhood, and all other cells are marked as passable; the value of passable cells in the ESDF map is calculated step by step; when the path planned by the global and local explorers passes through the area with a value of -1 in the ESDF map, a hybrid state is used to search for a local feasible path in the ESDF map; a unified trajectory optimizer is applied to refine the feasible path and control commands are issued to the ground robot for execution; Determine the polynomial representation of the exploration path; Based on the polynomial representation of the exploration path, the trajectory is optimized according to the safety, dynamic feasibility, and smoothness of the final trajectory.

5. The air-ground collaborative autonomous exploration method in a large-scale unknown environment according to claim 4, characterized in that, The optimization of the trajectory based on the polynomial representation of the exploration path, considering the safety, dynamic feasibility, and smoothness of the final trajectory, includes the following steps: After representing the trajectory segment using a polynomial, based on the linear equations in the equation MC = b, and considering the safety, dynamic feasibility, and smoothness of the final trajectory, an unconstrained optimization problem is proposed. The time of each trajectory segment is discretized into k. i There are time intervals, among which The position of the trajectory at the j-th interval is represented by equation x. i (t) Calculation yields the following expression for the optimization problem: Among them, f s ,f d and f m These represent penalty terms for safety, dynamic feasibility, and smoothness, respectively, with corresponding coefficients λ. s , λ d and λ m , This is a time penalty used to minimize the total execution time of the trajectory; Among them, the safety penalty is used to ensure that the UGV maintains a certain distance from the obstacle. min The minimum distance to avoid a collision, the expression for the safety penalty is: Where d(x) represents the distance from point x to the nearest obstacle; d min This represents the minimum distance the UGV maintains from the obstacle; The dynamic feasibility penalty is used to ensure that the trajectory segment does not exceed the physical limitations of the ground robot. The expression for the dynamic feasibility penalty is: f d =s v f d,v +s a f d,a , Where, σ v Represents the speed penalty parameter; f d,v Represents speed dynamics penalty; f d,a Represents acceleration dynamics penalty; The expression for the smoothness penalty is: in, Represents x i The first derivative; M represents that the trajectory is divided into M segments in total.

6. A system for implementing an air-ground cooperative autonomous exploration method in a large-scale unknown environment as described in any one of claims 1-5, characterized in that, include: The first module is used to enable the aerial robot to conduct autonomous exploration by constructing a bird's-eye view and using a coverage exploration strategy for the bird's-eye view; The second module is used for autonomous exploration of ground robots based on attention mechanisms; The third module is used to refine the trajectory of the ground robot based on the 2D ESDF map, and complete the autonomous exploration of air and ground cooperation.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

Images