Air-ground cooperative autonomous exploration method and system in large-scale unknown environment

Abstract

The invention discloses an air-ground cooperative autonomous exploration method and system in a large-scale unknown environment, and the method comprises the steps: firstly constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view, and carrying out the autonomous exploration of an aerial robot; then, based on an attention mechanism, autonomous exploration of the ground robot is carried out; and finally, according to the 2D ESDF map, the track of the ground robot is refined, and air-ground cooperative autonomous exploration is completed. The embodiment of the invention can quickly cover a high-yield area for efficient exploration, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an air-ground collaborative autonomous exploration method and system in a large-scale unknown environment. Background Art

[0002] During the robotic exploration process, the robot continuously calculates the potential gain at the viewpoint based on the obtained point cloud, and balances factors such as boundary coverage, robot posture, and driving distance. Then, they visit a series of high-gain viewpoints in sequence to explore the unknown area. This is an effective strategy for UGVs with only limited local information. However, due to the lack of global information, this may lead to repeated exploration of known areas or neglect of unexplored areas, seriously affecting the efficiency of autonomous exploration, especially in large-scale environments. To address this issue, recent developments have focused on providing robots with prior information about the area to be explored, enabling them to develop more informed global exploration strategies based on this prior knowledge. Although these methods improve exploration efficiency, they require a lot of manpower and effort to provide accurate prior information. Incorrect prior information may hinder rather than help the exploration process.

[0003] Furthermore, with the popularity of multi-robot systems, collaborative exploration through sharing and negotiating exploration areas is gaining attention. However, most studies focus on homogeneous robot clusters, while research on heterogeneous collaboration between ground and aerial robots is limited. Considering the characteristics of heterogeneous robots, ground robots have strong payload capacity and endurance, but lack the agility and maneuverability of aerial robots. In contrast, aerial robots are more agile and maneuverable, but are limited by their limited endurance. Leveraging the strengths of ground and aerial robots to improve exploration efficiency is an area worthy of further exploration.

[0004] In the related art, there are mainly the following problems:

[0005] 1. For UGVs with limited local information, when visiting a series of high-gain viewpoints in sequence to explore unknown areas, the lack of global information may lead to repeated exploration of known areas or neglect of unexplored areas, which seriously affects the efficiency of autonomous exploration, especially in large-scale environments;

[0006] 2. Most studies focus on homogeneous robot clusters, while there is limited research on heterogeneous collaboration between ground and aerial robots. Summary of the invention

[0007] The main purpose of the embodiments of the present invention is to propose an air-ground collaborative autonomous exploration method and system in a large-scale unknown environment, which can quickly cover high-yield areas for efficient exploration.

[0008] To achieve the above object, an embodiment of the present invention provides an air-ground collaborative autonomous exploration method in a large-scale unknown environment, comprising the following steps:

[0009] Conducting autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view;

[0010] Autonomous exploration of ground robots based on attention mechanism;

[0011] According to the 2D ESDF map, the trajectory of the ground robot is refined to complete the air-ground collaborative autonomous exploration.

[0012] In some embodiments, the autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view includes the following steps:

[0013] Construct a bird's-eye view based on the projections of different objects on the ground;

[0014] According to the bird's-eye view, an exploration strategy based on the bird's-eye view is constructed to complete the coverage of the area to be explored, thereby realizing autonomous exploration of the aerial robot.

[0015] In some embodiments, constructing a bird's-eye view according to projections of different objects on the ground includes the following steps:

[0016] As the drone moves, a series of images are captured by the onboard camera, and the semantic information in each image is used to generate a bird's-eye view by mapping different objects onto the ground;

[0017] Use Grounded-SAM and RAM to automatically mark and segment the object masks in the image. For the i-th image, after automatic annotation and segmentation, the mask and the corresponding label list can be obtained;

[0018] Each pixel in the image is converted to the camera coordinate system and then mapped to the corresponding coordinates; the conversion process from pixel coordinates to camera coordinate system is:

[0019]

[0020] Where h is the flight altitude of the drone, is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system; K is the intrinsic matrix of the camera, where f x and f y represents the focal length, c x and c y is the optical center; p c Represents the corresponding position in the camera coordinate system; x vRepresents the x-axis coordinate of the pixel pv in the image; y v Represents the y-axis coordinate of pixel pv in the image;

[0021] The conversion expression from the camera coordinate system to the world coordinate system is:

[0022]

[0023] in, and Represents the rotation matrix and translation vector from the drone body coordinate system to the world coordinate system; and Represents the rotation matrix and translation vector from the camera coordinate system to the drone body coordinate system;

[0024] The positions of object segments above the ground plane are determined by intersecting their continuous maps; a topology is established between ground segments and other object segments to locate the same object segments in consecutive images;

[0025] The topological structure is combined with the UAV positioning information to adjust the corresponding ground segment by stretching and shrinking.

[0026] In some embodiments, constructing an exploration strategy based on the bird's-eye view according to the bird's-eye view to complete the coverage of the area to be explored, thereby realizing autonomous exploration of the aerial robot, includes the following steps:

[0027] The area not covered by the drone’s field of view is defined as the unknown area, and the covered area is defined as the known area;

[0028] A level-restricted quadtree is constructed to decompose the bird's-eye view into a set of disjoint cells; during the decomposition, any cell at level i is divided into four sub-cells at level i+1 to represent the exploration area; after subdivision, the center of the cell is replaced by the center of the sub-cell;

[0029] A series of candidate waypoints are uniformly sampled along the boundary of the known area, excluding positions already occupied by obstacles;

[0030] Calculate the potential gain of each waypoint while gradually adding or removing candidate waypoints as you move;

[0031] For a quadtree with n cells, each cell Represented as a set of data in Indicates the location of the cell center, Indicates the size of the cell, Indicates the state of the cell;

[0032] Evaluate a given candidate waypoint w based on the number of new cells observed from the waypoint and the distance to the waypoint i The potential gain, potential benefit s(w i ) is calculated as:

[0033]

[0034] Among them, the function is an indicator function that takes the value 1 when the condition in the brackets is met, otherwise it takes the value 0; T s,j represents the Ts value of the jth cell; T p,j represents the Tp value of the jth cell; T o,j Represents the To value of the j-th cell.

[0035] In some embodiments, the autonomous exploration of the ground robot based on the attention mechanism includes the following steps:

[0036] Based on the attention mechanism, the ground robot performs global exploration; specifically, by checking whether the vector formed by two adjacent points is perpendicular to the gradient of a given point, the tolerance ∈ is calculated from the global direction to the navigation point. All closed-loop global wizard sub-tours of; function trilinearInterpolation() uses trilinear interpolation to estimate the gradient, and function nearestPoints() uses incremental kd-tree to quickly search for the two points closest to a given point; connect multiple closed-loop sub-paths to generate a global path function findTwoClosestPointsIndex() determines the two closest points between two closed-loop sub-tours; expand the TSP problem between the two closest points on both sides of each closed-loop sub-tour until the solution stabilizes;

[0037] Based on the local exploration trigger and sparse roadmap, the ground robot conducts local exploration;

[0038] Generate frontier information and viewpoint information; specifically: evaluate the information gain of the viewpoint by calculating the number of boundaries that can be observed directly from the viewpoint; when updating the information gain, only update to the newly generated boundaries whose distance is less than d max When the number of connected boundaries reaches zero, the viewpoint will be deleted, and the boundary will be marked as known only after it has been covered N times;

[0039] A local exploration path is constructed according to the frontier information and the viewpoint information.

[0040] In some embodiments, the local exploration by the ground robot based on the local exploration trigger and the sparse roadmap includes the following steps:

[0041] When the ground robot detects an unknown area that is not represented in the bird’s-eye view, a local exploration trigger is activated;

[0042] Determine whether a region is inside the ground segment by using a ray casting algorithm. Specifically, the number of intersections between the rays cast from the region and the edge of the ground segment is calculated. If the number of intersections is an odd number, it means that the region is inside the polygon. If it is outside, a local exploration trigger is triggered.

[0043] During the local exploration process, a sparse roadmap is used to store and query the 3D map. The sparse roadmap is a graph structure consisting of nodes and edges, represented as node are randomly sampled in the free space of the exploration area, and the topological connection between a node and its neighboring nodes is the edge ε;

[0044] The nodes generated near the adjacent nodes are generated if the distance is less than the threshold d th , they will be pruned, and only nodes that meet the distance requirements will be added to the sparse route map.

[0045] In some embodiments, the method of refining the trajectory of the ground robot according to the 2D ESDF map to complete the air-ground collaborative autonomous exploration includes the following steps:

[0046] Construct a 2D ESDF map and generate a local feasible path search; specifically, obstacles on the ground and in space as well as dynamic obstacles are mapped to the ESDF map with a corresponding value of -1; narrow passages that the ground robot cannot safely cross and cells with a height difference exceeding a threshold α are mapped to the ESDF map; th The area is also mapped to -1; considering the size of the UGV, the pixels with a value of -1 in the ESDF map are inflated to the 8-neighborhood, and all other pixels are marked as passable; the values ​​of the passable pixels in the ESDF map are calculated step by step; when the paths planned by the global and local explorers pass through the area with a value of -1 in the ESDF map, the hybrid state is used to search for a local feasible path in the ESDF map; a unified trajectory optimizer is applied to refine the feasible path, and control commands are issued to the ground robot for execution;

[0047] Determine the polynomial representation content of the exploration path;

[0048] Based on the polynomial representation of the explored path, the trajectory is optimized based on the safety, dynamic feasibility and smoothness of the final trajectory.

[0049] In some embodiments, the optimizing the trajectory based on the safety, dynamic feasibility and smoothness of the final trajectory according to the polynomial representation of the exploration path comprises the following steps:

[0050] After representing the trajectory segments with polynomials, an unconstrained optimization problem is proposed based on the linear equations in equation MC = b, considering the safety, dynamic feasibility and smoothness of the final trajectory. The time of each trajectory segment is discretized into k i time intervals, where represents the position of the trajectory at the jth interval, given by the equation x i (t) is calculated, and the expression of the optimization problem is:

[0051]

[0052] Among them, f s ,f d and f m They represent the penalty terms for safety, dynamic feasibility and smoothness, respectively, and the corresponding coefficients are λ s , d and λ m , is the time penalty, used to minimize the total execution time of the trajectory;

[0053] Among them, the safety penalty is used to ensure that the UGV maintains a distance from the obstacle. min The minimum distance to avoid collision, the expression of safety penalty is:

[0054]

[0055] Where d(x) represents the distance from point x to the nearest obstacle; d min Represents the minimum distance between the UGV and obstacles;

[0056] The dynamic feasibility penalty is used to ensure that the trajectory segment does not exceed the physical limitations of the ground robot. The expression of the dynamic feasibility penalty is:

[0057] f d =σ v f d,v +σ a f d,a ,

[0058] Among them, σ v represents the speed penalty parameter; f d,v represents the speed dynamics penalty; f d,a represents the acceleration dynamics penalty;

[0059] The expression of smoothness penalty is:

[0060]

[0061] in, represents the first-order derivative of x; M represents that the trajectory is divided into M segments in total.

[0062] Another aspect of the embodiment of the present invention further provides an air-ground collaborative autonomous exploration system in a large-scale unknown environment, including:

[0063] A first module is used to perform autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view;

[0064] The second module is used for autonomous exploration of ground robots based on the attention mechanism;

[0065] The third module is used to refine the trajectory of the ground robot according to the 2D ESDF map and complete the air-ground collaborative autonomous exploration.

[0066] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0067] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0068] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0069] The embodiments of the present invention include at least the following beneficial effects: The present invention provides an air-ground collaborative autonomous exploration method and system in a large-scale unknown environment. The scheme firstly constructs a bird's-eye view and a coverage exploration strategy for the bird's-eye view to conduct autonomous exploration of the aerial robot; then conducts autonomous exploration of the ground robot based on the attention mechanism; finally, refines the trajectory of the ground robot according to the 2DESDF map to complete the air-ground collaborative autonomous exploration. The embodiments of the present invention can quickly cover high-yield areas for efficient exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0071] Figure 2 It is an implementation process diagram of a specific scenario provided by an embodiment of the present invention;

[0072] Figure 3is a schematic diagram of a bird's-eye view BEV generation process provided by an embodiment of the present invention;

[0073] Figure 4 is a schematic diagram of the position of an object segment above the ground plane provided by an embodiment of the present invention;

[0074] Figure 5 is a schematic diagram of dividing the exploration area provided by an embodiment of the present invention;

[0075] Figure 6 is a schematic diagram of a global guidance waypoint provided by an embodiment of the present invention;

[0076] Figure 7 is a schematic diagram of the structure of a sparse roadmap provided by an embodiment of the present invention;

[0077] Figure 8 is a schematic diagram of generating viewpoints provided by an embodiment of the present invention;

[0078] Fig. 9 It is a schematic diagram of constructing an ESDF map provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0080] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0081] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0083] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0084] BEV: BEV stands for Bird's-Eye-View, which is a vivid expression used to describe the perspective obtained when looking down at the ground from a high altitude, just like the view a bird can see when flying in the air and looking down. In map making, the bird's-eye view is a special form of map representation, which simulates the effect of looking down at the earth's surface from directly above, and can clearly show the spatial relationship and layout between geographical elements such as terrain, buildings, and road networks.

[0085] Autonomous exploration: Robot autonomous exploration means that the robot can autonomously perceive the environment, plan paths, and execute actions through built-in intelligent algorithms and sensor systems without direct human guidance or control, in order to explore and understand the unknown environment. This exploration can occur in various fields, including but not limited to scientific research, industrial applications, military use, and daily life. In this process, the robot needs to have a certain degree of intelligence and autonomy, and be able to make adaptive decisions based on changes in the environment to achieve the completion of tasks and the achievement of goals.

[0086] The air-ground collaborative autonomous exploration method and system in a large-scale unknown environment provided by an embodiment of the present invention relate to the field of computer technology. The air-ground collaborative autonomous exploration method in a large-scale unknown environment provided by an embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a drone device, an intelligent robot, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an air-ground collaborative autonomous exploration method in a large-scale unknown environment, etc., but is not limited to the above forms.

[0087] The embodiment of the present invention provides an air-ground collaborative autonomous exploration method in a large-scale unknown environment. Figure 1 , Figure 1 The flowchart of the air-ground collaborative autonomous exploration method in a large-scale unknown environment applied to a server provided in an embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 1 , the method may include the following steps:

[0088] Conducting autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view;

[0089] Autonomous exploration of ground robots based on attention mechanism;

[0090] According to the 2D ESDF map, the trajectory of the ground robot is refined to complete the air-ground collaborative autonomous exploration.

[0091] In some embodiments, the autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view includes the following steps:

[0092] Construct a bird's-eye view based on the projections of different objects on the ground;

[0093] According to the bird's-eye view, an exploration strategy based on the bird's-eye view is constructed to complete the coverage of the area to be explored, thereby realizing autonomous exploration of the aerial robot.

[0094] In some embodiments, constructing a bird's-eye view according to projections of different objects on the ground includes the following steps:

[0095] As the drone moves, a series of images are captured by the onboard camera, and the semantic information in each image is used to generate a bird's-eye view by mapping different objects onto the ground;

[0096] Use Grounded-SAM and RAM to automatically mark and segment the object masks in the image. For the i-th image, after automatic annotation and segmentation, the mask and the corresponding label list can be obtained;

[0097] Each pixel in the image is converted to the camera coordinate system and then mapped to the corresponding coordinates; the conversion process from pixel coordinates to camera coordinate system is:

[0098]

[0099] Where h is the flight altitude of the drone, is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system; K is the intrinsic matrix of the camera, where fx and f y represents the focal length, c x and c y is the optical center; p c Represents the corresponding position in the camera coordinate system; x v Represents the x-axis coordinate of the pixel pv in the image; y v Represents the y-axis coordinate of pixel pv in the image;

[0100] The conversion expression from the camera coordinate system to the world coordinate system is:

[0101]

[0102] in, and Represents the rotation matrix and translation vector from the drone body coordinate system to the world coordinate system; and Represents the rotation matrix and translation vector from the camera coordinate system to the drone body coordinate system;

[0103] The positions of object segments above the ground plane are determined by intersecting their continuous maps; a topology is established between ground segments and other object segments to locate the same object segments in consecutive images;

[0104] The topological structure is combined with the UAV positioning information to adjust the corresponding ground segment by stretching and shrinking.

[0105] In some embodiments, constructing an exploration strategy based on the bird's-eye view according to the bird's-eye view to complete the coverage of the area to be explored, thereby realizing autonomous exploration of the aerial robot, includes the following steps:

[0106] The area not covered by the drone’s field of view is defined as the unknown area, and the covered area is defined as the known area;

[0107] A level-restricted quadtree is constructed to decompose the bird's-eye view into a set of disjoint cells; during the decomposition, any cell at level i is divided into four sub-cells at level i+1 to represent the exploration area; after subdivision, the center of the cell is replaced by the center of the sub-cell;

[0108] A series of candidate waypoints are uniformly sampled along the boundary of the known area, excluding positions already occupied by obstacles;

[0109] Calculate the potential gain of each waypoint while gradually adding or removing candidate waypoints as you move;

[0110] For a quadtree with n cells, each cell Represented as a set of data in Indicates the location of the cell center, Indicates the size of the cell. Indicates the state of the cell;

[0111] Evaluate a given candidate waypoint w based on the number of new cells observed from the waypoint and the distance to the waypoint i The potential gain, potential benefit s(w i ) is calculated as:

[0112]

[0113] Among them, the function is an indicator function that takes the value 1 when the condition in the brackets is met, otherwise it takes the value 0; T s,j represents the Ts value of the jth cell; T p,j represents the Tp value of the jth cell; T o,j Represents the To value of the j-th cell.

[0114] In some embodiments, the autonomous exploration of the ground robot based on the attention mechanism includes the following steps:

[0115] Based on the attention mechanism, the ground robot performs global exploration; specifically, by checking whether the vector formed by two adjacent points is perpendicular to the gradient of a given point, the tolerance ∈ is calculated from the global direction to the navigation point. All closed-loop global wizard sub-tours of; function trilinearInterpolation() uses trilinear interpolation to estimate the gradient, and function nearestPoints() uses incremental kd-tree to quickly search for the two points closest to a given point; connect multiple closed-loop sub-paths to generate a global path function findTwoClosestPointsIndex() determines the two closest points between two closed-loop sub-tours; expand the TSP problem between the two closest points on both sides of each closed-loop sub-tour until the solution stabilizes;

[0116] Based on the local exploration trigger and sparse roadmap, the ground robot conducts local exploration;

[0117] Generate frontier information and viewpoint information; specifically: evaluate the information gain of the viewpoint by calculating the number of boundaries that can be observed directly from the viewpoint; when updating the information gain, only update to the newly generated boundaries whose distance is less than d max When the number of connected boundaries reaches zero, the viewpoint will be deleted, and the boundary will be marked as known only after it has been covered N times;

[0118] A local exploration path is constructed according to the frontier information and the viewpoint information.

[0119] In some embodiments, the local exploration by the ground robot based on the local exploration trigger and the sparse roadmap includes the following steps:

[0120] When the ground robot detects an unknown area that is not represented in the bird’s-eye view, a local exploration trigger is activated;

[0121] Determine whether a region is inside the ground segment by using a ray casting algorithm. Specifically, the number of intersections between the rays cast from the region and the edge of the ground segment is calculated. If the number of intersections is an odd number, it means that the region is inside the polygon. If it is outside, a local exploration trigger is triggered.

[0122] During the local exploration process, a sparse roadmap is used to store and query the 3D map. The sparse roadmap is a graph structure consisting of nodes and edges, represented as node are randomly sampled in the free space of the exploration area, and the topological connection between a node and its neighboring nodes is the edge ε;

[0123] The nodes generated near the adjacent nodes are generated if the distance is less than the threshold d th , they will be pruned, and only nodes that meet the distance requirements will be added to the sparse route map.

[0124] In some embodiments, the method of refining the trajectory of the ground robot according to the 2D ESDF map to complete the air-ground collaborative autonomous exploration includes the following steps:

[0125] Construct a 2D ESDF map and generate a local feasible path search; specifically, obstacles on the ground and in space as well as dynamic obstacles are mapped to the ESDF map with a corresponding value of -1; narrow passages that the ground robot cannot safely cross and cells with a height difference exceeding a threshold α are mapped to the ESDF map; th The area is also mapped to -1; considering the size of the UGV, the pixels with a value of -1 in the ESDF map are inflated to the 8-neighborhood, and all other pixels are marked as passable; the values ​​of the passable pixels in the ESDF map are calculated step by step; when the paths planned by the global and local explorers pass through the area with a value of -1 in the ESDF map, the hybrid state is used to search for a local feasible path in the ESDF map; a unified trajectory optimizer is applied to refine the feasible path, and control commands are issued to the ground robot for execution;

[0126] Determine the polynomial representation content of the exploration path;

[0127] Based on the polynomial representation of the explored path, the trajectory is optimized based on the safety, dynamic feasibility and smoothness of the final trajectory.

[0128] In some embodiments, the optimizing the trajectory based on the safety, dynamic feasibility and smoothness of the final trajectory according to the polynomial representation of the exploration path comprises the following steps:

[0129] After representing the trajectory segments with polynomials, an unconstrained optimization problem is proposed based on the linear equations in the equation MC = b, considering the safety, dynamic feasibility and smoothness of the final trajectory. The time of each trajectory segment is discretized into k i time intervals, where represents the position of the trajectory at the jth interval, given by the equation x i (t) is calculated, and the expression of the optimization problem is:

[0130]

[0131] Among them, f s ,f d and f m They represent the penalty terms for safety, dynamic feasibility and smoothness, respectively, and the corresponding coefficients are λ s , d and λ m , is the time penalty, used to minimize the total execution time of the trajectory;

[0132] Among them, the safety penalty is used to ensure that the UGV maintains a distance from the obstacle. min The minimum distance to avoid collision, the expression of safety penalty is:

[0133]

[0134] Where d(x) represents the distance from point x to the nearest obstacle; d min Represents the minimum distance between the UGV and obstacles;

[0135] The dynamic feasibility penalty is used to ensure that the trajectory segment does not exceed the physical limitations of the ground robot. The expression of the dynamic feasibility penalty is:

[0136] f d =σ v f d,v +σ a f d,a ,

[0137] Among them, σ v represents the speed penalty parameter; f d,v represents the speed dynamics penalty; f d,a represents the acceleration dynamic penalty;

[0138] The expression of smoothness penalty is:

[0139]

[0140] in, represents the first-order derivative of x; M represents that the trajectory is divided into M segments in total.

[0141] The implementation process of the embodiment of the present invention in a specific application scenario is described in detail below with reference to the accompanying drawings of the specification:

[0142] In view of the problems existing in the prior art, the purpose of the present invention is to propose an efficient air-ground collaborative autonomous exploration framework.

[0143] This technology has the following features:

[0144] 1. UAVs quickly explore high-yield areas to produce a bird's-eye view BEV, and ground robots UGVs use BEVs for efficient exploration.

[0145] 2. Based on the constructed BEV, a UAV rapid coverage exploration strategy is proposed, which enables the UAV to quickly cover high-yield areas.

[0146] 3. A hierarchical UGV exploration strategy is introduced, in which the global explorer uses BEV and attention mechanism to build a global exploration path, allowing the UGV to plan the global optimal path and pay more attention to the texture area. The local explorer is triggered to explore the area not included in the BEV.

[0147] refer to Figure 2 , the collaborative air-ground detection system proposed in an embodiment of the present invention integrates the capabilities of drones and UGVs to improve exploration efficiency in large-scale unknown environments. A drone with an onboard camera captures images and uses Grounded-Segment-Anyth ing (Grounded-SAM) and the Recognize Anything Model (RAM) to segment and identify different objects and areas. These segmentation masks are further mapped to create a BEV, providing a comprehensive overview of the area and highlighting key areas for further UGV exploration. In addition, the BEV guides the rapid exploration of the drone by sampling viewpoints within a known area and calculating their potential benefits, thereby guiding the drone to quickly cover the scene.

[0148] During the UGV exploration process, the BEV shared by the drone makes the UGV's exploration more informed. The UGV adopts a hierarchical strategy: a high-level global explorer provides BEV-based global navigation, while a low-level local explorer handles areas not included in the BEV. The high-level global explorer finds the global optimal guidance route under given conditions from the real-time BEV. In order to focus the UGV's exploration on areas with rich environmental features rather than featureless flat ground, an attention mechanism is introduced to optimize the selection of global guidance routes. When encountering an unknown area not covered by the BEV, the local explorer is triggered, allowing the UGV to identify the boundary and generate a local exploration path to visit high-gain viewing points, ensuring comprehensive coverage of the unknown area.

[0149] Although global planning and local exploration planning are efficient, they may have sharp turns that require the UGV to slow down, affecting exploration efficiency. In addition, following the path may result in collisions with dynamic obstacles. Therefore, a trajectory optimizer is used to further refine the trajectory. This process involves analyzing the traversability of the UGV in the 3D environment and mapping the obstacles into a 2D Euclidean signed distance field (ESDF) that moves with the UGV. When an impassable obstacle is encountered, a locally feasible path is searched in the ESDF. Finally, a polynomial-based trajectory optimization problem is proposed to ensure the optimal and safe trajectory of the UGV, considering factors such as safety, dynamic feasibility, and smoothness.

[0150] The core steps of the embodiment of the present invention include:

[0151] Step 1: Autonomous exploration of aerial robots. This step mainly includes the construction process of BEV and proposes a fast coverage exploration algorithm based on BEV to promote the UAV to cover as many high-yield exploration areas as possible in a short time. The specific detailed steps are as follows:

[0152] Step 1: Build BEV:

[0153] The BEV used in the proposed framework only provides coarse prior information to the UAV and UGV, so it is defined as the projection of different objects on the ground. It can be obtained through a direct and fast mapping process, which is different from other high-precision BEVs that require complex data processing and fusion. As the UAV moves, the onboard camera captures a series of images Each image contains rich semantic information, which can be used to generate BEV by mapping different objects onto the ground. Figure 3 The BEV generation process is shown, where Grounded-SAM and RAM are used to automatically label and segment the object masks in the image. For the i-th image, after automatic labeling and segmentation, the mask O = {O 1 ,O 2 ,...,On} and the corresponding label Among them, ground masks are of great help to ground robots.

[0154] Without loss of generality, we assume that the ground is a plane and that the UAV and UGV share the same world coordinate system, with the UAV flying at a fixed altitude. v (x v ,y v ) can be converted to the camera coordinate system and then mapped to its corresponding coordinate p w (x w ,y w ). The conversion process from pixel coordinates to camera coordinate system is as follows:

[0155]

[0156] Where h is the flight altitude of the drone, is the z-component of the translation vector from the body coordinate system to the camera coordinate system. K is the intrinsic matrix of the camera, where f x and f y represents the focal length, c x and c y is the optical center. Then the transformation from the camera coordinate system to the world coordinate system is defined as:

[0157]

[0158] in, and represents the rotation matrix and translation vector from the drone body coordinate system to the world coordinate system. Similarly, and Represents the rotation matrix and translation vector from the camera coordinate system to the drone body coordinate system.

[0159] In the above mapping process, various object segments that do not belong to the ground plane are directly projected onto the ground plane, resulting in projection distortion in their positioning. To alleviate this problem, the positions of object segments above the ground plane are determined by intersecting their continuous mappings, such as Figure 4 The black cells at the intersection of the red and blue cells are shown in the figure. In addition, due to the limited voxel resolution of the image, distant object segments may cause obvious deviations after mapping. Therefore, a topology is established between the ground segments and other object segments, allowing the same object segments to be located in consecutive images. By combining this topology with the drone positioning information, the corresponding ground segments can be adjusted by stretching and shrinking, thereby improving the accuracy of BEV.

[0160] Step 2: Quickly explore coverage:

[0161] In the proposed framework, the UAV provides the UGV with the BEV of the area to be explored. In order to achieve autonomous and rapid coverage of high-yield areas, a BEV-based exploration strategy is proposed. The area not covered by the UAV's field of view FoV is defined as the unknown area, while the covered area is the known area, including free space and inaccessible obstacles. A level-restricted quadtree is used to decompose the BEV into a set of non-intersecting cells. Each cell of the quadtree can be further subdivided into four sub-cells at a higher level, forming cells of different resolutions from coarse to fine. During the decomposition process, any cell at level i can be divided into four sub-cells at level i+1, allowing for flexible representation of the exploration area. Figure 5 Illustrated division of the exploration area, where the red dots represent the centers of cells. After subdivision, the center of a cell is replaced by the centers of its subcells. Due to the limited number of levels, the highest level cells whose FoV covers more than half of the area are marked as known areas.

[0162] In order to achieve autonomous exploration, a series of candidate waypoints are uniformly sampled along the boundary of the known area, excluding positions already occupied by obstacles, expressed as like Figure 5 The UAV calculates the potential gain of each waypoint and gradually adds or removes candidate waypoints as it moves. For a quadtree with n cells Each cell Can be represented as a group in Indicates the location of the cell center, Indicates the size of the cell, Indicates the state of the cell (1 for known, 0 for unknown). Given a candidate waypoint The potential gain of is evaluated based on the number of new cells that can be observed from the waypoint and the distance to the waypoint. i ) can be calculated as follows:

[0163]

[0164] Among them, the function is an indicator function that takes the value 1 when the condition in the brackets is met, otherwise it takes the value 0. The drone selects the one with the largest gain. As the next target point. The drone's exploration process is constantly updated and moves to the candidate waypoint with the maximum potential gain. When the potential gain of any waypoint is below the threshold s th When

[0165] The process of building a BEV relies on the assumption that the ground is flat. When mapping object segments to the quadtree, some obstacles (such as trees and buildings) whose actual height exceeds the ground may occupy a large area in the quadtree due to projection deformation. Therefore, in actual operation, the size of the cells occupied by the mapped object segments is limited. Although this method will introduce some errors to the quadtree, our extensive testing shows that it will not significantly affect the exploration strategy of the UAV. In addition, the embodiments of the present invention do not consider obstacle avoidance during the flight of the UAV, because this is automatically handled by the low-level controller of the UAV.

[0166] Step 2: Autonomous exploration by ground robots. In order to achieve efficient exploration, the present invention designs a hierarchical framework for UGV, including a global resource manager and a local resource manager. The global explorer creates global guidance waypoints using the BEV shared from the UAV, thereby generating a high-level exploration route that avoids redundant exploration. When an unknown area that is not represented in the BEV is detected, the local explorer is activated. In addition, an embodiment of the present invention also introduces an attention mechanism that focuses the exploration of the UGV on areas that require fine reconstruction rather than open or featureless areas. The specific refinement steps are as follows:

[0167] Step 1: Global Exploration:

[0168] As the UGV gradually receives real-time BEV, its exploration becomes more intelligent, relying on prior information about the area to be explored. Although this information may be a bit rough at the edges of the BEV due to segmentation errors introduced by Grounded-SAM with RAM when processing images with limited resolution, it still provides important assistance to the UGV's exploration. In exploration tasks, the areas that need to be reconstructed are those with rich textures, rather than vast ground areas or large open spaces such as squares. It is worth noting that feature-rich areas are often located at the edges of the BEV ground segments. Therefore, an attention mechanism is introduced to focus the UGV's attention on the areas that need detailed reconstruction.

[0169] Each point p in a BEV ground segment is assigned an attention value, which is higher for points closer to the ground segment boundary. This means that the attention value of each point p is inversely proportional to the Euclidean distance d(p) from the point to the nearest boundary of the segment. Therefore, the attention value of a point p is calculated as follows:

[0170]

[0171] Where d(p) can be calculated using the incremental method. Global Guidance Waypoints With the same attention value ψ th The positions of are uniformly sampled, such as Figure 6For quick exploration, all guide waypoints can be Connected end-to-end, a global tour guide is constructed. Although this itinerary can be formulated as a traveling salesman problem (TSP) and solved using algorithms such as x-opt and Lin-Kernighan, the computational cost increases with the size of the TSP.

[0172] The embodiment of the present invention does not solve TSP directly, but first uses the attention value and the internal connectivity of the ground segment to sequentially connect the guide waypoints in each ground segment along the direction perpendicular to the attention gradient. Generate a closed global guided sub-tour. Then, formulate a local TSP between the two nearest points of each pair of guided sub-tours to establish a connection. The TSP gradually extends outward from the two nearest points until the solution stabilizes, eventually forming the final global guided tour, such as Figure 6 As shown in the red closed-loop tour.

[0173] The detailed algorithm is shown in Algorithm 1. Lines 1-15 calculate the gradient from the global navigation point with tolerance ∈ by checking whether the vector formed by two adjacent points is perpendicular to the given point. The function trilinearInterpolation() uses trilinear interpolation to estimate the gradient, while the function nearestPoints() uses an incremental kd-tree to quickly search for the two closest points to a given point. Lines 16-25 connect multiple closed sub-paths to generate a global path. The function findTwoClosestPointsIndex() determines the two closest points between two closed sub-tours. Finally, the algorithm expands the TSP problem between the two closest points on both sides of each closed sub-tour until the solution stabilizes. The TSP problem is solved using the Lin-Kern ighan heuristic to ensure efficient computation.

[0174] The global guide is continuously updated as the UGV moves and receives updates from the BEV. Overridden guide waypoints are removed from the global guidance path and the guide is reconnected accordingly. In addition, there may be multiple disconnected ground segments in the BEV due to obstacles in the drone’s FoV, such as Figure 6 If the UGV detects a connection between these segments, the same strategy described in Lines 16-27 of Algorithm 1 is applied to regenerate a global tour across multiple ground segments.

[0175]

[0176]

[0177] Step 2: Local exploration:

[0178] 1) Local Exploration Trigger

[0179] When the UGV detects an unknown area that is not represented in the BEV, a local exploration trigger is activated. A ray casting algorithm is used to determine if a region is inside a ground segment. This is done by counting the number of intersections between rays cast from the region and the edge of the ground segment. If the number of intersections is an odd number, it means that the region is inside the polygon; if it is outside, the local explorer is triggered.

[0180] 2) Sparse roadmap

[0181] During local exploration, UGV needs to frequently enter the explored area and find an efficient exploration path in the traversable area. Therefore, a sparse roadmap is used instead of directly storing and querying the 3D map to minimize the consumption of memory and computing resources. The sparse roadmap is a graph structure consisting of nodes and edges, represented as like Figure 7 The blue line in the figure shows the node The nodes are randomly sampled in the free space of the exploration area. The topological connection between a node and its neighboring nodes is an edge ε. To ensure the sparsity of the roadmap, nodes generated near neighboring nodes are generated if the distance is less than a threshold d. th , they will be pruned, and only nodes that meet the distance requirements will be added to the sparse route map.

[0182] 3) Frontier Generation and Viewpoint Generation

[0183] Based on FEP, the local explorer allows the UGV to quickly cover the boundary and thus explore unknown areas. The boundary is defined as the intersection between the unknown area and the known area, and is gradually updated as the UGV moves. To cover the boundary, the UGV moves to the specified viewpoints, which are uniformly sampled within the area centered on the UGV with a radius of d v , which is smaller than LiDAR d max The maximum sensing distance. Figure 8 As shown. At position p i The generated viewpoint is marked in blue, and at position p i+1 Newly generated viewpoints are marked in orange.

[0184] For efficient exploration, a local exploration path with high exploration gain should be planned to guide the UGV to continuously visit a series of viewpoints. However, as the scale of the exploration scene increases, the computational cost of calculating the viewpoint information gain will also increase. To reduce this cost, two strategies are adopted. First, similar to FAEL, the viewpoint information gain is evaluated by calculating the number of boundaries that can be observed directly from the viewpoint, rather than calculating the volume information gain. The boundaries are connected to the nearest visible viewpoint, such as Figure 8 As shown by the purple dashed line in , the unconnected viewpoints will be deleted. Secondly, when updating the information gain, it is only updated until the distance of the newly generated boundary is less than d max When the number of connected boundaries reaches zero, the viewpoint is deleted. In addition, since the error of LiDAR measurement becomes larger with the longer sensing distance, the boundary will be marked as known only after it is covered N times.

[0185] 4) Local Exploration Path

[0186] The construction of local exploration paths can be viewed as a TSP problem, which aims to minimize the total moving distance and turns while maximizing the exploration benefit. Given a set in is the current position of the UGV, is the viewpoint to be visited, and the optimization problem of the local exploration path is defined as:

[0187]

[0188] Where G(V i ) represents information gain, P(V i ) represents the penalty term, where c 1 and c 2 is the weight coefficient. The penalty term consists of two parts: the first part is the moving distance penalty term, which aims to minimize the total travel length. Function l(V 0 ,V i ) Calculate the position V from the UGV 0 To viewpoint V i The cumulative distance, where V i-1 and V i The distance between them is searched using A* in the sparse roadmap. The second part is the turning penalty, which aims to reduce the frequent turning of the UGV and select the tour with the least turning among multiple turns of the same length. Function g(V i-1 ,V i ) estimates the turn between two travel segments and is defined as follows:

[0189]

[0190] equation The optimization problem in is solved using a modified 2-opt search. Once the local explorer finishes exploring the region that does not exist in the BEV, it returns to the global exploration to continue exploring.

[0191] Although the paths planned by the global and local explorers are efficient and safe, they may include some sharp turns that require the UGV to slow down for tracking, such as Figure 7In addition, the presence of static and dynamic obstacles makes it impractical to directly follow the path. Therefore, further refinement and optimization are required to generate safe, dynamically feasible, and smooth trajectories.

[0192] Step 3: Trajectory refinement for ground robots. In this step, we address the need to improve and optimize the global and local exploration planning paths to ensure the generation of safe, dynamically feasible, and smooth trajectories for the UGV. In addition, in order to handle static and dynamic obstacles that may affect the UGV's exploration, a 2D ESDF map is introduced for local feasible path search. Finally, the trajectory optimization problem is formulated based on polynomial segmentation, considering multiple penalty terms.

[0193] Step 1: 2D ESDF map and local feasible path search.

[0194] Although the movement of the UGV is guided by the prior information received from the BEV, it needs to avoid collisions with obstacles, especially moving vehicles and pedestrians, during actual exploration. Therefore, local path replanning is necessary. Directly storing and querying 3D maps of large-scale scenes consumes a lot of memory and time, and typical 2D maps cannot contain spatial information, resulting in collisions with obstacles such as low shrubs. To solve this problem, a fixed-size 2D ESDF map is maintained and moves with the UGV. It is generated after analyzing the traversability in 3D space, and is used to search for local feasible paths when part of the path becomes inaccessible, and is used to optimize the trajectory.

[0195] The construction of the ESDF map consists of two steps. First, obstacles on the ground and in space as well as dynamic obstacles are mapped into the ESDF map with a corresponding value of -1. In addition, narrow passages that the UGV cannot safely traverse and the height difference between adjacent cells exceeds a threshold α th The area is also mapped to -1, such as Fig. 9 As shown in the purple cells in the figure, the size of the UGV cannot be ignored. Considering that the size of the UGV cannot be ignored, the pixel with a value of -1 in the ESDF map is inflated to the 8-neighborhood to ensure safety, while all other pixels are marked as passable. Subsequently, the values ​​of the passable pixels in the ESDF map are calculated step by step.

[0196] When the paths planned by the global and local explorers cross an area with a value of -1 in the ESDF map, a hybrid state A* is used to search for a local feasible path in the ESDF map. Subsequently, a unified trajectory optimizer is applied to refine the feasible path and issue control commands to the UGV for execution.

[0197] Step 2: Trajectory representation.

[0198] The m-dimensional paths planned by the global and local explorers can be represented in a unified form as M segments of trajectories, where each segment is a polynomial of order N = 2s - 1, s > 1. The ith segment of the trajectory can be expressed as:

[0199]

[0200] where β(t) = (1, t,... t N ) T is the basis function of the polynomial trajectory, denotes the coefficient matrix of the polynomial. Each segment corresponds to a specific time T = {T 1 , T 2 ,..., T M}. To ensure that all segments except the first and the last are connected at the start and end points, and the trajectory passes through the viewpoints in the planned path, the intermediate conditions Here, f i represents the number of specified intermediate states that must be reached at the intersection of two adjacent line segments, the intermediate conditions in the form of the coefficient matrix can be obtained from the following:

[0201]

[0202] where E i , F i ∈R 2s×2s :

[0203]

[0204] In the intermediate condition constraint (E i F i ), when f i = 0, it ensures that adjacent segments are connected and satisfy differentiability and continuity of order N. When 0 < f i < N, the intermediate conditions also guarantee that the connection points of adjacent segments must pass through the specified states.

[0205] In addition, the initial state X 0 ∈R s×m and the initial boundary condition F 0 = (β(0),..., β s-1 (0)) T ∈R s×2s are introduced to correspond to the current motion state of the UGV, ensuring the continuity and smoothness between the newly optimized trajectory and the ongoing trajectory. The terminal state X M ∈R s×m and the terminal boundary condition E i = (β(T M ),..., β (s-1) (TM )) T ∈R s×2s It is used to constrain the motion state required when the UGV reaches the end of the trajectory. Combined with equation (E i F i ), we can construct the following linear equations:

[0206] MC=b

[0207] Among them here M∈R 2Ms×2Ms and b∈R 2Ms×2m Defined as:

[0208]

[0209] Step 3: Trajectory optimization.

[0210] After representing the trajectory segments with polynomials, an unconstrained optimization problem is proposed based on the linear equations in the equation MC = b, considering the safety, dynamic feasibility and smoothness of the final trajectory. The time of each trajectory segment is discretized into k i time intervals, where represents the position of the trajectory at the jth interval, given by the equation x i (t) is calculated. The optimization problem is defined as follows:

[0211]

[0212] Among them, f s ,f d and f m They represent the penalty terms for safety, dynamic feasibility and smoothness, respectively, and the corresponding coefficients are λ s , d and λ m . is a time penalty that aims to minimize the total execution time of the trajectory.

[0213] 1) Safety penalty: Safety penalty ensures that the UGV maintains a distance from obstacles. min The minimum distance to avoid collision is defined as follows:

[0214]

[0215] Among them, d(x) represents the distance from point x to the nearest obstacle, which is calculated during the construction of the local ESDF map and does not require additional calculation. Γ(x 1 ,x 2 ) is a piecewise function defined as:

[0216]

[0217] 2) Dynamic feasibility penalty: Dynamic feasibility penalty ensures that the trajectory segment does not exceed the physical limitations of the UGV, such as the maximum speed v max , maximum acceleration a max and maximum angular velocity These penalties can be defined as follows:

[0218]

[0219] The final dynamic feasibility penalty is:

[0220] f d =σ v f d,v +σ a f d,a .

[0221] 3) Smoothness penalty: Smoothness penalty is used to make the UGV's movement smoother and reduce unnecessary speed adjustments. The penalty is defined as:

[0222]

[0223] In solving the equation After solving the optimization problem in the above algorithm, a transient optimal trajectory is obtained that takes into account safety, smoothness, and dynamic feasibility. This trajectory is then sent to the lower-level controller of the UGV for trajectory tracking, thereby promoting efficient exploration.

[0224] In summary, this paper proposes a collaborative air-ground robot autonomous exploration framework (CAGE) that leverages the complementary advantages of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) to improve the exploration efficiency and reconstruction details of UGVs in large-scale unknown environments. To the best of our knowledge, this is the first framework that uses UAVs to build a bird's-eye view (BEV) to assist UGV exploration. In this framework, UAVs with onboard cameras use Grounded Segmentation-Anything (Grounded-SAM) and Recognize-Anything Model (RAM) to segment images and build BEVs to identify key areas for ground exploration. To quickly cover high-yield areas, a quadtree-based exploration strategy is proposed to guide the UAV's motion. UGVs equipped with LiDAR perform detailed exploration using a hierarchical exploration mechanism based on the real-time BEV shared by the UAVs. To enhance the reconstruction details, an attention mechanism is proposed to guide the UGV to pay more attention to areas that need to be finely mapped rather than spacious and featureless areas. The effectiveness of the CAGE system is verified by real experiments, showing that the exploration efficiency is significantly improved compared to the state-of-the-art methods.

[0225] The present invention has the following characteristics:

[0226] 1. Drones quickly detect high-yield areas to generate BEVs, and ground robots use BEVs for efficient exploration.

[0227] 2. A UAV rapid coverage exploration strategy based on building BEV enables UAVs to quickly cover high-yield areas.

[0228] 3. A hierarchical UGV exploration strategy is introduced, in which the global explorer uses BEV and attention mechanism to build a global exploration path, allowing the UGV to plan the global optimal path and pay more attention to the texture area. The local explorer is triggered to explore the area not included in the BEV.

[0229] Compared with the prior art, the present invention has the following advantages:

[0230] 1. High efficiency: It reduces the autonomous exploration time and shortens the total moving distance of autonomous exploration.

[0231] 2. High precision: The introduction of the attention mechanism brings higher point cloud density and better detail retention.

[0232] 3. High maneuverability: The drone has high maneuverability and a wide field-of-view camera, which can quickly explore key areas and provide BEV priors for unmanned vehicle exploration.

[0233] 4. Long endurance: The driverless vehicle has a longer endurance and LiDAR equipment, which reconstructs the area in detail under the guidance of the proposed attention mechanism.

[0234] Another aspect of the embodiment of the present invention further provides an air-ground collaborative autonomous exploration system in a large-scale unknown environment, including:

[0235] A first module is used to perform autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view;

[0236] The second module is used for autonomous exploration of ground robots based on the attention mechanism;

[0237] The third module is used to refine the trajectory of the ground robot according to the 2D ESDF map and complete the air-ground collaborative autonomous exploration.

[0238] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0239] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned air-ground collaborative autonomous exploration method in a large-scale unknown environment when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0240] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0241] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned air-ground collaborative autonomous exploration method in a large-scale unknown environment.

[0242] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0243] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0244] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0245] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0246] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0247] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0249] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0250] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0251] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0252] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0253] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0254] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0255] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for air-ground collaborative autonomous exploration in a large-scale unknown environment, characterized in that: The following steps are involved: Conducting autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view; Autonomous exploration of ground robots based on attention mechanism; According to the 2D ESDF map, the trajectory of the ground robot is refined to complete the air-ground collaborative autonomous exploration.

2. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 1 is characterized in that: The autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view includes the following steps: Construct a bird's-eye view based on the projections of different objects on the ground; According to the bird's-eye view, an exploration strategy based on the bird's-eye view is constructed to complete the coverage of the area to be explored, thereby realizing autonomous exploration of the aerial robot.

3. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 2 is characterized in that: The method of constructing a bird's-eye view according to the projections of different objects on the ground includes the following steps: As the drone moves, a series of images are captured by the onboard camera, and the semantic information in each image is used to generate a bird's-eye view by mapping different objects onto the ground; Use Grounded-SAM and RAM to automatically mark and segment the object masks in the image. For the i-th image, after automatic annotation and segmentation, the mask and the corresponding label list can be obtained; Each pixel in the image is converted to the camera coordinate system and then mapped to the corresponding coordinates; the conversion process from pixel coordinates to camera coordinate system is: Where h is the flight altitude of the drone, is the z-axis component of the translation vector from the body coordinate system to the camera coordinate system; K is the intrinsic matrix of the camera, where f x and f y represents the focal length, c x and c y is the optical center; p c Represents the corresponding position in the camera coordinate system; x v Represents the x-axis coordinate of the pixel pv in the image; y v Represents the y-axis coordinate of pixel pv in the image; The conversion expression from the camera coordinate system to the world coordinate system is: in, and Represents the rotation matrix and translation vector from the drone body coordinate system to the world coordinate system; and Represents the rotation matrix and translation vector from the camera coordinate system to the drone body coordinate system; The positions of object segments above the ground plane are determined by intersecting their continuous maps; a topology is established between ground segments and other object segments to locate the same object segments in consecutive images; The topological structure is combined with the UAV positioning information to adjust the corresponding ground segment by stretching and shrinking.

4. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 2 is characterized in that: The step of constructing an exploration strategy based on the bird's-eye view to complete the coverage of the area to be explored, thereby realizing autonomous exploration of the aerial robot, includes the following steps: The area not covered by the drone’s field of view is defined as the unknown area, and the covered area is defined as the known area; A level-restricted quadtree is constructed to decompose the bird's-eye view into a set of disjoint cells; during the decomposition, any cell at level i is divided into four sub-cells at level i+1 to represent the exploration area; after subdivision, the center of the cell is replaced by the center of the sub-cell; A series of candidate waypoints are uniformly sampled along the boundary of the known area, excluding positions already occupied by obstacles; Calculate the potential gain of each waypoint while gradually adding or removing candidate waypoints as you move; For a quadtree with n cells, each cell Represented as a set of data in Indicates the location of the cell center, Indicates the size of the cell. Indicates the state of the cell; Evaluate a given candidate waypoint w based on the number of new cells observed from the waypoint and the distance to the waypoint i The potential gain, potential benefit s(w i ) is calculated as: Among them, the function is an indicator function that takes the value 1 when the condition in the brackets is met, otherwise it takes the value 0; T s,j represents the Ts value of the jth cell; T p,j represents the Tp value of the jth cell; T o,j Represents the To value of the j-th cell.

5. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 1 is characterized in that: The autonomous exploration of the ground robot based on the attention mechanism includes the following steps: Based on the attention mechanism, the ground robot performs global exploration; specifically, by checking whether the vector formed by two adjacent points is perpendicular to the gradient of a given point, the tolerance ∈ is calculated from the global direction to the navigation point. All closed-loop global wizard sub-tours of; function trilinearInterpolation() uses trilinear interpolation to estimate the gradient, and function nearestPoints() uses incremental kd-tree to quickly search for the two points closest to a given point; connect multiple closed-loop sub-paths to generate a global path function findTwoClosestPointsIndex() determines the two closest points between two closed-loop sub-tours; expand the TSP problem between the two closest points on both sides of each closed-loop sub-tour until the solution stabilizes; Based on the local exploration trigger and sparse roadmap, the ground robot conducts local exploration; Generate frontier information and viewpoint information; specifically: evaluate the information gain of the viewpoint by calculating the number of boundaries that can be observed directly from the viewpoint; when updating the information gain, only update to the newly generated boundaries whose distance is less than d max When the number of connected boundaries reaches zero, the viewpoint will be deleted, and the boundary will be marked as known only after it has been covered N times; A local exploration path is constructed according to the frontier information and the viewpoint information.

6. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 5 is characterized in that: The local exploration performed by the ground robot based on the local exploration trigger and the sparse roadmap includes the following steps: When the ground robot detects an unknown area that is not represented in the bird’s-eye view, a local exploration trigger is activated; Determine whether a region is inside the ground segment by using a ray casting algorithm. Specifically, the number of intersections between the rays cast from the region and the edge of the ground segment is calculated. If the number of intersections is an odd number, it means that the region is inside the polygon. If it is outside, a local exploration trigger is triggered. During the local exploration process, a sparse roadmap is used to store and query the 3D map. The sparse roadmap is a graph structure consisting of nodes and edges, represented as node are randomly sampled in the free space of the exploration area, and the topological connection between a node and its neighboring nodes is the edge ε; The nodes generated near the adjacent nodes are generated if the distance is less than the threshold d th , they will be pruned, and only nodes that meet the distance requirements will be added to the sparse route map.

7. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 1 is characterized in that: The method of refining the trajectory of the ground robot according to the 2D ESDF map and completing the air-ground collaborative autonomous exploration includes the following steps: Construct a 2D ESDF map and generate a local feasible path search; specifically, obstacles on the ground and in space as well as dynamic obstacles are mapped to the ESDF map with a corresponding value of -1; narrow passages that the ground robot cannot safely cross and cells with a height difference exceeding a threshold α are mapped to the ESDF map; th The area is also mapped to -1; considering the size of the UGV, the pixels with a value of -1 in the ESDF map are inflated to the 8-neighborhood, and all other pixels are marked as passable; the values ​​of the passable pixels in the ESDF map are calculated step by step; when the paths planned by the global and local explorers pass through the area with a value of -1 in the ESDF map, the hybrid state is used to search for a local feasible path in the ESDF map; a unified trajectory optimizer is applied to refine the feasible path, and control commands are issued to the ground robot for execution; Determine the polynomial representation content of the exploration path; Based on the polynomial representation of the explored path, the trajectory is optimized based on the safety, dynamic feasibility and smoothness of the final trajectory.

8. The method for air-ground collaborative autonomous exploration in a large-scale unknown environment according to claim 7 is characterized in that: The optimization of the trajectory based on the polynomial representation of the exploration path and the safety, dynamic feasibility and smoothness of the final trajectory includes the following steps: After representing the trajectory segments with polynomials, an unconstrained optimization problem is proposed based on the linear equations in the equation MC = b, considering the safety, dynamic feasibility and smoothness of the final trajectory. The time of each trajectory segment is discretized into k i time intervals, where represents the position of the trajectory at the jth interval, given by the equation x i (t) is calculated, and the expression of the optimization problem is: Among them, f s ,f d and f m They represent the penalty terms for safety, dynamic feasibility and smoothness, respectively, and the corresponding coefficients are λ s , d and λ m , is the time penalty, used to minimize the total execution time of the trajectory; Among them, the safety penalty is used to ensure that the UGV maintains a distance from the obstacle. min The minimum distance to avoid collision, the expression of safety penalty is: Where d(x) represents the distance from point x to the nearest obstacle; d min Represents the minimum distance between the UGV and obstacles; The dynamic feasibility penalty is used to ensure that the trajectory segment does not exceed the physical limitations of the ground robot. The expression of the dynamic feasibility penalty is: f d =s v f d,v +s a f d,a , Among them, σ v represents the speed penalty parameter; f d,v represents the speed dynamics penalty; f d,a represents the acceleration dynamics penalty; The expression of smoothness penalty is: in, represents the first-order derivative of x; M represents that the trajectory is divided into M segments in total.

9. An air-ground collaborative autonomous exploration system in a large-scale unknown environment, characterized in that: include: A first module is used to perform autonomous exploration of the aerial robot by constructing a bird's-eye view and a coverage exploration strategy for the bird's-eye view; The second module is used for autonomous exploration of ground robots based on the attention mechanism; The third module is used to refine the trajectory of the ground robot according to the 2D ESDF map and complete the air-ground collaborative autonomous exploration.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images