Air-ground cooperative detection method, system and device for limited environment, and medium

Abstract

The invention provides an air-ground cooperative detection method, system and device for a limited environment, and a medium, and belongs to the technical field of agent application. The method comprises the following steps: controlling a plurality of base station unmanned vehicles to enter a task area for base station deployment, wherein the plurality of base station unmanned vehicles comprise a target base station unmanned vehicle carrying a detection unmanned aerial vehicle; the detection unmanned vehicle is controlled to enter a task area for detection, and point cloud data acquisition is carried out through a laser radar carried by the detection unmanned vehicle; the detection unmanned aerial vehicle is controlled to fly away from the target base station unmanned vehicle and detect the task area, and image data acquisition is carried out through a depth camera carried by the detection unmanned aerial vehicle; in the detection process of the detection unmanned vehicle and the detection unmanned aerial vehicle, positioning information is provided by a plurality of base station unmanned vehicles; and performing two-dimensional mapping according to the point cloud data acquired by the laser radar, and performing three-dimensional mapping according to the image data acquired by the depth camera. According to the invention, the detection precision of the limited environment can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of intelligent agent application technology, and in particular to air-ground cooperative detection methods, systems, devices and media for confined environments. Background Technology

[0002] In existing technologies, a single type of intelligent agent is usually used to conduct detection in confined environments. However, this approach has obvious limitations. For example, ground unmanned vehicles (UGVs) have advantages in two-dimensional mapping and temporary communication and positioning network deployment, but they are not good at detecting targets at high altitudes. Unmanned aerial vehicles (UAVs) excel in three-dimensional environment modeling and target detection, but they are not adaptable to confined environments and are unable to cope with varied terrain and dynamic changes. Summary of the Invention

[0003] The main purpose of this application is to propose a method, system, equipment and medium for air-ground collaborative detection in confined environments, which can effectively improve the detection accuracy in confined environments.

[0004] To achieve the above objectives, one aspect of this application proposes an air-ground cooperative detection method for confined environments, the method comprising: Multiple unmanned vehicles (UAVs) are controlled to enter the mission area for base station deployment; wherein, the multiple UAVs include a target base station UAV, and the target base station UAV is equipped with a detection drone; The system controls an unmanned vehicle (UAV) to enter the mission area for exploration, while simultaneously collecting point cloud data using a lidar system mounted on the UAV; and controls a drone to fly away from the target base station UAV and explore the mission area, while simultaneously collecting image data using a depth camera mounted on the drone; furthermore, the UAV and the drone receive positioning information from the multiple base station UAVs during the exploration process. Two-dimensional mapping is performed based on the point cloud data collected by the lidar, and three-dimensional mapping is performed based on the image data collected by the depth camera.

[0005] Furthermore, the step of controlling multiple base station unmanned vehicles to enter the mission area for base station deployment includes: The first base station unmanned vehicle is deployed at the boundary location after entering the mission area; For each of the base station unmanned vehicles that subsequently enter the task area, the target location of the base station unmanned vehicle to be deployed is determined based on the signal coverage boundary formed by all the base station unmanned vehicles already deployed in the task area, and then the base station unmanned vehicle to be deployed is controlled to move to the target location.

[0006] Furthermore, the control of the unmanned detection vehicle to enter the mission area for detection includes: The task area is rasterized into a map, resulting in several grids; After the unmanned vehicle enters the mission area, a dynamic target point allocation algorithm is used for path planning to detect each grid. At the same time, an artificial potential field method and a preset speed calibration principle are used to control the unmanned vehicle to avoid obstacles.

[0007] Furthermore, controlling the detection drone to fly away from the target base station unmanned vehicle and to detect the mission area includes: After the detection drone flies away from the target base station unmanned vehicle, it uses an improved fast drone exploration algorithm to plan its path in order to detect each grid cell. The improved fast UAV exploration algorithm introduces a detection path optimization mechanism into the original fast UAV exploration algorithm. The detection path optimization mechanism is used to optimize the cost function generated based on the special areas identified by the detection UAV during the detection process, so as to generate a path that guides the detection UAV to prioritize the detection of the special areas. The special areas include narrow areas, near-boundary areas, and areas near obstacles.

[0008] Furthermore, the method also includes: When the detection drone completes its detection mission in the target area, it is controlled to land on the target base station unmanned vehicle according to the ArUco marker on the target base station unmanned vehicle.

[0009] Furthermore, before performing 3D mapping based on the image data acquired by the depth camera, the process also includes: The image data acquired by the depth camera is used to identify targets by a pre-trained target detection model so that the targets can be displayed after 3D mapping. The target detection model is built based on the YOLOv5 model.

[0010] To achieve the above objectives, another aspect of this application proposes an air-ground cooperative detection system for confined environments, the system comprising: The first control module is used to control multiple base station unmanned vehicles to enter the mission area for base station deployment; wherein, the multiple base station unmanned vehicles include a target base station unmanned vehicle, and the target base station unmanned vehicle is equipped with a detection drone; The second control module is used to control the unmanned vehicle to enter the task area for detection, and to collect point cloud data through the lidar on the unmanned vehicle; and to control the unmanned aerial vehicle to fly away from the target base station vehicle and detect the task area, and to collect image data through the depth camera on the unmanned aerial vehicle; and the unmanned vehicle and the unmanned aerial vehicle are provided with positioning information by the multiple base station vehicles during the detection process. The mapping module is used to perform two-dimensional mapping based on the point cloud data collected by the lidar, and to perform three-dimensional mapping based on the image data collected by the depth camera.

[0011] Furthermore, the second control module is also used to control the detection drone to land on the target base station unmanned vehicle according to the ArUco mark provided on the target base station unmanned vehicle when the detection drone completes the detection task of the task area.

[0012] To achieve the above objectives, another aspect of this application proposes an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described air-ground cooperative detection method for confined environments.

[0013] To achieve the above objectives, another aspect of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned air-ground cooperative detection method for confined environments.

[0014] This application includes at least the following beneficial effects: by controlling multiple base station unmanned vehicles to enter the mission area for base station deployment, a network can be built within the mission area, thereby providing reliable positioning information for the detection drones and detection unmanned vehicles, ensuring the normal execution of the detection mission; by controlling the detection unmanned vehicles and detection drones to conduct joint detection in the mission area, and then building maps based on the detection data obtained by the detection drones and detection unmanned vehicles respectively, a more efficient and accurate comprehensive perception and detection of the mission area can be achieved. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an air-ground cooperative detection method for confined environments provided in an embodiment of this application. Figure 2 This is a schematic diagram of the deployment scenario of base station unmanned vehicles, detection unmanned vehicles, and detection drones in the mission area provided in the embodiments of this application; Figure 3 This is a schematic diagram of the composition of an air-ground cooperative detection system for confined environments provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0018] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In existing technologies, a single type of intelligent agent is usually used to conduct detection in confined environments. However, this approach has obvious limitations. For example, ground unmanned vehicles (UGVs) have advantages in two-dimensional mapping and temporary communication and positioning network deployment, but they are not good at detecting targets at high altitudes. Unmanned aerial vehicles (UAVs) excel in three-dimensional environment modeling and target detection, but they are not adaptable to confined environments and are unable to cope with varied terrain and dynamic changes.

[0021] In view of this, embodiments of this application provide a method, system, device, and medium for air-ground cooperative detection in confined environments. This solution proposes deploying multiple base station unmanned vehicles (UAVs) within the mission area, establishing a network to provide reliable positioning information for the detection UAVs and UAVs, ensuring the normal execution of the detection mission. By controlling the UAVs and UAVs to conduct joint detection within the mission area, and then mapping based on the detection data obtained by each UAV and UAV, a more efficient and accurate comprehensive perception and detection of the mission area can be achieved. Specifically, the UAVs, with their flexibility and high-altitude field of view, can quickly detect large areas, while the UAVs can maneuver flexibly in complex terrain to perform close-range detection tasks. By detecting the mission area at different heights and angles, comprehensive information about the mission area environment can be obtained, ensuring all-round perception of the mission area environment. This complementary application can improve the accuracy and robustness of mission area detection.

[0022] This application provides an air-ground cooperative detection method for confined environments, relating to the field of intelligent agent application technology. It can be applied to terminals, servers, or software running on either a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the above method, but is not limited to these forms.

[0023] Please see Figure 1 , Figure 1 This is an optional flowchart illustrating an air-ground cooperative detection method for confined environments provided in an embodiment of this application. The method may include, but is not limited to, the following steps S101 to S103: Step S101: Control multiple base station unmanned vehicles to enter the mission area for base station deployment; wherein, the multiple base station unmanned vehicles include a target base station unmanned vehicle, which is equipped with a detection drone; Step S102: Control the unmanned vehicle to enter the mission area for detection, and collect point cloud data using the lidar on the unmanned vehicle; and control the unmanned aerial vehicle to fly away from the target base station vehicle and detect the mission area, and collect image data using the depth camera on the unmanned aerial vehicle; and the unmanned vehicle and the unmanned aerial vehicle are provided with positioning information by multiple base station vehicles during the detection process. Step S103: Perform two-dimensional mapping based on the point cloud data collected by the lidar, and perform three-dimensional mapping based on the image data collected by the depth camera.

[0024] Steps S101 to S103 as shown in the embodiments of this application involve using unmanned vehicles and drones to conduct environmental detection and mapping of the task area. During this process, multiple base station vehicles deployed in the task area are used for assisted positioning, which can improve the detection accuracy of the task area and increase the efficiency of collaborative operation between unmanned vehicles and drones.

[0025] In step S101 of some embodiments, regarding controlling multiple base station unmanned vehicles to enter the mission area for base station deployment, the corresponding implementation methods may include, but are not limited to, the following: After the first base station unmanned vehicle enters the task area, it is deployed at the boundary of the task area. For each base station unmanned vehicle that enters the task area later, the target position of the base station unmanned vehicle to be deployed is determined based on the signal coverage boundary formed by all base station unmanned vehicles already deployed in the task area, and then the base station unmanned vehicle to be deployed is controlled to move to the target position.

[0026] Specifically, first, determine the intersection points between the two signal coverage boundaries formed by every two deployed base station unmanned vehicles within the task area, and the intersection points between the signal coverage boundary formed by each deployed base station unmanned vehicle within the task area and the boundary of the task area. Merge the signal coverage boundaries at each intersection point that fall within the task area to form a new signal coverage boundary. Starting from the intersection point between one end of this new signal coverage boundary and the boundary of the task area, and ending at the other end of the new signal coverage boundary, begin uniformly segmenting the area into units of length L. The interval between each segment is the target point for the next deployment of base station unmanned vehicles. Here, L is variable, and its size determines the deployment density. To maximize signal coverage, L is set to 1.5 times the signal coverage radius of a single base station.

[0027] As an optional implementation, when the task area is a rectangular area, at least five base station unmanned vehicles are controlled to enter the task area for base station deployment. Four base station unmanned vehicles are deployed at the four corners of the task area, and the base station unmanned vehicle equipped with a detection drone is deployed in the center of the task area. (See [reference needed]). Figure 2 As shown; and in this application, under the WIFI local area network, the detection unmanned vehicle and the unmanned vehicles of each base station can communicate through a self-built LoRaMesh, and the detection drone and the unmanned vehicles of each base station can connect for data transmission through a socket communication protocol based on TCP / UDP.

[0028] In step S102 of some embodiments, regarding the control of the unmanned vehicle to enter the task area for exploration, the corresponding implementation may include, but is not limited to: rasterizing the task area to obtain several grids; controlling the unmanned vehicle to perform path planning using a dynamic target point allocation algorithm after entering the task area, so as to explore each grid, while using an artificial potential field method and a preset speed calibration principle to perform obstacle avoidance motion control on the unmanned vehicle.

[0029] Specifically, the artificial potential field method is an algorithm that simulates gravitational fields in physics. Its basic idea is to design the motion of the agent in its surrounding environment as motion within an abstract artificial gravitational field. The target point exerts an attractive force on the agent, while obstacles exert a repulsive force. Finally, by superimposing the attractive force at the target point and the repulsive force from the obstacles in the environment, the magnitude and direction of the resultant force on the agent's path to the target point can be obtained. Regarding the use of the artificial potential field method for obstacle avoidance motion control of this unmanned aerial vehicle, the corresponding implementation methods include the following: When the unmanned rover needs to move from its current position to a predetermined target position, the gravitational potential field formed between the target position and the current position can be calculated using the following mathematical expression: ; Next, gradient calculation is performed on the gravitational potential field formed between the target location and the current location to obtain the gravitational force acting on the unmanned vehicle. This can be achieved using the following mathematical expression: ; In the formula, This refers to the gravitational potential field formed between the target position and the current position of the unmanned reconnaissance vehicle. Let be the gravitational potential field constant. This is the current location of the unmanned reconnaissance vehicle. This is the target location for the unmanned reconnaissance vehicle. This represents the relative distance between the current position of the unmanned reconnaissance vehicle and the target position. The gravitational force acting on the unmanned probe; When obstacles exist around the unmanned reconnaissance vehicle, the obstacle closest to the vehicle is designated as the target obstacle, and its current position is recorded as... Since the distance between the unmanned vehicle and the target obstacle will affect the trajectory of the unmanned vehicle to varying degrees, a repulsion radius of the repulsive field centered on the unmanned vehicle is set and denoted as . Determine the Euclidean distance between the unmanned rover and the target obstacle and denote it as . ,when When it is determined that the unmanned vehicle will not be affected by the target obstacle, that is, there is no repulsion between the unmanned vehicle and the target obstacle, when When it is determined that the target obstacle falls within the safe radius of the unmanned vehicle, meaning there is a repulsive phenomenon between the unmanned vehicle and the target obstacle, the repulsive potential field formed between the unmanned vehicle and the target obstacle can be calculated using the following mathematical expression: ; Next, gradient calculation is performed on the repulsive potential field formed between the unmanned vehicle and the target obstacle to obtain the repulsive force experienced by the unmanned vehicle. This can be achieved using the following mathematical expression: ; In the formula, This refers to the repulsive potential field formed between the unmanned reconnaissance vehicle and the target obstacle. The position gain constant, This refers to the repulsive force experienced by the unmanned reconnaissance vehicle. The resultant force acting on the unmanned probe can be obtained by vector summing the gravitational and repulsive forces acting on it, which can be expressed by the following mathematical expression: ; In the formula, The resultant force acting on the unmanned exploration vehicle; Finally, the unmanned exploration vehicle is controlled to move along the direction of the resultant force acting on it to avoid obstacles.

[0030] It should be noted that when controlling multiple base station unmanned vehicles to enter the mission area for base station deployment, the artificial potential field method can also be used for obstacle avoidance movement between the multiple base station unmanned vehicles.

[0031] Specifically, regarding the obstacle avoidance motion control of the unmanned exploration vehicle using a preset speed calibration principle, the corresponding implementation methods include the following: The repulsion velocity of the probe vehicle relative to each base station vehicle entering its repulsion range can be calculated using the following mathematical expression: ; Based on the repulsion velocity of the detection drone relative to each base station drone entering its repulsion range, the total repulsion velocity of the detection drone can be calculated using the following mathematical expression: ; In the formula, Let be the repulsive velocity of the detection drone relative to the j-th base station drone that enters its repulsive force range. It is the repulsion strength coefficient and is greater than zero. The repulsion radius of the unmanned vehicle being probed. Let be the distance between the detection drone and the drone at the j-th base station. Let this be the position vector of the unmanned vehicle. Let j be the position vector of the unmanned vehicle at the j-th base station. This represents the unit position vector between the detection unmanned vehicle and the unmanned vehicle at the j-th base station. This represents the total repulsion velocity of the unmanned detection vehicle. The maximum safe braking speed of the probe vehicle relative to the unmanned vehicles at its adjacent base stations can be calculated using the following mathematical expression: ; in: ; Based on the maximum safe braking speed of the probe vehicle relative to each of its adjacent base station vehicles, the viscous velocity of the probe vehicle relative to each of its adjacent base station vehicles can be calculated using the following mathematical expression: ; Based on the viscous velocity of the probe drone relative to each of its adjacent base station drones, the total viscous velocity of the probe drone can be calculated using the following mathematical expression: ; In the formula, This represents the maximum safe braking speed of the unmanned vehicle relative to the kth adjacent base station unmanned vehicle. The preset lower limit of safe braking speed. Let the distance be the distance between the detection drone and the drone at the k-th base station. The optimal braking distance between the detection drone and the drone at the k-th base station is... The first preset acceleration parameter is used to reduce the speed of the unmanned probe to a safe range. Linear gain for velocity calibration Refers to the velocity decay function, Let the viscous velocity of the probe vehicle be relative to the k-th base station vehicle. For speed calibration coefficient, Let the speed of the unmanned vehicle be relative to the unmanned vehicle at the k-th base station. Let this be the velocity vector of the unmanned vehicle being probed. Let the velocity vector of the unmanned vehicle at the k-th base station be . This represents the unit velocity vector between the probe drone and the drone at the k-th base station. The total viscous velocity of the unmanned probe. , and All of these refer to parameters; When the boundary of the mission area is surrounded by a wall, the viscous velocity between the unmanned rover and the wall can be calculated using the following mathematical expression: ; Based on the viscous velocity between the unmanned vehicle and the wall, the repulsive velocity between them can be calculated using the following mathematical expression: ; In the formula, The probe's velocity relative to the wall is determined by the adhesive properties of the unmanned vehicle. The vertical distance between the unmanned vehicle and the wall is the detection distance. To determine the optimal braking distance between the unmanned vehicle and the wall, The second preset acceleration parameter is applied to the unmanned vehicle and the wall. The linear gain refers to the optimal braking curve representing the speed difference between the autonomous vehicle and the wall. The repulsion velocity between the unmanned vehicle and the wall was detected. The speed of the unmanned vehicle relative to the wall. The velocity vector of the wall; Based on the total repulsive velocity, total viscous velocity, and the repulsive velocity between the unmanned vehicle and the wall, the optimal velocity of the unmanned vehicle at the next moment can be calculated using the following mathematical expression: ; In the formula, To determine the optimal speed for the unmanned vehicle in the next moment, The initial velocity of the unmanned probe. The repulsive force between the unmanned vehicle and the obstacle, and Similarly, we can obtain; Finally, the unmanned rover is controlled to move at its optimal speed for obstacle avoidance in the next moment.

[0032] In this application, by introducing a velocity calibration principle based on the artificial potential field method, the unmanned rover can achieve high stability and low collision rate during the detection mission.

[0033] In step S102 of some embodiments, regarding controlling the detection drone to fly away from the target base station unmanned vehicle and to detect the task area, the corresponding implementation may include, but is not limited to: rasterizing the task area to obtain several grids; and after the detection drone flies away from the target base station unmanned vehicle, using an improved fast drone exploration algorithm to perform path planning in order to detect each grid.

[0034] The core of the original fast UAV exploration algorithm (i.e., the FUEL algorithm) is a hierarchical processing flow based on incremental updates, which includes steps such as map update, incremental front detection and clustering, front information structure (FIS) update and connection cost update (i.e. the motion cost between clusters and all other existing clusters). Then, a continuous flight trajectory is generated through global exploration path planning, local viewpoint refinement and dynamic planning.

[0035] Specifically, the improved fast UAV exploration algorithm introduces a detection path optimization mechanism into the original fast UAV exploration algorithm. This mechanism optimizes the cost function generated based on special regions identified by the UAV during the exploration process to generate a path that guides the UAV to prioritize exploration of these special regions, thereby improving coverage detection efficiency. These special regions include narrow areas, near-boundary areas, and areas near obstacles. The cost function can be understood as an overall exploration reward cost adjustment function, which is explained below: For regions near boundaries or obstacles, the first cost adjustment function is defined as follows: ; In the formula, Referring to the first cost adjustment function, it can be understood as the distance cost adjustment function. This is the distance cost weighting coefficient. Let be the Euclidean distance between the center point of cluster c and the boundary of the task region. Let be the Euclidean distance between the center point of cluster c and the nearest obstacle; Introducing the AABB (Axially Aligned Bounding Box) concept from the FUEL algorithm, the spatial extent of the leading cluster is approximated by the minimum axial bounding rectangle, and special geometric regions are identified through bounding box inflation. Based on this, the second cost adjustment function is defined as follows: ; In the formula, Refers to the second cost adjustment function. This represents the weighting coefficient of the second cost adjustment function within the overall exploration reward cost adjustment function. This represents the proportion of the boundary voxels of the task region within the inflated bounding box. A larger value indicates that cluster c is closer to the task region boundary. This represents the proportion of obstacle elements within the inflated bounding box; a larger value indicates that cluster c is closer to the obstacle. The indicator function for the first isolated region takes a value of 1 if there are no other cluster voxels within the inflated bounding box, and a value of 0 otherwise. It is used to identify isolated cluster regions. , and All are the weighting coefficients of each item and It is used to adjust the intensity of excitation for different environmental characteristics; For small areas, the exploration priority is adjusted primarily based on the size of the area occupied, and the third cost adjustment function is defined as follows: ; In the formula, The term "third cost adjustment function" can be understood as the area cost adjustment function. This is the area bonus coefficient. Let be the area of ​​the polygonal region formed by cluster c. This is the indicator function for the second isolated region. If there are multiple clusters in the narrow region and their area is less than the preset area threshold, the value is 1; otherwise, the value is 0. Integrating the first cost adjustment function, the second cost adjustment function, and the third cost adjustment function, we obtain the overall exploration reward cost adjustment function as follows: ; In the formula, Refers to the overall exploration reward cost adjustment function.

[0036] The implementation process of this improved fast UAV exploration algorithm may include, but is not limited to, the following steps: the first step is environmental perception; the second step is boundary detection and clustering update based on the output of the first step; the third step is viewpoint generation based on the output of the second step; the fourth step is updating the cluster connection cost based on the output of the third step; the fifth step is local and global search based on the outputs of the third and fourth steps; the sixth step is adjusting the detection priority of special areas based on the output of the fifth step; the seventh step is updating the viewpoint sequence based on the output of the sixth step; the eighth step is dynamic path planning based on the output of the seventh step; and the ninth step is motion control of the exploration UAV based on the output of the eighth step. Steps two, three, and four are executed within an incremental frontier information structure.

[0037] In this application, based on an improved fast UAV exploration algorithm, a hierarchical planning structure is established based on boundary information to drive the exploration UAV to efficiently explore the mission area considering viewpoint and dynamic constraints. It can quickly identify and locate obstacles and closely integrate obstacle avoidance strategy with path planning. It can not only avoid obstacles in real time, but also simultaneously construct the current environmental information of the mission area.

[0038] In step S102 of some embodiments, the detection unmanned vehicle, the detection drone, and each base station unmanned vehicle are equipped with a UWB module. The UWB module on the detection unmanned vehicle is configured as a first tag, and the UWB module on the detection drone is configured as a second tag. Regarding the fact that the detection unmanned vehicle and the detection drone receive positioning information from multiple base station unmanned vehicles during the detection process, this can be understood as follows: For the positioning of the detection unmanned vehicle during the detection process, the first distance information between the first tag and each base station unmanned vehicle is first measured using the DR mode (distributed ranging mode) built into the UWB module on the detection unmanned vehicle. Then, this first distance information is subjected to one-dimensional Kalman filtering. The initial position information of the first tag is then calculated using a preset iterative algorithm. This initial position information is then subjected to a two-dimensional Kalman filter to obtain the position information of the unmanned vehicle (UAV) in the mission area. For the UAV's positioning during the detection process, the second distance information between the second tag and each base station UAV is first measured using the DR mode of the UWB module on the UAV. This second distance information is then subjected to a one-dimensional Kalman filter and fed into the preset iterative algorithm to obtain the initial position information of the second tag. This initial position information is then subjected to a two-dimensional Kalman filter to obtain the position information of the UAV in the mission area. The preset iterative algorithm can employ nonlinear least squares methods such as the Gauss-Newton method or the Levenberg-Marquardt method. By using two Kalman filtering algorithms for relevant positioning data processing, the tag positioning accuracy can be improved.

[0039] Specifically, when performing two-dimensional Kalman filtering on the solution results output by the preset iterative algorithm, the Kalman filter structure parameters can be optimized and adjusted to improve the accuracy of the final output results. This is achieved by: increasing the process noise covariance matrix Q, preferably by a factor of 10, to make the filter more confident in the measured values; increasing the measurement noise covariance matrix R, preferably by a factor of 10, to make the filter more confident in the previous state estimates; setting initialization to only occur when the filter state estimate is at its initial value, preferably using the tagframe0Callback function to add conditions, thus avoiding resetting the state estimate with each callback and preventing repeated initialization; increasing the maximum number of iterations to 2000 to improve the accuracy of the solution; and reducing the tolerance to 10. -8 This is to make the convergence conditions more stringent.

[0040] In some embodiments, the above-described air-ground cooperative detection method for confined environments further includes: when the detection drone completes its detection task for the target area, controlling the detection drone to return and land on the target base station unmanned vehicle according to the ArUco marker provided on the target base station unmanned vehicle.

[0041] It should be noted that the reconnaissance drone is equipped with a forward-looking camera and a top-down camera. The forward-looking camera is the depth camera mentioned in step S102 above, used to collect overall environmental information of the mission area. The top-down camera is preferably a USB camera, used to identify the ArUco marker on the target base station unmanned vehicle during the landing phase of the reconnaissance drone. The ArUco marker is a square black and white pattern containing a unique binary code. This binary code makes the ArUco marker unique within a preset marker set. The four corners of the ArUco marker are black, which helps to quickly locate the ArUco marker in the image captured by the USB camera.

[0042] Specifically, ArUco marker recognition is performed first, which involves sequentially performing edge detection, contour extraction, marker recognition, and pose estimation on the image captured by the USB camera to obtain the ArUco marker's pose information. Then, the ArUco marker's pose information is processed to calibrate the pose of the detection drone and the target base station unmanned vehicle. This is specifically manifested as follows: The current position and speed of the detection drone are obtained, and the current position and speed of the target base station unmanned vehicle are determined based on the pose information of the ArUco marker. Obtain the predicted future location and speed of the detection drone, and obtain the predicted future location and speed of the target base station unmanned vehicle; Based on the current and predicted future positions of the detection drone and the target base station unmanned vehicle, the position error can be calculated using the following mathematical expression: ; In the formula, For positional error, This is the current location of the reconnaissance drone. This is the predicted future location of the reconnaissance drone. This indicates the current location of the unmanned vehicle at the target base station. The predicted future location of the unmanned vehicle at the target base station; Based on the current and predicted future speeds of the detection drone and the target base station unmanned vehicle, the speed error can be calculated using the following mathematical expression: ; In the formula, For speed error, The current speed of the reconnaissance drone. The predicted speed of the reconnaissance drone in the future, The current speed of the unmanned vehicle at the target base station. The predicted speed of the unmanned vehicle at the target base station; Based on the position error and the velocity error, the velocity adjustment amount for the detection drone and the target base station unmanned vehicle can be calculated using the following mathematical expression: , ; In the formula, This is the speed adjustment amount for the reconnaissance drone. The speed adjustment amount for the unmanned vehicle at the target base station. and All are proportional gains, where, The correction strength used to adjust the position error. The correction strength used to adjust speed error; Based on the speed adjustment of the detection drone and the target base station unmanned vehicle, motion control is performed on the detection drone and the target base station unmanned vehicle.

[0043] In step S103 of some embodiments, before performing 3D mapping based on the image data acquired by the depth camera, the method further includes: performing target recognition on the image data acquired by the depth camera using a pre-trained target detection model, so as to display the target location after 3D mapping. The target detection model is built based on the YOLOv5 model. It can be trained using Torch 2.4.0 and an NVIDIA RTX 4060 GPU, based on a training dataset containing multiple target object images and a preset loss function (including a CIoU loss function and a classification loss function). With a batch size of 24 and an iteration count of 100, the model undergoes multiple rounds of optimization to obtain the target detection model.

[0044] In step S103 of some embodiments, the two-dimensional map of the task area constructed based on the point cloud data collected by the lidar and the three-dimensional map of the task area constructed based on the image data collected by the depth camera are in the same coordinate system. The two-dimensional map can be used to monitor the environmental terrain features of the task area, and the three-dimensional map can be used to construct the high-altitude three-dimensional environment of the task area.

[0045] It should be noted that the above-mentioned air-ground collaborative detection method for restricted environments can be applied to restricted environments such as GPS satellite positioning communication information denial and pre-set positioning base station information denial within the mission area, and can play a role in scenarios such as emergency disaster relief, military reconnaissance, border patrol, and urban security.

[0046] Please see Figure 3 , Figure 3 This is a schematic diagram of an optional component of an air-ground cooperative detection system for confined environments provided in an embodiment of this application. This system can implement the aforementioned air-ground cooperative detection method for confined environments, and may include, but is not limited to, the following: The first control module 201 is used to control multiple base station unmanned vehicles to enter the mission area for base station deployment; wherein, the multiple base station unmanned vehicles include a target base station unmanned vehicle, which is equipped with a detection drone; The second control module 202 is used to control the unmanned vehicle (UAV) to enter the mission area for detection, and simultaneously collect point cloud data using the lidar on the UAV; and to control the unmanned aerial vehicle (UAV) to fly away from the target base station UAV and detect the mission area, while simultaneously collecting image data using the depth camera on the UAV; and to provide positioning information for the UAV and the UAV during the detection process using multiple base station UAVs; and to control the UAV to return and land on the target base station UAV when the UAV completes its detection mission in the mission area, based on the ArUco marker on the target base station UAV. The mapping module 203 is used to perform two-dimensional mapping based on the point cloud data collected by the lidar, and to perform three-dimensional mapping based on the image data collected by the depth camera.

[0047] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those implemented in the above method embodiments, and the beneficial effects achieved by this system embodiment are also the same as those achieved by the above method embodiments.

[0048] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned air-ground cooperative detection method for confined environments. The electronic device may include any smart terminal such as a tablet computer or laptop computer.

[0049] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those implemented by the above method embodiments, and the beneficial effects achieved by the present device embodiments are also the same as those achieved by the above method embodiments.

[0050] Please see Figure 4 , Figure 4 A schematic diagram illustrating the hardware structure of an electronic device according to another embodiment, the electronic device comprising the following: The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 302 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301. Input / output interface 303 is used to implement information input and output; The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304); The processor 301, memory 302, input / output interface 303 and communication interface 304 are connected to each other within the device via bus 305.

[0051] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described air-ground cooperative detection method for confined environments.

[0052] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented by this storage medium embodiment are the same as those implemented by the above method embodiments, and the beneficial effects achieved by this storage medium embodiment are also the same as those achieved by the above method embodiments.

[0053] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0055] Those skilled in the art will understand that the technical solutions illustrated in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than illustrated, or combine certain steps, or different steps. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0056] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0057] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0059] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0060] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0061] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0062] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for air-ground cooperative detection in a restricted environment, characterized in that, The method comprises: controlling a plurality of base station unmanned vehicles to enter a task area for base station deployment; wherein the plurality of base station unmanned vehicles comprises a target base station unmanned vehicle, and the target base station unmanned vehicle is loaded with a detection unmanned aerial vehicle; controlling a detection unmanned vehicle to enter the task area for detection, and simultaneously collecting point cloud data through a laser radar carried by the detection unmanned vehicle; and controlling the detection unmanned aerial vehicle to fly away from the target base station unmanned vehicle and detect the task area, and simultaneously collecting image data through a depth camera carried by the detection unmanned aerial vehicle; and the detection unmanned vehicle and the detection unmanned aerial vehicle are provided with positioning information by the plurality of base station unmanned vehicles during detection; performing two-dimensional mapping according to the point cloud data collected by the laser radar, and performing three-dimensional mapping according to the image data collected by the depth camera.

2. The method of claim 1, wherein, The control of the plurality of base station unmanned vehicles to enter the task area for base station deployment comprises: controlling a first base station unmanned vehicle to be deployed at a boundary position after entering the task area; for each base station unmanned vehicle entering the task area subsequently, determining a target position of the base station unmanned vehicle to be deployed according to a signal coverage boundary formed by all the base station unmanned vehicles already deployed in the task area, and then controlling the base station unmanned vehicle to be deployed to move to the target position.

3. The method of claim 1, wherein, The control of the detection unmanned vehicle to enter the task area for detection comprises: gridizing the task area to obtain a plurality of grids; controlling the detection unmanned vehicle to perform path planning by using a dynamic target point allocation algorithm after entering the task area, so as to detect each grid, and simultaneously performing obstacle avoidance motion control of the detection unmanned vehicle by using an artificial potential field method and a preset speed calibration principle.

4. The method of claim 3, wherein, The control of the detection unmanned aerial vehicle to fly away from the target base station unmanned vehicle and detect the task area comprises: controlling the detection unmanned aerial vehicle to perform path planning by using an improved fast unmanned aerial vehicle exploration algorithm after flying away from the target base station unmanned vehicle, so as to detect each grid; wherein the improved fast unmanned aerial vehicle exploration algorithm introduces a detection path optimization mechanism in the original fast unmanned aerial vehicle exploration algorithm, and the detection path optimization mechanism is used to optimize a cost function generated based on a special area identified by the detection unmanned aerial vehicle during detection, so as to generate a path guiding the detection unmanned aerial vehicle to preferentially detect the special area, the special area including a narrow area, a boundary adjacent area and an obstacle adjacent area.

5. The method of claim 1, wherein, The method further comprises: when the detection unmanned aerial vehicle completes the detection task of the task area, controlling the detection unmanned aerial vehicle to land on the target base station unmanned vehicle according to an ArUco marker provided on the target base station unmanned vehicle.

6. The method of claim 1, wherein, Before performing three-dimensional mapping according to the image data collected by the depth camera, the method further comprises: performing target identification on the image data collected by the depth camera through a pre-trained target detection model, so as to display the target after three-dimensional mapping; wherein the target detection model is constructed based on a YOLOv5 model.

7. An air-ground cooperative detection system for a restricted environment, characterized in that, The system comprises: The first control module is configured to control a plurality of base station unmanned vehicles to enter a task area to deploy base stations, wherein the plurality of base station unmanned vehicles include a target base station unmanned vehicle, and the target base station unmanned vehicle is loaded with a detection unmanned aerial vehicle; The second control module is configured to control a detection unmanned vehicle to enter the task area to perform detection, and simultaneously perform point cloud data collection through a laser radar carried by the detection unmanned vehicle; and control the detection unmanned aerial vehicle to fly away from the target base station unmanned vehicle and perform detection on the task area, and simultaneously perform image data collection through a depth camera carried by the detection unmanned aerial vehicle; and the detection unmanned vehicle and the detection unmanned aerial vehicle are provided with positioning information by the plurality of base station unmanned vehicles during the detection process; The mapping module is configured to perform two-dimensional mapping according to the point cloud data collected by the laser radar, and perform three-dimensional mapping according to the image data collected by the depth camera.

8. The air-ground collaborative detection system for a restricted environment according to claim 7, wherein, The second control module is further configured to, when the detection unmanned aerial vehicle completes the detection task on the task area, control the detection unmanned aerial vehicle to land on the target base station unmanned vehicle according to an ArUco marker arranged on the target base station unmanned vehicle.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the air-ground collaborative detection method for a restricted environment according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the air-ground collaborative detection method for a restricted environment according to any one of claims 1 to 6.

Images