Unmanned aerial vehicle autonomous exploration method and device in dynamic unknown environment

Abstract

The invention discloses an unmanned aerial vehicle autonomous exploration method and device in a dynamic unknown environment. The method comprises the steps that a current three-dimensional voxel map is updated through a map module; performing boundary processing on the current three-dimensional voxel map through a map module to obtain a target boundary area; updating a global route map through a map module; the sensing module processes the current environment image to obtain image key information; determining a target working mode of the unmanned aerial vehicle according to the binary image through a planning module; if the target working mode is a detection mode, controlling the unmanned aerial vehicle to detect a target detection object to obtain target detection information; and if the target working mode is an exploration mode, generating a target exploration path through a planning module according to the target boundary region, the global route map and the dynamic obstacle information. Seamless switching between the detection mode and the exploration mode can be realized, the efficient target searching capability of the unmanned aerial vehicle during autonomous exploration in a dynamic unknown environment is guaranteed, and the method can be widely applied to the technical field of unmanned aerial vehicles.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned aerial vehicles, and particularly to a method and device for autonomous exploration of unmanned aerial vehicles in a dynamic unknown environment. Background Art

[0002] Currently, the technology of autonomous exploration of unmanned aerial vehicles usually focuses on mapping unknown spaces, and realizes path planning and navigation functions by mapping unknown areas into passable areas and obstacle areas; while the target search technology emphasizes more on accurate detection and recognition of targets during the exploration process. However, for the target search problem during the exploration process, the core of traditional exploration methods lies in quickly constructing a map of the unknown environment. If this method is directly combined with a target detection algorithm, it is easy to result in incomplete target search. Moreover, due to the continuous change of the motion state of the unmanned aerial vehicle during exploration, the captured camera images become blurred, and directly performing target detection on the blurred images will lead to insufficient detection accuracy. In addition, the current autonomous target search algorithms attempt to check each obstacle in the environment one by one, which not only consumes a large amount of on-board computing resources, but also affects the exploration efficiency of the unmanned aerial vehicle when exploring large-scale scenes.

[0003] In summary, the technical problems existing in the related art need to be improved. Summary of the Invention

[0004] The embodiments of the present application aim to at least solve one of the technical problems in the related art to some extent. For this reason, the main purpose of the embodiments of the present application is to propose a method and device for autonomous exploration of unmanned aerial vehicles in a dynamic unknown environment, which can realize seamless switching between the detection mode and the exploration mode, and ensure the efficient target search ability of the unmanned aerial vehicle during autonomous exploration in a dynamic unknown environment.

[0005] To achieve the above object, on the one hand, an embodiment of the present application proposes a method for autonomous exploration of unmanned aerial vehicles in a dynamic unknown environment, which is applied to an unmanned aerial vehicle autonomous exploration system. The unmanned aerial vehicle autonomous exploration system includes a map module, a perception module, and a planning module. The method includes the following steps:

[0006] Obtain the current sensor data of the unmanned aerial vehicle; the current sensor data includes the current unmanned aerial vehicle pose information, the current unmanned aerial vehicle point cloud information, and the current environmental image;

[0007] Update the historical three-dimensional voxel map according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information through the map module to obtain the current three-dimensional voxel map;

[0008] Perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain the target boundary region;

[0009] The map module updates the current roadmap according to the target boundary area to obtain a global roadmap;

[0010] The perception module processes the current environmental image to obtain key image information; the key image information includes a binary image and dynamic obstacle information;

[0011] The planning module determines the target working mode of the drone according to the binary image; the target working mode includes a detection mode and an exploration mode;

[0012] If the target working mode is the detection mode, the drone is controlled to detect a target detection object in a dynamic unknown environment to obtain target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode;

[0013] If the target working mode is the exploration mode, the planning module generates a target exploration path for a target exploration point according to the target boundary area, the global roadmap, and the dynamic obstacle information.

[0014] To achieve the above object, on the other hand, an embodiment of the present application proposes a drone autonomous exploration device in a dynamic unknown environment, which is applied to a drone autonomous exploration system. The drone autonomous exploration system includes a map module, a perception module, and a planning module. The device includes the following modules:

[0015] A sensor data acquisition module, configured to acquire the current sensor data of the drone; the current sensor data includes the current drone pose information, the current drone point cloud information, and the current environmental image;

[0016] A three-dimensional voxel map update module, configured to update the historical three-dimensional voxel map according to the current drone pose information and the current drone point cloud information through the map module to obtain the current three-dimensional voxel map;

[0017] A boundary area extraction module, configured to perform heuristic boundary area detection processing on the current three-dimensional voxel map through the map module to obtain a target boundary area;

[0018] A global roadmap update module, configured to update the current roadmap according to the target boundary area through the map module to obtain a global roadmap;

[0019] An image processing module, configured to process the current environmental image through the perception module to obtain key image information; the key image information includes a binary image and dynamic obstacle information;

[0020] The UAV working mode determination module is used to determine the target working mode of the UAV according to the binarized image through the planning module; the target working mode includes a detection mode and an exploration mode;

[0021] The target object detection module is used to control the UAV to detect the target detection object in the dynamic unknown environment to obtain target detection information if the target working mode is the detection mode; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode;

[0022] The target exploration path generation module is used to generate a target exploration path for the target exploration point according to the target boundary area, the global roadmap and the dynamic obstacle information through the planning module if the target working mode is the exploration mode.

[0023] The embodiments of the present application at least include the following beneficial effects: The present application provides a method and device for autonomous exploration of drones in a dynamic unknown environment. The solution includes obtaining the current sensor data of the drone; the current sensor data includes the current pose information of the drone, the current point cloud information of the drone, and the current environmental image; updating the historical three-dimensional voxel map according to the current pose information and the current point cloud information of the drone through the map module to obtain the current three-dimensional voxel map; performing heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain the target boundary region; updating the current roadmap according to the target boundary region through the map module to obtain the global roadmap; performing image processing on the current environmental image through the perception module to obtain the key image information; the key image information includes the binary image and the dynamic obstacle information; determining the target working mode of the drone according to the binary image through the planning module; the target working mode includes the detection mode and the exploration mode; if the target working mode is the detection mode, controlling the drone to detect the target detection object in the dynamic unknown environment to obtain the target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode; if the target working mode is the exploration mode, generating a target exploration path for the target exploration point according to the target boundary region, the global roadmap, and the dynamic obstacle information through the planning module. The embodiments of the present application obtain the target boundary region through heuristic boundary region detection processing, and update the current roadmap based on the target boundary region and the current three-dimensional voxel map to obtain the global roadmap, which improves the generation speed of the global roadmap and reduces the calculation amount; through the planning module, seamless switching between the detection mode and the exploration mode of the drone is realized, which can balance the needs of autonomous exploration and target inspection of the drone, significantly improve the flexibility and efficiency of drone task execution, and at the same time ensure the high-efficiency target search ability of the drone during autonomous exploration in a dynamic unknown environment, making the target search more comprehensive, thereby improving the accuracy of target search and its target detection; according to the target boundary region, the global roadmap, and the dynamic obstacle information, the drone can plan the optimal exploration path through the planning module, reduce the time and energy consumption required for exploration, improve the exploration efficiency, and based on the dynamic obstacle information, reduce the risk of collision between the drone and dynamic obstacles, improving the safety of drone operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of a method for autonomous exploration of a drone in a dynamic unknown environment provided by an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of the framework of an autonomous target exploration system of a drone provided by an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of color segmentation provided by an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of an environmental image provided by an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of a binary image provided by an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of a target detection result provided by an embodiment of the present application;

[0030] Figure 7 is a schematic diagram of a target search process provided by an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a three-dimensional voxel map provided by an embodiment of the present application;

[0032] Figure 9 is a schematic diagram of a multi-layer two-dimensional map provided by an embodiment of the present application;

[0033] Figure 10 is a schematic diagram of a two-dimensional image provided by an embodiment of the present application;

[0034] Figure 11 is a schematic diagram of a boundary detection result provided by an embodiment of the present application;

[0035] Figure 12 is a schematic diagram of the current route map of a drone provided by an embodiment of the present application;

[0036] Figure 13 is a schematic diagram of an incremental route map update provided by an embodiment of the present application;

[0037] Figure 14 is a schematic diagram of a drone route map update provided by an embodiment of the present application;

[0038] Figure 15 is a schematic diagram of the structure of a drone autonomous exploration device in a dynamic unknown environment provided by an embodiment of the present application;

[0039] Figure 16 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0040] In order to make the objectives, technical solutions and advantages of this application more clearly understood, the following further details this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely used to explain this application and are not used to limit this application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of this application. They are merely examples of devices and methods that are consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0041] It can be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, the first information may also be referred to as the second information. Similarly, the second information may also be referred to as the first information. Depending on the context, the words "if", "when" as used herein may be interpreted as "when...", "while...", or "in response to determining".

[0042] The terms "at least one", "a plurality of", "each", "any one", etc. used in this application, at least one includes one, two or more than two, a plurality of includes two or more than two, each refers to each one of the corresponding plurality, and any one refers to any one of the plurality.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] As an example, drones are characterized by their small size and strong mobility, which can help humans explore unknown or dangerous environments. With the rapid development of drone technology and its wide application in fields such as logistics transportation, agricultural monitoring, and disaster relief, autonomous exploration in dynamic unknown environments has become one of the important research directions of drone technology. In these application scenarios, the unknown environment has a high degree of uncertainty. When drones explore unknown spaces, they need to avoid dynamic obstacles in a timely manner to ensure flight safety. For scenarios that require searching for target objects, drones also need to efficiently and accurately identify and locate specific targets during exploration to meet the requirements of complex tasks such as disaster area search and rescue and logistics transportation. However, the distribution of obstacles in dynamic unknown environments is complex and changeable, and the information is updated in real time, which poses a higher challenge to the autonomous exploration ability of drones. Currently, drone autonomous exploration technology usually focuses on mapping the unknown space. By mapping the unknown area into a passable area and an obstacle area, path planning and navigation functions are realized. The target search technology, on the other hand, emphasizes more on accurately detecting and identifying targets during the exploration process. However, for the target search problem during the exploration process, the core of traditional exploration methods is to quickly construct a map of the unknown environment. If this method is directly combined with a target detection algorithm, it is likely to result in incomplete target search. Moreover, due to the continuous change of the motion state of the drone during exploration, the captured camera images become blurred. Directly performing target detection on the blurred images will lead to insufficient detection accuracy. In addition, current autonomous target search algorithms attempt to check each obstacle in the environment one by one. This method not only consumes a large amount of on-board computing resources but also affects the exploration efficiency of the drone when exploring large-scale scenarios.

[0045] In view of this, an unmanned aerial vehicle (UAV) autonomous exploration method and device in a dynamic unknown environment are provided in an embodiment of the present application. The solution includes obtaining current sensor data of the UAV; the current sensor data includes current UAV pose information, current UAV point cloud information, and current environment images; updating a historical three-dimensional voxel map according to the current UAV pose information and the current UAV point cloud information through a map module to obtain a current three-dimensional voxel map; performing heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain a target boundary region; updating the current roadmap according to the target boundary region through the map module to obtain a global roadmap; performing image processing on the current environment images through a perception module to obtain image key information; the image key information includes a binary image and dynamic obstacle information; determining a target working mode of the UAV according to the binary image through a planning module; the target working mode includes a detection mode and an exploration mode; if the target working mode is the detection mode, controlling the UAV to detect a target detection object in the dynamic unknown environment to obtain target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode; if the target working mode is the exploration mode, generating a target exploration path for a target exploration point according to the target boundary region, the global roadmap, and the dynamic obstacle information through the planning module. In the embodiment of the present application, a target boundary region is obtained through heuristic boundary region detection processing, and the current roadmap is updated based on the target boundary region and the current three-dimensional voxel map to obtain a global roadmap, which improves the generation speed of the global roadmap and reduces the calculation amount; the seamless switching between the detection mode and the exploration mode of the UAV is realized through the planning module, which can balance the needs of UAV autonomous exploration and target inspection, significantly improves the flexibility and efficiency of UAV task execution, and at the same time ensures the high-efficiency target search ability of the UAV during autonomous exploration in a dynamic unknown environment, making the target search more comprehensive, and further improving the accuracy of target search and its target detection; according to the target boundary region, the global roadmap, and the dynamic obstacle information, the UAV can plan an optimal exploration path through the planning module, reduce the time and energy consumption required for exploration, improve the exploration efficiency, and moreover, based on the dynamic obstacle information, the risk of collision between the UAV and dynamic obstacles can be reduced, improving the safety of UAV operations.

[0046] The method for autonomous exploration of drones in a dynamic unknown environment provided by the embodiments of the present application relates to the field of drone technology. The method for autonomous exploration of drones in a dynamic unknown environment provided by the embodiments of the present application can be applied to a terminal, a server, or software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or as a server cluster or a distributed system composed of multiple physical servers, or as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method for autonomous exploration of drones in a dynamic unknown environment, etc., but is not limited to the above forms.

[0047] The present application can be used in many general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet-type devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (Personal Computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0048] Please refer to Figure 1 , Figure 1 which is an optional step flowchart of a method for autonomous exploration of drones in a dynamic unknown environment provided by the embodiments of the present application. Figure 1 The method in [[ ]] is applied to an autonomous drone exploration system, and the autonomous drone exploration system includes a map module, a perception module, and a planning module. Figure 1 The method in [[ ]] can include but is not limited to steps S101 to S108.

[0049] Step S101, obtain the current sensor data of the drone; the current sensor data includes the current drone pose information, the current drone point cloud information, and the current environmental image;

[0050] Among them, a method for autonomous exploration of drones in a dynamic unknown environment provided by an embodiment of the present application is applied to a drone autonomous exploration system. Please refer to Figure 2 , Figure 2 which is a schematic diagram of a framework of a drone autonomous target exploration system provided by an embodiment of the present application; as Figure 2 shown, the drone autonomous exploration system includes a map module, a perception module, and a planning module. Through the interaction of the map module, the perception module, and the planning module, the final drone autonomous target exploration trajectory can be obtained.

[0051] Optionally, the current sensor data of the drone includes the current drone pose information, the current drone point cloud information, and the current environmental image. Among them, the current sensor data is the sensor data corresponding to the current position where the drone is located at the current moment.

[0052] Among them, the drone pose information is mainly obtained by a visual inertial odometer composed of camera vision information and an inertial measurement unit. Its content mainly includes the position information and attitude information of the drone, and specifically may include information such as the coordinates of the three axes of the x-axis, y-axis, and z-axis, the yaw angle, the roll angle, and the pitch angle.

[0053] Among them, the drone point cloud information refers to a set of three-dimensional spatial data points obtained from the scene through a depth sensor (an RGB-D camera has a depth sensor). These points represent the distance from the drone to the surface of the object in the scene and are represented by a set of three-dimensional (X, Y, Z) coordinates. The coordinates of each point are deduced from the measurement results of the depth sensor and represent the specific position of the point in three-dimensional space. The point cloud information may specifically include the spatial coordinates of each point, the depth value (the distance between the point and the drone), the sparsity information, and the point cloud size, etc.

[0054] In specific implementation, during system initialization, first, a three-dimensional voxel map of the initial position of the drone can be constructed through the drone pose information and the drone point cloud information of the drone at the initial position, and then the three-dimensional voxel map can be updated in real time using the real-time pose data and point cloud information of the drone.

[0055] For the current environmental image, it refers to the two-dimensional visual information of the surrounding environment captured in real time by a camera or other image acquisition devices carried on the drone during the execution of the task by the drone. The airborne visual sensor used in the embodiment of the present application is an RGB-D camera.

[0056] Step S102: Update the historical three-dimensional voxel map according to the current UAV pose information and the current UAV point cloud information through the map module to obtain the current three-dimensional voxel map.

[0057] In some embodiments, before step S102, it may further include: obtaining the historical sensor data of the UAV; wherein the historical sensor data includes historical UAV pose information and historical UAV point cloud information; constructing a historical three-dimensional voxel map through the map module according to the historical UAV pose information and the historical UAV point cloud information.

[0058] For the historical sensor data, it refers to a series of data collected by the UAV through its carried sensors in the past period of time, which is used to construct the historical three-dimensional voxel map. It should be noted that the historical sensor data is the data obtained in a period of time before the current sensor data; the historical sensor data can also be the sensor data of the initial position of the UAV obtained during initialization. It can be understood that during system initialization, first, a three-dimensional voxel map of the initial position of the UAV can be constructed through the UAV pose information and the UAV point cloud information of the UAV at the initial position, and then the three-dimensional voxel map can be updated in real time using the real-time pose data and point cloud information of the UAV.

[0059] For the three-dimensional voxel map, it is a data structure used to represent the three-dimensional space environment, which divides the space into a series of small and uniform cubic units, and these cubic units are called voxels.

[0060] In a specific implementation, the historical three-dimensional voxel map can be updated according to the current UAV pose information and the current UAV point cloud information through the map module to obtain the current three-dimensional voxel map. As Figure 2 shown, the three-dimensional voxel map can be updated according to the real-time point cloud data and the real-time pose information obtained by the UAV through the map module.

[0061] Step S103: Perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain the target boundary region.

[0062] In some embodiments, step S103 may include: performing voxel map layering processing on the current three-dimensional voxel map through the map module to obtain the current two-dimensional voxel map; performing map imaging processing on the current two-dimensional voxel map through the map module to obtain the current two-dimensional map image; performing boundary region extraction processing on the current two-dimensional map image through the map module to obtain the target boundary region.

[0063] In a specific implementation, the heuristic boundary region detection processing flow consists of four steps: (1) Construction of a 3D (Three-Dimensional) occupancy voxel map (i.e., a three-dimensional voxel map): First, the drone generates a 3D occupancy voxel map using the point cloud data of the camera and the pose data of the drone to accurately represent the spatial distribution of the current environment. (2) Voxel map layering processing: For ease of processing, the 3D voxel map is segmented into multiple two-dimensional mapping maps according to height, and each height layer corresponds to a two-dimensional occupancy map. (3) Map image processing: The two-dimensional occupancy map is converted into a two-dimensional image to facilitate subsequent efficient image processing operations. (4) Boundary region extraction: Image processing is performed on the converted two-dimensional image to extract the boundary region, which is used to represent the key region to be explored, and most of the region is uncharted space. The extracted boundary region includes both free regions and occupied space regions to guide the exploration direction of the drone.

[0064] Specifically, a screening threshold is set to optimize the selection of the boundary region, and only the high-priority regions are retained for score calculation, reducing the number of boundary regions participating in the calculation, thereby reducing the computational burden while ensuring the effectiveness of exploration and improving the real-time performance and efficiency of the exploration algorithm.

[0065] Step S104, the map module updates the current roadmap according to the target boundary region to obtain a global roadmap;

[0066] In a specific implementation, the global roadmap is constructed incrementally based on the global situation. The movement trajectory of the UAV is planned based on the global roadmap. The global roadmap refers to a map that updates the current roadmap in real time until there are no boundary regions in the current roadmap. Specifically, each time the current roadmap is updated, the nodes on the current roadmap are updated starting from the current position of the UAV (when updating the current roadmap after reaching the target point, the UAV stays briefly, and after the update is completed, the UAV continues to navigate to the next target point). The navigation target point is calculated by the exploration algorithm based on the boundary regions extracted by the map module. Each target point that the UAV navigates to is the target boundary region to be explored, and the boundary region is calculated from the 3D occupancy voxel map. Therefore, the specific construction process of the global roadmap is as follows: First, the map module extracts boundary region information from the 3D occupancy voxel map (including free and occupied regions) to generate a two-dimensional image (including boundary regions); then, the UAV updates the current roadmap based on the current position of the UAV and the boundary region (that is, the UAV adds reasonable nodes around the current node according to the boundary region). After the nodes are updated, the exploration algorithm calculates the navigation target point based on the boundary regions extracted by the map module and plans a path to the navigation target point according to the current roadmap. After reaching the navigation target point, the above operations are repeated (starting from calculating the two-dimensional image from the 3D occupancy voxel map) until there are no boundary regions, and the task of constructing the global roadmap is completed. Among them, the 3D occupancy voxel map is updated in real time during the movement of the UAV.

[0067] Step S105: Process the current environmental image through the perception module to obtain key image information; the key image information includes a binary image and dynamic obstacle information;

[0068] In some embodiments, step S105 may include: performing feature segmentation processing on the current environmental image through the perception module to obtain the binary image in the key image information; performing depth detection processing on the current environmental image through the perception module to obtain the dynamic obstacle information in the key image information.

[0069] In some specific embodiments, performing feature segmentation processing on the current environmental image through the perception module to obtain the binary image in the key image information may include: performing color space conversion processing on the current environmental image through the perception module to obtain the environmental image to be segmented; performing feature segmentation processing on the environmental image to be segmented through the perception module according to a preset segmentation threshold to obtain the binary image in the key image information; wherein the binary image includes the target detection object in the dynamic unknown environment.

[0070] In specific implementation, since RGB images are vulnerable to factors such as natural light, occlusion, and shadows, directly performing image processing in the RGB color space may lead to problems such as decreased detection accuracy and feature mismatch. To improve the robustness of color segmentation, the embodiments of the present application convert the RGB color space into the HSV (Hue, Saturation, Value) color space. The HSV color space represents the hue, saturation, and lightness of a color respectively, and can more intuitively describe the color characteristics. Compared with the RGB color space, the HSV color space is more suitable for color comparison and segmentation, and thus is more efficient and accurate when detecting target objects with specific color characteristics. Specifically, first, the perception module performs HSV color space conversion processing on the current environmental image to obtain the environmental image to be segmented; then, in the HSV color space, by setting a segmentation threshold, the target object with a specific color is extracted from the environmental image to be segmented. After the segmentation process, a black-and-white image, that is, a binary image, will be generated. This feature segmentation processing method effectively filters out irrelevant regions while retaining the main features of the target of interest.

[0071] Step S106, determining the target working mode of the drone according to the binary image by the planning module; the target working mode includes a detection mode and an exploration mode;

[0072] In some embodiments, step S106 may include: calculating the white pixel points in the binary image by the planning module to obtain a set of white pixel points; if the number of the set of white pixel points is less than a preset pixel point threshold, determining that the target working mode of the drone is the exploration mode; if the number of the set of white pixel points is greater than or equal to the preset pixel point threshold, determining that the target working mode of the drone is the detection mode.

[0073] Step S107, if the target working mode is the detection mode, controlling the drone to detect the target detection object in the dynamic unknown environment to obtain target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode;

[0074] In specific implementation, when entering the planning module, the mode selector in the planning module determines the suitable working mode of the current drone according to the binary image obtained by the perception module. If in the detection mode, the drone conducts a detailed visual inspection of the region of interest in the environment, and after identifying the target object, prints the position information of the target object and immediately returns to the mode selector; otherwise, the detection mode returns to the mode selector after the set time ends.

[0075] Step S108, if the target working mode is the exploration mode, then the planning module generates a target exploration path for the target exploration point according to the target boundary area, the global roadmap, and the dynamic obstacle information.

[0076] In some embodiments, step S108 may include: if the target working mode is the exploration mode, then the planning module generates a candidate exploration path set according to the target boundary area and the global roadmap; the planning module performs information gain calculation on the candidate exploration path set to obtain an information gain calculation result; the planning module filters out a target information gain path set according to the information gain calculation result and a preset information gain screening threshold, and constructs a candidate global path set of the unmanned aerial vehicle based on the target information gain path set; the planning module selects a target global path from the candidate global path set; if no dynamic obstacle information is detected to obstruct the target global path in the dynamic unknown environment, then the target global path is used as the target exploration path for the target exploration point; if dynamic obstacle information is detected to obstruct the target global path in the dynamic unknown environment, then the planning module adjusts the target global path according to the dynamic replanning method to obtain the target exploration path for the target exploration point.

[0077] In some specific embodiments, if dynamic obstacle information is detected to obstruct the target global path in the dynamic unknown environment, then the planning module adjusts the target global path according to the dynamic replanning method to obtain the target exploration path for the target exploration point, which may include: if dynamic obstacle information is detected to obstruct the target global path in the dynamic unknown environment, then the planning module calculates the density of the dynamic obstacle information to obtain the obstacle density; the planning module calculates the target exploration distance between the unmanned aerial vehicle and the target exploration point; the planning module calculates the trajectory replanning frequency of the target exploration path according to the obstacle density and the target exploration distance, so as to adjust the target global path according to the trajectory replanning frequency to obtain the target exploration path for the target exploration point; the calculation formula of the trajectory replanning frequency is:

[0078]

[0079] where f replan represents the trajectory replanning frequency, β max represents the upper limit of the trajectory replanning frequency, γ base represents the scaling constant for adjusting the trajectory replanning frequency, fac obs represents the obstacle scaling factor, fac dis represents the distance scaling factor.

[0080] In some other embodiments, it may further include: if the target working mode is the exploration mode, the planning module generates a local exploration path for the target exploration point according to the current UAV position information and the current UAV heading information of the UAV; wherein, the local exploration path is used to optimize the target exploration path, and the current UAV heading information is set with a maximum yaw angle limit, and the expression of the maximum yaw angle limit is:

[0081]

[0082] wherein, angle change represents the yaw angle rotation angle when the UAV moves from the current position to the next waypoint, v next represents the direction vector of the UAV moving from the current position to the next waypoint, and v current represents the current direction vector of the UAV.

[0083] Optionally, the trajectory replanning frequency refers to the frequency of path planning. When the UAV explores in the environment, it is necessary to continuously plan the exploration path so that the UAV can complete the exploration task. If the frequency of path planning is too large, the trajectory change of the UAV will be more frequent, resulting in a decrease in exploration efficiency; if the frequency of path planning is too small, the effect of the UAV avoiding dynamic obstacles will be worse. Therefore, the embodiments of the present application provide a formula for the trajectory replanning frequency to flexibly adjust the frequency of path planning to obtain an optimal exploration path. Among them, the specific planned path is calculated by the exploration algorithm.

[0084] Among them, the target exploration path is also called the global exploration path. The global exploration path guides the forward direction of the UAV, but the global exploration path generated based on the exploration algorithm may not be reasonable, so it is necessary to use the local exploration path for adjustment; it can be understood that the local exploration path is used to refine the path in the forward direction guided by the global exploration path. Among them, a global exploration path is composed of several local exploration paths. The local exploration path needs to consider obstacle avoidance. At the same time, when generating the local exploration path, the maximum yaw angle limit is to make the UAV move in the general motion direction guided by the original global exploration path.

[0085] Steps S101 to S108 illustrated in the embodiments of the present application obtain the current sensor data of the drone; the current sensor data includes the current drone pose information, the current drone point cloud information, and the current environmental image; update the historical three-dimensional voxel map according to the current drone pose information and the current drone point cloud information through the map module to obtain the current three-dimensional voxel map; perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain the target boundary region; update the current roadmap according to the target boundary region through the map module to obtain the global roadmap; perform image processing on the current environmental image through the perception module to obtain the key image information; the key image information includes the binary image and the dynamic obstacle information; determine the target working mode of the drone according to the binary image through the planning module; the target working mode includes the detection mode and the exploration mode; if the target working mode is the detection mode, control the drone to detect the target detection object in the dynamic unknown environment to obtain the target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode; if the target working mode is the exploration mode, generate a target exploration path for the target exploration point according to the target boundary region, the global roadmap, and the dynamic obstacle information through the planning module. The embodiments of the present application obtain the target boundary region through heuristic boundary region detection processing, and update the current roadmap based on the target boundary region and the current three-dimensional voxel map to obtain the global roadmap, which improves the generation speed of the global roadmap and reduces the calculation amount; through the planning module, seamless switching between the detection mode and the exploration mode of the drone is realized, which can balance the needs of the drone's autonomous exploration and target inspection, significantly improve the flexibility and efficiency of the drone's task execution, and at the same time ensure the efficient target search ability of the drone during autonomous exploration in the dynamic unknown environment, making the target search more comprehensive, thereby improving the accuracy of target search and its target detection; through the planning module according to the target boundary region, the global roadmap, and the dynamic obstacle information, the drone can plan the optimal exploration path, reduce the time and energy consumption required for exploration, improve the exploration efficiency, and based on the dynamic obstacle information, reduce the risk of the drone colliding with dynamic obstacles, improving the safety of the drone operation.

[0086] To explain the principle of the technical solution of the present invention in detail, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0087] In specific implementation, in the limited three-dimensional space In it, the task of the UAV's autonomous exploration is to build an accurate three-dimensional voxel map of the unknown environment through an on-board vision sensor (RGB-D camera), and detect and identify an unknown number of target objects. The space is represented as a set of cubic voxels, and the occupancy probability P(v) of each voxel V is continuously updated, incrementally mapping the initial unknown space V unk = V into two parts: (free space) and (occupied space). Since the perception of most sensors stops on the surface, some hollow or corner spaces cannot be mapped, and these spaces are represented by V res . In the initial stage of exploration, the entire dynamic environment is unknown, and the UAV only has the initial mapped map M init of the nearby area. According to the initial mapped map M init , the UAV needs to iteratively generate collision-free trajectories to explore the unknown area and search for targets until the entire environment is explored and mapped as M env , at this time M env = (V free ∪ V occ ) \ V res , and the exploration task is completed. Since the target objects are located in the occupied space V occ , the UAV must observe all voxels to ensure the integrity of the search. In a dynamic environment, the UAV must cope with the changes of dynamic obstacles, continuously generate collision-free trajectories, such as fast-moving obstacles like pedestrians, ensure real-time map update and safe obstacle avoidance.

[0088] As Figure 2 shown, the algorithm framework of the UAV autonomous exploration method in the dynamic unknown environment provided by the embodiment of the present application is as Figure 2 shown, that is, the UAV autonomous exploration system includes three core parts: a map module, a perception module, and a planning module. The UAV autonomous exploration system provided by the embodiment of the present application can realize seamless switching between the exploration mode and the detection mode. Among them, the function of the map module is to dynamically update the three-dimensional voxel map using the current pose data and point cloud information of the UAV, and extract a new boundary area based on this three-dimensional voxel map; then, the map module updates the current roadmap according to the current pose information of the UAV and the new boundary area to obtain a global roadmap. After the global roadmap is updated successfully, the planning module can be entered. At the same time, the perception module performs color segmentation on the RGB image, extracts the features of the RGB image to generate a binary image; the perception module also performs dynamic detection on the depth image to obtain dynamic obstacle information, and generates a bounding box based on the dynamic obstacle information (such as Figure 2as shown in the "Dynamic Obstacle" box in the perception module). When entering the planning module, the mode selector in the planning module determines the suitable working mode of the current UAV according to the binary image obtained by the perception module: If in the detection mode, the UAV conducts a detailed visual inspection of the area of interest in the environment, and after identifying the target object, prints the position information of the target object and immediately returns to the mode selector; otherwise, the detection mode returns to the mode selector after the set time ends; If in the exploration mode, the planning module generates a safe and efficient exploration path based on the boundary area extracted from the map module, the updated global roadmap information, and the dynamic obstacle data provided by the perception module; among them, the default mode of the mode selector is the exploration mode, ensuring that the UAV is in the state of exploring the unknown environment most of the time. After generating the exploration path, the planning module updates the local trajectory in real time through the dynamic planner, enabling the UAV to quickly respond to environmental changes. During the path generation process, Path 2 ( Figure 2 sequence number 2 in) represents the candidate global path, Path 1 ( Figure 2 sequence number 1 in) is the selected target global path, and Path 3 ( Figure 2 sequence number 3 in) is the dynamically generated local path. Compared with the traditional exploration method relying on lidar, the embodiment of the present application significantly improves the flexibility and efficiency of task execution while reducing the system weight and hardware cost, providing an efficient and low-cost solution for the autonomous target search of UAVs in a dynamic unknown environment.

[0089] Specifically, as Figure 2 shown, the input data of the UAV autonomous exploration system are camera images, pose data, and point cloud data. First, the map module updates the global roadmap of the UAV according to the current pose data and point cloud information of the UAV; the map module also updates the three-dimensional voxel map according to the current pose data and point cloud information of the UAV, and then extracts a new boundary range based on the three-dimensional voxel map. Then, the perception module processes the camera image to obtain dynamic obstacle information and the binary image after feature extraction. Next, the mode selector of the planning module selects a suitable working mode according to the binary image. Finally, the dynamic planner generates a safe and efficient exploration path and adjusts the trajectory in real time according to the dynamic planner.

[0090] In specific implementation, the specific implementation process of the UAV autonomous exploration method in the dynamic unknown environment provided by the embodiment of the present application includes the following three steps (Step 1 to Step 3):

[0091] Step 1 (Step 1.1 to Step 1.3), target detection based on color segmentation.

[0092] To ensure the accuracy and efficiency of target search during exploration, the embodiments of this application design a target detector based on color segmentation and propose a mechanism for switching between autonomous exploration and target detection modes. By using color segmentation technology to extract semantic information from the environment, the drone can conduct detailed visual inspections only on target objects of interest, rather than comprehensively scanning all occupied spaces in the environment, thereby improving the efficiency of target search.

[0093] In specific implementation, first, the camera is used to preliminarily scan all voxels in the environment to obtain basic information within the global field of view, so as to make up for the incompleteness of traditional methods in target object search. Subsequently, color segmentation technology is used to process the RGB images obtained during the scanning process, and only the color space information related to the target object is extracted. Then, according to the color segmentation results, it is judged whether there are potential target objects. If potential target objects are detected, the corresponding detection viewpoints of the potential target objects are generated, and a detailed perspective inspection is carried out on the target objects. If no potential target objects are detected, the exploration algorithm is continued to be executed. It should be noted that the target search in the embodiments of this application refers to the target detection part, not the autonomous exploration part.

[0094] Step 1.1, color segmentation.

[0095] Please refer to Figures 3 to 6 , Figure 3 which is a schematic diagram of color segmentation provided by the embodiments of this application, Figure 4 which is a schematic diagram of an environmental image provided by the embodiments of this application, Figure 5 which is a schematic diagram of a binary image provided by the embodiments of this application, Figure 6 which is a schematic diagram of target detection results provided by the embodiments of this application; the process of color segmentation is as Figure 3 shown. During the process of the drone exploring an unknown environment, the drone's autonomous exploration system scans the objects in the space by continuously adjusting the drone's perspective to obtain RGB images (as Figure 4 shown).

[0096] However, RGB images are vulnerable to factors such as natural light, occlusion, and shadows. If image processing is directly carried out in the RGB color space, problems such as decreased detection accuracy and feature mismatch may occur. To improve the robustness of color segmentation, the embodiments of this application convert the RGB color space to the HSV (Hue, Saturation, Value) color space. The HSV color space represents the hue, saturation, and lightness of colors respectively, and can more intuitively describe color characteristics. Compared with the RGB color space, the HSV color space is more suitable for color comparison and segmentation, and thus is more efficient and accurate when detecting target objects with specific color characteristics.

[0097] Specifically, in the HSV color space, by setting a segmentation threshold (as shown in Figure 3 ), the target object with a specific color is extracted from the image; after the segmentation process, a black-and-white image, i.e., a binary image (as shown in Figure 5 ), is generated. Among them, the white area shown in Figure 5 represents the set of pixel points that meet the segmentation threshold characteristics. This feature segmentation method effectively filters out the irrelevant areas while retaining the main features of the target of interest.

[0098] Step 1.2, generating a detection viewpoint.

[0099] Calculate the binary image after color segmentation in Step 1.1. If the number of white pixel points pix count is less than the threshold pix thre , it is considered that there is no potential target object in the current field of view of the drone, and no detailed visual inspection is required; if the number of white pixel points pix count is greater than or equal to the threshold pix thre , it is considered that there is a potential target object. Adjust the yaw angle of the drone to make the object to be detected at the center of the field of view, and at the same time conduct a visual inspection on the potential object. The calculation formula for the yaw angular velocity ω yaw of the drone is as follows:

[0100]

[0101] Among them, ω 0 represents the angular velocity adjustment parameter, which needs to be set in advance according to the complexity of the environment; ω init represents the original angular velocity of the drone; img width represents the width of the camera image frame; pix avg represents the abscissa of the centroid of the target pixel points. Among them, the calculation formula for the abscissa pix avg of the target pixel points is as follows:

[0102]

[0103] Among them, n represents the total number of pixel points pix.

[0104] In the specific implementation, by summing the abscissas pix x of each pixel point and then dividing by the total number of pixel points pix n , the centroid coordinate pix avg of the potential target is obtained.

[0105] Specifically, while conducting a detailed visual inspection of potential target objects, in order not to affect the exploration efficiency, the drone needs to always remain in a moving state, that is, continuously exploring the unknown environment. In the embodiments of the present application, the angular velocity is continuously changed to adjust the potential target object to be within the field of view of the drone. During the movement of the drone, multiple-angle scanning and detection of the area where the potential target object is located can be achieved. Multiple-angle scanning can update the three-dimensional voxel map, and multiple-angle detection can improve the detection accuracy of the object.

[0106] Step 1.3, target object detection.

[0107] Since the motion state of the drone in the unknown environment is constantly changing, the camera's field of view also changes rapidly, resulting in blurred images being captured. The target detector designed in the embodiments of the present application can dynamically adjust the yaw angular velocity of the drone according to the color segmentation result to ensure that the camera's field of view always focuses on the target area, thereby achieving accurate recognition of the target object (such as Figure 6 the "spor is ball" mark shown).

[0108] The method for autonomous exploration of a drone in a dynamic unknown environment provided by the embodiments of the present application uses a three-dimensional bounded search space R 3 as the exploration range and discretizes it into voxel grids v i , then constructs an initial environment model according to the occupancy probability P(v i ), and then initializes the unknown area V unk to the entire space V, initializes the free area V free and the occupied area V occ to the empty set, and at the same time retains the restricted area V res . Please refer to Figure 7 , Figure 7 is a schematic diagram of a target search process provided by the embodiments of the present application; the entire target search process (target detection process) is as shown in Figure 7 . Specifically, after the parameter initialization is successful, first construct a 3D occupancy voxel map according to the pose information of the drone and the point cloud data, then perform heuristic boundary area detection, and generate candidate exploration target points based on the exploration method based on the boundary area range. At the same time, the path planning framework updates the roadmap according to the updated 3D occupancy voxel map and plans a global path according to the selected exploration target points. On the other hand, during the process of the drone going to the exploration target point, the mode selector will judge whether to enter the target detection mode according to the binary image (such as Figure 7In the autonomous exploration framework shown in [reference], when the drone detects a target color area in its field of view during exploration, it switches to the target detection mode. In the target detection mode, the color segmentation module calculates the center point of the target area and adjusts the yaw angular velocity of the drone based on the center point to keep the center of the camera's field of view on the target area to be detected. At the same time, the system uses a target detection algorithm (such as the traditional detection algorithm YOLO-v3) to identify the target (as shown in [reference]). Figure 7 In the target search framework shown in [reference], after the identification is completed, the drone switches back to the autonomous exploration mode and continues to scan the unknown space (as shown in [reference]). Figure 7 In the autonomous exploration framework shown in [reference]. When the exploration is not completed, the path planning framework updates the current roadmap M based on the free areas and occupied areas of the 3D occupancy voxel map. env According to the roadmap and the dynamic obstacle information obtained by the camera, a collision-free global path to the exploration target point is generated. In each iteration, the planning algorithm updates the obstacle information in the environment map in real time, checks the validity of the drone's path, and uses the environment perception trajectory prediction method based on the Markov chain to predict the trajectories of dynamic obstacles in the environment. If it detects that the trajectory of a dynamic obstacle conflicts with the current path, the system will trigger trajectory replanning to change the current path of the drone to ensure safety (as shown in the path planning framework in [reference]). Figure 7 In the path planning framework shown in [reference].

[0109] Among them, the exploration algorithm can be HIRE (Heuristic-based Incremental Probabilistic Roadmap for Efficient UAV Exploration in Dynamic Environments), and the exploration target point to be finally visited is selected based on this exploration algorithm. The planning algorithm can adopt (A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera). It should be noted that the exploration algorithm and the planning algorithm can be selected according to the actual situation, and the embodiments of the present application do not limit this.

[0110] Step 2 (Steps 2.1 to 2.2), construction of an incremental roadmap of the heuristic boundary.

[0111] Step 2.1, detection of the heuristic boundary area.

[0112] Please refer to Figures 8 to 11 , Figure 8 which is a schematic diagram of a three-dimensional voxel map provided by an embodiment of the present application.Figure 9 It is a schematic diagram of a multi - layer two - dimensional map provided by an embodiment of the present application. Figure 10 It is a schematic diagram of a two - dimensional image provided by an embodiment of the present application. Figure 11 It is a schematic diagram of a boundary detection result provided by an embodiment of the present application; in a specific implementation, the heuristic boundary region detection processing flow consists of four steps: (1) Construction of a 3D occupancy voxel map (i.e., a three - dimensional voxel map): First, the drone generates a 3D occupancy voxel map (as shown in Figure 8 ) using the point cloud data of the camera and the pose data of the drone to accurately represent the spatial distribution of the current environment. (2) Stratification processing of the voxel map: For ease of processing, the three - dimensional voxel map is divided into multiple two - dimensional mapping maps according to height (as shown in Figure 9 ), and each height layer corresponds to a two - dimensional occupancy map. (3) Map imaging processing: The two - dimensional occupancy map is converted into a two - dimensional image (as shown in Figure 10 ) to facilitate subsequent efficient image processing operations. (4) Boundary region extraction: Image processing is performed on the converted two - dimensional image to extract the boundary region (as shown in Figure 11 ), where, Figure 11 the boundary region shown is marked with a red circle, F represents the center of the circle, and R represents the radius. This boundary region is used to represent the key region to be explored, and most of the region is un - explored space. The extracted boundary region contains both free regions and occupied space regions to guide the exploration direction of the drone.

[0113] Specifically, a screening threshold is set to optimize the selection of the boundary region, and only high - priority regions are retained for score calculation, reducing the number of boundary regions participating in the calculation, thereby reducing the computational burden while ensuring the effectiveness of exploration and improving the real - time performance and efficiency of the exploration algorithm.

[0114] Step 2.2, Incremental roadmap construction.

[0115] After detecting the heuristic boundary region, a probabilistic roadmap is gradually constructed in each planning iteration. The goal is to ensure that the nodes in the roadmap can be evenly distributed in the free space, and the newly sampled nodes can make the roadmap grow towards the un - explored regions. Please refer to Figures 12 to 14 , Figure 12 It is a schematic diagram of the current roadmap of the drone provided by an embodiment of the present application. Figure 13 It is a schematic diagram of incremental roadmap update provided by an embodiment of the present application. Figure 14 It is a schematic diagram of the drone roadmap update provided by an embodiment of the present application; the process of heuristic incremental roadmap construction is as shown in Figures 12 to 14 , Figure 12 represents the current roadmap of the drone, Figure 13 represents the incremental roadmap, Figure 14Represents the updated UAV roadmap; specifically, the UAV takes the current roadmap ( Figure 12 ), the 2D map image ( Figure 10 ), and its own pose as the input of the incremental roadmap, and finally obtains the heuristic incremental roadmap.

[0116] In the specific implementation, initially set the number of heuristic sampling failures N fail and perform sampling until this value exceeds the threshold N max . For the sampling of roadmap nodes, first perform weighted sampling on the heuristic boundary n f in the heuristic boundary region S f , and then find its neighborhood N in the roadmap; then, for each neighbor n i in the neighborhood N, obtain the candidate roadmap node n i by extending the neighbor n f towards the heuristic boundary n by a user-defined distance δ i.next ; subsequently, update the roadmap so that the roadmap can efficiently extend to unknown areas and provide reliable path point references for UAV path planning.

[0117] During the sampling process, through the validity check, ensure that the path points are in the free space and maintain them within an acceptable distance range from their nearest neighbor nodes. This check process can ensure that the roadmap nodes are evenly distributed, thereby improving the overall exploration efficiency and planning accuracy.

[0118] (3) Step 3 (Steps 3.1 to 3.3), the implementation path planner for dynamic scenarios.

[0119] This planner dynamically adjusts the path planning frequency according to the density of current obstacles and the distance between the UAV and the target exploration point. Through this mechanism, the UAV can autonomously explore the unknown environment while timely responding to changes in obstacles in the environment to ensure flight safety. In addition, to prevent the UAV from repeatedly entering the explored area, the embodiment of the present application makes a maximum limit range for the heading deflection of the UAV trajectory to reduce the sharp change in direction during flight. This path planner combines the current position of the UAV, the yaw angle deflection range, and environmental information to generate a safe and efficient exploration trajectory.

[0120] Step 3.1, global path generation.

[0121] As described in step 2, the drone continuously updates the roadmap nodes during exploration. When planning the exploration path each time, for the exploration target points to be visited, the path planner needs to generate several optimal candidate paths based on the roadmap. The path score is calculated based on the cumulative information gain of the nodes obtained by the drone from the current position to the next target point. By integrating the path points with the maximum information gain in the roadmap, the global candidate path of the drone is constructed. The information gain calculation formula for each path node in the generated path is as follows:

[0122]

[0123] where the sensor range function sensor Range returns the on-board camera range of the path point n at the direction angle of the drone, the information gain function IG calculates the number of unknown voxels within the sensor range of the perspective node, v represents a voxel (such as a voxel in a 3D voxel map), and V i represents the unknown area. Then, the candidate paths are scored and screened, and the one with the highest score is selected as the global path of the drone. unk During the execution of the exploration path, if the camera detects that there may be dynamic obstacles temporarily occupying the roadmap nodes on the generated path, resulting in an unsafe flight trajectory, the planner needs to re-plan the path. By re-selecting waypoints and dynamically adjusting the path, the obstacles occupying the nodes are bypassed to ensure that the drone can continue to fly to the exploration area to be detected. If the planner fails to generate a path, it is considered that there is no reasonable path to the target area, that is, it is unreachable based on the current roadmap. In this case, a plan will be made to fly to other exploration areas.

[0124] Step 3.2, Dynamic replanning.

[0125] As described in step 3.1, when it is detected that the obstacles may hinder the current path of the drone, the path needs to be re-planned in a timely manner to ensure flight safety. The core idea of the dynamic replanning method proposed in this embodiment of the application is to dynamically adjust the frequency of trajectory planning according to the obstacle density and the distance to the target, so that the drone can quickly adapt to environmental changes.

[0126] where the dynamic replanning frequency is a function of the number of obstacles and the distance between the drone and the exploration target point. By comprehensively analyzing the current obstacle situation and the relative position to the exploration target point, the drone can autonomously determine the replanning frequency. This method can not only update the flight path in a timely manner to avoid potential collision risks, but also optimize the computing resources for path planning, ensuring that the drone completes the exploration task in an efficient and safe manner.

[0127] Specifically, the dynamic replanning frequency is determined by the number of detected obstacles N

[0128] ​obs and the Euclidean distance D to the target goal The interaction between them determines. The scaling factor calculation formula based on the number of dynamic obstacles and the distance between the UAV and the exploration target point is as follows:

[0129]

[0130] where α obs and α dis are respectively constant parameters preset according to environmental characteristics and UAV mission requirements. The combined influence of these factors constructs a new replanning frequency, and the new replanning frequency is defined as the following expression:

[0131]

[0132] where f replan represents the trajectory replanning frequency, β max represents the upper limit of the trajectory replanning frequency, γ base represents the scaling constant for adjusting the trajectory replanning frequency, fac obs represents the obstacle scaling factor, fac dis represents the distance scaling factor. The calculation formula of the new replanning frequency ensures that the replanning frequency can dynamically adapt to the obstacle density in the environment and the distance between the UAV and the exploration target point. When more obstacles are detected (i.e., N obs increases), the replanning frequency will increase accordingly, enabling the UAV to update the trajectory more frequently and thus quickly respond to potential risks; while as the UAV approaches the exploration target point (i.e., D goal decreases), the replanning frequency will gradually decrease, which can reduce the computational load while ensuring the exploration efficiency.

[0133] The dynamic replanning mechanism designed in the embodiments of this application is applicable to dynamic environments where obstacles may appear in an unpredictable manner. Through this method, the UAV can, on the basis of enhancing the environmental perception ability, achieve efficient and safe flight path planning, thereby improving the success rate of target search and the robustness of the system during the autonomous exploration process.

[0134] Step 3.3, local path planning.

[0135] In a dynamic unknown environment, the ability of the UAV to generate efficient trajectories is the key to achieving efficient exploration. The path planner designed in the embodiments of this application proposes an improved trajectory generation method, which further introduces a maximum yaw angle limit while taking into account the path information gain, thereby ensuring the smoothness and coherence of the trajectory.

[0136] Since the global path executed by the drone is not a straight line but includes multiple waypoints, it is necessary to continuously adjust the local trajectory to reach these waypoints during the flight. Therefore, the local path generation process is as follows: based on the real-time position and heading information of the drone, the subsequent waypoints are dynamically determined. The goal of local path generation is to ensure the continuity of the trajectory while satisfying the constraint conditions of the yaw angle change and avoiding unnecessary energy consumption and delay caused by sharp turns. The embodiments of this application aim to generate a smooth and efficient exploration path, and by restricting the maximum yaw angle change between path points, the naturalness of the trajectory and the stability of the flight are ensured.

[0137] Among them, the calculation expression of the local trajectory adjustment process (i.e., the expression of the maximum yaw angle limit) is as follows:

[0138]

[0139] Among them, angle change represents the yaw angle rotation angle when the drone moves from the current position to the next waypoint, v next represents the direction vector of the drone when moving from the current position to the next waypoint, v current represents the current direction vector of the drone.

[0140] Specifically, when the yaw angle rotation angle exceeds the set threshold, it is considered that the drone will repeat the exploration of the known area, eliminate the selected waypoint, and reselect a new waypoint. By restricting the yaw angle deflection range of the drone, the drone can be prevented from going to the explored area and the repeated path can be reduced. This angle constraint can ensure that the drone follows the desired flight path, reduce unnecessary heading angle deflections, and promote efficient exploration.

[0141] In the field of drone autonomous exploration, the autonomous exploration method requires the drone to be able to identify and distinguish the occupied space and the free space in a dynamic unknown environment and accurately observe the target area. And the embodiments of this application propose a data perception system based on an RGB-D camera, which abandons the dependence on lidar, enabling the system to run in a lightweight manner and is particularly suitable for small drone platforms with limited resources.

[0142] It should be noted that this embodiment only briefly illustrates the general process of the drone autonomous exploration method in a dynamic unknown environment. For the detailed description of each step, reference can be made to the relevant content in the foregoing embodiments, and details are not described herein. It can be understood that the present invention places no restrictions on this.

[0143] In the embodiments of the present application, the current sensor data of the unmanned aerial vehicle (UAV) is obtained; the current sensor data includes the current UAV pose information, the current UAV point cloud information, and the current environmental image; the map module updates the historical three-dimensional voxel map according to the current UAV pose information and the current UAV point cloud information to obtain the current three-dimensional voxel map; the map module performs heuristic boundary region detection processing on the current three-dimensional voxel map to obtain the target boundary region; the map module updates the current roadmap according to the target boundary region to obtain the global roadmap; the perception module performs image processing on the current environmental image to obtain the key image information; the key image information includes the binary image and the dynamic obstacle information; the planning module determines the target working mode of the UAV according to the binary image; the target working mode includes the detection mode and the exploration mode; if the target working mode is the detection mode, the UAV is controlled to detect the target detection object in the dynamic unknown environment to obtain the target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode; if the target working mode is the exploration mode, the planning module generates the target exploration path for the target exploration point according to the target boundary region, the global roadmap, and the dynamic obstacle information. In the embodiments of the present application, the target boundary region is obtained through heuristic boundary region detection processing, and the current roadmap is updated based on the target boundary region and the current three-dimensional voxel map to obtain the global roadmap, which improves the generation speed of the global roadmap and reduces the computational amount; the seamless switching between the detection mode and the exploration mode of the UAV is realized through the planning module, which can balance the needs of the UAV's autonomous exploration and target inspection, significantly improve the flexibility and efficiency of the UAV's task execution, and at the same time ensure the high-efficiency target search ability of the UAV during autonomous exploration in the dynamic unknown environment, making the target search more comprehensive, and further improving the accuracy of target search and its target detection; the UAV can plan the optimal exploration path according to the target boundary region, the global roadmap, and the dynamic obstacle information through the planning module, reduce the time and energy consumption required for exploration, improve the exploration efficiency, and moreover, based on the dynamic obstacle information, the risk of collision between the UAV and the dynamic obstacle can be reduced, and the safety of the UAV operation is improved.

[0144] In summary, the UAV autonomous exploration method in the dynamic unknown environment proposed in the embodiments of the present application has the following advantages:

[0145] (1) The heuristic incremental probabilistic roadmap technology is adopted to construct the global roadmap, which improves the generation speed of the roadmap and reduces the computational amount. Combining with the dynamic replanning mechanism proposed in the embodiments of the present application, the path update frequency is dynamically adjusted according to the obstacle density and the target position, so as to quickly adjust the flight trajectory of the UAV and ensure the navigation safety and path stability of the UAV.

[0146] (2) A multi-task collaborative UAV autonomous exploration system is designed. This system realizes seamless switching between the exploration mode and the detection mode. Moreover, it effectively combines the mechanism of dynamically adjusting the yaw angular velocity with the target detection technology, not only improving the accuracy of target recognition but also ensuring the efficient target search ability of the UAV during autonomous exploration in a dynamic unknown environment.

[0147] (3) By introducing the maximum yaw angle constraint and comprehensively considering the exploration coverage rate, path yaw angle, and flight distance, it aims to minimize the repeated visits of the UAV to the explored areas and improve the exploration efficiency of the UAV. Also, camera sensing technology is used instead of lidar to reduce the hardware cost while meeting the requirements of efficient sensing and navigation in a dynamic environment.

[0148] Please refer to Figure 15 , the embodiment of this application also provides a UAV autonomous exploration device 1500 in a dynamic unknown environment, which is applied to the UAV autonomous exploration system. The UAV autonomous exploration system includes a map module, a sensing module, and a planning module, and can implement the above-mentioned UAV autonomous exploration method in a dynamic unknown environment. This device includes the following modules:

[0149] The sensor data acquisition module 1501 is used to acquire the current sensor data of the UAV; the current sensor data includes the current UAV pose information, the current UAV point cloud information, and the current environment image;

[0150] The 3D voxel map update module 1502 is used to update the historical 3D voxel map according to the current UAV pose information and the current UAV point cloud information through the map module to obtain the current 3D voxel map;

[0151] The boundary area extraction module 1503 is used to perform heuristic boundary area detection processing on the current 3D voxel map through the map module to obtain the target boundary area;

[0152] The global roadmap update module 1504 is used to update the current roadmap according to the target boundary area through the map module to obtain the global roadmap;

[0153] The image processing module 1505 is used to perform image processing on the current environment image through the sensing module to obtain the key image information; the key image information includes the binary image and the dynamic obstacle information;

[0154] The UAV working mode determination module 1506 is used to determine the target working mode of the UAV according to the binary image through the planning module; the target working mode includes the detection mode and the exploration mode;

[0155] The target object detection module 1507 is configured to, if the target working mode is the detection mode, control the drone to detect a target detection object in a dynamic unknown environment, so as to obtain target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode.

[0156] The target exploration path generation module 1508 is configured to, if the target working mode is the exploration mode, generate a target exploration path for a target exploration point through the planning module according to the target boundary area, the global roadmap, and the dynamic obstacle information.

[0157] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0158] The embodiments of the present application further provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned method for autonomous exploration of a drone in a dynamic unknown environment. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0159] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0160] Please refer to Figure 16 , Figure 16 which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0161] The processor 1601 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present application.

[0162] The memory 1602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1602 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1602 and are called by the processor 1601 to execute the autonomous exploration method of the unmanned aerial vehicle in the dynamically unknown environment of this application embodiment;

[0163] The input / output interface 1603 is used to implement information input and output;

[0164] The communication interface 1604 is used to implement communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0165] The bus 1605 transmits information between various components of the device (such as the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604);

[0166] Among them, the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604 achieve communication connections with each other inside the device through the bus 1605.

[0167] The embodiments of this application also provide a computer-readable storage medium. This computer-readable storage medium stores a computer program, and when this computer program is executed by a processor, it implements the above-mentioned autonomous exploration method of the unmanned aerial vehicle in the dynamically unknown environment.

[0168] It can be understood that the content in the above method embodiments is applicable to the embodiments of this storage medium. The functions specifically implemented by the embodiments of this storage medium are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0169] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The method and device for autonomous exploration of an unmanned aerial vehicle (UAV) in a dynamic unknown environment provided by the embodiments of the present application obtain the current sensor data of the UAV; the current sensor data includes the current pose information of the UAV, the current point cloud information of the UAV, and the current environmental image; update the historical three-dimensional voxel map according to the current pose information and the current point cloud information of the UAV through the map module to obtain the current three-dimensional voxel map; perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain the target boundary region; update the current roadmap according to the target boundary region through the map module to obtain the global roadmap; perform image processing on the current environmental image through the perception module to obtain the key image information; the key image information includes the binary image and the dynamic obstacle information; determine the target working mode of the UAV according to the binary image through the planning module; the target working mode includes the detection mode and the exploration mode; if the target working mode is the detection mode, control the UAV to detect the target detection object in the dynamic unknown environment to obtain the target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode; if the target working mode is the exploration mode, generate the target exploration path for the target exploration point according to the target boundary region, the global roadmap, and the dynamic obstacle information through the planning module. The embodiments of the present application obtain the target boundary region through heuristic boundary region detection processing, and update the current roadmap based on the target boundary region and the current three-dimensional voxel map to obtain the global roadmap, which improves the generation speed of the global roadmap and reduces the computational amount; through the planning module, seamless switching between the detection mode and the exploration mode of the UAV is realized, which can balance the requirements of autonomous exploration and target inspection of the UAV, significantly improve the flexibility and efficiency of UAV mission execution, and at the same time ensure the efficient target search ability of the UAV during autonomous exploration in a dynamic unknown environment, making the target search more comprehensive, thereby improving the accuracy of target search and its target detection; through the planning module according to the target boundary region, the global roadmap, and the dynamic obstacle information, the UAV can plan the optimal exploration path, reduce the time and energy consumption required for exploration, improve the exploration efficiency, and moreover, based on the dynamic obstacle information, the risk of collision between the UAV and dynamic obstacles can be reduced, improving the safety of UAV operation.

[0171] The embodiments described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0172] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown, or combine certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware and their appropriate combinations.

[0175] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A method for autonomous exploration of unmanned aerial vehicles in a dynamic unknown environment, characterized in that: Applied to an unmanned aerial vehicle autonomous exploration system, the unmanned aerial vehicle autonomous exploration system includes a map module, a perception module and a planning module, and the method includes the following steps: Acquire current sensor data of the drone; the current sensor data includes current drone position information, current drone point cloud information and current environment image; The map module updates the historical three-dimensional voxel map according to the current UAV posture information and the current UAV point cloud information to obtain a current three-dimensional voxel map; Performing heuristic boundary area detection processing on the current three-dimensional voxel map by the map module to obtain a target boundary area; The map module updates the current route map according to the target boundary area to obtain a global route map; Performing image processing on the current environment image through the perception module to obtain image key information; the image key information includes a binary image and dynamic obstacle information; Determining the target working mode of the UAV according to the binary image by the planning module; the target working mode includes a detection mode and an exploration mode; If the target working mode is the detection mode, the drone is controlled to detect the target detection object in the dynamic unknown environment to obtain target detection information; wherein, after the target detection object is detected, the detection mode is automatically switched to the exploration mode; If the target working mode is the exploration mode, a target exploration path for a target exploration point is generated by the planning module according to the target boundary area, the global roadmap and the dynamic obstacle information.

2. The method according to claim 1, characterized in that Before the map module updates the historical three-dimensional voxel map according to the current drone pose information and the current drone point cloud information to obtain the current three-dimensional voxel map, the method further includes: Acquire historical sensor data of the drone; wherein the historical sensor data includes historical drone posture information and historical drone point cloud information; The historical three-dimensional voxel map is constructed according to the historical drone posture information and the historical drone point cloud information through the map module.

3. The method according to claim 1, characterized in that The step of performing heuristic boundary region detection processing on the current three-dimensional voxel map by the map module to obtain a target boundary region includes: Performing voxel map layering processing on the current three-dimensional voxel map by the map module to obtain a current two-dimensional voxel map; Performing map imaging processing on the current two-dimensional voxel map by the map module to obtain a current two-dimensional map image; The map module performs boundary area extraction processing on the current two-dimensional map image to obtain the target boundary area.

4. The method according to claim 1, characterized in that: The performing image processing on the current environment image by the perception module to obtain key image information includes: Performing feature segmentation processing on the current environment image through the perception module to obtain the binary image in the image key information; The current environment image is subjected to depth detection processing by the perception module to obtain the dynamic obstacle information in the image key information.

5. The method according to claim 4, characterized in that The performing feature segmentation processing on the current environment image by the perception module to obtain the binary image in the image key information includes: Performing color space conversion processing on the current environment image through the perception module to obtain an environment image to be segmented; The perception module performs feature segmentation processing on the environment image to be segmented according to a preset segmentation threshold to obtain the binary image in the image key information; wherein the binary image contains the target detection object in a dynamic unknown environment.

6. The method according to claim 1, characterized in that Determining the target working mode of the UAV according to the binary image by the planning module includes: The white pixel points in the binary image are calculated by the planning module to obtain a set of white pixel points; If the number of the white pixel set is less than a preset pixel threshold, determining that the target working mode of the drone is the exploration mode; If the number of the white pixel point set is greater than or equal to a preset pixel point threshold, it is determined that the target working mode of the drone is the detection mode.

7. The method according to claim 1, characterized in that If the target working mode is the exploration mode, generating a target exploration path for the target exploration point according to the target boundary area, the global roadmap and the dynamic obstacle information through the planning module, including: If the target working mode is the exploration mode, generating a candidate exploration path set according to the target boundary area and the global roadmap through the planning module; Performing information gain calculation on the candidate exploration path set by the planning module to obtain an information gain calculation result; The planning module selects a target information gain path set according to the information gain calculation result and a preset information gain screening threshold, and constructs a candidate global path set of the UAV based on the target information gain path set; Selecting a target global path from the candidate global path set by the planning module; If the dynamic obstacle information is not detected to have a path obstruction to the target global path in the dynamic unknown environment, the target global path is used as the target exploration path for the target exploration point; If it is detected in a dynamic unknown environment that the dynamic obstacle information has a path obstruction to the target global path, the target global path is adjusted by the planning module according to a dynamic replanning method to obtain the target exploration path for the target exploration point.

8. The method according to claim 7, characterized in that If it is detected in a dynamic unknown environment that the dynamic obstacle information has a path obstruction to the target global path, the target global path is adjusted by the planning module according to a dynamic replanning method to obtain the target exploration path for the target exploration point, including: If it is detected in a dynamic unknown environment that the dynamic obstacle information has a path obstruction to the target global path, the planning module performs density calculation on the dynamic obstacle information to obtain an obstacle density; Calculating the target exploration distance between the drone and the target exploration point by the planning module; The trajectory replanning frequency of the target exploration path is calculated by the planning module according to the obstacle density and the target exploration distance, so as to adjust the target global path according to the trajectory replanning frequency to obtain the target exploration path for the target exploration point; the calculation formula of the trajectory replanning frequency is: Among them, f replan represents the trajectory replanning frequency, β max represents the upper limit of trajectory replanning frequency, γ base represents the scaling constant for adjusting the trajectory replanning frequency, fac obs represents the obstacle scaling factor, fac dis Represents the distance scaling factor.

9. The method according to claim 1, characterized in that: The method further comprises: If the target working mode is the exploration mode, a local exploration path for the target exploration point is generated by the planning module according to the current drone position information and the current drone heading information of the drone; wherein the local exploration path is used to optimize the target exploration path, and the current drone heading information is set with a maximum yaw angle limit, and the expression of the maximum yaw angle limit is: Among them, angle change Indicates the yaw rotation angle of the drone from the current position to the next waypoint, v next Indicates the direction vector from the current position of the drone to the next waypoint, v current Represents the current direction vector of the drone.

10. An autonomous exploration device for unmanned aerial vehicles in a dynamic unknown environment, characterized in that: Applied to the autonomous exploration system of unmanned aerial vehicles, the autonomous exploration system of unmanned aerial vehicles includes a map module, a perception module and a planning module, and the device includes the following modules: A sensor data acquisition module is used to acquire the current sensor data of the drone; the current sensor data includes the current drone posture information, the current drone point cloud information and the current environment image; A three-dimensional voxel map updating module, used to update the historical three-dimensional voxel map according to the current UAV posture information and the current UAV point cloud information through the map module to obtain a current three-dimensional voxel map; A boundary region extraction module, used for performing heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain a target boundary region; A global route map updating module, used to update the current route map according to the target boundary area through the map module to obtain a global route map; An image processing module, used to perform image processing on the current environment image through the perception module to obtain key image information; the key image information includes a binary image and dynamic obstacle information; A UAV working mode determination module, used to determine the target working mode of the UAV according to the binary image through the planning module; the target working mode includes a detection mode and an exploration mode; A target object detection module, for controlling the UAV to detect a target detection object in a dynamic unknown environment to obtain target detection information if the target working mode is the detection mode; wherein the detection mode is automatically switched to the exploration mode after the target detection object is detected; A target exploration path generation module is used to generate a target exploration path for a target exploration point through the planning module according to the target boundary area, the global roadmap and the dynamic obstacle information if the target working mode is the exploration mode.

Images