An unmanned aerial vehicle autonomous exploration method and device in a dynamic unknown environment

Abstract

The application discloses a kind of unmanned aerial vehicle autonomous exploration method and device under dynamic unknown environment, method includes: by map module updates current three-dimensional voxel map;By map module, the target boundary region is obtained by boundary processing to current three-dimensional voxel map;By map module, global route map is updated;By perception module, image key information is obtained by current environmental image processing;By planning module, the target working mode of unmanned aerial vehicle is determined according to binary image;If target working mode is detection mode, then control unmanned aerial vehicle detects target detection object and obtains target detection information;If target working mode is exploration mode, then target exploration path is generated by planning module according to target boundary region, global route map and dynamic obstacle information.This application can realize seamless switching of detection mode and exploration mode, guarantee the efficient target search capability of unmanned aerial vehicle when autonomous exploration in dynamic unknown environment, and can be widely applied in unmanned aerial vehicle technical field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicles, and in particular to an unmanned aerial vehicle autonomous exploration method and device in a dynamic unknown environment. BACKGROUND

[0002] Currently, the autonomous exploration technology of unmanned aerial vehicles usually takes mapping of unknown space as the core, realizes path planning and navigation functions by mapping the unknown area into passable areas and obstacle areas, and the target search technology emphasizes more on accurate detection and identification of the target in the exploration process. However, for the target search problem in the exploration process, the core of the traditional exploration method is to quickly build a map of the unknown environment. If this method is directly combined with a target detection algorithm, it is easy to cause the target search to be incomplete, and because the motion state of the unmanned aerial vehicle changes constantly during exploration, the camera images collected are blurred, and directly detecting the blurred images will result in insufficient detection accuracy. In addition, the current autonomous target search algorithm attempts to check all obstacles in the environment one by one, which not only consumes a large amount of on-board computing resources, but also affects the exploration efficiency of the unmanned aerial vehicle when exploring large scenes.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] Embodiments of the present application aim to at least partially solve one of the technical problems in the related art. To this end, the main purpose of the embodiments of the present application is to propose an unmanned aerial vehicle autonomous exploration method and device in a dynamic unknown environment, which can realize seamless switching between detection mode and exploration mode, and guarantee the efficient target search capability of the unmanned aerial vehicle in the dynamic unknown environment.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application proposes an unmanned aerial vehicle autonomous exploration method in a dynamic unknown environment, applied to an unmanned aerial vehicle autonomous exploration system, the unmanned aerial vehicle autonomous exploration system comprising a map module, a perception module and a planning module, and the method comprising the following steps:

[0006] Obtaining current sensor data of the unmanned aerial vehicle; the current sensor data comprising current unmanned aerial vehicle pose information, current unmanned aerial vehicle point cloud information and current environment images;

[0007] Updating a historical three-dimensional voxel map according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information by the map module to obtain a current three-dimensional voxel map;

[0008] Performing heuristic boundary region detection processing on the current three-dimensional voxel map by the map module to obtain a target boundary region;

[0009] updating a current route map according to the target boundary region through the map module, to obtain a global route map;

[0010] obtaining image key information by performing image processing on the current environment image through the perception module; the image key information includes a binary image and dynamic obstacle information;

[0011] determining a target working mode of the UAV according to the binary image through the planning module; the target working mode includes a detection mode and an exploration mode;

[0012] if the target working mode is the detection mode, detecting a target detection object in a dynamic unknown environment to obtain target detection information; and automatically switching the detection mode to the exploration mode after the target detection object is detected;

[0013] if the target working mode is the exploration mode, generating a target exploration path for a target exploration point according to the target boundary region, the global route map and the dynamic obstacle information through the planning module.

[0014] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a UAV autonomous exploration device in a dynamic unknown environment, applied to a UAV autonomous exploration system, the UAV autonomous exploration system including a map module, a perception module and a planning module, and the device includes the following modules:

[0015] a sensor data acquisition module, configured to acquire current sensor data of the UAV; the current sensor data includes current UAV pose information, current UAV point cloud information and a current environment image;

[0016] a three-dimensional voxel map updating module, configured to update a historical three-dimensional voxel map according to the current UAV pose information and the current UAV point cloud information through the map module, to obtain a current three-dimensional voxel map;

[0017] a boundary region extraction module, configured to perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module, to obtain a target boundary region;

[0018] a global route map updating module, configured to update a current route map according to the target boundary region through the map module, to obtain a global route map;

[0019] an image processing module, configured to perform image processing on the current environment image through the perception module, to obtain image key information; the image key information includes a binary image and dynamic obstacle information;

[0020] An unmanned aerial vehicle working mode determination module is configured to determine a target working mode of the unmanned aerial vehicle according to the binary image by using the planning module; the target working mode includes a detection mode and an exploration mode;

[0021] A target object detection module is configured to, if the target working mode is the detection mode, control the unmanned aerial vehicle to detect a target detection object in a dynamic unknown environment, and obtain target detection information; wherein, after the target detection object completes detection, the detection mode is automatically switched to the exploration mode;

[0022] A target exploration path generation module is configured to, if the target working mode is the exploration mode, generate a target exploration path for a target exploration point according to the target boundary region, the global route map and the dynamic obstacle information by using the planning module.

[0023] The embodiments of the present application at least have the following beneficial effects: the present application provides a method and device for autonomous exploration of a UAV in a dynamic unknown environment, the method comprises the following steps: obtaining current sensor data of the UAV; the current sensor data comprises current UAV pose information, current UAV point cloud information and a current environment image; updating a historical three-dimensional voxel map according to the current UAV pose information and the current UAV point cloud information by a map module, to obtain a current three-dimensional voxel map; performing heuristic boundary region detection processing on the current three-dimensional voxel map by the map module, to obtain a target boundary region; updating a current route map according to the target boundary region by the map module, to obtain a global route map; performing image processing on the current environment image by a perception module, to obtain image key information; the image key information comprises a binary image and dynamic obstacle information; determining a target working mode of the UAV according to the binary image by a planning module; the target working mode comprises a detection mode and an exploration mode; if the target working mode is the detection mode, the UAV is controlled to detect a target detection object in the dynamic unknown environment, to obtain target detection information; wherein, the detection mode is automatically switched to the exploration mode after the target detection object completes the detection; if the target working mode is the exploration mode, a target exploration path for a target exploration point is generated according to the target boundary region, the global route map and the dynamic obstacle information by the planning module. The embodiments of the present application obtain the target boundary region through heuristic boundary region detection processing, and update the current route map according to the target boundary region and the current three-dimensional voxel map to obtain the global route map, thereby improving the generation speed of the global route map and reducing the calculation amount; the planning module realizes seamless switching of the detection mode and the exploration mode of the UAV, can balance the demand for autonomous exploration and target inspection of the UAV, significantly improves the flexibility and efficiency of task execution of the UAV, and at the same time guarantees the efficient target search capability of the UAV in autonomous exploration in the dynamic unknown environment, so that the target search is more comprehensive, and the accuracy of target search and target detection is improved; the planning module can plan the optimal exploration path according to the target boundary region, the global route map and the dynamic obstacle information, so as to reduce the time and energy consumption required for exploration, improve the exploration efficiency, and reduce the risk of collision between the UAV and the dynamic obstacle based on the dynamic obstacle information, thereby improving the safety of UAV operation. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flowchart of a method for autonomous exploration of a UAV in a dynamic unknown environment provided by the embodiments of the present application;

[0025] Figure 2 is a schematic diagram of a UAV autonomous target exploration system framework provided by the embodiments of the present application;

[0026] Figure 3 is a color segmentation schematic diagram provided by the embodiments of the present application;

[0027] Figure 4 is a kind of environment image schematic diagram provided by the embodiment of the application;

[0028] Figure 5 is a kind of binary image schematic diagram provided by the embodiment of the application;

[0029] Figure 6 is a kind of target detection result schematic diagram provided by the embodiment of the application;

[0030] Figure 7 is a kind of target search flow schematic diagram provided by the embodiment of the application;

[0031] Figure 8 is a kind of three-dimensional voxel map schematic diagram provided by the embodiment of the application;

[0032] Figure 9 is a kind of multi-layer two-dimensional map schematic diagram provided by the embodiment of the application;

[0033] Figure 10 is a kind of two-dimensional image schematic diagram provided by the embodiment of the application;

[0034] Figure 11 is a kind of boundary detection result schematic diagram provided by the embodiment of the application;

[0035] Figure 12 is a kind of current route map schematic diagram of unmanned aerial vehicle provided by the embodiment of the application;

[0036] Figure 13 is a kind of incremental route map update schematic diagram provided by the embodiment of the application;

[0037] Figure 14 is a kind of unmanned aerial vehicle route map update schematic diagram provided by the embodiment of the application;

[0038] Figure 15 is a kind of unmanned aerial vehicle autonomous exploration device structure schematic diagram in dynamic unknown environment provided by the embodiment of the application;

[0039] Figure 16 is a kind of hardware structure schematic diagram of electronic equipment provided by the embodiment of the application. DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the present application clearer, further described are the present application in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0041] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the embodiments of the present application. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determination" or "in response to a determination".

[0042] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0044] As an example, the unmanned aerial vehicle has the characteristics of small size and strong maneuverability, which can help humans explore unknown or dangerous environments. With the rapid development of unmanned aerial vehicle technology and its wide application in logistics transportation, agricultural monitoring, disaster rescue and other fields, autonomous exploration in dynamic unknown environments has become one of the important directions of unmanned aerial vehicle technology research. In these application scenarios, there is great uncertainty in the unknown environment, and the unmanned aerial vehicle needs to avoid dynamic obstacles in time when exploring unknown space to ensure flight safety; for scenarios that need to search for target objects, the unmanned aerial vehicle also needs to efficiently and accurately identify and locate specific targets when exploring, in order to meet the complex task requirements of disaster area search and rescue, logistics transportation, etc. However, the distribution of obstacles in dynamic unknown environments is complex and variable, and the information update is real-time, which poses higher challenges to the autonomous exploration capability of unmanned aerial vehicles. At present, the autonomous exploration technology of unmanned aerial vehicles usually takes mapping of unknown space as the core, and realizes path planning and navigation function by mapping unknown areas into passable areas and obstacle areas; while the target search technology emphasizes more on accurate detection and identification of targets during exploration. However, for the target search problem in the exploration process, the core of the traditional exploration method is to quickly build a map of the unknown environment, and if this method is directly combined with a target detection algorithm, it is easy to cause the target search to be incomplete, and because the motion state of the unmanned aerial vehicle changes constantly during exploration, the camera images collected appear blurred, and directly detecting the blurred images will result in insufficient detection accuracy, and the current autonomous target search algorithm attempts to check all obstacles in the environment one by one, which not only consumes a lot of on-board computing resources, but also affects the exploration efficiency of the unmanned aerial vehicle when exploring large scenes.

[0045] In view of this, the embodiment of the present application provides a method and device for autonomous exploration of a UAV in a dynamic unknown environment. The method comprises the following steps: acquiring current sensor data of the UAV; the current sensor data comprises current UAV pose information, current UAV point cloud information, and a current environment image; updating a historical three-dimensional voxel map according to the current UAV pose information and the current UAV point cloud information by a map module, to obtain a current three-dimensional voxel map; performing heuristic boundary region detection processing on the current three-dimensional voxel map by the map module, to obtain a target boundary region; updating a current route map according to the target boundary region by the map module, to obtain a global route map; performing image processing on the current environment image by a perception module, to obtain image key information; the image key information comprises a binary image and dynamic obstacle information; determining a target working mode of the UAV according to the binary image by a planning module; the target working mode comprises a detection mode and an exploration mode; if the target working mode is the detection mode, controlling the UAV to detect a target detection object in the dynamic unknown environment, to obtain target detection information; wherein, the detection mode is automatically switched to the exploration mode after the target detection object completes the detection; if the target working mode is the exploration mode, generating a target exploration path for a target exploration point according to the target boundary region, the global route map, and the dynamic obstacle information by the planning module. The embodiment of the present application obtains the target boundary region through heuristic boundary region detection processing, and updates the current route map according to the target boundary region and the current three-dimensional voxel map to obtain the global route map, thereby improving the generation speed of the global route map and reducing the calculation amount. The planning module realizes seamless switching of the detection mode and the exploration mode of the UAV, which can balance the needs of autonomous exploration and target inspection of the UAV, significantly improve the flexibility and efficiency of task execution of the UAV, and also guarantee the efficient target search capability of the UAV in autonomous exploration in the dynamic unknown environment, so that the target search is more comprehensive, thereby improving the accuracy of target search and target detection. The planning module can plan the optimal exploration path for the UAV according to the target boundary region, the global route map, and the dynamic obstacle information, thereby reducing the time and energy consumption required for exploration, improving the exploration efficiency, and reducing the risk of collision between the UAV and the dynamic obstacle based on the dynamic obstacle information, thereby improving the safety of UAV operation.

[0046] The unmanned aerial vehicle autonomous exploration method in a dynamic unknown environment provided by the embodiment of the present application relates to the technical field of unmanned aerial vehicles. The unmanned aerial vehicle autonomous exploration method in a dynamic unknown environment provided by the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can further be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can further be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; and the software can be an application that implements the unmanned aerial vehicle autonomous exploration method in a dynamic unknown environment, and the like, but is not limited to the above forms.

[0047] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs (Personal Computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0048] Please refer to Figure 1 , Figure 1 is an optional step flowchart of the unmanned aerial vehicle autonomous exploration method in a dynamic unknown environment provided by the embodiment of the present application, Figure 1 The method in the above method is applied to an unmanned aerial vehicle autonomous exploration system, and the unmanned aerial vehicle autonomous exploration system includes a map module, a perception module, and a planning module. Figure 1 The method in the above method can include, but is not limited to, steps S101 to S108.

[0049] In step S101, current sensor data of the UAV is acquired; the current sensor data includes current UAV pose information, current UAV point cloud information, and a current environment image.

[0050] In the embodiment of the present application, the method for autonomous exploration of the UAV in a dynamic unknown environment is applied to a UAV autonomous exploration system, please refer to Figure 2 , Figure 2 is a schematic diagram of a UAV autonomous target exploration system framework provided by the embodiment of the present application; as shown in Figure 2 , the UAV autonomous exploration system includes a map module, a perception module, and a planning module, and through the interaction of the map module, the perception module, and the planning module, the final UAV autonomous target exploration trajectory can be obtained.

[0051] Optionally, the current sensor data of the UAV includes current UAV pose information, current UAV point cloud information, and a current environment image. The current sensor data is sensor data corresponding to the current position of the UAV at the current time.

[0052] The UAV pose information is mainly obtained by a visual-inertial odometer composed of camera vision information and an inertial measurement unit, and the content mainly includes position information and attitude information of the UAV, and specifically can include the coordinates of the x-axis, y-axis, and z-axis, the yaw angle, the roll angle, and the pitch angle, etc.

[0053] The UAV point cloud information refers to a set of three-dimensional spatial data points obtained from the scene by a depth sensor (the RGB-D camera has a depth sensor), and these points represent the distance from the UAV to the surface of the object in the scene, and are represented by a set of three-dimensional (X, Y, Z) coordinates. The coordinates of each point are calculated from the measurement results of the depth sensor, which represent the specific position of the point in the three-dimensional space. The point cloud information specifically can include the spatial coordinates of each point, the depth value (the distance between the point and the UAV), the sparsity information, and the point cloud size, etc.

[0054] In the specific implementation, at the system initialization, first, the three-dimensional voxel map of the initial position of the UAV can be constructed by the UAV pose information and the UAV point cloud information of the UAV at the initial position, and then the three-dimensional voxel map is updated in real time by using the real-time pose data and the point cloud information of the UAV.

[0055] The current environment image refers to the two-dimensional visual information of the surrounding environment captured in real time by the camera or other image acquisition devices carried on the UAV during the execution of the task. The airborne visual sensor adopted in the embodiment of the present application is an RGB-D camera.

[0056] In step S102, the historical three-dimensional voxel map is updated according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information by the map module, and a current three-dimensional voxel map is obtained.

[0057] In some embodiments, before step S102, the method can further include: obtaining historical sensor data of the unmanned aerial vehicle; wherein the historical sensor data includes historical unmanned aerial vehicle pose information and historical unmanned aerial vehicle point cloud information; and constructing a historical three-dimensional voxel map according to the historical unmanned aerial vehicle pose information and the historical unmanned aerial vehicle point cloud information by the map module.

[0058] For the historical sensor data, it refers to a series of data collected by the sensors carried by the unmanned aerial vehicle in the past period of time, which is used to construct the historical three-dimensional voxel map. It should be noted that the historical sensor data is data obtained in a period of time before the current sensor data; the historical sensor data can also be sensor data obtained at the initial position of the unmanned aerial vehicle at the initialization time. It can be understood that at the initialization time of the system, the three-dimensional voxel map of the initial position of the unmanned aerial vehicle can be first constructed by the unmanned aerial vehicle pose information and the unmanned aerial vehicle point cloud information of the initial position of the unmanned aerial vehicle, and then the three-dimensional voxel map is updated in real time by using the real-time pose data and the point cloud information of the unmanned aerial vehicle.

[0059] For the three-dimensional voxel map, it is a data structure used to represent a three-dimensional space environment, which divides the space into a series of small, uniform cubic units, which are called voxels.

[0060] In a specific implementation, the historical three-dimensional voxel map can be updated according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information by the map module, and the current three-dimensional voxel map is obtained. As shown in FIG. 4, the three-dimensional voxel map can be updated by the map module according to the real-time point cloud data and the real-time pose information obtained by the unmanned aerial vehicle. Figure 2

[0061] In step S103, the current three-dimensional voxel map is subjected to heuristic boundary region detection processing by the map module, and a target boundary region is obtained.

[0062] In some embodiments, step S103 can include: performing voxel map layering processing on the current three-dimensional voxel map by the map module, and obtaining a current two-dimensional voxel map; performing map imaging processing on the current two-dimensional voxel map by the map module, and obtaining a current two-dimensional map image; and performing boundary region extraction processing on the current two-dimensional map image by the map module, and obtaining the target boundary region.

[0063] ​In a specific implementation, the heuristic boundary region detection process flow is composed of four steps: (1) construction of a 3D (Three-Dimensional, three-dimensional) occupancy voxel map (i.e., a three-dimensional voxel map): first, the unmanned aerial vehicle generates a three-dimensional occupancy voxel map using the point cloud data of the camera and the pose data of the unmanned aerial vehicle to accurately represent the spatial distribution of the current environment. (2) voxel map layering processing: to facilitate processing, the three-dimensional voxel map is divided into multiple two-dimensional mapping maps according to the height, and each height layer corresponds to a two-dimensional occupancy map. (3) map image processing: converting the two-dimensional occupancy map into a two-dimensional image facilitates subsequent efficient image processing operations. (4) boundary region extraction: image processing is performed on the converted two-dimensional image to extract a boundary region, which is used to represent a key region to be explored, and most of the region is unexplored space. The extracted boundary region contains both free areas and occupied space areas to guide the exploration direction of the unmanned aerial vehicle.

[0064] Specifically, a screening threshold is set to optimize the selection of the boundary region, and only high-priority regions are retained for score calculation, reducing the number of boundary regions participating in the calculation, thereby reducing the computational burden while ensuring the effectiveness of the exploration, and improving the real-time performance and efficiency of the exploration algorithm.

[0065] In step S104, the map module updates the current route map according to the target boundary region to obtain a global route map.

[0066] In a specific implementation, the global roadmap is incrementally constructed based on the global roadmap, and the motion trajectory of the UAV is planned based on the global roadmap. The global roadmap refers to a map that updates the current roadmap in real time until there is no boundary region in the current roadmap. Specifically, each time the current roadmap is updated, the nodes on the current roadmap are updated from the current position of the UAV (the UAV is temporarily stopped when the current roadmap is updated after reaching the target point, and the UAV continues to navigate to the next target point after the update is completed). The navigation target point is calculated by the exploration algorithm based on the boundary region extracted by the map module. Each time the UAV navigates to the target point, it is the target boundary region that needs to be explored, and the boundary region is calculated from the 3D occupancy voxel map. Therefore, the specific construction process of the global roadmap is as follows: first, the map module extracts the boundary region information from the 3D occupancy voxel map (including free and occupied regions) to generate a two-dimensional image (including boundary regions); then, the UAV updates the current roadmap according to the current position of the UAV and the boundary region (i.e., the UAV adds reasonable nodes around the current node according to the boundary region), and after the node update, the exploration algorithm calculates the navigation target point based on the boundary region extracted by the map module, and plans a path to the navigation target point based on the current roadmap. After reaching the navigation target point, repeat the above operation (starting from calculating the two-dimensional image from the 3D occupancy voxel map), until there is no boundary region, and the task of constructing the global roadmap is completed. During the movement of the UAV, the 3D occupancy voxel map is updated in real time.

[0067] In step S105, the perception module processes the current environment image to obtain image key information; the image key information includes a binary image and dynamic obstacle information.

[0068] In some embodiments, step S105 can include: the perception module performs feature segmentation processing on the current environment image to obtain the binary image in the image key information; and the perception module performs depth detection processing on the current environment image to obtain the dynamic obstacle information in the image key information.

[0069] In some specific embodiments, the perception module performs feature segmentation processing on the current environment image to obtain the binary image in the image key information can include: the perception module performs color space conversion processing on the current environment image to obtain a to-be-segmented environment image; and the perception module performs feature segmentation processing on the to-be-segmented environment image according to a preset segmentation threshold to obtain the binary image in the image key information; wherein the binary image contains target detection objects in a dynamic unknown environment.

[0070] In a specific implementation, since the RGB image is susceptible to natural light, occlusion, and shadow, etc., if image processing is directly performed in the RGB color space, it may lead to problems such as detection accuracy decline and feature mismatch. In order to improve the robustness of color segmentation, the RGB color space is converted into the HSV (Hue, Saturation, Value) color space in the embodiments of the present application, the HSV color space represents the hue, saturation, and value of the color respectively, and can more intuitively describe the color characteristics. Compared with the RGB color space, the HSV color space is more suitable for color contrast and segmentation, and therefore is more efficient and accurate when detecting target objects with specific color characteristics. Specifically, first, the current environment image is subjected to HSV color space conversion processing by the perception module to obtain a to-be-segmented environment image; then, in the HSV color space, the target object with a specific color is extracted from the to-be-segmented environment image by setting a segmentation threshold, and after the segmentation processing, a black-and-white image, i.e., a binary image, is generated. This feature segmentation processing mode effectively filters irrelevant areas while retaining the main features of the target of interest.

[0071] In step S106, the target working mode of the UAV is determined according to the binary image by the planning module; the target working mode includes a detection mode and an exploration mode.

[0072] In some embodiments, step S106 can include: calculating the white pixel points in the binary image by the planning module to obtain a white pixel point set; if the number of the white pixel point set is less than a preset pixel point threshold, it is determined that the target working mode of the UAV is the exploration mode; if the number of the white pixel point set is greater than or equal to the preset pixel point threshold, it is determined that the target working mode of the UAV is the detection mode.

[0073] In step S107, if the target working mode is the detection mode, the UAV detects the target detection object in a dynamic unknown environment to obtain target detection information; wherein the detection mode is automatically switched to the exploration mode after the target detection object completes the detection.

[0074] In a specific implementation, when entering the planning module, the mode selector in the planning module determines the working mode suitable for the current UAV according to the binary image obtained by the perception module, if in the detection mode, the UAV performs detailed visual inspection on the region of interest in the environment, and after identifying the target object, the position information of the target object is printed and immediately returned to the mode selector, otherwise the detection mode is returned to the mode selector after a set time ends.

[0075] Step S108, if the target working mode is the exploration mode, generating a target exploration path for the target exploration point by the planning module according to the target boundary region, the global roadmap and the dynamic obstacle information.

[0076] In some embodiments, step S108 can include: if the target working mode is the exploration mode, generating a candidate exploration path set by the planning module according to the target boundary region and the global roadmap; performing information gain calculation on the candidate exploration path set by the planning module to obtain an information gain calculation result; screening a target information gain path set from the information gain calculation result and a preset information gain screening threshold by the planning module, and constructing a candidate global path set of the UAV based on the target information gain path set; selecting a target global path from the candidate global path set by the planning module; if no dynamic obstacle information is detected to exist path obstruction to the target global path in the dynamic unknown environment, taking the target global path as the target exploration path for the target exploration point; and if dynamic obstacle information is detected to exist path obstruction to the target global path in the dynamic unknown environment, adjusting the target global path according to a dynamic re-planning method by the planning module to obtain the target exploration path for the target exploration point.

[0077] In some specific embodiments, if dynamic obstacle information is detected to exist path obstruction to the target global path in the dynamic unknown environment, adjusting the target global path according to a dynamic re-planning method by the planning module to obtain the target exploration path for the target exploration point can include: if dynamic obstacle information is detected to exist path obstruction to the target global path in the dynamic unknown environment, performing density calculation on the dynamic obstacle information by the planning module to obtain an obstacle density; calculating a target exploration distance between the UAV and the target exploration point by the planning module; calculating a trajectory re-planning frequency of the target exploration path according to the obstacle density and the target exploration distance by the planning module to adjust the target global path according to the trajectory re-planning frequency to obtain the target exploration path for the target exploration point; and a calculation formula of the trajectory re-planning frequency is:

[0078]

[0079] wherein f replan represents the trajectory re-planning frequency, β max represents an upper limit of the trajectory re-planning frequency, γ base represents a scaling constant for adjusting the trajectory re-planning frequency, fac obs represents an obstacle scaling factor, fac dis represents a distance scaling factor.

[0080] In some embodiments, the method further comprises: if the target working mode is the exploration mode, generating, by the planning module, a local exploration path for the target exploration point according to current UAV position information and current UAV heading information of the UAV; wherein the local exploration path is used to optimize the target exploration path, the current UAV heading information is provided with a maximum yaw angle limit, and an expression of the maximum yaw angle limit is:

[0081]

[0082] wherein angle change represents a yaw angle rotation angle of the UAV from a current position to a next waypoint, v next represents a direction vector of the UAV from the current position to the next waypoint, v current represents a current direction vector of the UAV.

[0083] Optionally, the trajectory re-planning frequency refers to a frequency of path planning. The UAV needs to constantly plan an exploration path when exploring in an environment, so as to complete an exploration task. If the frequency of path planning is higher, the trajectory of the UAV changes more frequently, thereby reducing exploration efficiency. If the frequency of path planning is lower, the UAV has a poorer effect on dynamic obstacle avoidance. Therefore, the embodiments of the present application provide a formula of the trajectory re-planning frequency to flexibly adjust the frequency of path planning to obtain an optimal exploration path. Specifically, the planning path is calculated by an exploration algorithm.

[0084] wherein the target exploration path is also referred to as a global exploration path. The global exploration path guides a forward direction of the UAV. However, the global exploration path generated based on the exploration algorithm is not necessarily reasonable, and thus needs to be adjusted by using a local exploration path. It can be understood that the local exploration path is used to refine a path in the forward direction guided by the global exploration path. One global exploration path is composed of a plurality of local exploration paths. The local exploration path needs to consider obstacle avoidance. When the local exploration path is generated, the maximum yaw angle limit is used to enable the UAV to move in a general movement direction guided by the global exploration path.

[0085] The steps S101 to S108 shown in the embodiments of the present application are as follows: current sensor data of the unmanned aerial vehicle is acquired; the current sensor data includes current unmanned aerial vehicle pose information, current unmanned aerial vehicle point cloud information, and a current environment image; a historical three-dimensional voxel map is updated according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information by a map module, to obtain a current three-dimensional voxel map; a target boundary region is obtained by performing heuristic boundary region detection processing on the current three-dimensional voxel map by the map module; a global route map is obtained by updating a current route map according to the target boundary region by the map module; image key information is obtained by performing image processing on the current environment image by a perception module; the image key information includes a binary image and dynamic obstacle information; a target working mode of the unmanned aerial vehicle is determined according to the binary image by a planning module; the target working mode includes a detection mode and an exploration mode; if the target working mode is the detection mode, the unmanned aerial vehicle is controlled to detect a target detection object in a dynamic unknown environment, to obtain target detection information; wherein, after the target detection object completes the detection, the detection mode is automatically switched to the exploration mode; if the target working mode is the exploration mode, a target exploration path for a target exploration point is generated according to the target boundary region, the global route map, and the dynamic obstacle information by the planning module. The embodiments of the present application obtain the target boundary region through heuristic boundary region detection processing, and update the current route map to obtain the global route map based on the target boundary region and the current three-dimensional voxel map, thereby improving the generation speed of the global route map and reducing the calculation amount; the planning module is used to realize seamless switching of the detection mode and the exploration mode of the unmanned aerial vehicle, which can balance the needs of autonomous exploration and target inspection of the unmanned aerial vehicle, significantly improve the flexibility and efficiency of task execution of the unmanned aerial vehicle, and also ensure the efficient target search capability of the unmanned aerial vehicle in autonomous exploration in the dynamic unknown environment, so that the target search is more comprehensive, and the accuracy of target search and target detection is improved; the planning module is used to generate the optimal exploration path according to the target boundary region, the global route map, and the dynamic obstacle information, so as to reduce the time and energy consumption required for exploration, improve the exploration efficiency, and reduce the risk of collision between the unmanned aerial vehicle and the dynamic obstacle based on the dynamic obstacle information, thereby improving the safety of the unmanned aerial vehicle operation.

[0086] To explain the principle of the technical scheme of the present application in detail, the overall process of the present application will be described below in combination with some specific embodiments. It should be easily understood that the following is an explanation of the technical principle of the present application and cannot be regarded as a limitation of the present application.

[0087] In a specific implementation, in a limited three-dimensional space In the middle, the task of the UAV autonomous exploration is to establish an accurate three-dimensional voxel map of the unknown environment through the on-board visual sensor (RGB-D camera), and to detect and identify an unknown number of target objects. The space is represented as a set of cubic voxels, and the occupancy probability P(v) of each voxel V is continuously updated, incrementally mapping the initial unknown space V unk = V to two parts: (free space) and (occupied space). Since the perception of most sensors stops at the surface, some hollow or corner spaces cannot be mapped, which are represented by V res In the initial stage of exploration, the entire dynamic environment is unknown, and the UAV only has an initial mapping map M init of the nearby area, and according to the initial mapping map M init The UAV needs to iteratively generate a collision-free trajectory to explore the unknown area and search for the target until the entire environment is explored and mapped as M env , at which time M env = (V free ∪ V occ )\V res , the exploration task is completed. Since the target object is located in the occupied space V occ , the UAV must observe all voxels to ensure the integrity of the search. In a dynamic environment, the UAV must cope with changes in dynamic obstacles and constantly generate collision-free trajectories, such as fast-moving obstacles such as pedestrians, to ensure real-time updating of the map and safe obstacle avoidance.

[0088] As shown in Figure 2 , the algorithm framework of the UAV autonomous exploration method in a dynamic unknown environment provided by the embodiments of the present application is as shown in Figure 2 , that is, the UAV autonomous exploration system includes three core parts of a map module, a perception module and a planning module. The UAV autonomous exploration system provided by the embodiments of the present application can seamlessly switch between exploration mode and detection mode. Among them, the function of the map module is to dynamically update the three-dimensional voxel map using the current pose data and point cloud information of the UAV, and extract a new boundary region based on the three-dimensional voxel map. Then, the map module updates the current route map according to the current pose information of the UAV and the new boundary region to obtain a global route map. After the global route map is successfully updated, the planning module can be entered. At the same time, the perception module performs color segmentation on the RGB image to extract the features of the RGB image and generate a binary image; the perception module also dynamically detects the depth image to obtain dynamic obstacle information, and generates a labeled box based on the dynamic obstacle information (such as Figure 2"dynamic obstacle" block in the perception module). When entering the planning module, the mode selector in the planning module determines the appropriate working mode for the current UAV according to the binary image obtained by the perception module: if in the detection mode, the UAV performs a detailed visual inspection of the region of interest in the environment and prints the target object position information after identifying the target object and returns to the mode selector immediately, otherwise the detection mode returns to the mode selector after a set time; if in the exploration mode, the planning module generates a safe and efficient exploration path according to the boundary region extracted in the map module, the updated global route map information and the dynamic obstacle data provided by the perception module; wherein the default mode of the mode selector is the exploration mode, which ensures that the UAV is in the exploration of unknown environment most of the time. After generating the exploration path, the planning module updates the local trajectory in real time through the dynamic planner, so that the UAV can quickly respond to environmental changes. In the path generation process, path 2( Figure 2 in sequence number 2) represents a candidate global path, path 1( Figure 2 in sequence number 1) is the selected target global path, and path 3( Figure 2 in sequence number 3) is a dynamically generated local path. Compared with the traditional exploration method relying on laser radar, the embodiment of the present application reduces the weight and hardware cost of the system while significantly improving the flexibility and efficiency of task execution, providing an efficient and low-cost solution for autonomous target search of UAV in dynamic unknown environment.

[0089] Specifically, as shown in Figure 2 , the input data of the UAV autonomous exploration system is camera image, pose data and point cloud data. First, the map module updates the global route map of the UAV according to the current pose data and point cloud information of the UAV; the map module also updates the three-dimensional voxel map according to the current pose data and point cloud information of the UAV, and extracts a new boundary range based on the three-dimensional voxel map. Then, the perception module processes the camera image to obtain dynamic obstacle information and a binary image after feature extraction. Next, the mode selector of the planning module selects the appropriate working mode according to the binary image. Finally, the dynamic planner generates a safe and efficient exploration path, and adjusts the trajectory in real time according to the dynamic planner.

[0090] In a specific implementation, the specific implementation process of the UAV autonomous exploration method in a dynamic unknown environment provided by the embodiment of the present application includes the following three steps (steps 1 to 3) :

[0091] Step 1 (steps 1.1 to 1.3), target detection based on color segmentation.

[0092] In order to ensure the accuracy and efficiency of the target search in the exploration, an embodiment of the present application designs a target detector based on color segmentation, and proposes a mechanism for switching between autonomous exploration and target detection mode. Through the color segmentation technology, the semantic information in the environment is extracted, so that the unmanned aerial vehicle only performs detailed visual inspection on the target object of interest, rather than comprehensive scanning of all occupied spaces in the environment, thereby improving the efficiency of target search.

[0093] In a specific implementation, first, all voxels in the environment are preliminarily scanned by the camera to obtain the basic information in the global field of view, so as to make up for the incompleteness of the traditional method in target object search; then, the color segmentation technology is used to process the RGB image obtained in the scanning process, and only the color space information related to the target object is extracted; then, according to the color segmentation result, it is judged whether there is a potential target object: if a potential target object is detected, a detection viewpoint corresponding to the potential target object is generated, and the target object is subjected to detailed angle inspection; if no potential target object is detected, the exploration algorithm is continued. It should be noted that the target search in the embodiment of the present application refers to the target detection part, not the autonomous exploration part.

[0094] Step 1.1, color segmentation.

[0095] Please refer to Figure 3 to Figure 6 , Figure 3 is a color segmentation diagram provided by an embodiment of the present application, Figure 4 is an environment image diagram provided by an embodiment of the present application, Figure 5 is a binary image diagram provided by an embodiment of the present application, Figure 6 is a target detection result diagram provided by an embodiment of the present application; the color segmentation process is as shown in Figure 3 . In the process of unmanned aerial vehicle exploring unknown environment, the unmanned aerial vehicle autonomous exploration system scans the objects in the space by continuously adjusting the angle of the unmanned aerial vehicle, and obtains the RGB image (as shown in Figure 4 ).

[0096] However, the RGB image is easily affected by natural light, shielding and shadow, etc. If the image is directly processed in the RGB color space, it may cause problems such as detection accuracy decline and feature mismatch. In order to improve the robustness of color segmentation, the RGB color space is converted to the HSV (Hue, Saturation, Value) color space in the embodiment of the present application. The HSV color space respectively represents the hue, saturation and brightness of the color, and can more intuitively describe the color characteristics. Compared with the RGB color space, the HSV color space is more suitable for color contrast and segmentation, so it is more efficient and accurate when detecting target objects with specific color characteristics.

[0097] Specifically, in HSV color space, the target object with specific color is extracted from the image by setting a segmentation threshold (as shown in Figure 3 ), and a black and white image, i.e., a binary image (as shown in Figure 5 ), is generated after the segmentation process, wherein Figure 5 the white area represents a set of pixel points meeting the characteristics of the segmentation threshold. This feature segmentation processing method effectively filters out irrelevant areas while retaining the main features of the target of interest.

[0098] Step 1.2, generate a detection viewpoint.

[0099] Calculate the binary image after color segmentation in step 1.1. If the number of white pixel points pix count is less than the threshold pix thre , it is considered that there is no potential target object in the current field of view of the UAV, and no careful visual inspection is needed; if the number of white pixel points pix count is greater than or equal to the threshold pix thre , it is considered that there is a potential target object, and the yaw angle of the UAV is adjusted to place the object to be detected in the center of the field of view, and the potential object is visually inspected. The calculation formula of the yaw angle speed ω yaw of the UAV is as follows:

[0100]

[0101] where ω0 represents an angular velocity adjustment parameter that needs to be set in advance according to the complexity of the environment; ω init represents the original angular velocity of the UAV; img width represents the width of the camera image frame; pix avg represents the centroid horizontal coordinate of the target pixel point. The calculation formula of the centroid horizontal coordinate pix avg of the target pixel point is as follows:

[0102]

[0103] where n represents the total number of pixel points pix.

[0104] In specific implementation, the centroid coordinate pix avg of the potential target is obtained by summing the horizontal coordinate pix x of each pixel point and dividing the total number of pixel points pix n .

[0105] Specifically, while conducting a detailed visual inspection of the potential target object, the UAV needs to keep moving, i.e., constantly exploring the unknown environment, without affecting the exploration efficiency. The embodiments of the present application adjust the potential target object to be in the field of view of the UAV through constant changes in the angular velocity, so that the UAV can achieve multi-angle scanning and detection of the region where the potential target object is located during movement. The multi-angle scanning can update the three-dimensional voxel map, and the multi-angle detection can improve the detection accuracy of the object.

[0106] Step 1.3, target object detection.

[0107] Since the motion state of the UAV in the unknown environment is constantly changing, the field of view of the camera also changes rapidly, resulting in blurred images. The target detector designed in the embodiments of the present application can dynamically adjust the yaw angular velocity of the UAV according to the color segmentation result, so as to ensure that the field of view of the camera is always concentrated in the target region, thereby achieving accurate identification of the target object (such as Figure 6 “spor is ball” mark).

[0108] The UAV autonomous exploration method in a dynamic unknown environment provided by the embodiments of the present application uses a three-dimensional bounded search space R 3 to explore the range and discretize it into a voxel grid v i , then constructs an initial environment model according to the occupancy probability P(v i ) of each voxel, and initializes the unknown region V unk to the entire space V, the idle region V free and the occupied region V occ to empty sets, while retaining the restricted region V res . Please refer to Figure 7 , Figure 7 is a schematic diagram of a target search process provided by the embodiments of the present application; the entire target search process (target detection process) is shown in Figure 7 , specifically, after successful parameter initialization, first construct a 3D occupancy voxel map according to the pose information and point cloud data of the UAV, then perform heuristic boundary region detection, and generate candidate exploration target points based on the boundary region range according to the exploration method. At the same time, the path planning framework updates the route map according to the updated 3D occupancy voxel map, and plans a global path according to the selected exploration target point. On the other hand, the mode selector will determine whether to enter the target detection mode (such as Figure 7the autonomous exploration framework shown in FIG. 1), when the UAV detects a target color region in the field of view during exploration, it will switch to the target detection mode, in which the color segmentation module calculates the center point of the target region and adjusts the yaw angular velocity of the UAV according to the center point, so that the center of the camera field of view remains in the target region to be detected. At the same time, the system uses a target detection algorithm (such as the traditional detection algorithm YOLO-v3) to identify the target (such as Figure 7 the target search framework shown in FIG. 1), after the identification is completed, the UAV will switch back to the autonomous exploration mode and continue to scan the unknown space (such as Figure 7 the autonomous exploration framework shown in FIG. 1). During the exploration, the path planning framework updates the current route map M env based on the free and occupied regions of the 3D occupancy voxel map, generates a collision-free global path to the exploration target point according to the route map and the dynamic obstacle information obtained by the camera, and in each iteration process, the planning algorithm updates the obstacle information in the environment map in real time, checks the effectiveness of the UAV path, predicts the trajectory of the dynamic obstacle in the environment by using a Markov chain-based environment perception trajectory prediction method, and if the detected dynamic obstacle trajectory conflicts with the current path, the system will trigger trajectory re-planning to change the current UAV path to ensure safety (such as Figure 7 the path planning framework shown in FIG. 1).

[0109] wherein the exploration algorithm can be HIRE (Heuristic-based Incremental Probabilistic Roadmap for Efficient UAV Exploration in Dynamic Environments), and the final exploration target point is selected based on the exploration algorithm. The planning algorithm can use (A real-time dynamic obstacle tracking and mapping system for UAV navigation and collision avoidance with an RGB-D camera). It should be noted that the exploration algorithm and the planning algorithm can be selected according to actual conditions, and the embodiments of the present application do not limit this.

[0110] Step 2 (step 2.1 to step 2.2), incremental roadmap construction of heuristic boundary.

[0111] Step 2.1, heuristic boundary region detection.

[0112] Please refer to Figure 8 to Figure 11 , Figure 8 is a three-dimensional voxel map schematic diagram provided by an embodiment of the present application,Figure 9 is a multi-layer two-dimensional map schematic diagram provided by an embodiment of the present application, Figure 10 is a two-dimensional image schematic diagram provided by an embodiment of the present application, Figure 11 is a boundary detection result schematic diagram provided by an embodiment of the present application; in a specific implementation, the heuristic boundary region detection process is composed of four steps: (1) construction of a 3D occupancy voxel map (i.e., a three-dimensional voxel map): first, the unmanned aerial vehicle generates a three-dimensional occupancy voxel map (as shown in Figure 8 ) using the point cloud data of the camera and the pose data of the unmanned aerial vehicle, to accurately represent the spatial distribution of the current environment. (2) voxel map layering processing: to facilitate processing, the three-dimensional voxel map is divided into multiple two-dimensional mapping maps according to the height (as shown in Figure 9 ), each height layer corresponds to a two-dimensional occupancy map. (3) map image processing: convert the two-dimensional occupancy map into a two-dimensional image (as shown in Figure 10 ), to facilitate subsequent efficient image processing operations. (4) boundary region extraction: perform image processing on the converted two-dimensional image to extract a boundary region (as shown in Figure 11 ), wherein, Figure 11 the boundary region shown in is marked with a red circle, F represents the center, and R represents the radius. This boundary region is used to represent the key region to be explored, and most of the region is unexplored space. The extracted boundary region contains both free regions and occupied space regions, to guide the exploration direction of the unmanned aerial vehicle.

[0113] Specifically, a screening threshold is set to optimize the selection of the boundary region, only high-priority regions are retained for score calculation, and the number of boundary regions participating in the calculation is reduced, thereby reducing the computational burden while ensuring the effectiveness of the exploration, and improving the real-time performance and efficiency of the exploration algorithm.

[0114] Step 2.2, incremental route map construction.

[0115] After detecting the heuristic boundary region, a probability route map is gradually constructed in each planning iteration. The goal is to ensure that the nodes in the route map can be uniformly distributed in the free space, and the newly sampled nodes can make the route map grow towards unexplored areas. Please refer to Figure 12 to Figure 14 , Figure 12 is a current route map of an unmanned aerial vehicle provided by an embodiment of the present application, Figure 13 is an incremental route map update schematic diagram provided by an embodiment of the present application, Figure 14 is an unmanned aerial vehicle route map update schematic diagram provided by an embodiment of the present application; the heuristic incremental route map construction process is as shown in Figure 12 to Figure 14 , Figure 12 represents the current route map of the unmanned aerial vehicle, Figure 13 represents the incremental route map, Figure 14represents an updated route map of the UAV; specifically, the UAV takes the current route map (R Figure 12 ), the 2D map image (M Figure 10 ) and its own pose as inputs of the incremental route map, and finally obtains the heuristic incremental route map.

[0116] In a specific implementation, the initial number of heuristic sampling failures N fail is set, and sampling is performed until the value exceeds the threshold N max . For route map node sampling, first, weighted sampling is performed on the heuristic boundary n f in the heuristic boundary region S f , and then its neighborhood N is found in the route map; then, for each neighbor n i in the neighborhood N, a candidate route map node n i is obtained by extending the neighbor n f to the heuristic boundary n i.next by a user-defined distance δ; subsequently, the route map is updated so that the route map can be efficiently extended to unknown regions, providing reliable path point references for UAV path planning.

[0117] During the sampling process, the effectiveness check ensures that the path points are within the free space and remain within an acceptable distance range from their nearest neighbor nodes. This checking process can ensure uniform distribution of route map nodes, thereby improving overall exploration efficiency and planning accuracy.

[0118] (3) Step 3 (Steps 3.1 to 3.3), an implementation path planner for dynamic scenarios.

[0119] The planner dynamically adjusts the path planning frequency according to the current obstacle density and the distance between the UAV and the target exploration point. Through this mechanism, the UAV can autonomously explore unknown environments while responding to changes in obstacles in the environment in a timely manner, ensuring flight safety. In addition, to prevent the UAV from repeatedly entering explored areas, the present application embodiment makes a maximum limit range for the heading deflection of the UAV trajectory, reducing the sharp changes in direction during flight. The path planner combines the current position of the UAV, the yaw angle deflection range and environmental information to generate a safe and efficient exploration trajectory.

[0120] Step 3.1, global path generation.

[0121] As described in step 2, the UAV continuously updates the roadmap nodes during exploration. At each planning of the exploration path, the path planner needs to generate several optimal candidate paths based on the roadmap for the exploration target point to be reached. The path score is calculated based on the node cumulative information gain obtained by the UAV from the current position to the next target point. By integrating the path points with the maximum information gain in the roadmap, the global candidate path of the UAV is constructed. The information gain calculation formula of each path node in the generated path is as follows:

[0122]

[0123] Wherein, the sensor range function sensor Range returns the onboard camera range of the UAV at the path point n i with the direction angle The information gain function IG calculates the number of unknown voxels within the sensor range of the perspective node, v represents the voxel (such as the voxel in the 3D voxel map), and V unk represents the unknown region. Then the candidate paths are scored and screened, and the highest score is selected as the global path of the UAV.

[0124] In the execution of the exploration path, if the camera detects a dynamic obstacle that may temporarily occupy the roadmap node on the generated path, causing the flight trajectory to be unsafe, the planner needs to re-plan the path by re-selecting the waypoints and dynamically adjusting the path to bypass the obstacle occupying node, ensuring that the UAV can continue to the exploration area to be detected. If the planner does not generate a path, it is considered that there is no reasonable path to the target area, i.e. it is unreachable based on the current roadmap, and at this time it will plan to go to other exploration areas.

[0125] Step 3.2, dynamic re-planning.

[0126] As described in step 3.1, when an obstacle is detected that may hinder the current path of the UAV, the path needs to be re-planned in time to ensure the safety of flight. The core idea of the dynamic re-planning method proposed by the embodiments of the present application is to dynamically adjust the frequency of trajectory planning according to the obstacle density and the distance to the target, so that the UAV can quickly adapt to changes in the environment.

[0127] Wherein, the dynamic re-planning frequency is a function of the number of obstacles and the distance of the UAV to the exploration target point. By comprehensively analyzing the current obstacle situation and the relative position to the exploration target point, the UAV can autonomously determine the re-planning frequency. This method not only updates the flight path in time to avoid potential collision risks, but also optimizes the calculation resources of path planning to ensure that the UAV completes the exploration task in an efficient and safe manner.

[0128] Specifically, the dynamic re-planning frequency is determined by the number of detected obstacles Nobs and the Euclidean distance D to the target goal The interaction between the number of dynamic obstacles and the distance between the UAV and the exploration target point determines the new replanning frequency f

[0129]

[0130] where α obs and α dis are constant parameters preset according to the environmental characteristics and the task requirements of the UAV. The comprehensive influence of these factors builds a new replanning frequency, which is defined as the following expression:

[0131]

[0132] where f replan represents the trajectory replanning frequency, β max represents the upper limit of the trajectory replanning frequency, γ base represents the scaling constant for adjusting the trajectory replanning frequency, fac obs represents the obstacle scaling factor, and fac dis represents the distance scaling factor. The calculation formula of the new replanning frequency ensures that the replanning frequency can dynamically adapt to the obstacle density in the environment and the distance between the UAV and the exploration target point. When more obstacles are detected (i.e., N obs increases), the replanning frequency will be correspondingly increased, so that the UAV can update the trajectory more frequently and quickly respond to potential risks; and as the UAV approaches the exploration target point (i.e., D goal decreases), the replanning frequency will gradually decrease, which can reduce the computational load while ensuring the efficiency of exploration.

[0133] The dynamic replanning mechanism designed in the embodiments of the present application is applicable to dynamic environments where obstacles may appear in unpredictable ways. Through this method, the UAV can achieve efficient and safe flight path planning on the basis of enhancing environmental perception ability, thereby improving the success rate of target search and the robustness of the system in the autonomous exploration process.

[0134] Step 3.3, local path planning.

[0135] In a dynamic unknown environment, the ability of the UAV to generate an efficient trajectory is the key to achieving efficient exploration. The path planner designed in the embodiments of the present application proposes an improved trajectory generation method, which further introduces a maximum yaw angle limit on the basis of considering the path information gain, thereby ensuring the smoothness and continuity of the trajectory.

[0136] Since the global path performed by the UAV is not a straight line, but includes multiple waypoints, the local trajectory needs to be constantly adjusted to go to these waypoints during flight. Therefore, the local path generation process is to dynamically determine the subsequent waypoints based on the real-time position and heading information of the UAV. The goal of local path generation is to ensure the continuity of the trajectory while satisfying the constraint condition of the change of yaw angle, avoiding unnecessary energy consumption and delay caused by sharp turning. The embodiments of the present application aim to generate a smooth and efficient exploration path, which ensures the naturalness of the trajectory and the stability of the flight by limiting the maximum yaw angle change between path points.

[0137] The calculation expression of the local trajectory adjustment process (i.e., the expression of the maximum yaw angle limit) is as follows:

[0138]

[0139] where angle change represents the yaw rotation angle of the UAV from the current position to the next waypoint, v next represents the direction vector of the UAV from the current position to the next waypoint, v current represents the current direction vector of the UAV.

[0140] Specifically, when the yaw rotation angle exceeds the set threshold, it is considered that the UAV will repeatedly explore the known area, and the selected waypoint is removed and a new waypoint is selected. By limiting the yaw angle deflection range of the UAV, the UAV can avoid going to the explored area and reduce repeated paths. This angle constraint can ensure that the UAV follows the expected flight path, reduces unnecessary yaw angle deflection, and promotes efficient exploration.

[0141] In the field of autonomous exploration of UAVs, the autonomous exploration method needs the UAV to be able to identify and distinguish between occupied space and free space in a dynamic unknown environment, and to accurately observe the target area. The embodiments of the present application propose a data perception system based on an RGB-D camera, which eliminates the dependence on laser radar, making the system lightweight and suitable for small UAV platforms with limited resources.

[0142] It should be noted that the present embodiment only briefly illustrates the general flow of the autonomous exploration method of the UAV in a dynamic unknown environment, and the detailed description of each step can refer to the related content in the foregoing embodiments, which will not be repeated here. It can be understood that the present application does not limit this.

[0143] The embodiment of the present application obtains current sensor data of the unmanned aerial vehicle; the current sensor data includes current unmanned aerial vehicle pose information, current unmanned aerial vehicle point cloud information and current environment image; the map module updates the historical three-dimensional voxel map according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information to obtain a current three-dimensional voxel map; the map module performs heuristic boundary region detection processing on the current three-dimensional voxel map to obtain a target boundary region; the map module updates the current route map according to the target boundary region to obtain a global route map; the perception module performs image processing on the current environment image to obtain image key information; the image key information includes a binary image and dynamic obstacle information; the planning module determines a target working mode of the unmanned aerial vehicle according to the binary image; the target working mode includes a detection mode and an exploration mode; if the target working mode is the detection mode, the unmanned aerial vehicle detects a target detection object in a dynamic unknown environment to obtain target detection information; wherein the detection mode is automatically switched to the exploration mode after the target detection object completes detection; if the target working mode is the exploration mode, the planning module generates a target exploration path for a target exploration point according to the target boundary region, the global route map and the dynamic obstacle information. The embodiment of the present application obtains the target boundary region through heuristic boundary region detection processing, and updates the current route map based on the target boundary region and the current three-dimensional voxel map to obtain the global route map, thereby improving the generation speed of the global route map and reducing the calculation amount; the planning module realizes seamless switching of the detection mode and the exploration mode of the unmanned aerial vehicle, can balance the needs of autonomous exploration and target inspection of the unmanned aerial vehicle, significantly improves the flexibility and efficiency of task execution of the unmanned aerial vehicle, and at the same time guarantees the efficient target search capability of the unmanned aerial vehicle in autonomous exploration in the dynamic unknown environment, so that target search is more comprehensive, and the accuracy of target search and target detection is improved; the planning module can plan the optimal exploration path for the unmanned aerial vehicle according to the target boundary region, the global route map and the dynamic obstacle information, thereby reducing the time and energy consumption required for exploration, improving the exploration efficiency, and based on the dynamic obstacle information, the risk of collision between the unmanned aerial vehicle and the dynamic obstacle is reduced, and the safety of the unmanned aerial vehicle operation is improved.

[0144] In summary, the unmanned aerial vehicle autonomous exploration method in a dynamic unknown environment proposed in the embodiment of the present application has the following advantages:

[0145] (1) The heuristic incremental probability route map technology is used for global route map construction, which improves the generation speed of the route map and reduces the calculation amount. In combination with the dynamic re-planning mechanism proposed in the embodiment of the present application, the path update frequency is dynamically adjusted according to the obstacle density and the target position, so that the flight trajectory of the unmanned aerial vehicle can be quickly adjusted to ensure the navigation safety and path stability of the unmanned aerial vehicle.

[0146] (2) A multi-task cooperative unmanned aerial vehicle autonomous exploration system is designed, which realizes seamless switching between exploration mode and detection mode, and effectively combines the mechanism of dynamically adjusting yaw angular velocity and target detection technology, not only improving the accuracy of target identification, but also ensuring the efficient target search ability of the unmanned aerial vehicle in dynamic unknown environment.

[0147] (3) By introducing the maximum yaw angle constraint, considering the exploration coverage, path yaw angle and flight distance, the unmanned aerial vehicle is designed to minimize repeated visits to the explored area, and improve the exploration efficiency of the unmanned aerial vehicle. And use camera sensing technology instead of laser radar to reduce hardware cost, while meeting the needs of efficient sensing and navigation in dynamic environment.

[0148] Please refer to Figure 15 The embodiment of the application also provides an unmanned aerial vehicle autonomous exploration device 1500 in a dynamic unknown environment, which is applied to an unmanned aerial vehicle autonomous exploration system, and the unmanned aerial vehicle autonomous exploration system comprises a map module, a perception module and a planning module, and can realize the unmanned aerial vehicle autonomous exploration method in the dynamic unknown environment. The device comprises the following modules:

[0149] A sensor data acquisition module 1501 is configured to acquire current sensor data of the unmanned aerial vehicle; the current sensor data comprises current unmanned aerial vehicle pose information, current unmanned aerial vehicle point cloud information and current environment image;

[0150] A three-dimensional voxel map updating module 1502 is configured to update a historical three-dimensional voxel map according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information through the map module, to obtain a current three-dimensional voxel map;

[0151] A boundary region extraction module 1503 is configured to perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module, to obtain a target boundary region;

[0152] A global route map updating module 1504 is configured to update a current route map according to the target boundary region through the map module, to obtain a global route map;

[0153] An image processing module 1505 is configured to perform image processing on the current environment image through the perception module, to obtain image key information; the image key information comprises a binary image and dynamic obstacle information;

[0154] An unmanned aerial vehicle working mode determination module 1506 is configured to determine a target working mode of the unmanned aerial vehicle according to the binary image through the planning module; the target working mode comprises a detection mode and an exploration mode;

[0155] The target object detection module 1507 is configured to, if the target working mode is the detection mode, control the UAV to detect a target detection object in a dynamic unknown environment to obtain target detection information; and automatically switch the detection mode to the exploration mode after the target detection object completes the detection.

[0156] The target exploration path generation module 1508 is configured to, if the target working mode is the exploration mode, generate a target exploration path for a target exploration point according to the target boundary region, the global route map and the dynamic obstacle information through the planning module.

[0157] It can be understood that the content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0158] The embodiments of the present application further provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method for autonomous exploration of a UAV in a dynamic unknown environment when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0159] It can be understood that the content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0160] Please refer to Figure 16 , Figure 16 The electronic device of another embodiment is illustrated, which includes:

[0161] The processor 1601 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute a related program to implement the technical solutions provided by the embodiments of the present application.

[0162] The memory 1602 can be implemented in the form of a Read-Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 1602 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1602 and are called and executed by the processor 1601 to perform the method of autonomous exploration of the unmanned aerial vehicle in the dynamic unknown environment according to the embodiments of the present application;

[0163] The input / output interface 1603 is configured to realize information input and output.

[0164] The communication interface 1604 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0165] The bus 1605 is configured to transmit information between various components (for example, the processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604) of the device.

[0166] The processor 1601, the memory 1602, the input / output interface 1603, and the communication interface 1604 are connected to each other through the bus 1605 to realize the communication connection between the device.

[0167] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method of autonomous exploration of the unmanned aerial vehicle in the dynamic unknown environment.

[0168] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiments, the functions specifically realized by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those of the above method embodiments.

[0169] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0170] The unmanned aerial vehicle autonomous exploration method and device in a dynamic unknown environment provided by the embodiments of the present application obtain current sensor data of the unmanned aerial vehicle; the current sensor data includes current unmanned aerial vehicle pose information, current unmanned aerial vehicle point cloud information, and a current environment image; a map module updates a historical three-dimensional voxel map according to the current unmanned aerial vehicle pose information and the current unmanned aerial vehicle point cloud information to obtain a current three-dimensional voxel map; the map module performs heuristic boundary region detection processing on the current three-dimensional voxel map to obtain a target boundary region; the map module updates a current route map according to the target boundary region to obtain a global route map; a perception module performs image processing on the current environment image to obtain image key information; the image key information includes a binary image and dynamic obstacle information; a planning module determines a target working mode of the unmanned aerial vehicle according to the binary image; the target working mode includes a detection mode and an exploration mode; if the target working mode is the detection mode, the unmanned aerial vehicle detects a target detection object in the dynamic unknown environment to obtain target detection information; wherein, the detection mode is automatically switched to the exploration mode after the target detection object completes the detection; if the target working mode is the exploration mode, the planning module generates a target exploration path for a target exploration point according to the target boundary region, the global route map, and the dynamic obstacle information. The embodiments of the present application obtain the target boundary region through heuristic boundary region detection processing, and update the current route map based on the target boundary region and the current three-dimensional voxel map to obtain the global route map, thereby improving the generation speed of the global route map and reducing the calculation amount; the planning module realizes seamless switching of the detection mode and the exploration mode of the unmanned aerial vehicle, can balance the needs of unmanned aerial vehicle autonomous exploration and target inspection, significantly improves the flexibility and efficiency of unmanned aerial vehicle task execution, and at the same time guarantees the efficient target search capability of the unmanned aerial vehicle in autonomous exploration in the dynamic unknown environment, so that the target search is more comprehensive, and the accuracy of target search and target detection is improved; the planning module can plan the optimal exploration path for the unmanned aerial vehicle according to the target boundary region, the global route map, and the dynamic obstacle information, thereby reducing the time and energy consumption required for exploration, improving the exploration efficiency, and reducing the risk of collision between the unmanned aerial vehicle and the dynamic obstacle based on the dynamic obstacle information, thereby improving the safety of unmanned aerial vehicle operation.

[0171] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs.

[0174] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0175] The preferred embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the right of the present application is not limited thereto. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the present application shall be within the scope of the right of the present application.

Claims

1. A method for autonomous exploration by unmanned aerial vehicles (UAVs) in a dynamic, unknown environment, characterized in that, The method, applied to an autonomous exploration system for unmanned aerial vehicles (UAVs), which includes a map module, a perception module, and a planning module, comprises the following steps: Acquire the current sensor data of the drone; the current sensor data includes the current drone pose information, the current drone point cloud information, and the current environmental image; The map module updates the historical 3D voxel map based on the current UAV pose information and the current UAV point cloud information to obtain the current 3D voxel map. The target boundary region is obtained by performing heuristic boundary region detection processing on the current 3D voxel map through the map module. The map module updates the current route map based on the target boundary area to obtain a global route map; The perception module performs image processing on the current environment image to obtain key image information, which includes a binarized image and dynamic obstacle information. The planning module determines the target operating mode of the UAV based on the binarized image; the target operating mode includes a detection mode and an exploration mode. If the target working mode is the detection mode, then the UAV is controlled to detect the target object in a dynamic unknown environment and obtain target detection information; wherein, after the target object is detected, the detection mode is automatically switched to the exploration mode. If the target working mode is the exploration mode, then the planning module generates a target exploration path for the target exploration point based on the target boundary area, the global route map, and the dynamic obstacle information.

2. The method according to claim 1, characterized in that, Before updating the historical 3D voxel map based on the current UAV pose information and the current UAV point cloud information using the map module to obtain the current 3D voxel map, the method further includes: Acquire historical sensor data of the UAV; wherein, the historical sensor data includes historical UAV pose information and historical UAV point cloud information; The map module constructs the historical 3D voxel map based on the historical UAV pose information and the historical UAV point cloud information.

3. The method according to claim 1, characterized in that, The step of performing heuristic boundary region detection processing on the current 3D voxel map through the map module to obtain the target boundary region includes: The map module performs voxel layering processing on the current three-dimensional voxel map to obtain the current two-dimensional voxel map. The map module performs map image processing on the current two-dimensional voxel map to obtain the current two-dimensional map image; The target boundary region is obtained by performing boundary region extraction processing on the current two-dimensional map image through the map module.

4. The method according to claim 1, characterized in that, The step of processing the current environment image through the perception module to obtain key image information includes: The current environment image is segmented using the perception module to obtain the binarized image from the key information of the image. The perception module performs depth detection processing on the current environment image to obtain the dynamic obstacle information in the key information of the image.

5. The method according to claim 4, characterized in that, The step of performing feature segmentation processing on the current environment image through the perception module to obtain the binarized image from the key information of the image includes: The current environment image is processed by color space conversion through the perception module to obtain the environment image to be segmented; The perception module performs feature segmentation processing on the environment image to be segmented according to a preset segmentation threshold to obtain the binarized image in the key information of the image; wherein, the binarized image contains the target detection object in a dynamic unknown environment.

6. The method according to claim 1, characterized in that, The step of determining the target operating mode of the UAV based on the binarized image through the planning module includes: The planning module calculates the white pixels in the binarized image to obtain a set of white pixels. If the number of white pixels is less than a preset pixel threshold, then the target working mode of the drone is determined to be the exploration mode. If the number of white pixels is greater than or equal to a preset pixel threshold, then the target working mode of the drone is determined to be the detection mode.

7. The method according to claim 1, characterized in that, If the target working mode is the exploration mode, then the planning module generates a target exploration path for the target exploration point based on the target boundary area, the global route map, and the dynamic obstacle information, including: If the target working mode is the exploration mode, then the planning module generates a set of candidate exploration paths based on the target boundary region and the global route map; The information gain is calculated by the planning module on the candidate exploration path set to obtain the information gain calculation result; The planning module filters out a target information gain path set based on the information gain calculation results and a preset information gain filtering threshold, and constructs a candidate global path set for the UAV based on the target information gain path set. The planning module selects the target global path from the set of candidate global paths. If no dynamic obstacle information is detected to obstruct the target global path in a dynamic unknown environment, then the target global path is taken as the target exploration path for the target exploration point. If the dynamic obstacle information is detected to obstruct the target global path in a dynamic unknown environment, the planning module adjusts the target global path according to the dynamic replanning method to obtain the target exploration path for the target exploration point.

8. The method according to claim 7, characterized in that, If, in a dynamically unknown environment, the dynamic obstacle information is detected as obstructing the target global path, the planning module adjusts the target global path using a dynamic replanning method to obtain the target exploration path for the target exploration point, including: If the dynamic obstacle information is detected to obstruct the target global path in a dynamic and unknown environment, the planning module calculates the density of the dynamic obstacle information to obtain the obstacle density. The planning module calculates the target exploration distance between the UAV and the target exploration point. The planning module calculates the trajectory replanning frequency of the target exploration path based on the obstacle density and the target exploration distance. The global path of the target is then adjusted according to this trajectory replanning frequency to obtain the target exploration path for the target exploration point. The formula for calculating the trajectory replanning frequency is: Among them, f replan β represents the trajectory replanning frequency. max γ represents the upper limit of the trajectory replanning frequency. base fac represents the scaling constant used to adjust the frequency of trajectory replanning. obs fac represents the obstacle scaling factor. dis This represents the distance scaling factor.

9. The method according to claim 1, characterized in that, The method further includes: If the target operating mode is the exploration mode, then the planning module generates a local exploration path for the target exploration point based on the current UAV position information and current UAV heading information; wherein, the local exploration path is used to optimize the target exploration path, and the current UAV heading information is set with a maximum yaw angle limit, the expression of which is: Among them, angle change This represents the yaw angle rotation angle that the drone will rotate from its current position to the next waypoint. next v represents the direction vector of the drone from its current position to the next waypoint. current This represents the current direction vector of the drone.

10. An autonomous exploration device for unmanned aerial vehicles (UAVs) in a dynamic, unknown environment, characterized in that: An application is made in an autonomous exploration system for unmanned aerial vehicles (UAVs), which includes a map module, a perception module, and a planning module. The device includes the following modules: The sensor data acquisition module is used to acquire the current sensor data of the UAV; the current sensor data includes the current UAV pose information, the current UAV point cloud information, and the current environmental image. The 3D voxel map update module is used to update the historical 3D voxel map based on the current UAV pose information and the current UAV point cloud information through the map module to obtain the current 3D voxel map; The boundary region extraction module is used to perform heuristic boundary region detection processing on the current three-dimensional voxel map through the map module to obtain the target boundary region; The global route map update module is used to update the current route map based on the target boundary area through the map module to obtain a global route map; The image processing module is used to perform image processing on the current environment image through the perception module to obtain key image information; the key image information includes a binarized image and dynamic obstacle information; The UAV operating mode determination module is used to determine the target operating mode of the UAV based on the binarized image through the planning module; the target operating mode includes a detection mode and an exploration mode. The target object detection module is used to control the UAV to detect target objects in a dynamic unknown environment and obtain target detection information if the target working mode is the detection mode; wherein, after the target object detection is completed, the detection mode is automatically switched to the exploration mode. The target exploration path generation module is used to generate a target exploration path for the target exploration point by means of the planning module, based on the target boundary area, the global route map and the dynamic obstacle information, if the target working mode is the exploration mode.

Images