Unmanned aerial vehicle flight control method and system based on passable path determination

Abstract

The application provides a kind of unmanned aerial vehicle flight control method and system based on passable path determination, comprising: obtaining the environmental image information of unmanned aerial vehicle traffic direction and the IMU information of unmanned aerial vehicle;Through the environmental image information and IMU information, the unmanned aerial vehicle is positioned and the environment is simultaneously mapped, and the region range of the target to be passed through is identified;According to the region range, determine the passable path, and judge whether the unmanned aerial vehicle can pass through, if yes, control unmanned aerial vehicle to pass through;If not, the passable path is determined again.The method proposed in the application can quickly find the passable path for the unmanned aerial vehicle in unknown environment, has higher accuracy, can identify false passable path, and avoid errors in unmanned aerial vehicle flight when false information is collected.

Description

Technical Field

[0001] This invention relates to the field of UAV perception, identification, and positioning technology, specifically to a UAV flight control method and system based on traversable path determination. More particularly, it relates to a method and system for a UAV to determine the traversable range of a target gate in an unknown environment. Background Technology

[0002] With the continuous development of technology, drone technology has become increasingly mature, and its application areas are becoming more and more extensive. For example, drones are used to explore unknown environments and to achieve long-distance target tracking. Among the various applications in different environments and fields, the use of drones in unknown indoor environments has become a hot topic in recent years. For drones to perform tasks indoors, they must be able to identify and determine their own position and the traversable paths within the environment.

[0003] In unknown environments, the task of determining a passable path during drone flight involves localization, perception, and decision-making simultaneously, and existing methods are relatively independent. To identify the image region of a door using a camera, object detection methods are needed. Object detection is one of the core problems in computer vision; its core task is to find all objects of interest in an image and determine their category and location. Because objects have different appearances, shapes, and states in various environments, and are also affected by factors such as lighting and occlusion, object detection has always been a challenging problem.

[0004] In an unknown indoor environment, identifying a passable path can be simplified to a drone finding a path to another room, or in other words, locating an open door. To find a passable door, the first step is to locate it. In indoor environments, drones typically use onboard cameras for identification and positioning during flight. Locating the door involves identifying the area containing the door within the image frames captured by the camera, and then determining its specific coordinates in the spatial area. The degree to which the door is open also needs to be calculated.

[0005] Patent document CN114326821A discloses an autonomous obstacle avoidance system and method for unmanned aerial vehicles (UAVs) based on deep reinforcement learning. This autonomous obstacle avoidance method adopts a deep reinforcement learning model based on policy iteration. It uses the original RGB images captured by the UAV's monocular camera as training data, without the need for complex point clouds or other 3D information. It obtains depth image information by training the original RGB images through a fully convolutional neural network, and then analyzes and predicts the images through a policy iteration-based reinforcement learning method to predict the UAV's flight actions in advance, thereby achieving autonomous obstacle avoidance.

[0006] However, patent document CN114326821A has the problem that it encounters false images, which makes it impossible to correctly identify the obstacle and thus leads to incorrect obstacle avoidance actions.

[0007] Patent document CN104765360A discloses an image recognition-based autonomous flight system for unmanned aerial vehicles (UAVs), which is applied to autonomous flight missions consisting of n UAVs and a ground command center. Its features include: a flight communication module, a ground communication module, an input module, an FFMPEG module, a video display module, an image recognition module, a command generation module, and an output module. Both the flight communication module and the ground communication module include UDP 5556 and TCP 5555 ports.

[0008] However, patent document CN104765360A has the problem of not being able to fly autonomously, and requires the existence of a ground command center. Summary of the Invention

[0009] In view of the deficiencies in the prior art, the purpose of this invention is to provide a flight control method and system for unmanned aerial vehicles (UAVs) based on the determination of passable paths.

[0010] A UAV flight control method based on traversable path determination provided by the present invention includes:

[0011] Step S1: Acquire environmental image information and the UAV's IMU information in the direction the UAV is traveling;

[0012] Step S2: Locate the UAV and simultaneously map the environment using the environmental image information and IMU information, while identifying the area range of the target to be passed;

[0013] Step S3: Determine a passable path based on the area range and determine whether the drone can pass through. If yes, control the drone to pass; if not, re-determine a passable path.

[0014] Preferably, the target to be traversed includes interior doors, tunnels, and caves;

[0015] The environmental image information includes image frames in front of the UAV for communication, and the IMU information includes the acceleration and angular velocity of the UAV itself.

[0016] Preferably, identifying the area range of the target to be passed includes the following sub-steps:

[0017] Step S2.1: Build a deep learning model;

[0018] Step S2.2: Use the annotation program to annotate the current image frame to obtain the bounding box of the target to be passed;

[0019] Step S2.3: Use the bounding box of the target to be passed and the COCO dataset as the training data of the model to train the model and obtain a model about the target to be passed;

[0020] Step S2.4: Identify the area range of the target to be passed through using the model of the target to be passed.

[0021] Preferably, image information is received at preset intervals and stored in local storage space for comparison and judgment when a target to be passed is identified; the preset interval includes 1 second.

[0022] Preferably, determining the passable path based on the area range includes the following sub-steps:

[0023] Step S3.1: Mark the spatial locations of the targets to be passed within the area on the established map using the location determination module;

[0024] Step S3.2: Control the UAV to fly using the flight control module to obtain images of the target to be passed from different angles;

[0025] Step S3.3: Compare the image feature points to determine whether the current state of the target to be passed is passable. If yes, trigger step S3.4; otherwise, re-determine the passable path.

[0026] Step S3.4: Calculate the size of the currently passable range by using the distance between the feature point set of the target to be passed and the current position of the UAV, and determine whether the current range is sufficient for the UAV to pass. If yes, issue a decision command to control the UAV to pass; otherwise, re-determine the passable path.

[0027] A UAV flight control system based on traversable path determination according to the present invention includes:

[0028] Module M1: Acquires environmental image information and IMU information of the UAV in the direction of travel;

[0029] Module M2: Uses the environmental image information and IMU information to locate the UAV and simultaneously map the environment, while identifying the area range of the target to be passed;

[0030] Module M3: Determines a passable path based on the area range and determines whether the drone can pass through. If so, it controls the drone to pass through; otherwise, it re-determines a passable path.

[0031] Preferably, the target to be traversed includes interior doors, tunnels, and caves;

[0032] The environmental image information includes image frames in front of the UAV for communication, and the IMU information includes the acceleration and angular velocity of the UAV itself.

[0033] Preferably, the area range for identifying the target to be passed includes the following sub-modules:

[0034] Module M2.2: Uses an annotation program to annotate the current image frame to obtain the bounding box of the target to be passed;

[0035] Module M2.3: The bounding box of the target to be passed and the COCO dataset are used as training data for the model to train the model and obtain a model about the target to be passed.

[0036] Module M2.4: Identifies the area range of the target to be passed through using the model of the target to be passed.

[0037] Preferably, image information is received at preset intervals and stored in local storage space for comparison and judgment when a target to be passed is identified; the preset interval includes 1 second.

[0038] Preferably, the step of determining the passable path based on the area range includes the following sub-modules:

[0039] Module M3.1: The location determination module marks the spatial locations of the targets to be passed within the area on the established map;

[0040] Module M3.2: Controls the UAV's flight via the flight control module to obtain images of the target to be passed from different angles;

[0041] Module M3.3: Compare the image feature points to determine whether the current target to be passed is passable. If yes, then trigger module M3.4; otherwise, re-determine the passable path.

[0042] Module M3.4: Calculates the size of the currently passable range by using the distance between the set of feature points of the target to be passed and the current position of the UAV, and determines whether the current range is sufficient for the UAV to pass. If so, it issues a decision command to control the UAV to pass; otherwise, it redetermines the passable path.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The method proposed in this invention enables UAVs to quickly find a passageway in unknown environments with high accuracy, and can also identify false passageways, thus avoiding errors in UAV flight when false information is collected.

[0045] 2. The method proposed in this invention supports recognition using only a camera, which can be adapted to most devices and reduce the limitations caused by the device.

[0046] 3. The method proposed in this invention is easy to use, requires no complicated operation, avoids operational errors, and enables UAVs to fly autonomously, locate and identify in unknown indoor environments. Attached Figure Description

[0047] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a schematic diagram illustrating the data interaction between the modules of the method for determining a passable path for an unmanned aerial vehicle in an unknown environment provided by the present invention.

[0049] Figure 2 This is a schematic diagram of the workflow of the present invention.

[0050] Figure 3 This is a schematic diagram of the network structure of the prediction model in this invention.

[0051] Figure 4 This is an example diagram of the method used in this invention to determine the size range of a passable path. Detailed Implementation

[0052] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0053] This invention comprises two phases: an information acquisition phase and a path search phase. The information acquisition phase involves the drone operating in an unknown environment, acquiring environmental information through sensors, using an IMU and onboard camera to obtain the drone's angular velocity, acceleration, and image information, and simultaneously performing localization and mapping of the surroundings. The path search phase involves transmitting the image information acquired by the drone back to the ROS system, processing the image, identifying the path, determining whether the drone can traverse the path, locating targets in the image using sensor information, finding their specific spatial coordinates, and then controlling the drone to traverse the path.

[0054] According to the present invention, a flight control method for unmanned aerial vehicles (UAVs) based on traversable path determination is provided, such as... Figure 1 and Figure 2 As shown, it includes:

[0055] Step S1: Acquire environmental image information and IMU information of the drone's travel direction. The environmental image information includes image frames in front of the drone, and the IMU information includes the drone's acceleration and angular velocity. This information can be acquired using the drone's onboard camera.

[0056] Step S2: The UAV is located and the environment is simultaneously mapped using the environmental image information and IMU information, while the area of ​​the target to be passed is identified. The target includes indoor doors, tunnels, and caves. Identifying the area of ​​the target includes the following sub-steps:

[0057] Step S2.1: Build a deep learning model.

[0058] Step S2.2: Use the annotation program to annotate the current image frame to obtain the bounding box of the target to be passed. Image information is received at preset time intervals and stored in local storage space for comparison and judgment when the target to be passed is identified; the preset time interval includes 1 second.

[0059] Step S2.3: Use the bounding box of the target to be passed and the COCO dataset as the training data of the model to train the model and obtain a model about the target to be passed.

[0060] Step S2.4: Identify the area range of the target to be passed through using the model of the target to be passed.

[0061] Step S3: Determine a passable path based on the defined area range and determine whether the drone can pass. If yes, control the drone to pass; otherwise, re-determine the passable path. Determining the passable path based on the defined area range includes the following sub-steps:

[0062] Step S3.1: Mark the spatial location of the target to be passed within the area on the established map using the location determination module.

[0063] Step S3.2: Control the UAV to fly using the flight control module to obtain images of the target to be passed from different angles.

[0064] Step S3.3: Compare the image feature points to determine whether the current state of the target to be passed is passable. If not, redetermine the passable path. If yes, calculate the current passable range based on the distance between the feature point set of the target to be passed and the current position of the UAV, and determine whether the current range is sufficient for the UAV to pass. If yes, issue a decision command to control the UAV to pass. If not, redetermine the passable path.

[0065] Furthermore, assuming the current target is an interior door, combined with the attached... Figure 3 The present invention is described in detail below:

[0066] First, the drone flies autonomously in an unknown environment, collecting information through its onboard camera and IMU. The collected information includes image frames from the front of the drone and the drone's own acceleration and angular velocity. These parameters are then published through the ROS system in formats such as " / image_raw" and " / imu / data". These formats are customizable, ensuring the topic names are clear and easy to understand. Furthermore, the " / image_raw" message is published in a matrix format storing images, and real-time image information is displayed via Rviz.

[0067] Then, after acquiring the information, the backend system subscribes to published images, acceleration, and angular velocity data from the ROS system to the topic, using this data for SLAM simultaneous localization and mapping. Simultaneously, this acquired information is also used by the target recognition module. In this example, the Mask R-CNN algorithm is used for target recognition, resulting in a mask matrix related to the target. This matrix uses 0 and 1 to represent pixels of the target and non-target respectively. In this example, the recognition function is `draw_mask()`. The prediction model required by the algorithm in this example is trained on a Mask R-CNN network, with the network structure as follows: Figure 3 The diagram illustrates the basic connections of each layer in the network. The Convolutional backbone represents a series of convolutional layers, the RPN represents the Region Proposal Network, the feature map represents the feature map itself, the RoIAlign layer represents a region feature aggregation method in the algorithm, the Fixed-size feature map represents the feature map of the anchor box, the Fully connected layers represent fully connected layers that provide the Box Regression coordinates and Classification labels for the anchor boxes, and the Maskbranch represents a branch of the algorithm's output Mask image. An annotation program was used to annotate the training images collected in the pre-defined environment, marking the target bounding boxes. The model was then trained using the COCO dataset to obtain a relatively accurate model for target gates, used for target recognition. Because the speed of target recognition differs from that of image acquisition, image information is received every 1 second and stored locally for comparison and judgment when target gates are identified.

[0068] Next, the drone's flight control module controls the drone to move in a certain direction to locate the target door. Once the target door is identified in the acquired images, the position determination module marks its spatial location on the established map. By controlling the drone's flight through the flight control module, images of the target door from different angles are obtained. The open / closed state of the door is determined by comparing the differences in feature points within the door's mask between the two images. In this example, ORB feature points are used to quickly acquire feature information; when the difference in feature points between the two images is too large, the door is determined to be open.

[0069] Next, in the UAV flight decision module, when the door is open, the size of the door's opening range is determined. The feature points of the door's opening range are denoted as set P, and the feature points of the door's edge are denoted as set D. Through the UAV's movement, the depth information between the current position and the midpoint of set D can be obtained from image information; that is, the distance between the current position and the UAV. Based on the distance between the midpoint of set P and the UAV, the spatial size of the area occupied by set P is calculated, which is the size of the door's opening range. The calculation process is as follows:

[0070]

[0071] Combination Figure 4 Image represents the captured image. In the formula, L represents the size of the passable area, L1 is the distance between the leftmost point and the drone, L2 is the distance between the rightmost point and the drone, θ is the angle between L1 and L2, which also represents the camera's field of view. a represents the pixel width of the passable area, and b is the pixel width of the entire image.

[0072] Finally, when the calculated range is suitable for the drone to pass through, a decision command is issued to control the drone to traverse the area.

[0073] The present invention also provides a UAV flight control system based on traversable path determination. Those skilled in the art can implement the UAV flight control system based on traversable path determination by executing the steps of the UAV flight control method based on traversable path determination. That is, the UAV flight control method based on traversable path determination can be understood as a preferred embodiment of the UAV flight control system based on traversable path determination.

[0074] A UAV flight control system based on traversable path determination according to the present invention includes:

[0075] Module M1: Acquires environmental image information and IMU information of the UAV's travel direction. The environmental image information includes image frames in front of the UAV, and the IMU information includes the acceleration and angular velocity of the UAV itself.

[0076] Module M2: Uses the environmental image information and IMU information to locate the UAV and simultaneously map the environment, while identifying the area range of targets to be passed. These targets include indoor doors, tunnels, and caves. The identification of the target area includes the following sub-modules, which receive image information at preset time intervals and store it in local storage for comparison and judgment when a target is identified; the preset time interval includes 1 second.

[0077] Module M2.2: Uses an annotation program to annotate the current image frame to obtain the bounding box of the target to be passed.

[0078] Module M2.3: The bounding box of the target to be passed and the COCO dataset are used as training data for the model to train the model and obtain a model about the target to be passed.

[0079] Module M2.4: Identifies the area range of the target to be passed through using the model of the target to be passed.

[0080] Module M3: Determines a passable path based on the defined area range and determines whether the drone can pass. If so, it controls the drone to proceed; otherwise, it re-determines the passable path. The process of determining a passable path based on the defined area range includes the following sub-modules:

[0081] Module M3.1: The location determination module marks the spatial location of the target to be passed within the area on the established map.

[0082] Module M3.2: Controls the UAV's flight through the flight control module, obtaining images of the target to be passed from different angles.

[0083] Module M3.3: Compare the image feature points to determine whether the current state of the target to be passed is passable. If not, redetermine the passable path. If yes, calculate the current passable range based on the distance between the feature point set of the target to be passed and the current position of the UAV, and determine whether the current range is sufficient for the UAV to pass. If yes, issue a decision command to control the UAV to pass. If not, redetermine the passable path.

[0084] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0085] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A UAV flight control method based on traversable path determination, characterized in that, include: Step S1: Acquire environmental image information and the UAV's IMU information in the direction the UAV is traveling; Step S2: Locate the UAV and simultaneously map the environment using the environmental image information and IMU information, while identifying the area range of the target to be passed; Step S3: Determine a passable path based on the area range and determine whether the drone can pass through. If yes, control the drone to pass; if not, re-determine a passable path. Determining a passable path based on the area includes the following sub-steps: Step S3.1: Mark the spatial locations of the targets to be passed within the area on the established map using the location determination module; Step S3.2: Control the UAV to fly using the flight control module to obtain images of the target to be passed from different angles; Step S3.3: Compare the image feature points to determine whether the current state of the target to be passed is passable. If yes, trigger step S3.4; otherwise, re-determine the passable path. Step S3.4: Calculate the size of the currently passable range by using the distance between the feature point set of the target to be passed and the current position of the UAV, and determine whether the current range is sufficient for the UAV to pass. If yes, issue a decision command to control the UAV to pass; otherwise, re-determine the passable path.

2. The UAV flight control method based on passable path determination according to claim 1, characterized in that, The targets to be traversed include interior doors, tunnels, and caves; The environmental image information includes image frames in front of the UAV for communication, and the IMU information includes the acceleration and angular velocity of the UAV itself.

3. The UAV flight control method based on passable path determination according to claim 1, characterized in that, The process of identifying the area of ​​the target to be passed includes the following sub-steps: Step S2.1: Build a deep learning model; Step S2.2: Use the annotation program to annotate the current image frame to obtain the bounding box of the target to be passed; Step S2.3: Use the bounding box of the target to be passed and the COCO dataset as the training data of the model to train the model and obtain a model about the target to be passed; Step S2.4: Identify the area range of the target to be passed through using the model of the target to be passed.

4. The UAV flight control method based on traversable path determination according to claim 3, characterized in that, Image information is received at preset intervals and stored in local storage space for comparison and judgment when a target to be passed is identified; the preset interval includes 1 second.

5. A UAV flight control system based on traversable path determination, characterized in that, include: Module M1: Acquires environmental image information and IMU information of the UAV in the direction of travel; Module M2: Uses the environmental image information and IMU information to locate the UAV and simultaneously map the environment, while identifying the area range of the target to be passed; Module M3: Determines a passable path based on the area range and determines whether the drone can pass through. If so, it controls the drone to pass; otherwise, it re-determines a passable path. The step of determining a passable path based on the area range includes the following sub-modules: Module M3.1: The location determination module marks the spatial locations of the targets to be passed within the area on the established map; Module M3.2: Controls the UAV's flight via the flight control module to obtain images of the target to be passed from different angles; Module M3.3: Compare the image feature points to determine whether the current state of the target to be passed is passable. If so, trigger module M3.

4. If not, then redetermine the passable path; Module M3.4: Calculates the size of the currently passable range by using the distance between the set of feature points of the target to be passed and the current position of the UAV, and determines whether the current range is sufficient for the UAV to pass. If so, it issues a decision command to control the UAV to pass; otherwise, it redetermines the passable path.

6. The UAV flight control system based on passable path determination according to claim 5, characterized in that, The targets to be traversed include interior doors, tunnels, and caves; The environmental image information includes image frames in front of the UAV for communication, and the IMU information includes the acceleration and angular velocity of the UAV itself.

7. The UAV flight control system based on traversable path determination according to claim 5, characterized in that, The area range for identifying the target to be passed includes the following sub-modules: Module M2.1: Building a deep learning model; Module M2.2: Uses an annotation program to annotate the current image frame to obtain the bounding box of the target to be passed; Module M2.3: The bounding box of the target to be passed and the COCO dataset are used as training data for the model to train the model and obtain a model about the target to be passed. Module M2.4: Identifies the area range of the target to be passed through using the model of the target to be passed.

8. The UAV flight control system based on passable path determination according to claim 7, characterized in that, Image information is received at preset intervals and stored in local storage space for comparison and judgment when a target to be passed is identified; the preset interval includes 1 second.

Images