Dense vision-based unmanned aerial vehicle autonomous obstacle avoidance method and system, and unmanned aerial vehicle

Abstract

The invention relates to an unmanned aerial vehicle autonomous obstacle avoidance method and system based on dense vision and an unmanned aerial vehicle, and belongs to the technical field of unmanned aerial vehicle obstacle avoidance. After dense visual predicted collision time of each frame of image and a corresponding unmanned aerial vehicle predicted flight state are obtained based on the image frame sequence by applying a first network model, the dense visual predicted collision time of each frame of image is converted into relative collision time; inputting the normalized relative collision time and the corresponding predicted flight state of the unmanned aerial vehicle into a second network model to obtain a plurality of candidate points of each frame of image, selecting optimal track points from the candidate points of each frame of image in an inverted order according to a shooting time sequence, sequentially connecting the optimal track points to obtain an obstacle avoidance path, and outputting the obstacle avoidance path to the unmanned aerial vehicle; the problems that in the prior art, unmanned aerial vehicle obstacle avoidance is large in calculation amount, low in efficiency and incapable of being practically applied are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle obstacle avoidance, and in particular to a method, system and unmanned aerial vehicle autonomous obstacle avoidance system based on dense vision. Background Art

[0002] In complex environments, drones usually use cameras carried on the drone to collect video streams to form drone vision. Based on drone vision, a three-dimensional obstacle avoidance map of the scene is constructed in real time through the conversion between the drone vision image frame coordinate system, camera coordinate system, and world coordinate system. Based on the real-time flight status of the drone, the motion planning algorithm is run in the three-dimensional obstacle avoidance map to achieve real-time obstacle avoidance.

[0003] In practical applications, the processing of map building based on existing technologies is very complicated, the amount of calculation is too large, and it is greatly limited for unknown or dynamic environments, resulting in incomplete maps, untimely calculations, and other problems, resulting in poor obstacle avoidance effects. On the other hand, in the existing technology, although the obstacle avoidance technology based on deep reinforcement learning bypasses the construction of obstacle avoidance maps and establishes a mapping relationship between drone vision and drone flight status through neural network training, deep learning relies on trial and error learning, which is inefficient and has poor generalization ability, and cannot be used in practical applications. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method, system and drone for autonomous obstacle avoidance of drones based on dense vision, so as to solve the problems in the prior art of large amount of computation, low efficiency and impractical application of autonomous obstacle avoidance of drones.

[0005] On the one hand, an embodiment of the present invention provides a method for autonomous obstacle avoidance of a drone based on dense vision, wherein the drone periodically shoots a video and cyclically executes the following process, which specifically includes:

[0006] S1: obtaining an image frame sequence arranged in shooting time sequence based on the current cycle video;

[0007] S2: inputting the image frame sequence into a first network model;

[0008] S3: The first network model identifies and outputs the predicted collision time and the corresponding predicted flight state of the UAV corresponding to each pixel point in each image;

[0009] S4: transforming the predicted collision time of the pixel point to obtain a normalized relative collision time of the pixel point;

[0010] S5: Obtaining the camera coordinate system coordinates of the corresponding pixel points based on the normalized relative collision time of the pixel points;

[0011] S6: inputting the image frame sequence, the normalized relative collision time of each pixel point of each image in the image frame sequence, the corresponding predicted flight state of the UAV, and the coordinates of the camera coordinate system into a second network model, and the second network model identifies and outputs multiple candidate points of each frame of the image;

[0012] S7: Find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, then connect the optimal trajectory points in sequence to obtain an obstacle avoidance path, and then output the obstacle avoidance path to the drone.

[0013] Based on the further improvement of the above method, the predicted collision time and the corresponding predicted flight status of the UAV corresponding to each pixel point of each image output by the first network model and the multiple candidate points of each image output by the second network model are sorted in chronological order according to the video shooting time.

[0014] Based on the further improvement of the above method, the first network model includes a first sub-network, a second sub-network, and a third sub-network, wherein:

[0015] Adjacent images in the image frame sequence are sequentially input into the first sub-network and the second sub-network in pairs;

[0016] The first sub-network sequentially merges each pair of adjacent images in the image frame sequence and performs feature extraction to obtain a first predicted collision time of each pixel point in each image and a corresponding predicted flight state of the UAV;

[0017] The second sub-network sequentially extracts features from each pair of adjacent images in the image frame sequence to obtain a second predicted collision time for each pixel point in each image and a corresponding predicted flight state of the UAV;

[0018] The first sub-network also obtains the third predicted collision time and the corresponding predicted flight state of the UAV for each pixel point in each corresponding image based on the second predicted collision time and the corresponding predicted flight state of the UAV and each pair of adjacent images in the image frame sequence;

[0019] The third sub-network fuses the first predicted collision time and the corresponding UAV predicted flight state, the third predicted collision time and the corresponding UAV predicted flight state to obtain the predicted collision time and the corresponding UAV predicted flight state of each pixel point in each corresponding image.

[0020] Based on the further improvement of the above method, the pixel point predicted collision time vector is transformed to obtain a normalized pixel point relative collision time vector, and the pixel point predicted collision time is transformed to obtain a normalized pixel point relative collision time, which specifically includes:

[0021] Calculating the relative collision time of the pixel point based on the predicted collision time of the pixel point;

[0022] The pixel relative collision time is normalized to obtain a normalized pixel relative collision time.

[0023] Based on the further improvement of the above method, the relative collision time of the pixel point is calculated by the following formula based on the predicted collision time of the pixel point, and the formula is specifically expressed as:

[0024] In the formula,

[0025] TTC is the relative collision time of pixels, t2-t1 is the time difference between adjacent image frames, TTC f is the predicted collision time of the pixel.

[0026] Based on the further improvement of the above method, the camera coordinate system coordinates of the corresponding pixel point are obtained by the following formula based on the normalized relative collision time of the pixel point, and the formula is specifically expressed as:

[0027]

[0028] In the formula,

[0029] is the pixel point camera coordinate system coordinate,

[0030] is the inverse of the intrinsic parameter matrix corresponding to the pixel point, where f x is the focal length of the camera on the x-axis, f y is the focal length of the camera on the y-axis, c x 、c y is the coordinate of the center point of the camera image,

[0031] It is the coordinate of the pixel point in the image frame plane coordinate system.

[0032] Based on the further improvement of the above method, the second network model extracts a group of images with a preset number of frames from the input image sequence in sequence according to the shooting sequence, and recognizes the same number of candidate points as the preset number of frames for each frame of the input image.

[0033] Based on the further improvement of the above method, the optimal trajectory point is found from the candidate points of each frame image in reverse order of shooting time, and then the optimal trajectory points are connected in sequence to obtain the obstacle avoidance path, which specifically includes:

[0034] S81: taking the candidate point of the last frame image in the shooting sequence as the candidate point of the current frame image, and finding the candidate point with the longest predicted collision time among the candidate points of the current frame image as the optimal trajectory point of the current frame;

[0035] S82: Filtering candidate points from a previous frame of the current frame image, at which the drone can safely fly to the optimal trajectory point of the current frame image;

[0036] S83: taking the candidate point with the longest predicted collision time among the selected candidate points as the optimal trajectory point of the previous frame image of the current frame image;

[0037] S84: taking the previous frame image of the current frame image as the current frame image, and returning to execute S82-S83 until the optimal trajectory point is obtained for each frame image;

[0038] S85: The route obtained by sequentially connecting the optimal trajectory points of each frame image in time sequence is the obstacle avoidance path.

[0039] On the other hand, an embodiment of the present invention provides an autonomous obstacle avoidance system for a UAV based on dense vision, the system comprising a data acquisition and preprocessing module, a first neural network, a data conversion module, a second neural network, and an obstacle avoidance path generation module, wherein:

[0040] The data acquisition and preprocessing module is used to collect videos taken by drone cameras, obtain a time-sequentially arranged image frame sequence based on the videos, and then input the image frame sequence into the first network model;

[0041] The first network model is used to obtain the predicted collision time and the corresponding predicted flight state of the UAV for each pixel point of each frame of the image based on the image frame sequence recognition;

[0042] The data conversion module is used to transform the predicted collision time of each pixel point in each frame of the image into a normalized relative collision time of the pixel point, and then obtain the camera coordinate system coordinate of the corresponding pixel point based on the normalized pixel point collision time and output it to the second network model;

[0043] The second network model is used to identify and output multiple candidate points in each frame of the image based on the normalized relative collision time of each pixel point in each image and the corresponding predicted flight state of the UAV and the corresponding camera coordinate system coordinates;

[0044] The obstacle avoidance path generation module is used to find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, and then connect the optimal trajectory points in sequence to obtain the obstacle avoidance path.

[0045] An embodiment of the present invention further provides a drone capable of autonomous obstacle avoidance based on dense vision, wherein the drone achieves autonomous obstacle avoidance by executing a method for autonomous obstacle avoidance of a drone based on dense vision disclosed in an embodiment of the present invention.

[0046] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0047] 1. There is no need to generate an obstacle avoidance map. The first network model is used to identify the video image frames periodically taken by the drone to obtain the predicted collision time vector of each pixel point in each frame (i.e., the dense visual predicted collision time vector). Based on the dense visual predicted collision time vector, the second network model is used to identify multiple candidate points for each frame, and then the optimal trajectory points found from the candidate points of each frame in reverse time order are connected in sequence to obtain the obstacle avoidance path. Not only is there no need to generate an obstacle avoidance map, but the processing of the drone's periodically shot video through the first network model and the second network model significantly reduces the amount of data processing compared to the existing technology, generates obstacle avoidance paths faster, makes obstacle avoidance path planning more reasonable, and enables the drone to respond to obstacle avoidance more promptly.

[0048] 2. The predicted collision time of each pixel point in each frame image obtained by the first network model, that is, the dense visual predicted collision time, is converted into a relative collision time and normalized, which further reduces the recognition processing speed and accuracy of the second network model and improves the speed of generating obstacle avoidance paths.

[0049] 3. In the first network model, the first subnetwork obtains the first collision time and the corresponding predicted UAV flight state corresponding to each pixel of each image by merging each pair of adjacent images in the input image frame sequence through convolution and deconvolution. The second subnetwork obtains the second collision time and the corresponding predicted UAV flight state corresponding to each pixel of each image by convolution and deconvolution on each pair of adjacent images in the input image frame sequence, and then inputs the second collision time and the corresponding predicted UAV flight state corresponding to each pixel of each image into the first subnetwork for identification. The first collision time and the corresponding predicted UAV flight state, the third collision time and the corresponding predicted UAV flight state are then fused to obtain the predicted collision time and the corresponding predicted UAV flight state of each pixel of each image. The recognition processing speed is fast and the result accuracy is high, which helps to obtain the obstacle avoidance path faster.

[0050] 4. The second network model processes the predicted collision time and the corresponding predicted flight status of the drone and the camera coordinate system coordinates of the corresponding pixels in each frame of the input image in batches, extracts a certain number of image frames in each batch according to the shooting sequence, and finds the optimal trajectory points from each frame based on the extracted candidate points of the image frames, wherein the number of image frames extracted in each batch is equal to the preset number of frames; the second network model no longer performs further learning and recognition on the image features, but directly recognizes and processes the predicted collision time and the corresponding camera coordinate system coordinates of the pixels in each frame of the input image after normalization to obtain multiple candidate points for each frame of the image, so that the data processing amount converges quickly, significantly reduces the calculation amount and complexity of the obstacle avoidance path, and improves the generation speed of the obstacle avoidance path.

[0051] 5. The reverse optimization method of finding the optimal trajectory point from the candidate points in each frame of the image in reverse order of shooting reduces the exponential collection calculation amount to logarithmic calculation amount compared to traditional linear optimization and forward optimization, further reducing the calculation amount and complexity of the obstacle avoidance path and improving the speed of generating the obstacle avoidance path.

[0052] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0054] Figure 1 This is a flow chart of the method of Example 1 of the present invention. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0056] A specific embodiment of the present invention discloses a method for autonomous obstacle avoidance of a UAV based on dense vision, such as Figure 1 shown.

[0057] The drone periodically shoots videos and cyclically executes the following process, which specifically includes:

[0058] S1: obtaining an image frame sequence arranged in shooting time sequence based on the current cycle video;

[0059] S2: inputting the image frame sequence into a first network model;

[0060] S3: The first network model identifies and outputs the predicted collision time and the corresponding predicted flight state of the UAV corresponding to each pixel point in each image;

[0061] S4: transforming the predicted collision time of the pixel point to obtain a normalized relative collision time of the pixel point;

[0062] S5: Obtaining the camera coordinate system coordinates of the corresponding pixel points based on the normalized relative collision time of the pixel points;

[0063] S6: inputting the image frame sequence, the normalized relative collision time of each pixel point of each image in the image frame sequence, the corresponding predicted flight state of the UAV, and the coordinates of the camera coordinate system into a second network model, and the second network model identifies and outputs multiple candidate points of each frame of the image;

[0064] S7: Find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, then connect the optimal trajectory points in sequence to obtain an obstacle avoidance path, and then output the obstacle avoidance path to the drone.

[0065] Each pixel in each frame of the video shot by the drone contains the image information reflecting the environmental factors that can be seen from the perspective of the drone camera at the time the image frame is generated. The image features of all pixels in each frame are called dense vision. The dense vision of the video shot by the drone not only corresponds to the perspective of the drone camera, but also corresponds to the flight status of the drone. There is a corresponding conversion relationship between the image coordinate system where the drone image is located, the pixel coordinate system of each pixel of the drone image frame, the drone camera coordinate system, and the world coordinate system. Therefore, based on the analysis of the drone's dense vision, it is possible to predict which pixel in the image frame corresponds to the obstacle avoidance point that should be selected for the next frame of the drone's perspective. The obstacle avoidance path is obtained by finding the pixels used for obstacle avoidance in all the image frames of the video and connecting them.

[0066] Specifically, the video captured by the drone is processed periodically, and the prediction results are identified and output, and then the autonomous obstacle avoidance path is generated by optimization; wherein, the video stream captured by the drone is input in sequence according to the set period length so that the length of the current period video processed can respond quickly and timely, and the longest video can be processed at one time to ensure the efficiency of data processing. The autonomous obstacle avoidance path based on the current period video is generated through the following process, which specifically includes:

[0067] Step S1: obtaining an image frame sequence arranged in shooting time sequence based on the current period video.

[0068] The current cycle video is preprocessed into a data form that can be recognized and processed by the first network model. Specifically, image frames arranged in the shooting sequence are obtained based on the current cycle video, and then each pair of adjacent image frames are grouped as an image frame according to the shooting sequence and arranged into an image frame sequence.

[0069] Step S2: Input the image frame sequence into the first network model.

[0070] Specifically, the first network model is mainly used to predict the dense vision predicted collision time and the corresponding UAV predicted flight state based on the input image frame sequence, that is, for each frame of the input image, the UAV predicted flight state and predicted collision time corresponding to each pixel in each frame of the image are predicted. The first network model has been trained in advance and has learned the dense vision and UAV flight state prediction capabilities, as well as the dense vision collision time prediction capabilities. The prior training of the first network model is specifically to construct a training set composed of a data set consisting of video images collected during the manual control of the UAV to perform obstacle avoidance operations, the UAV obstacle avoidance flight actions, and the UAV's world coordinates, and input the data into the first network model for training.

[0071] Specifically, the training of the first network model includes:

[0072] Obstacle points are obtained based on the collected video image recognition, and the projection distance from the obstacle point to the imaging plane of the drone camera is calculated. Then, according to the difference in the projection distance from the corresponding obstacle point to the camera imaging plane in the adjacent image frames, the time difference of the adjacent image frames is compared to obtain the visual collision time information of each pixel point in the adjacent image frames; at the same time, since the flight state information of the drone is accurately recorded when the obstacle avoidance action is executed, the drone flight state information specifically includes the speed, acceleration, angular velocity, angular acceleration and other information corresponding to each pixel point in the adjacent image frames. Therefore, the visual collision time information and the corresponding drone flight state information are saved in pairs to establish a training set and input into the first network model, so that the first network model can learn the relationship between the collision time of each pixel point in each pair of adjacent image frames in the video image frame and the corresponding drone predicted flight state and the corresponding drone flight state, and obtain prediction ability. After the first network model is trained with a training set established with a large amount of data, the first network model is strengthened by back propagation. The first network model finally obtained after training can obtain the predicted collision time of each pixel point of each frame of the image based on the image frame recognition of the captured video and can predict the flight state to be adopted by the corresponding drone for obstacle avoidance.

[0073] Step S3: The first network model identifies and outputs the predicted collision time and the corresponding predicted UAV flight status corresponding to each pixel point in each image.

[0074] Furthermore, the first network model includes a first sub-network, a second sub-network, and a third sub-network, wherein:

[0075] Adjacent images in the image frame sequence are sequentially input into the first sub-network and the second sub-network in pairs;

[0076] The first sub-network sequentially merges each pair of adjacent images in the image frame sequence and performs feature extraction to obtain a first predicted collision time of each pixel point in each image and a corresponding predicted flight state of the UAV;

[0077] The second sub-network sequentially extracts features from each pair of adjacent images in the image frame sequence to obtain a second predicted collision time for each pixel point in each image and a corresponding predicted flight state of the UAV;

[0078] The first sub-network also obtains the third predicted collision time and the corresponding predicted flight state of the UAV for each pixel point in each corresponding image based on the second predicted collision time and the corresponding predicted flight state of the UAV and each pair of adjacent images in the image frame sequence;

[0079] The third sub-network fuses the first predicted collision time and the corresponding UAV predicted flight state, the third predicted collision time and the corresponding UAV predicted flight state to obtain the predicted collision time and the corresponding UAV predicted flight state of each pixel point in each corresponding image.

[0080] Since the dense vision predicts the collision time for each pixel of the input image, that is, the image resolution and image features correspond to the dense vision resolution and predicted collision time results, that is to say, the first network model recognizes and processes the image frame sequence obtained based on the current period video, and predicts the dense vision collision time and the corresponding predicted drone flight state with the same image resolution based on the image features and the correspondence between the image and collision time and the drone flight state obtained by pre-training.

[0081] Specifically, the first network model includes a first sub-network, a second sub-network, and a third sub-network. Each pair of adjacent images in the image frame sequence is input into the first network model in the order of shooting, wherein each pair of adjacent images is simultaneously input into both the first sub-network and the second sub-network. The first sub-network and the second sub-network respectively identify and predict each frame image of the image frame sequence, wherein,

[0082] The first sub-network adopts an end-to-end Net-S network structure. The first sub-network merges each pair of adjacent input images into one image and then performs 6 consecutive convolution operations. Each convolution operation reduces the image resolution and increases the number of channels to extract high-order features. The merged feature map output after 6 convolution operations has fully learned the relationship between adjacent images and the positional relationship between each pixel in each image in the adjacent images. Therefore, 4 deconvolution operations are performed on this basis. Each deconvolution operation increases the resolution of the input feature map, reduces the number of channels, and generates the predicted collision time of each pixel in the image and the corresponding predicted flight status of the drone as the input for the next deconvolution. Each pixel predicted by the deconvolution operation is The collision time, the flight direction and the flight status of the UAV are all predicted simultaneously; each pixel of each image obtained after the deconvolution operation not only contains the predicted collision time and the corresponding predicted flight status of the UAV, but also contains the corresponding relationship with other pixel points in the same image, and the corresponding relationship with the pixel points at the corresponding position in the adjacent image of the image where the pixel point is located. Finally, the predicted collision time and the corresponding predicted flight status of the pixel points in the image that have not obtained the predicted collision time and the corresponding predicted flight status of the UAV are interpolated, and then the first predicted collision time and the corresponding predicted flight status of the UAV are obtained for each pixel point in each image through bilinear upsampling.

[0083] The second sub-network adopts a Net-C network structure that includes an association relationship. The second sub-network first performs three consecutive convolution operations on each image of each pair of adjacent images input. The convolution operations on adjacent images are performed in parallel. Each convolution operation reduces the resolution of the input image and increases the number of channels to extract high-order features. The feature maps of adjacent images output after the three convolutions are then associated and merged, specifically by matching and comparing the corresponding high-order features and then merging them, thereby retaining the association relationship between the feature maps of adjacent images. The merged feature maps are then subjected to six consecutive convolution operations to further extract local features. Each convolution operation still reduces the resolution of the input feature map and increases the number of channels. Four deconvolution operations are then performed, wherein the deconvolution operation of the second sub-network has the same level and function as the deconvolution operation of the first sub-network. The second sub-network adopts a network structure containing an associative relationship. It extracts features from each adjacent image through three layers of convolution and then merges them. It then extracts the merged features through three layers of convolution and then performs back propagation and collision time and corresponding UAV flight state prediction through four layers of deconvolution to obtain the second predicted collision time and corresponding UAV predicted flight state.

[0084] Furthermore, the second predicted collision time and the corresponding predicted flight state of the UAV output by the second sub-network are also input into the first sub-network together with the corresponding adjacent images to obtain the third predicted collision time and the corresponding predicted flight state of the UAV. After the first sub-network merges the input adjacent image pairs, when recognizing again, since the second predicted collision time and the corresponding predicted flight state of the UAV are input at the same time, the first sub-network can combine the predicted collision time and the corresponding predicted flight state of the corresponding image pixel points in the second predicted collision time and the corresponding predicted flight state of the UAV when recognizing and predicting the collision time and the corresponding predicted flight state of the UAV after the adjacent image pairs are merged, thereby obtaining the third predicted collision time and the corresponding predicted flight state of the UAV which are different from the first predicted collision time and the corresponding predicted flight state of the UAV.

[0085] The third sub-network fuses the first predicted collision time and the corresponding UAV predicted flight state, the third predicted collision time and the corresponding UAV predicted flight state to obtain the predicted collision time and the corresponding UAV predicted flight state of each pixel point in each corresponding image.

[0086] The predicted collision time of each pixel point of each image, that is, dense vision predicts the collision time, that is, each pixel point of each image corresponds to a predicted collision time, these collision times are arranged according to the positions of corresponding pixels to form a time matrix, each image corresponds to a time matrix, and all time matrices are sorted according to the shooting sequence of the corresponding image to form a time matrix sequence; the resolution of the matrix is ​​the same as the resolution of each frame of the image.

[0087] Step S4: transforming the pixel point predicted collision time to obtain a normalized pixel point relative collision time.

[0088] Further, the pixel point predicted collision time is transformed to obtain a normalized pixel point relative collision time, specifically including:

[0089] Calculating the relative collision time of the pixel point based on the predicted collision time of the pixel point;

[0090] The pixel relative collision time is normalized to obtain a normalized pixel relative collision time.

[0091] Specifically, the data lengths of the predicted collision time corresponding to each pixel point of each image predicted by the first network model vary greatly. Directly inputting them into the second network model is likely to cause a larger amount of calculation and error transmission, which is not conducive to obtaining accurate prediction results. Therefore, the actual predicted value of the predicted time in the predicted collision time is converted into a relative predicted value, which is convenient for processing and helps to eliminate errors.

[0092] Furthermore, the relative collision time of the pixel point is calculated based on the predicted collision time of the pixel point by the following formula, and the formula is specifically expressed as:

[0093] In the formula,

[0094] TTC is the relative collision time of pixels, t2-t1 is the time difference between adjacent image frames, TTC f is the predicted collision time of the pixel.

[0095] Then, the pixel relative collision time is calculated by the following formula to obtain the normalized pixel relative collision time, which is specifically expressed as:

[0096] Where, TTC z is the normalized relative collision time of pixels, min(TTC) is the shortest relative collision time of pixels in the current image frame, and max(TTC) is the longest relative collision time of pixels in the current image frame.

[0097] Since the processing of the predicted collision time vector does not involve the processing of the UAV's flight direction and flight state, but only the processing of the predicted collision time, it is only necessary to normalize the relative collision time to obtain the normalized pixel relative collision time. The predicted UAV flight direction and UAV flight state of the corresponding pixel do not change. In order to facilitate subsequent data processing, the relative collision time of each pixel is normalized to a constant between [0,100].

[0098] Step S5: obtaining the camera coordinate system coordinates of the corresponding pixel points based on the normalized relative collision time of the pixel points.

[0099] Specifically, the pixel collision time is a time amount defined in camera coordinates. The pixel predicted collision time vector obtained by the first network model based on image frame sequence recognition corresponds to the pixel coordinates. Since the pixel coordinate system is constantly changing for different images, the corresponding pixel camera coordinate system coordinates are obtained based on the pixel predicted collision time, which is equivalent to unifying each pixel in each image from each image coordinate system to the camera coordinate system, and the pixel predicted collision time vector can be used directly, which is beneficial to reducing the amount of subsequent obstacle avoidance path calculations.

[0100] Furthermore, based on the normalized relative collision time of the pixel point, the camera coordinate system coordinates of the corresponding pixel point are obtained by the following formula, which is specifically expressed as:

[0101]

[0102] In the formula,

[0103] is the pixel point camera coordinate system coordinate,

[0104] is the inverse of the intrinsic parameter matrix corresponding to the pixel point, where f x is the focal length of the camera on the x-axis, f y is the focal length of the camera on the y-axis, c x 、c y is the coordinate of the center point of the camera image,

[0105] It is the coordinate of the pixel point in the image frame plane coordinate system.

[0106] Step S6: The image frame sequence, the normalized relative collision time of each pixel point of each image in the image frame sequence, the corresponding predicted flight state of the UAV, and the camera coordinate system coordinates are input into the second network model, and the second network model identifies and outputs multiple candidate points for each frame of the image.

[0107] Furthermore, the second network model extracts a group of images with a preset number of frames from the input image sequence in sequence according to the shooting sequence, and recognizes the same number of candidate points as the preset number of frames for each frame of the input image.

[0108] Specifically, the second network model is an end-to-end structure composed of multiple convolutional layers, wherein the images in the image frame sequence are divided into a group of images with a preset number of frames in accordance with the shooting time sequence, and each time a group of images and each pixel point corresponding to each image in the group of images are normalized and predicted to have a collision time and the corresponding predicted flight state of the drone and the corresponding camera coordinate system coordinates are input into the second network model; the second network model recognizes the input group of images and identifies and outputs the candidate points with the same number of preset frames from each frame of the image. The second network model processes each group of input images in turn and outputs the candidate points of each frame of the image, forming a candidate point data chain arranged in time sequence.

[0109] Specifically, based on the principle of dense vision, although the direction of the change of the drone's viewing angle is the same as the change of the video shooting time sequence, the field of view and the image frame size are different. At the current moment of each image frame, the field of view formed by the projection area of ​​the cone formed by the drone camera observation point to the image frame at the current moment, in theory, a certain pixel point in the field of view should be the position point that the drone needs to fly to at the next moment. Therefore, based on dense vision optimization, it is necessary to first determine the field of view of the drone in the image frame. Generally, for each frame of the image, the drone's cone angle is the same, and the field of view is a regular plane shape field of view obtained by projecting the same cone angle on the image frame, preferably a rectangle or square. However, based on the difference in the drone's flight speed and direction at each moment of the image frame, the vertices and edges of the field of view graphics of each frame of the image correspond to the image. All pixel points need to be selected from the pixels of the same image, and a group of images with a preset number of frames are extracted from the input image sequence in sequence based on the second network model according to the shooting sequence. Then, for the input images with the preset number of frames, the corresponding field of view range of each frame is first determined; next, multiple points are identified and selected as candidate points for each pixel point of each frame of the input images with the preset number of frames, wherein the number of candidate points is equal to the preset number of frames, and the preset number of candidate points are identified for each frame of the image and the multiple candidate points corresponding to each image frame are output in sequence, wherein the second network model will also exclude candidate points that are not within the field of view range from the selected candidate points based on the field of view range corresponding to each frame of the image, to ensure that the multiple candidate points identified by the second network model for each frame of the image are within the field of view range of the image.

[0110] Furthermore, the predicted collision time and the corresponding predicted UAV flight status corresponding to each pixel point of each image output by the first network model and the multiple candidate points of each image output by the second network model are sorted in chronological order according to the video shooting time.

[0111] Specifically, the processing order of the image frame sequence based on the first network model and the second network model does not change the arrangement order of the original images, and the candidate points of each frame image obtained after the second network model recognizes each group of input images in the image frame sequence are also output in time sequence. Based on the recognition of the second network model, the number of pixels of the normalized predicted collision time of each pixel point of each input image is suddenly reduced to the preset number of frames, and the exponential optimization based on image resolution will be quickly reduced to logarithmic optimization, so that the amount of calculation of the obstacle avoidance path is greatly reduced, and the response time of the generation of the obstacle avoidance path is shorter.

[0112] Step S7: Find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, then connect the optimal trajectory points in sequence to obtain an obstacle avoidance path, and then output the obstacle avoidance path to the drone.

[0113] Specifically, based on the second network model, each image candidate point is output in sequence according to the shooting order, and the optimization process is started from the last frame of image output.

[0114] Furthermore, the step of finding the optimal trajectory point from the candidate points of each frame of the image in reverse order of the shooting time, and then sequentially connecting the optimal trajectory points to obtain the obstacle avoidance path specifically includes:

[0115] S81: taking the last frame image in the second image sequence as the current frame image, and finding the candidate point with the longest predicted collision time in the current frame image as the optimal trajectory point of the current frame;

[0116] S82: Filtering out candidate points from the previous frame image of the current frame image, at which the drone can safely fly to the optimal trajectory point of the current frame image;

[0117] S83: taking the candidate point with the longest predicted collision time among the selected candidate points as the optimal trajectory point of the previous frame image of the current frame image;

[0118] S84: taking the previous frame image of the current frame image as the current frame image, and returning to execute S82-S83 until the optimal trajectory point is obtained for each frame image;

[0119] S85: The route obtained by sequentially connecting the optimal trajectory points of each frame image in time sequence is the obstacle avoidance path.

[0120] Specifically, the optimization of the optimal obstacle avoidance path based on the candidate points of the last frame of image as the starting point is a reverse search of the data chain composed of the candidate points of each frame of image arranged in time sequence. Since the candidate points of each frame of image include the camera pose information at the corresponding moment and the information transmitted in the historical frame about the predicted collision time of each pixel at the current moment, the method of searching forward frame by frame from the last frame is equivalent to a linear optimization process that iterates forward frame by frame based on the knowledge of the camera pose and predicted collision time information of the candidate points in each image frame of the current cycle video.

[0121] When the optimal trajectory points are obtained from the candidate points of each image frame, the route obtained by connecting them in time sequence is the obstacle avoidance path, which can be output to the drone.

[0122] The drone continuously shoots videos. After the video shot in the current cycle is processed by S1-S7 to obtain the corresponding obstacle avoidance path, the video of the next cycle will be immediately input and S1-S7 will be executed. Similarly, each cycle video shot will be processed in turn and the corresponding obstacle avoidance path will be generated, so that the shooting video and the obstacle avoidance path are continuously generated.

[0123] In this embodiment, the drone continuously shoots video and periodically captures videos of the same length to obtain a sequence of image frames arranged in time sequence, and after applying the first network model to obtain the dense visual predicted collision time of each frame and the corresponding predicted flight state of the drone based on the image frame sequence, the dense visual predicted collision time of each frame is converted into a relative collision time and normalized, and then the normalized relative collision time and the corresponding predicted flight state of the drone are input into the second network model to obtain multiple candidate points for each frame, and then the optimal trajectory points are selected from the candidate points of each frame in reverse order of the shooting time, and are connected in sequence to obtain an obstacle avoidance path and output to the drone. Compared with the existing technology, not only does it not need to generate an obstacle avoidance map, but it also has a small amount of calculation, high execution efficiency, and can be put into practical application.

[0124] Embodiment 2:

[0125] A second specific embodiment of the present invention discloses an autonomous obstacle avoidance system for a UAV based on dense vision.

[0126] The system includes a data acquisition and preprocessing module, a first neural network, a data conversion module, a second neural network, and an obstacle avoidance path generation module, wherein:

[0127] The data acquisition and preprocessing module is used to collect videos taken by drone cameras, obtain a time-sequentially arranged image frame sequence based on the videos, and then input the image frame sequence into the first network model;

[0128] The first network model is used to obtain the predicted collision time and the corresponding predicted flight state of the UAV for each pixel point of each frame of the image based on the image frame sequence recognition;

[0129] The data conversion module is used to transform the predicted collision time of each pixel point in each frame of the image into a normalized relative collision time of the pixel point, and then obtain the camera coordinate system coordinate of the corresponding pixel point based on the normalized pixel point collision time and output it to the second network model;

[0130] The second network model is used to identify and output multiple candidate points in each frame of the image based on the normalized relative collision time of each pixel point in each image and the corresponding predicted flight state of the UAV and the corresponding camera coordinate system coordinates;

[0131] The obstacle avoidance path generation module is used to find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, and then connect the optimal trajectory points in sequence to obtain the obstacle avoidance path.

[0132] Generally, a drone is provided with a camera, a processor, and a positioning module, wherein the camera is used to shoot video to obtain dense vision and send it to the processor, and the positioning module can obtain the real-time flight status of the drone and send it to the processor, wherein the processor obtains an obstacle avoidance path based on the dense vision and the real-time flight status of the drone by executing the drone autonomous obstacle avoidance method based on dense vision disclosed in Example 1.

[0133] Specifically, the present embodiment discloses an autonomous obstacle avoidance system for a UAV based on dense vision, which specifically relates to a processor for a UAV based on shot videos and real-time flight status information. The system is deployed in the processor of the UAV. Without changing the functions and structural composition of the existing UAV, the implementation of the autonomous obstacle avoidance system for a UAV based on dense vision can be completed by improving the program deployed in the processor. Compared with the existing technology, the system can be quickly put into practical use without changing the existing configuration of the UAV, and can quickly obtain an obstacle avoidance path based on the shot video.

[0134] Embodiment 3:

[0135] A third specific embodiment of the present invention discloses a UAV capable of autonomous obstacle avoidance based on dense vision. The UAV achieves autonomous obstacle avoidance by executing a method for autonomous obstacle avoidance of a UAV based on dense vision disclosed in Example 1.

[0136] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0137] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for autonomous obstacle avoidance of a UAV based on dense vision, characterized in that: The drone periodically shoots videos and cyclically executes the following process, which specifically includes: S1: obtaining an image frame sequence arranged in shooting time sequence based on the current cycle video; S2: inputting the image frame sequence into a first network model; S3: The first network model identifies and outputs the predicted collision time and the corresponding predicted flight state of the UAV corresponding to each pixel point in each image; S4: transforming the predicted collision time of the pixel point to obtain a normalized relative collision time of the pixel point; S5: Obtaining the camera coordinate system coordinates of the corresponding pixel points based on the normalized relative collision time of the pixel points; S6: inputting the image frame sequence, the normalized relative collision time of each pixel point of each image in the image frame sequence, the corresponding predicted flight state of the UAV, and the coordinates of the camera coordinate system into a second network model, and the second network model identifies and outputs multiple candidate points of each frame of the image; S7: Find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, then connect the optimal trajectory points in sequence to obtain an obstacle avoidance path, and then output the obstacle avoidance path to the drone.

2. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 1, characterized in that: The predicted collision time and the corresponding predicted flight state of the UAV corresponding to each pixel point of each image output by the first network model and the multiple candidate points of each image output by the second network model are sorted in chronological order according to the video shooting time.

3. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 2 is characterized in that: The first network model includes a first sub-network, a second sub-network, and a third sub-network, wherein: Adjacent images in the image frame sequence are sequentially input into the first sub-network and the second sub-network in pairs; The first sub-network sequentially merges each pair of adjacent images in the image frame sequence and performs feature extraction to obtain a first predicted collision time of each pixel point in each image and a corresponding predicted flight state of the UAV; The second sub-network sequentially extracts features from each pair of adjacent images in the image frame sequence to obtain a second predicted collision time for each pixel point in each image and a corresponding predicted flight state of the UAV; The first sub-network also obtains the third predicted collision time and the corresponding predicted flight state of the UAV for each pixel point in each corresponding image based on the second predicted collision time and the corresponding predicted flight state of the UAV and each pair of adjacent images in the image frame sequence; The third sub-network fuses the first predicted collision time and the corresponding UAV predicted flight state, the third predicted collision time and the corresponding UAV predicted flight state to obtain the predicted collision time and the corresponding UAV predicted flight state of each pixel point in each corresponding image.

4. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 3 is characterized in that: Transforming the pixel point predicted collision time to obtain a normalized pixel point relative collision time specifically includes: Calculating the relative collision time of the pixel point based on the predicted collision time of the pixel point; The pixel relative collision time is normalized to obtain a normalized pixel relative collision time.

5. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 4 is characterized in that: Based on the predicted collision time of the pixel point, the relative collision time of the pixel point is calculated by the following formula, and the formula is specifically expressed as: In the formula, TTC is the relative collision time of pixels, t2-t1 is the time difference between adjacent image frames, TTC f is the predicted collision time of the pixel.

6. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 5, characterized in that: Based on the normalized relative collision time of the pixel point, the camera coordinate system coordinates of the corresponding pixel point are obtained by the following formula, which is specifically expressed as: In the formula, is the pixel point camera coordinate system coordinate, is the inverse of the intrinsic parameter matrix corresponding to the pixel point, where f x is the component of the camera focal length on the x-axis, f y is the component of the camera focal length on the y-axis, c x 、c y is the coordinate of the center point of the camera image, It is the coordinate of the pixel point in the image frame plane coordinate system.

7. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 6, characterized in that: The second network model extracts a group of images with a preset number of frames from the input image sequence in sequence according to the shooting sequence, and recognizes each frame of the input image to obtain candidate points with the same number of preset frames.

8. The method for autonomous obstacle avoidance of a UAV based on dense vision according to claim 7, characterized in that: The step of finding the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, and then connecting the optimal trajectory points in sequence to obtain an obstacle avoidance path specifically includes: S81: taking the candidate point of the last frame image in the shooting sequence as the candidate point of the current frame image, and finding the candidate point with the longest predicted collision time among the candidate points of the current frame image as the optimal trajectory point of the current frame; S82: Filtering candidate points from a previous frame of the current frame image, at which the drone can safely fly to the optimal trajectory point of the current frame image; S83: taking the candidate point with the longest predicted collision time among the selected candidate points as the optimal trajectory point of the previous frame image of the current frame image; S84: taking the previous frame image of the current frame image as the current frame image, and returning to execute S82-S83 until the optimal trajectory point is obtained for each frame image; S85: The route obtained by sequentially connecting the optimal trajectory points of each frame image in time sequence is the obstacle avoidance path.

9. An autonomous obstacle avoidance system for unmanned aerial vehicles based on dense vision, characterized in that: The system includes a data acquisition and preprocessing module, a first neural network, a data conversion module, a second neural network, and an obstacle avoidance path generation module, wherein: The data acquisition and preprocessing module is used to collect videos taken by drone cameras, obtain a time-sequentially arranged image frame sequence based on the videos, and then input the image frame sequence into the first network model; The first network model is used to obtain the predicted collision time and the corresponding predicted flight state of the UAV for each pixel point of each frame of the image based on the image frame sequence recognition; The data conversion module is used to transform the predicted collision time of each pixel point in each frame of the image into a normalized relative collision time of the pixel point, and then obtain the camera coordinate system coordinate of the corresponding pixel point based on the normalized pixel point collision time and output it to the second network model; The second network model is used to identify and output multiple candidate points in each frame of the image based on the normalized relative collision time of each pixel point in each image and the corresponding predicted flight state of the UAV and the corresponding camera coordinate system coordinates; The obstacle avoidance path generation module is used to find the optimal trajectory point from the candidate points of each frame image in reverse order of shooting time, and then connect the optimal trajectory points in sequence to obtain the obstacle avoidance path.

10. A UAV based on dense vision autonomous obstacle avoidance, characterized in that: The drone achieves autonomous obstacle avoidance by executing a drone autonomous obstacle avoidance method based on dense vision as described in any one of claims 1 to 8.

Images