Unmanned aerial vehicle route identification method

Abstract

The invention discloses an unmanned aerial vehicle route identification method, and belongs to the technical field of unmanned aerial vehicle route identification, and the technical scheme of the unmanned aerial vehicle route identification method comprises the following steps: S1, route mapping; s2, probability model training; s3, probability estimation and processing; s4, calculating the main direction of the route; s5, Rect area probability data are extracted in the main direction; s6, extracting a route; according to the unmanned aerial vehicle route identification method, the route information is associated with the image, the unmanned aerial vehicle can be guided to fly along the route through image identification, results at all moments are mutually independent, no accumulated error influence exists, and the unmanned aerial vehicle route identification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV route recognition, and specifically provides a UAV route recognition method. Background Technique

[0002] UAV navigation and positioning technology plays a crucial role in the flight safety of UAVs. The most commonly used method is to use satellite positioning systems such as GPS and Beidou to provide the aircraft's position. However, in some scenarios with terrain, meteorological, and complex electromagnetic field environments, satellite positioning will be interfered with and fail.

[0003] For example, the patent with the publication number CN109211241A discloses a UAV autonomous positioning method based on visual SLAM, which consists of a feature extraction and matching for motion solution part, an image and inertial measurement unit imu fusion part, and an estimation of 3D point depth part; in the motion solution part, a strategy of combining the feature point method and the direct method is adopted. After selecting key frames, point features and line features are extracted, and then the relative pose is calculated by minimizing the error; in the image and inertial measurement unit imu fusion part, it is fused by minimizing the error; the last step is the estimation of 3D point depth part. Based on the matching of the selected feature points, a triangulation method is used to solve the 3D position of the point, that is, the depth value of the point is obtained.

[0004] Another example is the patent with the publication number CN119845257A, which discloses a vision-inertial autonomous positioning method and system for fixed-wing UAVs that fuses satellite positioning, in which satellite positioning, visual sensors, and IMU data are fused to obtain the pose for fixed-wing UAVs; however, in the case of satellite denial, long-distance flight will cause cumulative errors, resulting in the gradual divergence of the positioning results and affecting the accurate recognition of the UAV route.

[0005] In view of the above problems, there is an urgent need to innovate and design on the basis of the original UAV route recognition method. Summary of the Invention

[0006] The purpose of the present invention is to provide a UAV route recognition method to solve the problem that in the case of satellite denial, long-distance flight will cause cumulative errors, resulting in the gradual divergence of the positioning results and affecting the accurate recognition of the UAV route as proposed in the above background technique.

[0007] To achieve the above purpose, the present invention provides the following technical solution: A UAV route recognition method, and the technical solution includes the following steps:

[0008] S1. Route Mapping

[0009] Extract the set of route trajectory points of the aircraft , where n is the number of trajectory points.

[0010] Retrieve all the images captured during the flight and the corresponding pose information, and project the trajectory points onto each image.

[0011] S2. Probability model training

[0012] Construct a dataset containing the original image set and the probability atlas. For an original image M, create an image M' of the same size. Using the original image set as the input and the probability image set as the output, train a neural network to obtain a probability model. Take the unet as an example for model training.

[0013] S3. Probability estimation and processing

[0014] Given an input image M0 to be predicted, use the above probability model for prediction and output the probability map M1.

[0015] Perform Gaussian filtering on the probability map M1 to remove obvious outliers.

[0016] Binarize M1, set the pixels greater than or equal to the threshold to 1 and the pixels less than the threshold to 0 to obtain the image M2.

[0017] Then perform dilation and erosion operations on M2 to obtain M3.

[0018] S4. Calculation of the main flight path direction

[0019] Extract the contours from M3, traverse and find the contour with the largest area contour; calculate the minimum bounding rectangle and direction of the largest contour.

[0020] S5. Extract probability data of the Rect region along the main direction

[0021] Calculate the corresponding key point sets before and after extracting the Rect region and .

[0022] Calculate the transformation matrix T1 and the inverse transformation matrix T2.

[0023] Use T1 to transform the image M3 and fill the boundary region with the value V to obtain the image M4, which is the probability data map of the Rect region and is a matrix with width and height w and h respectively.

[0024] S6. Extract the flight path

[0025] For the image M4, extract the left and right edge points of the binary image from top to bottom and calculate the corresponding flight path point coordinates.

[0026] Preferably, in the S1, for the image M and the k-th trajectory point , calculate The projection point on M Coordinates of:

[0027]

[0028] where K is the camera parameter matrix, is the camera rotation matrix at the image capture moment, is the camera position at the image capture moment.

[0029] Preferably, the width W and height H of the image are obtained.

[0030] If 0 <= u < W and 0 <= v < H, then the point is within the current image and is a valid point; otherwise, the point is discarded .

[0031] After projecting all the trajectory points, the set of valid points is the set of projection points of the flight path on the image M .

[0032] Each image has a corresponding set of projection points.

[0033] Preferably, in S2, the calculation method of each pixel value of the image M' is as follows:

[0034]

[0035] where u and v are the (u + 1)-th column and (v + 1)-th row of the image respectively, is the standard deviation, and M' is the probability map corresponding to the original image M.

[0036] Preferably, in S4, the minimum bounding rectangle Rect = minAreaRect of the contour is obtained, and minAreaRect is the function for calculating the minimum bounding rectangle.

[0037] where Rect contains three pieces of information about the rectangle: the center point (x, y), the width and height (w, h), and the angle θ.

[0038] Preferably, if w > h, the rectangle is corrected in direction as follows:

[0039]

[0040] Then update .

[0041] Preferably, in S5, a point set is constructed as follows:

[0042]

[0043]

[0044] Among them is the rotation matrix, .

[0045] Preferably, the construction point set, there is:

[0046]

[0047] .

[0048] Preferably, calculate the transformation matrix T1 and the inverse transformation matrix T2:

[0049] T1 = getPerspectiveTransform(src_pts, dst_ptst)

[0050] T2 = getPerspectiveTransform(dst_pts, src_pts)

[0051] Among them, getPerspectiveTransform is the transformation matrix calculation function.

[0052] Preferably, in the S6, the calculation method of the route point coordinates is as follows:

[0053] (1) For the j-th row, traverse from left to right, and use q i to represent the value of the i-th position point, and left and right respectively represent the positions of the candidate left and right side line points.

[0054] (2) If i = 0 and q0 = 1, then i is the left side line point, and update left = i.

[0055] (3) If 0 < i < w - 1:

[0056] a. If q i is equal to q i-1 : Ordinary point, skip.

[0057] b. If q i is not equal to q i-1 .

[0058] a) If q i-1 = V or q i = V, skip.

[0059] b) If q i-1 = 0, mark the i-th point as the left side line point, and update left = i.

[0060] c) If q i= 0, mark the \(i\)-th point as a right border point, update \(right = i\), and add \((left, right)\) to the set of candidate border point pairs.

[0061] (4) If \(i = w - 1\) and \(p_i = 1\), then the \(i\)-th point is a right border point, update \(right = i\), and add \((left, right)\) to the set of candidate border point pairs.

[0062] (5) Traverse the set of candidate point pairs, calculate the length \(L = right - left\), and find the candidate point pair \((left, right)\) corresponding to the maximum length. There is:

[0063] a. The coordinates of the flight path point corresponding to this line are \((j, (right + left) / 2)\).

[0064] b. The position mapped to the image \(M_0\) is:

[0065]

[0066]

[0067] The flight path extracted from the image \(M_0\) is represented as a set of a series of points , where \(m\) is the number of flight path points.

[0068] Compared with the prior art, the beneficial effects of the present invention are:

[0069] (1) Associating the flight path information with the image can guide the UAV to fly along the flight path through image recognition.

[0070] (2) The results at each moment are independent of each other, and there is no cumulative error effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the technical solution for flight path recognition of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0073] Embodiment 1: The technical solution includes the following steps:

[0074] S1. Flight path mapping

[0075] Extract the set of flight path trajectory points of the aircraft , where n is the number of trajectory points.

[0076] Retrieve all the images captured during the flight and the corresponding pose information, and project the trajectory points onto each image.

[0077] S2. Probability model training

[0078] Construct a dataset containing the original image set and the probability atlas. For an original image M, create an image M' of the same size. Using the original image set as the input and the probability image set as the output, train a neural network to obtain a probability model. Take the unet as an example for model training.

[0079] S3. Probability estimation and processing

[0080] Given an input image M0 to be predicted, use the above probability model for prediction and output the probability map M1.

[0081] Perform Gaussian filtering on the probability map M1 to remove obvious outliers.

[0082] Binarize M1, set the pixels greater than or equal to the threshold to 1 and the pixels less than the threshold to 0 to obtain the image M2.

[0083] Then perform dilation and erosion operations on M2 to obtain M3.

[0084] S4. Calculation of the main direction of the flight path

[0085] Extract the contours from M3, traverse and find the contour with the largest area contour; calculate the minimum bounding rectangle and direction of the largest contour.

[0086] S5. Extract probability data of the Rect region along the main direction

[0087] Calculate the corresponding key point sets before and after extracting the Rect region and .

[0088] Calculate the transformation matrix T1 and the inverse transformation matrix T2.

[0089] Use T1 to transform the image M3 and fill the boundary region with the value V to obtain the image M4, which is the probability data map of the Rect region and is a matrix with width and height w and h respectively.

[0090] S6. Extract the flight path

[0091] For the image M4, extract the left and right edge points of the binary image from top to bottom and calculate the corresponding flight path point coordinates.

[0092] In S1, for the image M and the k-th trajectory point , calculate the projection point of on M:

[0093]

[0094] where K is the camera parameter matrix, is the camera rotation matrix at the image capture moment, is the camera position at the image capture moment.

[0095] Obtain the width W and height H of the image.

[0096] If 0 <= u < W and 0 <= v < H, then the point is within the current image and is a valid point; otherwise, discard the point .

[0097] After projecting all the trajectory points, the set of valid points is the set of projection points of the flight path on image M .

[0098] Each image has a corresponding set of projection points.

[0099] In S2, the calculation method of each pixel value of image M' is as follows:

[0100]

[0101] where u and v are the (u + 1)-th column and (v + 1)-th row of the image respectively, is the standard deviation, and M' is the probability map corresponding to the original image M.

[0102] In S4, obtain the minimum bounding rectangle of contour Rect = minAreaRect, where minAreaRect is the function for calculating the minimum bounding rectangle.

[0103] where Rect contains three pieces of information about the rectangle: the center point (x, y), width and height (w, h), and angle θ.

[0104] If w > h, then correct the direction of the rectangle, and we have:

[0105]

[0106] Then update .

[0107] In S5, construct point set, and we have:

[0108]

[0109]

[0110] where is the rotation matrix, .

[0111] Construct a point set, there is:

[0112]

[0113] .

[0114] Preferably, calculate the transformation matrix T1 and the inverse transformation matrix T2:

[0115] T1 = getPerspectiveTransform(src_pts, dst_ptst)

[0116] T2 = getPerspectiveTransform(dst_pts, src_pts)

[0117] where getPerspectiveTransform is the transformation matrix calculation function.

[0118] In S6, the calculation method of the route point coordinates is as follows:

[0119] (1) For the j-th row, traverse from left to right, use q i to represent the value of the i-th position point, and left and right respectively represent the positions of the candidate left and right side points.

[0120] If i = 0 and q0 = 1, then i is the left side point, update left = i.

[0121] If 0 < i < w - 1:

[0122] a. If q i is equal to q i-1 : Ordinary point, skip.

[0123] b. If q i is not equal to q i-1 .

[0124] a) If q i-1 = V or q i = V, skip.

[0125] b) If q i-1 = 0, mark the i-th point as the left side point, update left = i.

[0126] c) If q iIf \(= 0\), mark the \(i\)-th point as a right edge point, update \(right = i\), and add \((left, right)\) to the set of candidate edge point pairs.

[0127] (4) If \(i = w - 1\) and \(p_i = 1\), then the \(i\)-th point is a right edge point, update \(right = i\), and add \((left, right)\) to the set of candidate edge point pairs.

[0128] (5) Traverse the set of candidate point pairs, calculate the length \(L = right - left\), and find the candidate point pair \((left, right)\) corresponding to the maximum length. There is:

[0129] a. The coordinates of the route point corresponding to this line are \((j, (right + left) / 2)\).

[0130] b. The position mapped to the image \(M_0\) is:

[0131]

[0132]

[0133] The route represented by the image \(M_0\) is expressed as a set of a series of points , where \(m\) is the number of route points

[0134] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0135] Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying the flight path of an unmanned aerial vehicle, the technical solution including the following steps: S1. Flight path mapping Extract the set of route trajectory points of the aircraft , where n is the number of trajectory points; Take out all the images and corresponding pose information taken during the flight, and project the trajectory points into each image; S2. Probability model training Construct a data set, including the original image set and the probability atlas. For an original image M, create an image M' of the same size. Using the original image set as the input and the probability image set as the output, train with a neural network to obtain a probability model. Take the unet as an example for model training; S3. Probability estimation and processing Given an input image M0 to be predicted, use the above probability model for prediction and output the probability map M1; Perform Gaussian filtering on the probability map M1 to remove obvious outliers; Binarize M1, set the pixels greater than or equal to the threshold to 1, and the pixels less than the threshold to 0 to obtain the image M2; Then perform dilation and erosion operations on M2 to obtain M3; S4. Calculation of the main direction of the flight path Extract the contour from M3, traverse and find the contour with the largest area contour; calculate the minimum bounding rectangle and direction of the largest contour; S5. Extract the probability data of the Rect area along the main direction Calculate the corresponding key point sets before and after extracting the Rect area and ; Calculate the transformation matrix T1 and the inverse transformation matrix T2; Use T1 to transform the image M3 and fill the boundary area with the value V to obtain the image M4, which is the probability data map of the Rect area, and is a matrix with width and height w and h respectively; S6. Extract the flight path For the image M4, extract the left and right edge points of the binary map from top to bottom and calculate the corresponding flight path point coordinates.

2. The method for identifying an unmanned aerial vehicle flight path according to claim 1, wherein: In S1, for the image M and the k-th trajectory point , calculate the coordinates of the projection point on M: ； where K is the camera parameter matrix, is the camera rotation matrix at the image capture time, is the camera position at the image capture time.

3. The method for identifying a UAV flight path according to claim 2, wherein: Obtain the width W and height H of the image; If 0 <= u < W and 0 <= v < H, then the point is in the current image and is a valid point; Otherwise, discard the point ; After projecting all the trajectory points, the set of valid points is the set of projection points of the flight path on the image M ; Each image has a corresponding set of projection points.

4. A method for identifying an unmanned aerial vehicle flight path according to claim 1, characterized in that: In S2, the calculation method of each pixel value of the image M' is as follows: ； where u and v are the (u + 1)-th column and (v + 1)-th row of the image respectively, is the standard deviation, and M' is the probability map corresponding to the original image M.

5. A method for identifying a drone flight path according to claim 1, characterized in that: In S4, obtain the minimum bounding rectangle Rect = minAreaRect of the contour, where minAreaRect is the function for calculating the minimum bounding rectangle; Among them, Rect contains three pieces of information about the rectangle: the center point (x, y), width and height (w, h), and angle θ.

6. A method for identifying a UAV flight path according to claim 5, characterized in that, If w > h, then correct the direction of the rectangle, and there is: ； Then update .

7. A method for identifying a UAV flight path according to claim 1, characterized in that: In S5, construct a point set, and there is: ； ； wherein is a rotation matrix .

8. A method for identifying an unmanned aerial vehicle flight path according to claim 7, characterized in that, Construct a point set, there is: ； 。 9. A method for identifying an unmanned aerial vehicle flight path according to claim 8, characterized in that, Calculate the transformation matrix T1 and the inverse transformation matrix T2: T1 = getPerspectiveTransform(src_pts, dst_ptst) T2 = getPerspectiveTransform(dst_pts, src_pts) Among them, getPerspectiveTransform is the function for calculating the transformation matrix.

10. A method for identifying an unmanned aerial vehicle flight path according to claim 1, wherein: In S6, the calculation method of the flight path point coordinates is as follows: (1)For the j-th row, traverse from left to right, and use q i to represent the value of the i-th position point, and left and right respectively represent the positions of the left and right line points of the candidate; (2) If i = 0 and q0 = 1, then i is the left edge point, update left = i; (3) If 0 < i < w - 1: a. If q i is equal to q i-1 : Ordinary point, skip; b. If q i is not equal to q i-1 : a) If q i-1 = V or q i = V, skip; b) If q i-1 = 0, mark the i-th point as a left edge point and update left = i; c) If q i = 0, mark the i-th point as a right edge point, update right = i, and add (left, right) to the set of candidate edge point pairs; (4) If i = w - 1 and pi = 1, then the i-th point is the right edge point, update right = i, and add (left, right) to the set of candidate edge point pairs; (5) Traverse the set of candidate point pairs, calculate the length L = right - left, and find the candidate point pair (left, right) corresponding to the maximum length. We have: a. The coordinates of the waypoint corresponding to this line are (j, (right + left) / 2); b. The position mapped to the image M0 is: ； ； The flight path extracted from the image M0 is represented as a set of a series of points , where m is the number of flight path points.

Images