A computer vision-based unmanned aerial vehicle autonomous along-line navigation method

Abstract

The application provides a computer vision-based unmanned aerial vehicle autonomous along-line navigation method. The method comprises the following steps: continuously shooting video data of a track line by an onboard camera of an unmanned aerial vehicle, segmenting the video data into continuous multi-frame track line images based on a lightweight neural network; constructing a smooth track center line based on the multi-frame track line images; calculating a desired trajectory, a flight speed and an onboard sensor attitude angle of the unmanned aerial vehicle based on the track center line, and realizing autonomous navigation and smooth and stable flight of the unmanned aerial vehicle along the track line through unmanned aerial vehicle flight speed control and onboard sensor attitude control, and a safe distance from the track line can be maintained.

Description

Technical Field

[0001] This invention relates to the field of intelligent navigation technology for unmanned aerial vehicles (UAVs), and more particularly to a method for autonomous route navigation of UAVs based on computer vision. Background Technology

[0002] Ensuring the safety of the rail transit operating environment is a crucial aspect of the rail transit system. Due to its significant advantages in efficiency and safety, drone-based autonomous track inspection has become the current development trend of track inspection solutions.

[0003] Currently, most existing autonomous navigation methods for unmanned aerial vehicles (UAVs) rely on GPS (Global Positioning System) for navigation and positioning. The drawbacks of this method include sparse orbital coordinate information and the inherent errors in GPS position information. Summary of the Invention

[0004] The embodiments of the present invention provide a computer vision-based method for autonomous navigation of unmanned aerial vehicles (UAVs) along a track, so as to effectively realize autonomous navigation of UAVs along a track.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A computer vision-based autonomous route navigation method for unmanned aerial vehicles (UAVs) includes:

[0007] Video data of the track line is continuously captured from above by the onboard camera of the drone, and the video data is segmented into continuous multi-frame track line images based on a lightweight neural network.

[0008] Construct a smooth track centerline based on multi-frame track line images;

[0009] Based on the centerline of the orbit, the desired trajectory, flight speed, and attitude angles of the UAV are calculated using the UAV body coordinate system to achieve UAV navigation and control.

[0010] Preferably, the video data of the track line continuously captured from above by the airborne camera of the UAV, and the video data being segmented into continuous multi-frame track line images based on a lightweight neural network, includes:

[0011] Video data of the railway line is continuously captured from above by an airborne camera on a drone. The video data is segmented into continuous multi-frame images of the railway line. These continuous multi-frame images are used for training to increase contextual information. Track structure similarity features are extracted from each frame using a hollow spatial pyramid pooling network and a many-to-many bidirectional recurrent convolutional neural network. An output feature map is constructed based on the features extracted from all frames. Based on the feature map, an attention mechanism module and upsampling are used to obtain the railway line segmentation image result.

[0012] Preferably, the construction of a smooth track centerline based on multiple frames of track images includes:

[0013] The segmented track image is divided into multiple discretized trapezoidal block nodes. A one-way directed graph of trapezoidal block nodes is established. The distance between different trapezoidal block nodes is calculated. Based on the distance between the trapezoidal block nodes, each trapezoidal block node is clustered.

[0014] The nodes are divided into strong and weak nodes. The trapezoidal block containing a pair of tracks and surrounded by track ballast is identified as a strong node, and the remaining trapezoidal blocks are identified as weak nodes. Low-confidence segmentation regions are filtered out, that is, candidate track lines whose length is less than half of the image height or candidate track lines whose number of strong nodes is less than half of the total number of nodes.

[0015] By fitting a quadratic polynomial curve to represent the track centerline and a linear function to represent the track width, the filtered discretized trapezoidal block nodes are reconstructed into a continuous and smooth curve representing the track centerline.

[0016] Preferably, the step of calculating the desired trajectory and flight speed of the UAV using the UAV body coordinate system based on the orbit centerline includes:

[0017] Based on the orbital centerline, a coordinate transformation is performed from the camera coordinate system to the UAV body coordinate system. Let U = (u, v) be a point on the orbital centerline L in the image, and let the camera focal length be f. A point in the camera coordinate system is represented as X. c = (f, u, v), the altitude of the drone is h. p The rotation matrix of the camera coordinate system relative to the UAV body coordinate system is R, and the coordinates of point U in the UAV body coordinate system are given by equation (1):

[0018]

[0019] In the formula It is the indicator vector of the Z-axis coordinate in the UAV body coordinate system;

[0020] Ignore UAV body coordinate system O bodyThe Z-axis is the coordinate system, with the UAV's position at the origin. The nearest control point between the orbital centerline and the UAV is X = (x, y), and the tangent direction of the control point is n1 = (x, y). n1 ,y n1 The normal direction of the control point is n2 = (x n2 ,y n2 According to geometric principles, vectors Perpendicular to vector vector The unit vector is vector sum vector For equations (2) and (3):

[0021]

[0022]

[0023] In the formula, L1 is the horizontal distance from the UAV to the control point, k = {-1, 1} is a fixed parameter set according to the UAV's position and detection direction, and L1 is the tangent line of the orbital centerline at the control point. L1 is along the vector... The expected safe distance d for translation th The desired trajectory L2 of the drone is obtained;

[0024] The speed of the drone is divided into: along the tangential direction. Navigation speed V g and along the normal direction Correction speed V a As shown in equation (4):

[0025] V = V g +V a =k g n1+k a n2(4)

[0026] In the formula, V g k represents the navigation velocity vector along the tangential direction. g V is a scalar representing velocity in the tangential direction. a k represents the corrected velocity vector along the normal direction. a A scalar representing the velocity in the direction of the normal;

[0027] Let D max The radius of the target selection range is defined by the radius of the target area. The intersection point V between the target direction and L2 is the selected target point. If there is no intersection point, the UAV needs to proceed along the normal direction. Approaching the desired trajectory at full speed, k g and k a As in equations (5) and (6):

[0028] k g =ωmax(D max +d th -d p ,0)(5)

[0029]

[0030] In the formula, ω is the proportionality coefficient between distance and speed.

[0031] Preferably, the step of calculating the onboard sensor attitude angles of the UAV based on the UAV body coordinate system using the UAV centerline to realize UAV navigation control includes:

[0032] The shooting reference point is controlled to be centered in the image by changing the pitch, yaw, and roll angles of the drone's onboard camera. The pitch angle of the drone's onboard camera is a fixed value β = -π / 2, the normal to the control point passes through the origin, and the drone's flight altitude is h. p The control point is (x, y), and the tangent direction of the control point is (n). x ,n y The target yaw angle α and roll angle γ of the UAV's onboard sensors are calculated using equations (7) and (8):

[0033]

[0034]

[0035] As can be seen from the technical solutions provided by the embodiments of the present invention above, the method of the embodiments of the present invention enables the UAV to autonomously navigate along the orbital line through real-time, high-precision orbital target detection.

[0036] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram illustrating the implementation principle of an autonomous flight navigation method for unmanned aerial vehicles (UAVs) based on visual recognition of a virtual centerline of an orbit, provided in an embodiment of the present invention.

[0039] Figure 2This is a schematic diagram of the processing flow of an autonomous flight navigation method for unmanned aerial vehicles based on visual recognition of virtual centerline of orbit, provided in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram illustrating the relative relationship between an image coordinate system and an unmanned aerial vehicle (UAV) body coordinate system, provided as an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of a UAV body coordinate system after eliminating the Z-axis, provided as an embodiment of the present invention. Detailed Implementation

[0042] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0043] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0044] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0045] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0046] This invention enables autonomous UAV navigation along a target track through real-time, high-precision target detection. During navigation, the UAV's sensors are locked onto the target track to continuously track the trajectory. A point located on the virtual centerline of the track and closest to the UAV is selected as a control point. Based on the control point's position and orientation relative to the UAV, a flight and sensor attitude control strategy for the UAV is designed.

[0047] The implementation principle diagram of an autonomous flight navigation method for unmanned aerial vehicles based on visual recognition of virtual centerline of orbit provided by this invention is shown in the figure below. Figure 1 As shown, the specific processing flow is as follows: Figure 2 As shown, the processing steps include the following:

[0048] Step S1: Using video data of the track line taken continuously from above by a drone, the video data is segmented into continuous multi-frame track line images based on a lightweight neural network;

[0049] Step S2: Construct a smooth track centerline based on multiple frames of track images;

[0050] Step S3: Calculate the desired trajectory, flight speed, and attitude angles of the UAV using the centerline of the orbit to achieve UAV navigation control.

[0051] Preferably, step S1 specifically includes: taking continuous aerial photos of the track line using a drone, segmenting the video data into continuous multi-frame track line images, using the continuous multi-frame track line images for training to increase contextual information, extracting track structure similarity features from each of the above frames using a hollow spatial pyramid pooling network and a many-to-many bidirectional recurrent convolutional neural network, constructing an output feature map based on the features extracted from all frames, and obtaining the railway line segmentation image result based on the feature map through an attention mechanism module and upsampling.

[0052] Preferably, step S2 specifically includes: dividing the segmented track image into multiple discretized trapezoidal block nodes, establishing a unidirectional directed graph of the trapezoidal block nodes, calculating the distance between different trapezoidal block nodes, and performing clustering processing on each trapezoidal block node based on the distance between the trapezoidal block nodes.

[0053] The nodes are divided into strong and weak nodes. The trapezoidal block containing a pair of tracks and surrounded by track ballast is identified as a strong node, and the remaining trapezoidal blocks are identified as weak nodes. Low-confidence segmentation regions are filtered out, that is, candidate track lines whose length is less than half of the image height or candidate track lines whose number of strong nodes is less than half of the total number of nodes.

[0054] By fitting a quadratic polynomial curve to represent the track centerline and a linear function to represent the track width, the filtered discretized trapezoidal block nodes are reconstructed into a continuous and smooth curve representing the track centerline.

[0055] Preferably, step S3 specifically includes: performing coordinate transformation from the image coordinate system to the UAV body coordinate system; selecting the point located on the centerline of the track and closest to the UAV as the control point, calculating the desired trajectory and desired flight speed to achieve flight control; calculating the attitude angle of the onboard sensor to achieve attitude control, keeping the target within the sensor's field of view, and achieving continuous tracking of the track line.

[0056] (1) Perform coordinate transformation from the camera coordinate system to the UAV body coordinate system.

[0057] Drone navigation requires obtaining the position information of the orbital centerline in the drone's body coordinate system. The relative relationship between the image coordinate system and the drone's body coordinate system is as follows: Figure 3 As shown.

[0058] Suppose U = (u, v) is a point on the center line L of the track in the image, the camera focal length is f, and a point in the camera coordinate system is represented as X. c = (f, u, v). The altitude of the drone is h. p The rotation matrix of the PTZ (Pan-Tilt-Zoom) camera coordinate system relative to the UAV body coordinate system is R, and the coordinates of point U in the UAV body coordinate system are given by equation (1):

[0059]

[0060] In the formula It is the indicator vector of the Z-axis coordinate.

[0061] (2) Calculate the desired trajectory and flight speed of the UAV to achieve flight control.

[0062] Since all orbital targets are on the ground, the UAV body coordinate system O can be ignored. body Z-axis, Figure 4 This is a schematic diagram of a UAV body coordinate system after eliminating the Z-axis, provided as an embodiment of the present invention. Figure 4 As shown, The unmanned aerial vehicle body coordinate system O body There is no coordinate system without a Z-axis. The UAV's position is at the origin. The control point is X = (x, y), and the tangent direction is n1 = (x, y). n1 ,y n1 The normal direction is n2 = (x) n2 ,y n2 ).

[0063] Since the control point is the closest control point between the orbital centerline and the UAV, according to geometric principles, vectors... Perpendicular to vector vector The unit vector is vector sum vector For equation (2) (3):

[0065]

[0066]

[0067] In the formula, It is the horizontal distance from the UAV to the control point, and k = {-1, 1} is a fixed parameter set according to the UAV's position and detection direction.

[0068] exist Figure 4 In the diagram, L1 is the tangent to the track centerline at the control point. Due to the large turning radius of the track, the track centerline within the field of view can be approximated by this tangent. L1 is along the vector... The expected safe distance d for translation th The desired safe trajectory L2 is obtained, such as Figure 4 The dotted line L2 in the lower right corner is shown.

[0069] To control the drone to approach the desired trajectory, the drone's speed can be divided into two parts: along the tangential direction. Navigation speed V g and along the normal direction Correction speed V a As shown in equation (4):

[0070] V = V g +V a =k g n1+k a n2(4)

[0071] In the formula, V g k represents the navigation velocity vector along the tangential direction. g V is a scalar representing velocity in the tangential direction. a k represents the corrected velocity vector along the normal direction. a A scalar representing the velocity in the direction of the normal.

[0072] It is positively correlated with the distance from the desired trajectory of the drone. When the distance is small, k a Smaller and k g The drone is relatively large, and it mainly moves along the track line; when k a Larger and k gWhen the drone is small, it will quickly move toward the desired trajectory.

[0073] The speed of the drone is controlled by the distance to the target point in each step. Figure 3 With a median radius of D max The dashed circle represents the target point selection range, and the intersection point V between the drone and L2 along the target direction is the selected target point. If there is no intersection point, the drone needs to proceed along the normal direction. Approaching the desired trajectory at full speed. g and k a As in equations (5) and (6):

[0074] k g =ωmax(D max +d th -d p ,0)(5)

[0075]

[0076] In the formula, ω is the proportionality coefficient between distance and speed. Equations (5) and (6) ensure that the speed of the UAV remains constant and that the UAV can move smoothly along the desired trajectory.

[0077] (3) Calculate the attitude angles of the UAV's onboard sensors to achieve attitude control.

[0078] To keep the target trajectory line within the field of view of the airborne camera, the reference point is controlled to be centered in the image by changing the pitch, yaw, and roll angles of the PTZ.

[0079] The pitch angle is a fixed value β = -π / 2, therefore the normal to the control point will pass through the origin. For the UAV's flight altitude h... p Control point (x, y) and its tangent direction (n) x ,n y The target yaw angle α and roll angle γ of the sensor are given by equations (7)(8):

[0080]

[0081]

[0082] Considering the non-negligible short time delay between image acquisition and target pose calculation, directly using the angle defined by the equation would lead to overshoot. Therefore, a proportional-integral-derivative (PID) algorithm is used to update the actual control angle.

[0083] In summary, the real-time visual UAV navigation system for track inspection, as demonstrated in this embodiment of the invention and after field testing, identifies and segments the track structure and reconstructs the virtual centerline of the track. Using a smooth curve as the desired flight trajectory for the UAV, and through UAV speed control and attitude control via onboard sensors, it achieves autonomous navigation and smooth, stable flight along the track line while maintaining a safe distance. This invention enables autonomous UAV navigation along the track, serving as a prerequisite and foundation for autonomous UAV track inspection schemes.

[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0085] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0086] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A computer vision-based autonomous route navigation method for unmanned aerial vehicles (UAVs), characterized in that, include: Video data of the track line is continuously captured from above by the onboard camera of the drone, and the video data is segmented into continuous multi-frame track line images based on a lightweight neural network. Construct a smooth track centerline based on multi-frame track images; Based on the centerline of the orbit, the desired trajectory, flight speed, and attitude angles of the UAV are calculated using the UAV body coordinate system to achieve UAV navigation and control; The method of calculating the desired trajectory and flight speed of the UAV using the UAV body coordinate system based on the orbit centerline includes: Based on the centerline of the orbit, a coordinate transformation is performed from the camera coordinate system to the UAV body coordinate system. It is a point on the center line L of the track in the image, and the focal length of the camera is... A point in the camera coordinate system is represented as The drone's altitude is The rotation matrix of the camera coordinate system relative to the UAV body coordinate system is R, and point... The coordinates in the UAV body coordinate system are given by equation (1): （1） In the formula It is the indicator vector of the Z-axis coordinate in the UAV body coordinate system; Ignore UAV body coordinate system The Z-axis is the coordinate system, with the drone's position at the origin. The nearest control point between the drone and the orbital centerline is... The tangent direction of the control point is The normal direction of the control point is According to geometric principles, vectors Perpendicular to vector ,vector The unit vector is ,vector sum vector For equations (2) and (3): （2） （3） In the formula, It is the horizontal distance from the drone to the control point. These are fixed parameters set based on the drone's position and detection direction. It is the tangent to the centerline of the track at the control point. Along the vector Expected safe distance for translation To obtain the desired trajectory of the drone ; The speed of the drone is divided into: along the tangential direction. navigation speed and along the normal direction Correction speed As in equation (4): （4） In the formula, This represents the navigation velocity vector along the tangent direction. A scalar representing velocity in the tangential direction. This represents the corrected velocity vector along the normal direction. A scalar representing the velocity in the direction of the normal; set up The radius of the selected range for the target point, along the target direction and... Intersection point V is the selected target point. If there is no intersection point, the drone needs to travel along the normal direction. Approaching the desired trajectory at full speed, and As in equations (5) and (6): （5） （6） In the formula, It is the ratio coefficient between distance and speed; The method of calculating the onboard sensor attitude angles of the UAV based on the UAV body coordinate system using the UAV centerline to achieve UAV navigation control includes: The shooting reference point is controlled to be centered in the image by changing the pitch, yaw, and roll angles of the drone's onboard camera, while the pitch angle of the drone's onboard camera is a fixed value. The normal to the control point passes through the origin, and the drone's flight altitude is... The control point is The tangent direction of the control point is The target yaw angle of the drone's onboard sensors and roll angle The calculation method is given by equations (7) and (8): （7） （8）。 2. The method according to claim 1, characterized in that, The video data of the track line captured continuously from an aerial camera on a drone, and the segmentation of the video data into continuous multi-frame track line images based on a lightweight neural network, includes: Video data of the railway line is continuously captured from above by an airborne camera on a drone. The video data is segmented into continuous multi-frame images of the railway line. These continuous multi-frame images are used for training to increase contextual information. Track structure similarity features are extracted from each frame using a hollow spatial pyramid pooling network and a many-to-many bidirectional recurrent convolutional neural network. An output feature map is constructed based on the features extracted from all frames. Based on the feature map, an attention mechanism module and upsampling are used to obtain the railway line segmentation image result.

3. The method according to claim 1, characterized in that, The method of constructing a smooth track centerline based on multi-frame track images includes: The segmented track image is divided into multiple discretized trapezoidal block nodes. A one-way directed graph of trapezoidal block nodes is established. The distance between different trapezoidal block nodes is calculated. Based on the distance between the trapezoidal block nodes, each trapezoidal block node is clustered. The nodes are divided into strong and weak nodes. The trapezoidal block containing a pair of tracks and surrounded by track ballast is identified as a strong node, and the remaining trapezoidal blocks are identified as weak nodes. Low-confidence segmentation regions are filtered out, that is, candidate track lines whose length is less than half of the image height or candidate track lines whose number of strong nodes is less than half of the total number of nodes. By fitting a quadratic polynomial curve to represent the track centerline and a linear function to represent the track width, the filtered discretized trapezoidal block nodes are reconstructed into a continuous and smooth curve representing the track centerline.

Images