End-to-End UAV Autonomous Flight Navigation Method and System Along Railway Based on Skeleton Recognition

Abstract

The present invention provides an end-to-end autonomous flight navigation method and system for an unmanned aerial vehicle (UAV) along a railway based on skeleton recognition, belonging to the technical field of intelligent control of UAVs. The railway skeleton is detected in real time based on a multi-scale direction hybrid linear attention parallel network; the flight speed and attitude angle are calculated based on speed smoothing linear interpolation to achieve autonomous navigation control of the UAV. In view of the linear invariant features of the railway and the UAV flight mission, the present invention designs a multi-scale multi-direction hybrid linear attention mechanism to enhance the extraction of railway skeleton features, and uses a semantic segmentation branch parallel to the skeleton detection branch to reduce misidentification, realizing high-precision railway skeleton detection; for edge computing platforms with limited computing resources, a multi-direction attention parameter sharing mechanism is designed to reduce the number of network parameters and improve the skeleton detection speed; for the railway skeleton obtained by network detection, smooth and stable autonomous high-speed flight control of the UAV along the line is achieved by using speed smoothing linear interpolation calculation.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of unmanned aerial vehicles, and particularly to an end-to-end method and system for autonomous flight navigation of an unmanned aerial vehicle along a railway based on skeleton recognition. Background Art

[0002] To improve the inspection efficiency and ensure personnel safety, the railway line inspection scheme based on unmanned aerial vehicles is the mainstream development trend to replace the current manual inspection. In order to avoid the distance limitation of remote control and reduce the dependence on GPS positioning, the implementation of real-time detection technology for high-precision railway targets based on vision is a better solution for the autonomous navigation of unmanned aerial vehicles along railway lines.

[0003] After using an image segmentation neural network to segment the railway track, post-processing processes such as fitting are required to obtain the flight path of the unmanned aerial vehicle. However, the target result is affected by empirical parameters. Therefore, an end-to-end detection network that directly obtains the flight trajectory and meets the requirements of the unmanned aerial vehicle computing device is needed. Summary of the Invention

[0004] The purpose of the present invention is to provide an end-to-end method and system for autonomous flight navigation of an unmanned aerial vehicle along a railway based on skeleton recognition, so as to solve at least one of the technical problems existing in the above background art.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] In the first aspect, the present invention provides an end-to-end method for autonomous flight navigation of an unmanned aerial vehicle along a railway based on skeleton recognition, including:

[0007] Obtain railway images;

[0008] Process the obtained railway images by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway images; wherein, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to traverse and extract railway features of different directions that may appear by combining multiple discrete attention directions; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton.

[0009] Extract smooth target discrete points at a fixed distance interval according to the obtained railway skeleton; select the forward nearest smooth point as the target point from the candidate points according to the positions of the smooth target discrete points and the current position, and combine the smooth linear velocity interpolation method to control the target flight speed and target attitude angle of the unmanned aerial vehicle in real time.

[0010] Furthermore, the parameter sharing mechanism of the attention direction is used to reduce the number of parameters, and the parameters are shared when calculating all attention directions, including: in the attention module of each stage, first calculate the Query-Key and Value of the attention. In all attention directions traversed in the current stage, two groups of parameters are shared. The Query-Key and Value are split with the same width but different directions and combined into new dimensions to calculate multiple groups of attention feature maps, and the attention features of this stage are obtained by merging.

[0011] Furthermore, based on the railway skeleton in the detected image, according to the custom control strategy of flying along the right railway, the rightmost skeleton in the image is selected as the target skeleton. To ensure smooth flight, several discrete points are evenly extracted on the target skeleton with an empirical value of a fixed pixel distance, and after coordinate transformation, they are used as candidate flight trajectory points.

[0012] Furthermore, according to the discrete target position and the current position, the forward nearest smooth point is selected as the target point among the candidate points, including:

[0013] Assume that the UAV coordinates when the neural network detects the skeleton image are O1, and the target point coordinates are A i , where i is the target point index, and the UAV coordinates when the control algorithm uses this target point are O2. The distance vector from the current position of the UAV to the target point is:

[0014]

[0015] Considering the flight oscillation caused by the change of the UAV speed, to ensure smooth flight, according to the magnitude and direction of the distance vector of the candidate target point, the nearest smooth point is selected as the flight target point of the current image inference result.

[0016] Furthermore, the smooth linear velocity interpolation method is used to real-time control the target flight speed and target attitude angle of the UAV, including:

[0017] Assume that the current flight speed is (v N0 , v E0 , v D0 ), the target point direction angle is Φ, the magnitude of the target flight speed is v, and the flight speed of the target point is (v N1 , v E1 , v D1 ). To ensure the smooth change of speed during flight, according to the smooth speed linear interpolation, the flight speed (v Nj , v Ej , v Dj ) at each time t with a set control frequency is calculated, and the flight speed is adjusted according to the target point direction angle.

[0018] Furthermore,

[0019]

[0020] The vector angle Θ between candidate target points, that is, the angle of the railway skeleton, is the angle Θ between the yaw angle of the target attitude angle of the UAV and the yaw angle of the pan-tilt camera:

[0021]

[0022] Under the set control frequency and the speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to achieve autonomous navigation control of the UAV.

[0023] In a second aspect, the present invention provides an end-to-end UAV autonomous flight navigation system along a railway based on skeleton recognition, including:

[0024] An acquisition module for acquiring railway images;

[0025] A positioning module for processing the acquired railway images by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway images; wherein, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features, wherein, by using a multi-scale attention mechanism, railway features of different widths that may appear are extracted at different scales, and by using a multi-directional attention mechanism, a combination of multiple discrete attention directions is adopted to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton;

[0026] A control module for extracting smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; selecting the forward nearest smooth point as the target point from the candidate points according to the positions of the smooth target discrete points and the current position, and combining the smooth linear speed interpolation method to control the target flight speed and target attitude angle of the UAV in real time.

[0027] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, and the non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the end-to-end UAV autonomous flight navigation method based on skeleton recognition as described in the first aspect is implemented.

[0028] Fourthly, the present invention provides a computer device, including a memory and a processor, where the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the end-to-end drone autonomous flight navigation method based on skeleton recognition as described in the first aspect.

[0029] Fifthly, the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device runs, the processor executes the computer program stored in the memory, so that the electronic device executes instructions to implement the end-to-end drone autonomous flight navigation method based on skeleton recognition as described in the first aspect.

[0030] Advantages of the present invention: The multi-scale direction hybrid linear attention parallel network tested by comparative experiments and ablation experiments detects high-precision railway skeletons through the multi-scale multi-direction hybrid linear attention mechanism, reduces computing resources and improves computing speed through the multi-direction attention parameter sharing mechanism, and reduces the misrecognition rate through the parallel semantic segmentation branch. The end-to-end drone autonomous flight navigation system based on skeleton recognition tested through field experiments calculates the flight speed and attitude of the drone through speed smoothing linear interpolation, and realizes the smooth, stable and fast autonomous navigation along the line of the drone.

[0031] The advantages of the additional aspects of the present invention will be more clearly given in the following description part, or understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0033] Figure 1 It is a schematic algorithm flow diagram of the end-to-end drone autonomous flight navigation method based on railway skeleton recognition described in the embodiments of the present invention.

[0034] Figure 2 It is a schematic hardware structure diagram of the end-to-end drone autonomous flight navigation method based on railway skeleton recognition described in the embodiments of the present invention.

[0035] Figure 3 It is a schematic diagram of the multi-scale direction hybrid linear attention parallel network described in the embodiments of the present invention.

[0036] Figure 4Schematic diagram of different attention direction angles in the multi-scale multi-direction hybrid linear attention mechanism according to the embodiments of the present invention.

[0037] Figure 5 Schematic diagram of the multi-direction attention parameter sharing mechanism in the multi-scale multi-direction hybrid linear attention mechanism according to the embodiments of the present invention. Detailed implementation manners

[0038] The following details the implementation manners of the present invention. Examples of the implementation manners are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The implementation manners described through the drawings are exemplary and are only used to explain the present invention and cannot be construed as a limitation to the present invention.

[0039] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art in the field to which the present invention belongs.

[0040] It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined as such herein.

[0041] Those skilled in the art of the present technology can understand that, unless specifically stated, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or their groups.

[0042] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0043] To facilitate the understanding of the present invention, the following further explains the present invention with specific embodiments in conjunction with the drawings, and the specific embodiments do not constitute a limitation to the embodiments of the present invention.

[0044] Those skilled in the art should understand that the accompanying drawings are only schematic diagrams of the embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0045] Embodiment 1

[0046] In this Embodiment 1, first, an end-to-end UAV autonomous flight navigation system based on skeleton recognition along a railway is provided, including: an acquisition module for acquiring railway images; a positioning module for processing the acquired railway images using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway images. Among them, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network. In the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, a multi-scale attention mechanism is used to extract different-width railway features that may appear at different scales, and a multi-directional attention mechanism is used to traverse and extract different-direction railway features that may appear by combining multiple discrete attention directions. In the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton. A control module for extracting smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; selecting the forward nearest smooth point as the target point from the candidate points according to the positions of the smooth target discrete points and the current position, and combining the smooth linear velocity interpolation method to control the target flight speed and target attitude angle of the UAV in real time.

[0047] In this embodiment, the above system can be used to implement an end-to-end UAV autonomous flight navigation method based on skeleton recognition along a railway, including: acquiring railway images; processing the acquired railway images using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway images. Among them, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network. In the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, a multi-scale attention mechanism is used to extract different-width railway features that may appear at different scales, and a multi-directional attention mechanism is used to traverse and extract different-direction railway features that may appear by combining multiple discrete attention directions. In the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton; extracting smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; selecting the forward nearest smooth point as the target point from the candidate points according to the positions of the smooth target discrete points and the current position, and combining the smooth linear velocity interpolation method to control the target flight speed and target attitude angle of the UAV in real time.

[0048] Among them, the attention direction parameter sharing mechanism is used to reduce the number of parameters, and the parameters are shared when calculating all attention directions, including: in the attention module of each stage, first calculate the Query-Key and Value of the attention. In all attention directions traversed in the current stage, two groups of parameters are shared. The Query-Key and Value are respectively split into different directions with the same width and combined into new dimensions, multiple groups of attention feature maps are calculated, and the attention features of this stage are merged.

[0049] Based on the detected railway skeleton in the image, according to the custom control strategy of flying along the right railway, the rightmost skeleton in the image is selected as the target skeleton. To ensure smooth flight, several discrete points are evenly extracted on the target skeleton with an empirical value of a fixed pixel distance, and after coordinate transformation, they are used as candidate flight trajectory points. According to the discrete target position and the current position, the forward nearest smooth point is selected as the target point among the candidate points, including:

[0050] Assume that the UAV coordinate when the neural network detects the skeleton image is O1, and the target point coordinate is A i , where i is the target point index, the UAV coordinate when the control algorithm uses this target point is O2, and the distance vector from the current position of the UAV to the target point is:

[0051]

[0052] Considering the flight oscillation caused by the change of the UAV speed, to ensure smooth flight, according to the magnitude and direction of the distance vector of the candidate target point, the nearest smooth point is selected as the flight target point of the current image inference result.

[0053] Using the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time, including:

[0054] Assume that the current flight speed is (v N0 , v E0 , v D0 ), the target point direction angle is Φ, the magnitude of the target flight speed is v, and the flight speed of the target point is (v N1 , v E1 , v D1 ). To ensure the smooth change of speed during flight, according to the smooth speed linear interpolation, the flight speed (v Nj , v Ej , v Dj ) at each time t under the set control frequency is calculated, and the flight speed is adjusted according to the target point direction angle.

[0055]

[0056] The vector angle Θ between candidate target points, i.e., the angle of the railway skeleton, is such that the yaw angle of the UAV target attitude angle and the yaw angle of the pan-tilt camera are angle Θ:

[0057]

[0058] Under the set control frequency and the speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to achieve autonomous navigation control of the UAV.

[0059] Embodiment 2

[0060] In this Embodiment 2, an end-to-end UAV autonomous flight navigation method based on skeleton recognition is provided, including the following steps: S1. Detect the railway skeleton based on a multi-scale direction hybrid linear attention parallel network; S2. Calculate the flight speed and attitude angle based on speed smoothing linear interpolation to achieve UAV autonomous navigation.

[0061] Among them, the method for detecting the railway skeleton based on the multi-scale direction hybrid linear attention parallel network in the step S1 is as follows: Based on the linear invariant features of the railway and the UAV flight mission, use the multi-scale multi-direction hybrid linear attention mechanism to improve the detection accuracy. In the four-stage linear attention, design different combinations of attention widths and attention directions, segment and reorganize the image into different dimensions for attention calculation; use the attention direction parameter sharing mechanism to reduce the number of parameters and improve the inference speed. In each stage of the attention module, share parameters when traversing the calculations of all attention directions; use the semantic segmentation branch network parallel to the skeleton detection branch to reduce the misrecognition rate, and the skeleton branch obtains the mixed railway skeleton through multiple side output layers.

[0062] The method for calculating the flight speed and attitude angle based on speed smoothing linear interpolation to achieve UAV autonomous navigation in the step S2 is as follows: Extract smooth target discrete points at fixed distance intervals according to a custom control strategy, select the forward nearest smooth point as the flight target point among the candidate points, use the speed smoothing linear interpolation method to calculate the target flight speed and target attitude angle, and update the control quantity in real time to achieve UAV autonomous navigation.

[0063] In this embodiment, the multi-scale direction hybrid linear attention parallel network tested through comparative experiments and ablation experiments detects high-precision railway skeletons through the multi-scale multi-direction hybrid linear attention mechanism, reduces computing resources and improves the computing speed through the multi-direction attention parameter sharing mechanism, reduces the misrecognition rate through the parallel semantic segmentation branch, and the end-to-end UAV autonomous flight navigation system based on skeleton recognition tested through field experiments calculates the UAV flight speed and attitude through speed smoothing linear interpolation, achieving smooth, stable and fast autonomous along-line navigation of the UAV.

[0064] Specifically, the algorithm flow of the end-to-end UAV autonomous flight navigation method based on railway skeleton recognition in this embodiment is as follows Figure 1 shown, and the hardware structure is as follows Figure 2 shown, and the multi-scale direction hybrid linear attention parallel network structure is as follows Figure 3 shown.

[0065] Step S1. Detect the railway skeleton based on the multi-scale direction hybrid linear attention parallel network, including:

[0066] (1) According to the UAV flight mission, an innovative multi-scale and multi-direction hybrid linear attention for the invariant features of the railway is proposed. Through this attention mechanism, the extraction of railway skeleton features is enhanced.

[0067] Within the field of view of the UAV, different from features such as color that are vulnerable to the environment, the shape of the railway always maintains linear invariant features. In this embodiment, the feature map is segmented into linear stripes through linear attention and recombined into new-dimensional features.

[0068] In the UAV autonomous line patrol mission, the width of the target railway in the image changes with the flight height of the UAV and the position of the obstacle. In this embodiment, a multi-scale attention mechanism is designed. The width of the segmented stripes increases with the increase of the network depth, and railway features of different widths that may appear are extracted at different scales. Figure 3 In stage 1 shown as follows, the feature map size is 256×128, the receptive field size is 7×7, and the attention width is 2; in stage 2, the feature map size is 128×64, the receptive field size is 11×11, and the attention width is 4; in stage 3, the feature map size is 64×32, the receptive field size is 19×19, and the attention width is 8; in stage 4, the feature map size is 64×32, the receptive field size is 35×35, and the attention width is 8. As the network depth increases, through the downsampling module, the resolution of the feature map decreases while the receptive field size increases. Increasing the attention width can improve the ability to extract large target features, and at the same time, the number of segmented stripes decreases, which is consistent with the actual situation that the wider the target railway in the image, the fewer the number.

[0069] Similarly, the direction of the target railway in the image changes with the flight direction of the UAV and the position of the obstacle. The present invention designs a multi-direction attention mechanism. In four stages, multiple discrete attention direction combinations are used to traverse and extract railway features of different directions that may appear. To adapt to edge computing platforms with different computing powers, in this embodiment, different attention angle combinations and four sizes of attention models are designed. The designed attention directions are as follows Figure 4 shown, and the attention angle combination is formula (1):

[0070]

[0071] The network structures of the four sizes of attention models are all as followsFigure 3 As shown, it is basically the same except for the multi-scale multi-directional hybrid linear attention in the attention module. The attention modules in the four stages of the small attention model all adopt the s1 attention angle combination; the attention modules in the four stages of the medium attention model all adopt the s2 attention angle combination; the attention modules in the four stages of the large attention model all adopt the s3 attention angle combination; in the hybrid attention model, the attention module in stage 1 adopts the s1 attention angle combination, the attention module in stage 2 adopts the s2 attention angle combination, and the attention modules in stages 3 and 4 both adopt the s3 attention angle combination. In each attention module, all the set angle values are traversed in sequence, the feature map is segmented and recombined, and the combined multiple feature maps are the mapped features of this attention module.

[0072] Therefore, the multi-scale multi-directional hybrid linear attention mechanism designed in this embodiment divides the image into new dimensions according to the linear shape of the set hybrid width and direction, and extracts railway features.

[0073] (2) In this embodiment, according to the real-time flight requirements of the drone, an innovative multi-directional attention parameter sharing mechanism is proposed, and the number of model parameters is reduced and the calculation speed is improved through this method.

[0074] The multi-directional attention parameter sharing mechanism is as Figure 5 shown. In the attention module of each stage, first calculate the Query-Key and Value of the attention. In all the attention directions traversed in the current stage, two groups of parameters are shared. The Query-Key and Value are segmented and combined into new dimensions with the same width but different directions respectively, multiple groups of attention feature maps are calculated, and the attention features of this stage are obtained by merging.

[0075] (3) According to the complex environment of the railway line, by adding a semantic segmentation branch parallel to the skeleton detection branch, the misrecognition rate is reduced.

[0076] When extracting the railway skeleton, since there are few effective features in the skeleton, similar objects around the railway line may be misidentified as detection targets, such as roads, bridges, stairs, edges, shadows, etc. To reduce the misidentification rate, the present invention adds a semantic segmentation branch parallel to the skeleton detection branch to assist in locating railway targets; the network Backbone and network Neck of the semantic segmentation branch are the same as those of the skeleton detection branch. In the network Head, the four side output feature maps of the network Neck are converted to the same size through multiple convolutions and upsamplings, and then the sum of the feature maps is regularized and convolved to obtain a binary image of railway segmentation. After passing through the four-stage multi-scale multi-direction hybrid linear attention network Backbone and the feature pyramid upsampling network Neck, the network Head includes a parallel skeleton detection branch and semantic segmentation branch. The skeleton branch extracts railway skeletons and hybrid railway skeletons of different layers through a combination of multiple side output layers.

[0077] Step S2. Calculate the flight speed and attitude angle based on the skeleton points to achieve autonomous navigation of the UAV, including:

[0078] (1) According to the custom control strategy, smooth target discrete points are extracted at fixed distance intervals from the railway skeleton obtained in step S1;

[0079] After detecting the railway skeleton in the image through step S1, according to the custom control strategy of flying along the right railway, the rightmost skeleton in the image is selected as the target skeleton. To ensure smooth flight, based on the empirical value of the fixed pixel distance, several discrete points are evenly extracted on the target skeleton and used as candidate flight trajectory points after coordinate transformation.

[0080] (2) According to the discrete target position and the current position, the forward nearest smooth point is selected as the target point among the candidate points;

[0081] Assume that the UAV coordinate when the neural network detects the skeleton image is O1, and the target point coordinate is A i , where i is the target point index, and the UAV coordinate when the control algorithm uses this target point is O2. The distance vector of the UAV from the current position to the target point is Equation (2):

[0082]

[0083] Considering the flight oscillation caused by the change of the UAV speed, to ensure smooth flight, according to the magnitude and direction of the distance vector of the candidate target point, the nearest smooth point is selected as the flight target point of the current image inference result.

[0084] (3) According to the UAV speed control method, a smooth linear speed interpolation method is proposed to control the target flight speed and target attitude angle of the UAV in real time;

[0085] Assume the current flight speed is (v N0 , v E0 , v D0 ), the direction angle of the target point is Φ, the magnitude of the target flight speed is v, and the flight speed of the target point (v N1 , v E1 , v D1 ) is Equation (3). To ensure the smooth change of speed during flight, according to the linear interpolation of the smooth speed, calculate the flight speed (v Nj , v Ej , v Dj ) at each time t under the set control frequency, and adjust the flight speed according to the direction angle of the target point, where j is the control segment index:

[0086]

[0087] The vector angle Θ between candidate target points is Equation (5), that is, the angle of the railway skeleton. The yaw angle of the target attitude angle of the UAV and the yaw angle of the pan-tilt camera are angle Θ.

[0088]

[0089] Under the set control frequency and the speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to achieve the autonomous navigation control of the UAV.

[0090] Embodiment 3

[0091] This Embodiment 3 provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the end-to-end UAV autonomous flight navigation method based on skeleton recognition as described above is implemented. The method includes:

[0092] Obtain railway images;

[0093] Process the obtained railway images by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway images. Among them, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network. In the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, by using a multi-scale attention mechanism, railway features of different widths that may appear are extracted at different scales, and by using a multi-directional attention mechanism, multiple discrete attention direction combinations are used to traverse and extract railway features of different directions that may appear. In the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton;

[0094] According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals; according to the positions of the smooth target discrete points and the current position, the forward nearest smooth point is selected as the target point from the candidate points, and combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

[0095] Embodiment 4

[0096] This Embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the above-mentioned end-to-end UAV autonomous flight navigation method based on skeleton recognition. The method includes:

[0097] Obtain railway images;

[0098] The obtained railway images are processed by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway images; wherein, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, by using a multi-scale attention mechanism, different-width railway features that may appear at different scales are extracted, and by using a multi-direction attention mechanism, multiple discrete attention directions are combined to traverse and extract different-direction railway features that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton.

[0099] According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals; according to the positions of the smooth target discrete points and the current position, the forward nearest smooth point is selected as the target point from the candidate points, and combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

[0100] Embodiment 5

[0101] This Embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device runs, the processor executes the computer program stored in the memory so that the electronic device executes the instructions for implementing the above-mentioned end-to-end UAV autonomous flight navigation method based on skeleton recognition. The method includes:

[0102] Obtain railway images;

[0103] Process the acquired railway image using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image. Among them, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network. In the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, using a multi-scale attention mechanism, different-width railway features that may appear are extracted at different scales, and using a multi-directional attention mechanism, multiple discrete attention direction combinations are adopted to traverse and extract different-direction railway features that may appear. In the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton.

[0104] Extract smooth target discrete points at fixed distance intervals according to the acquired railway skeleton. According to the positions of the smooth target discrete points and the current position, select the forward nearest smooth point as the target point among the candidate points, and combine with the smooth linear velocity interpolation method to real-time control the target flight speed and target attitude angle of the unmanned aerial vehicle.

[0105] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a machine for realizing the Figure 1 a process or multiple processes and / or blocks Figure 1 a device for the functions specified in one or more blocks.

[0107] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the Figure 1 a process or multiple processes and / or blocks Figure 1 a device for the functions specified in one or more blocks.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, where they perform a series of operational steps to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 or steps for implementing the functions specified in a plurality of blocks.

[0109] Although the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative efforts should be covered within the scope of protection of the present invention.

Claims

1. An end-to-end autonomous flight navigation method for drones along railways based on skeleton recognition, characterized in that, Including: Obtain a railway image; Process the obtained railway image using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image. Among them, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network. In the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, a multi-scale attention mechanism is used to extract different-width railway features that may appear at different scales, and a multi-directional attention mechanism is used to traverse and extract different-direction railway features that may appear by combining multiple discrete attention directions. In the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton; Extract smooth target discrete points at fixed distance intervals according to the obtained railway skeleton. According to the positions of the smooth target discrete points and the current position, select the forward nearest smooth point as the target point among the candidate points, and combine the smooth linear velocity interpolation method to control the target flight speed and target attitude angle of the UAV in real time.

2. The end-to-end autonomous flight navigation method of an unmanned aerial vehicle along a railway based on skeleton recognition according to claim 1, characterized in that Use the attention direction parameter sharing mechanism to reduce the number of parameters and share parameters when calculating all attention directions. It includes: in each stage of the attention module, first calculate the Query-Key and Value of the attention, share two sets of parameters among all attention directions traversed in the current stage, divide the Query-Key and Value by the same width but different directions and combine them into a new dimension, calculate multiple sets of attention feature maps, and merge them to obtain the attention feature of this stage.

3. The end-to-end autonomous flight navigation method of an unmanned aerial vehicle along a railway based on skeleton recognition according to claim 1, wherein Based on the railway skeleton in the detected image, according to the custom control strategy of flying along the right railway, select the rightmost skeleton in the image as the target skeleton. To ensure smooth flight, extract several discrete points evenly on the target skeleton with an empirical value of a fixed pixel distance, and use them as candidate flight trajectory points after coordinate transformation.

4. The end-to-end autonomous flight navigation method of an unmanned aerial vehicle along a railway based on skeleton recognition according to claim 3, wherein According to the discrete target position and the current position, select the forward nearest smooth point as the target point among the candidate points, including: Assume that the UAV coordinates are O1 and the target point coordinates are A when the neural network detects the skeleton image i , where i is the target point index. The UAV coordinates are O2 when the control algorithm uses this target point. The distance vector from the current position of the UAV to the target point is: Considering the flight oscillation caused by the change of the UAV speed, to ensure smooth flight, select the nearest smooth point as the flight target point of the current image inference result according to the magnitude and direction of the distance vector of the candidate target point.

5. The end-to-end autonomous flight navigation method of an unmanned aerial vehicle along a railway based on skeleton recognition according to claim 4, wherein, The smooth linear velocity interpolation method to control the target flight speed and target attitude angle of the UAV in real time, including: Assume the current flight speed is (v N0 , v E0 , v D0 ), the target point direction angle is Φ, the magnitude of the target flight speed is v, and the flight speed of the target point is (v N1 , v E1 , v D1 ). To ensure the smooth change of speed during flight, according to the linear interpolation of smooth speed, calculate the flight speed (v Nj , v Ej , v Dj ) at each time t under the set control frequency, and adjust the flight speed according to the target point direction angle.

6. The end-to-end UAV autonomous flight navigation method along the railway based on skeleton recognition according to claim 5, characterized in that: The vector angle Θ between candidate target points, that is, the angle of the railway skeleton, and the yaw angle of the UAV target attitude angle and the yaw angle of the pan-tilt camera are angle Θ: Under the set control frequency and the speed control mode of the UAV, update the target flight speed and target attitude angle of the UAV in real time to achieve autonomous navigation control of the UAV.

7. An end-to-end autonomous flight navigation system for an unmanned aerial vehicle along a railway based on skeleton recognition, characterized in that, Including: An acquisition module for acquiring a railway image; A positioning module, which is used to process the acquired railway image by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image; wherein, the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and recombined into new-dimensional features to enhance the extraction of railway skeleton features. Among them, a multi-scale attention mechanism is used to extract different-width railway features that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract different-direction railway features that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in positioning the railway skeleton. A control module, which is used to extract smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; select the forward nearest smooth point as the target point from the candidate points according to the positions of the smooth target discrete points and the current position, and combine the smooth linear velocity interpolation method to control the target flight speed and target attitude angle of the UAV in real time.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the end-to-end UAV autonomous flight navigation method based on skeleton recognition according to any one of claims 1-6 is implemented.

9. A computer device, characterized in that, It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the end-to-end UAV autonomous flight navigation method based on skeleton recognition according to any one of claims 1-6.

10. An electronic device, characterized in that, Comprising: A processor, a memory, and a computer program; wherein, the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device runs, the processor executes the computer program stored in the memory so that the electronic device executes the instructions for implementing the end-to-end UAV autonomous flight navigation method based on skeleton recognition according to any one of claims 1-6.

Images