End-to-end unmanned aerial vehicle autonomous flight navigation method and system along railway based on skeleton identification

Abstract

The invention provides an end-to-end unmanned aerial vehicle autonomous flight navigation method and system along a railway based on skeleton recognition, belongs to the technical field of unmanned aerial vehicle intelligent control, and detects a railway skeleton in real time based on a multi-scale direction mixed linear attention parallel network. And calculating a flight speed and an attitude angle based on speed smooth linear interpolation to realize autonomous navigation control of the unmanned aerial vehicle. According to the method, a multi-scale multi-direction mixed linear attention mechanism is designed for railway linear invariant features and unmanned aerial vehicle flight tasks to enhance railway skeleton feature extraction, and a semantic segmentation branch parallel to a skeleton detection branch is utilized to reduce wrong recognition, so that high-precision railway skeleton detection is realized; aiming at an edge computing platform with limited computing resources, a multi-direction attention parameter sharing mechanism is designed, the network parameter quantity is reduced, and the skeleton detection speed is improved; for a railway skeleton obtained by network detection, smooth and stable unmanned aerial vehicle autonomous high-speed flight control along the line is realized by utilizing speed smooth linear interpolation calculation.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) intelligent control, and in particular to an end-to-end UAV autonomous flight navigation method and system along a railway based on skeleton recognition. Background Art

[0002] In order to improve inspection efficiency and ensure personnel safety, the UAV-based railway line inspection solution is the mainstream development trend to replace the current manual inspection. In order to avoid the distance limitation of remote control and reduce the dependence on GPS positioning, the realization of high-precision railway target real-time detection technology based on vision is a better solution for UAVs to autonomously navigate along the railway line.

[0003] After using the image segmentation neural network to segment the railway track, post-processing such as fitting is required to obtain the UAV flight path, but the target result will be affected by empirical parameters. Therefore, an end-to-end detection network is needed to directly obtain the flight trajectory and meet the requirements of UAV computing equipment. Summary of the invention

[0004] The object of the present invention is to provide an end-to-end UAV autonomous flight navigation method and system along a railway based on skeleton recognition, so as to solve at least one technical problem existing in the above-mentioned background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides an end-to-end UAV autonomous flight navigation method along a railway based on skeleton recognition, comprising:

[0007] Acquire railway images;

[0008] The acquired railway image is processed by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton;

[0009] According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals. According to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point among the candidate points. Combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

[0010] Furthermore, the attention direction parameter sharing mechanism is used to reduce the number of parameters, and the parameters are shared when traversing the calculations of all attention directions, including: in the attention module of each stage, the attention Query-Key and Value are first calculated, and two sets of parameters are shared in all attention directions traversed in the current stage. The Query-Key and Value are split with the same width but different directions and combined into new dimensions, and multiple sets of attention feature maps are calculated and merged to obtain the attention features of this stage.

[0011] Furthermore, by detecting the railway skeleton in the image, according to the customized control strategy of flying along the right railway, the rightmost skeleton in the image is selected as the target skeleton. To ensure smooth flight, a number of discrete points are extracted on the target skeleton on average with an empirical value of a fixed pixel distance, and the coordinate system is converted to serve as candidate flight trajectory points.

[0012] Furthermore, according to the discrete target position and the current position, the nearest forward smooth point is selected as the target point from the candidate points, including:

[0013] Assume that the coordinates of the drone when the neural network detects the skeleton image are O1 and the coordinates of the target point are A i , where i is the target point index, the coordinates of the drone when the control algorithm uses this target point are O2, and the distance vector from the current position of the drone to the target point is:

[0014]

[0015] Taking into account the flight oscillation caused by the change of the UAV speed, in order to ensure smooth flight, the nearest smooth point is selected as the flight target point of the current image inference result according to the size and direction of the distance vector of the candidate target point.

[0016] Furthermore, the smooth linear velocity interpolation method controls the target flight velocity and target attitude angle of the UAV in real time, including:

[0017] Assume that the current flight speed is (v N0 ,v E0 ,v D0 ), the target point direction angle is Φ, the target flight speed is v, and the target point flight speed is (v N1 ,v E1 ,v D1 ), in order to ensure the smooth change of speed during flight, the flight speed (v Nj ,v Ej ,v Dj ) and adjust the flight speed according to the direction angle of the target point.

[0018] Furthermore,

[0019]

[0020] The vector angle θ between the candidate target points, that is, the angle of the railway skeleton, the yaw angle of the drone target attitude angle and the yaw angle of the gimbal camera is angle θ:

[0021]

[0022] Under the set control frequency and speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to realize autonomous navigation control of the UAV.

[0023] In a second aspect, the present invention provides an end-to-end UAV autonomous flight navigation system along a railway based on skeleton recognition, comprising:

[0024] An acquisition module, used for acquiring railway images;

[0025] A positioning module is used to process the acquired railway image using a pre-trained railway skeleton detection model to obtain a railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton;

[0026] The control module is used to extract smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; according to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point from the candidate points, and the target flight speed and target attitude angle of the UAV are controlled in real time in combination with the smooth linear velocity interpolation method.

[0027] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the end-to-end unmanned aerial vehicle autonomous flight navigation method along the railway based on skeleton recognition as described in the first aspect is implemented.

[0028] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the end-to-end unmanned aerial vehicle autonomous flight navigation method along a railway based on skeleton recognition as described in the first aspect.

[0029] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the end-to-end unmanned aerial vehicle autonomous flight navigation method along the railway based on skeleton recognition as described in the first aspect.

[0030] The beneficial effects of the present invention are as follows: a multi-scale directional hybrid linear attention parallel network tested by comparative experiments and ablation experiments detects high-precision railway skeletons through a multi-scale and multi-directional hybrid linear attention mechanism, reduces computing resources and improves computing speed through a multi-directional attention parameter sharing mechanism, reduces misrecognition rate through parallel semantic segmentation branches, and an end-to-end UAV autonomous flight navigation system based on skeleton recognition tested by field experiments calculates the flight speed and posture of the UAV through speed smoothing linear interpolation, thereby achieving smooth, stable and rapid autonomous navigation of the UAV along the route.

[0031] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0033] Figure 1 The present invention is a schematic diagram of an algorithm flow of an end-to-end UAV autonomous flight navigation method based on railway skeleton recognition according to an embodiment of the present invention.

[0034] Figure 2 The figure is a schematic diagram of the hardware structure of the end-to-end UAV autonomous flight navigation method based on railway skeleton recognition described in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of a multi-scale directional hybrid linear attention parallel network according to an embodiment of the present invention.

[0036] Figure 4This is a schematic diagram of different attention direction angles in the multi-scale and multi-directional hybrid linear attention mechanism described in an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of the multi-directional attention parameter sharing mechanism in the multi-scale and multi-directional hybrid linear attention mechanism described in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0039] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0040] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0041] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0042] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0043] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0044] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0045] Example 1

[0046] In this embodiment 1, an end-to-end unmanned aerial vehicle autonomous flight navigation system along the railway based on skeleton recognition is first provided, including: an acquisition module for acquiring railway images. A positioning module for processing the acquired railway images using a pre-trained railway skeleton detection model to obtain a railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to use a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton. A control module is used to extract smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; according to the position of the smooth target discrete points and the current position, the forward nearest smooth point is selected from the candidate points as the target point, and the target flight speed and target attitude angle of the unmanned aerial vehicle are controlled in real time in combination with the smooth linear velocity interpolation method.

[0047] In this embodiment, the above-mentioned system can be used to implement an end-to-end unmanned aerial vehicle autonomous flight navigation method along a railway based on skeleton recognition, including: obtaining a railway image; using a pre-trained railway skeleton detection model to process the obtained railway image to obtain a railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to use a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton; according to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals; according to the position of the smooth target discrete point and the current position, the forward nearest smooth point is selected from the candidate points as the target point, and the target flight speed and target attitude angle of the unmanned aerial vehicle are controlled in real time in combination with the smooth linear velocity interpolation method.

[0048] Among them, the attention direction parameter sharing mechanism is used to reduce the number of parameters, and the parameters are shared when traversing the calculation of all attention directions, including: in the attention module of each stage, the Query-Key and Value of the attention are first calculated, and two sets of parameters are shared in all attention directions traversed in the current stage. The Query-Key and Value are split with the same width but different directions and combined into a new dimension, and multiple sets of attention feature maps are calculated and merged to obtain the attention features of this stage.

[0049] The railway skeleton in the detected image is selected as the target skeleton according to the custom control strategy of flying along the right railway. To ensure smooth flight, several discrete points are extracted on the target skeleton with an empirical value of fixed pixel distance, and the coordinate system is converted as candidate flight trajectory points. According to the discrete target position and the current position, the nearest smooth point in the forward direction is selected as the target point among the candidate points, including:

[0050] Assume that the coordinates of the drone when the neural network detects the skeleton image are O1 and the coordinates of the target point are A i , where i is the target point index, the coordinates of the drone when the control algorithm uses this target point are O2, and the distance vector from the current position of the drone to the target point is:

[0051]

[0052] Taking into account the flight oscillation caused by the change of the UAV speed, in order to ensure smooth flight, the nearest smooth point is selected as the flight target point of the current image inference result according to the size and direction of the distance vector of the candidate target point.

[0053] Using the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time, including:

[0054] Assume that the current flight speed is (v N0 ,v E0 ,v D0 ), the target point direction angle is Φ, the target flight speed is v, and the target point flight speed is (v N1 ,v E1 ,v D1 ), in order to ensure the smooth change of speed during flight, the flight speed (v Nj ,v Ej ,v Dj ) and adjust the flight speed according to the direction angle of the target point.

[0055]

[0056] The vector angle θ between the candidate target points, that is, the angle of the railway skeleton, the yaw angle of the drone target attitude angle and the yaw angle of the gimbal camera is angle θ:

[0057]

[0058] Under the set control frequency and speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to realize autonomous navigation control of the UAV.

[0059] Example 2

[0060] In this embodiment 2, an end-to-end UAV autonomous flight navigation method along a railway based on skeleton recognition is provided, comprising the following steps: S1. detecting the railway skeleton based on a multi-scale directional hybrid linear attention parallel network; S2. calculating the flight speed and attitude angle based on speed smoothing linear interpolation to realize autonomous navigation of the UAV.

[0061] Among them, the railway skeleton detection method based on a multi-scale directional hybrid linear attention parallel network described in step S1 is: based on the linear invariant features of the railway and the flight mission of the drone, a multi-scale multi-directional hybrid linear attention mechanism is used to improve the detection accuracy. In the four-stage linear attention, different attention widths and different attention direction combinations are designed, and the image is segmented and reorganized into different dimensions to perform attention calculations; the attention direction parameter sharing mechanism is used to reduce the number of parameters and improve the reasoning speed. In the attention module of each stage, parameters are shared when traversing the calculations of all attention directions; a semantic segmentation branch network parallel to the skeleton detection branch is used to reduce the error recognition rate, and the skeleton branch obtains a hybrid railway skeleton through multiple side output layers.

[0062] The method for realizing autonomous navigation of the UAV by calculating the flight speed and attitude angle based on speed smooth linear interpolation described in step S2 is: extracting smooth target discrete points at fixed distance intervals according to a custom control strategy, selecting the nearest forward smooth point as the flight target point among the candidate points, calculating the target flight speed and target attitude angle using the speed smooth linear interpolation method, updating the control amount in real time, and realizing autonomous navigation of the UAV.

[0063] In this embodiment, a multi-scale directional hybrid linear attention parallel network that has been tested by comparative experiments and ablation experiments detects high-precision railway skeletons through a multi-scale and multi-directional hybrid linear attention mechanism, reduces computing resources and improves computing speed through a multi-directional attention parameter sharing mechanism, reduces the misrecognition rate through parallel semantic segmentation branches, and an end-to-end UAV autonomous flight navigation system based on skeleton recognition that has been tested in field experiments calculates the flight speed and posture of the UAV through speed smooth linear interpolation, thereby achieving smooth, stable and rapid autonomous navigation of the UAV along the line.

[0064] Specifically, the algorithm flow of the end-to-end UAV autonomous flight navigation method based on railway skeleton recognition described in this embodiment is as follows: Figure 1 As shown, the hardware structure is as follows Figure 2 As shown, the multi-scale direction mixed linear attention parallel network structure is as follows Figure 3 shown.

[0065] Step S1. Detecting railway skeletons based on a multi-scale directional hybrid linear attention parallel network, including:

[0066] (1) According to the UAV flight mission, an innovative multi-scale and multi-directional hybrid linear attention mechanism is proposed for the invariant features of railways. The railway skeleton feature extraction is enhanced through this attention mechanism.

[0067] Within the field of view of the drone, unlike features such as color that are easily affected by the environment, the shape of the railway always maintains a linear invariant feature. In this embodiment, the feature map is segmented into linear stripes through linear attention and reorganized into new dimensional features.

[0068] In the UAV autonomous line inspection mission, the width of the target railway in the image varies with the flight altitude of the UAV and the position of obstacles. This embodiment designs a multi-scale attention mechanism. The width of the segmented stripes increases with the depth of the network. The railway features of different widths that may appear at different scales are extracted. Figure 3 In the stage 1 shown, the feature map size is 256×128, the receptive field size is 7×7, and the attention width is 2; the feature map size in stage 2 is 128×64, the receptive field size is 11×11, and the attention width is 4; the feature map size in stage 3 is 64×32, the receptive field size is 19×19, and the attention width is 8; the feature map size in stage 4 is 64×32, the receptive field size is 35×35, and the attention width is 8. As the network depth increases, through the downsampling module, the feature map resolution decreases and the receptive field size increases. Increasing the attention width can improve the ability to extract large target features, while reducing the number of segmented stripes, which is consistent with the actual situation that the larger the width of the target railway in the image, the fewer the number.

[0069] Similarly, the direction of the target railway in the image changes with the flight direction of the drone and the position of the obstacle. The present invention designs a multi-directional attention mechanism. In the four stages, multiple discrete attention direction combinations are used to traverse and extract railway features in different directions that may appear. In order to adapt to edge computing platforms with different computing powers, different attention angle combinations and four sizes of attention models are designed in this embodiment. The designed attention directions are as follows: Figure 4 As shown, the attention angle combination is formula (1):

[0070]

[0071] The network structures of the four attention models are the same Figure 3 As shown in the figure, except for the multi-scale and multi-directional hybrid linear attention in the attention module, they are basically the same. The attention modules in the four stages of the small attention model all use the s1 attention angle combination; the attention modules in the four stages of the medium attention model all use the s2 attention angle combination; the attention modules in the four stages of the large attention model all use the s3 attention angle combination; in the hybrid attention model, the attention module in stage 1 uses the s1 attention angle combination, the attention module in stage 2 uses the s2 attention angle combination, and the attention modules in stages 3 and 4 use the s3 attention angle combination. In each attention module, all the set angle values ​​are traversed in turn, the feature maps are segmented and reorganized, and the reorganized multiple feature maps are combined to form the mapping features of the attention module.

[0072] Therefore, the multi-scale and multi-directional mixed linear attention mechanism designed in this embodiment divides the image into new dimensions according to the linear shape of the set mixed width and direction to extract railway features.

[0073] (2) In this embodiment, according to the real-time flight requirements of the UAV, an innovative multi-directional attention parameter sharing mechanism is proposed. This method reduces the number of model parameters and improves the calculation speed.

[0074] The multi-directional attention parameter sharing mechanism is as follows Figure 5 As shown in the figure, in the attention module of each stage, the Query-Key and Value of the attention are first calculated, and two sets of parameters are shared in all attention directions traversed in the current stage. The Query-Key and Value are split with the same width in different directions and combined into new dimensions. Multiple sets of attention feature maps are calculated and merged to obtain the attention features of this stage.

[0075] (3) According to the complex environment of railway lines, the misrecognition rate is reduced by adding a semantic segmentation branch in parallel with the skeleton detection branch.

[0076] When extracting the railway skeleton, since the skeleton contains fewer effective features, similar objects around the railway line may be mistakenly identified as detection targets, such as roads, bridges, stairs, edges, shadows, etc. In order to reduce the misidentification rate, the present invention adds a semantic segmentation branch parallel to the skeleton detection branch to assist in locating the railway target; the network Backbone and network Neck of the semantic segmentation branch are the same as the skeleton detection branch. In the network Head, the four side output feature maps of the network Neck are converted to the same size through multiple convolutions and upsampling, and the sum of the feature maps is regularized and convolved to obtain a binary image of the railway segmentation. After the four-stage multi-scale multi-directional hybrid linear attention network Backbone and the feature pyramid upsampling network Neck, the network Head includes a parallel skeleton detection branch and a semantic segmentation branch, and the skeleton branch extracts railway skeletons and hybrid railway skeletons of different layers through a combination of multiple side output layers.

[0077] Step S2. Calculate the flight speed and attitude angle based on the skeleton points to achieve autonomous navigation of the UAV, including:

[0078] (1) According to the railway skeleton obtained from step S1 according to the custom control strategy, it is proposed to extract smooth target discrete points at fixed distance intervals;

[0079] After the railway skeleton in the image is detected by step S1, the rightmost skeleton in the image is selected as the target skeleton according to the customized control strategy of flying along the right railway. To ensure smooth flight, several discrete points are extracted on the target skeleton on average with an empirical value of a fixed pixel distance, and the coordinate system is converted to serve as candidate flight trajectory points.

[0080] (2) According to the discrete target position and the current position, the nearest forward smooth point is selected from the candidate points as the target point;

[0081] Assume that the coordinates of the drone when the neural network detects the skeleton image are O1 and the coordinates of the target point are A i , where i is the target point index, the coordinates of the UAV when the control algorithm uses the target point are O2, and the distance vector from the current position of the UAV to the target point is formula (2):

[0082]

[0083] Taking into account the flight oscillation caused by the change of the UAV speed, in order to ensure smooth flight, the nearest smooth point is selected as the flight target point of the current image inference result according to the size and direction of the distance vector of the candidate target point.

[0084] (3) Based on the UAV speed control method, a smooth linear velocity interpolation method is proposed to control the target flight speed and target attitude angle of the UAV in real time;

[0085] Assume that the current flight speed is (v N0 ,v E0 ,v D0 ), the target point direction angle is Φ, the target flight speed is v, the target point flight speed (v N1 ,v E1 ,v D1 ) is formula (3). To ensure smooth speed change during flight, the flight speed (v) of each time period t under the set control frequency is calculated according to the linear interpolation of the smooth speed. Nj ,v Ej ,v Dj ) is formula (4) and the flight speed is adjusted according to the direction angle of the target point, where j is the control segment index:

[0086]

[0087] The vector angle θ between the candidate target points is given by equation (5), i.e., the angle of the railway skeleton, the yaw angle of the drone target attitude angle and the yaw angle of the gimbal camera are angle θ.

[0088]

[0089] Under the set control frequency and speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to realize autonomous navigation control of the UAV.

[0090] Example 3

[0091] This embodiment 3 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned end-to-end UAV autonomous flight navigation method along a railway based on skeleton recognition is implemented. The method includes:

[0092] Acquire railway images;

[0093] The acquired railway image is processed by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton;

[0094] According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals. According to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point among the candidate points. Combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

[0095] Example 4

[0096] This embodiment 4 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned end-to-end unmanned aerial vehicle autonomous flight navigation method along a railway based on skeleton recognition, the method comprising:

[0097] Get railway images;

[0098] The acquired railway image is processed by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton;

[0099] According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals. According to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point among the candidate points. Combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

[0100] Example 5

[0101] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the above-mentioned end-to-end unmanned aerial vehicle autonomous flight navigation method along a railway based on skeleton recognition, the method comprising:

[0102] Get railway images;

[0103] The acquired railway image is processed by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton;

[0104] According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals. According to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point among the candidate points. Combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

[0105] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end UAV autonomous flight navigation method along a railway based on skeleton recognition, characterized in that: include: Get railway images; The acquired railway image is processed by using a pre-trained railway skeleton detection model to obtain the railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton; According to the obtained railway skeleton, smooth target discrete points are extracted at fixed distance intervals. According to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point among the candidate points. Combined with the smooth linear velocity interpolation method, the target flight speed and target attitude angle of the UAV are controlled in real time.

2. The end-to-end UAV autonomous flight navigation method along the railway based on skeleton recognition according to claim 1 is characterized in that: The attention direction parameter sharing mechanism is used to reduce the number of parameters. The parameters are shared when traversing the calculations of all attention directions, including: in the attention module of each stage, the Query-Key and Value of the attention are first calculated, and two sets of parameters are shared in all attention directions traversed in the current stage. The Query-Key and Value are split with the same width but different directions and combined into new dimensions, and multiple sets of attention feature maps are calculated and merged to obtain the attention features of this stage.

3. The end-to-end UAV autonomous flight navigation method along the railway based on skeleton recognition according to claim 1 is characterized in that: By detecting the railway skeleton in the image, according to the customized control strategy of flying along the right railway, the rightmost skeleton in the image is selected as the target skeleton. To ensure smooth flight, several discrete points are extracted on the target skeleton with an empirical value of fixed pixel distance on average, and the coordinate system is converted to serve as candidate flight trajectory points.

4. The end-to-end UAV autonomous flight navigation method along the railway based on skeleton recognition according to claim 3 is characterized in that: According to the discrete target position and the current position, the nearest forward smooth point is selected as the target point from the candidate points, including: Assume that the coordinates of the drone when the neural network detects the skeleton image are O1 and the coordinates of the target point are A i , where i is the target point index, the coordinates of the drone when the control algorithm uses this target point are O2, and the distance vector from the current position of the drone to the target point is: Taking into account the flight oscillation caused by the change of the UAV speed, in order to ensure smooth flight, the nearest smooth point is selected as the flight target point of the current image inference result according to the size and direction of the distance vector of the candidate target point.

5. The end-to-end UAV autonomous flight navigation method along the railway based on skeleton recognition according to claim 4 is characterized in that: Smooth linear velocity interpolation method, real-time control of the target flight speed and target attitude angle of the UAV, including: Assume that the current flight speed is (v N0 ,v E0 ,v D0 ), the target point direction angle is Φ, the target flight speed is v, and the target point flight speed is (v N1 ,v E1 ,v D1 ), in order to ensure the smooth change of speed during flight, the flight speed (v Nj ,v Ej ,v Dj ) and adjust the flight speed according to the direction angle of the target point.

6. The end-to-end UAV autonomous flight navigation method along a railway based on skeleton recognition according to claim 5 is characterized in that: The vector angle θ between the candidate target points, that is, the angle of the railway skeleton, the yaw angle of the drone target attitude angle and the yaw angle of the gimbal camera is angle θ: Under the set control frequency and speed control mode of the UAV, the target flight speed and target attitude angle of the UAV are updated in real time to realize autonomous navigation control of the UAV.

7. An end-to-end UAV autonomous flight navigation system along a railway based on skeleton recognition, characterized in that: include: An acquisition module, used for acquiring railway images; A positioning module is used to process the acquired railway image using a pre-trained railway skeleton detection model to obtain a railway skeleton in the railway image; wherein the railway skeleton detection model includes a skeleton detection branch network and a semantic segmentation branch network; in the skeleton detection branch network, the feature map is segmented into linear stripes and reorganized into new dimensional features to enhance the railway skeleton feature extraction, wherein a multi-scale attention mechanism is used to extract railway features of different widths that may appear at different scales, and a multi-directional attention mechanism is used to adopt a combination of multiple discrete attention directions to traverse and extract railway features of different directions that may appear; in the semantic segmentation branch network, a feature pyramid upsampling network is used to assist in locating the railway skeleton; The control module is used to extract smooth target discrete points at fixed distance intervals according to the acquired railway skeleton; according to the position of the smooth target discrete points and the current position, the nearest forward smooth point is selected as the target point from the candidate points, and the target flight speed and target attitude angle of the UAV are controlled in real time in combination with the smooth linear velocity interpolation method.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the end-to-end unmanned aerial vehicle autonomous flight navigation method along the railway based on skeleton recognition is implemented as described in any one of claims 1-6.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the end-to-end unmanned aerial vehicle autonomous flight navigation method along the railway based on skeleton recognition as described in any one of claims 1-6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the end-to-end UAV autonomous flight navigation method along the railway based on skeleton recognition as described in any one of claims 1-6.

Images