Unmanned aerial vehicle route planning and pose calibration method and system

Abstract

The invention provides an unmanned aerial vehicle route planning and pose calibration method and system, and belongs to the technical field of unmanned aerial vehicle control based on computer vision. In the flight process of the unmanned aerial vehicle, a track picture is obtained every time the unmanned aerial vehicle reaches a specified waypoint, the vertical distance between the position and a track line is recognized and calculated, whether the unmanned aerial vehicle is within an error distance or not is judged, that is, whether the unmanned aerial vehicle yaws or not is judged, and if yes, the unmanned aerial vehicle is corrected to the correct position; if not, whether the waypoint is the last waypoint or not is judged, if not, the unmanned aerial vehicle continues to fly to the next waypoint, and if yes, the unmanned aerial vehicle stops advancing and arrives at the terminal point. According to the invention, track identification is optimized, and the track identification accuracy is improved; an unmanned aerial vehicle route is drawn through a track line, external factors such as a map and a train timetable are comprehensively considered, and the optimal unmanned aerial vehicle route is drawn under the condition that the lowest cost is guaranteed; automatic track calibration of the unmanned aerial vehicle in the air is achieved, and therefore errors in the flight process are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle control based on computer vision, and in particular to a method and system for unmanned aerial vehicle route planning and posture calibration. Background Art

[0002] At present, traditional railway inspection drones are basically manually operated. Workers control the drones to reach the positioning point, and the inspectors need to follow the drones into the tunnel in real time. This technology consumes a lot of human resources and increases the learning cost of workers. At the same time, a series of safety hazards may be caused by human misjudgment or untimely information transmission.

[0003] Drone inspection can be applied to the inspection and maintenance of various facilities, such as power lines, bridges, wind farms, petrochemical facilities, etc. These locations usually require special requirements such as high-altitude perspective, high safety and high data accuracy, and drones just meet these requirements. Traditional drone line inspection flight cannot integrate external factors such as terrain and trains in route planning. It relies solely on track lines to achieve track planning, which greatly increases the possibility of accidents during the flight of drones; secondly, the drone may deviate from the planned route due to wind or external factors during flight. In terms of drone track inspection, drone flight personnel need to undergo various technical training and emergency training for emergencies, which consumes a lot of human resource training costs, and has low work efficiency and is difficult to handle multiple tasks at the same time; in terms of drone track inspection route design, the existing track planning methods have low accuracy and are difficult to comprehensively consider many other factors such as terrain and trains, which often makes it difficult to implement autonomous track inspection as planned.

[0004] An existing method for identifying railway tracks by unmanned aerial vehicles uses an onboard camera of a unmanned aerial vehicle to obtain a video image of a railway track and perform preprocessing; uses a pulse coupled neural network to identify the track route; and uses a third-order Bezier curve fitting to implement flight calculation of local target points. This technology uses a method for calculating local target points on the fly, which requires the measurement of the route during the flight of the unmanned aerial vehicle and the calculation of the subsequent flight route while flying. This method has a large amount of calculation, requires higher endurance for the unmanned aerial vehicle, and greatly increases the calculation cost of the unmanned aerial vehicle. This technology only uses local target waypoints to be planned during flight, and cannot guarantee that the unmanned aerial vehicle can effectively avoid obstacles during flight, which increases the risk of accidents during unmanned aerial vehicle flight. This technology does not explore how to select the target waypoint for the next flight of the unmanned aerial vehicle when a fork in the road occurs. When encountering a fork in the track route, the unmanned aerial vehicle has difficulty in choosing a route. This technology has high requirements for real-time performance, and it is difficult to achieve a balance between real-time performance and accuracy. With the continuous increase in the amount of unmanned aerial vehicle image data, it is difficult to store and process this data efficiently.

[0005] Another existing method, device, storage medium and electronic device for UAV trajectory planning first plans the path through k-means++ combined with the different importance of each target point, then completes the coverage within the range limit based on the UAV's endurance and obstacles as constraints, and finally solves through the DDQN algorithm to achieve path planning and improve the path planning effect. This technology sets different target points for local calculation of the track, which has a large amount of calculation and is difficult to balance with the UAV's endurance. This technology uses path planning in the early stage of the UAV's patrol, which takes a long time to calculate and has certain limitations on applicable scenarios. The local calculation of this technology causes the amount of image data stored by the UAV to be large, making it difficult to store and process this data efficiently. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for unmanned aerial vehicle route planning and posture calibration to solve at least one technical problem existing in the above-mentioned background technology.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for unmanned aerial vehicle route planning and posture calibration, comprising:

[0009] Obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, determine whether to implement multi-UAV dispatch;

[0010] Obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; quantify the priority of the reference objects according to their selection priority order, and form a reference point selection function based on the reference point distance, select the optimal reference point and set it as the drone observation point;

[0011] The initial route is drawn based on the track line and the determined observation points. The area that affects the flight of the drone is marked by combining the map information and the drone flight parameters. If there are obstacles that affect the flight of the drone in the straight-line distance between the drone and the observation point, the marked area will be avoided in the subsequent track planning as the final route.

[0012] During the flight of the drone, each time it reaches a specified waypoint, it obtains a track picture, identifies and calculates the vertical distance between the drone's location and the track line, and determines whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

[0013] Furthermore, it is determined whether it is necessary to call multiple drones for detection based on the acquired track map; among them, it is determined whether the track is a one-way track or a two-way track based on the train timetable. If it is a one-way track, a fork in the road judgment is performed; if it is a two-way track, the train timetable is used to determine whether the direction of the incoming vehicle is one-way or two-way within the limited time. If it is one-way, the fork in the road judgment module is performed, and if it is two-way, multiple drone dispatch is prepared.

[0014] Furthermore, based on the obtained railway route map, it is determined whether there is an intersection in the map and thus whether there is a fork in the track; if there is no fork, the route is directly drawn; if there is a fork, it is determined whether the fork exists within the distance range of the drone flight target based on the known drone flight distance parameters. If not, the useless fork track line is cleared. If so, it is determined in combination with the train timetable module whether there will be vehicles coming from different directions at the fork within the limited time. If not, the line without vehicles is considered to be a useless fork. If so, it is determined to draw multiple drone routes, and the drone routes of the two track lines after the fork are drawn separately, and multi-drone scheduling is started.

[0015] Furthermore, map information within the observation range is obtained and visual recognition is performed, all selectable reference objects within the flight range of the UAV are marked on the map, and the distance between each marked point and the UAV launch point is measured; secondly, according to the selection priority order of the reference objects, their priorities are quantified, and a reference point selection function is formed in combination with the reference point distance, the optimal reference point is selected and set as the UAV observation point.

[0016] Furthermore, for the selected track line, points are taken along the line, the length S of the track line is calculated, the number of waypoints n required is preliminarily determined based on the length S, and the waypoint interval S / n for the route that is approximately a straight line is calculated; it is determined whether there are turning points along the line, and if there are turning points, the curvature of the turning points is determined by calculation, and then the number of waypoints required for turning points with different curvatures is determined according to parameters, and then the points are taken; if there are no turning points, the points are taken according to the calculated point-taking interval.

[0017] Furthermore, the judgment of the turning point of the drone is to calculate the curvature K according to the curvature calculation formula. The curve that does not reach the turning point curvature threshold is regarded as an approximate straight line, and the curve that exceeds the turning point threshold needs to further judge the curvature size; the number of required turning points is determined according to different curvature sizes and then the points are selected.

[0018] In a second aspect, the present invention provides a UAV route planning and posture calibration system, comprising:

[0019] The multi-UAV dispatch module is used to obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, it is determined whether to implement multi-UAV dispatch;

[0020] The observation point selection module is used to obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; according to the selection priority order of the reference objects, quantify their priorities, and form a reference point selection function in combination with the reference point distance, select the optimal reference point and set it as the drone observation point;

[0021] The route drawing module is used to draw the initial route according to the track line and the determined observation points; the map information and the UAV flight parameters are combined to mark the area that affects the UAV flight. If there are obstacles that affect the UAV flight in the straight-line distance between the UAV and the observation point, the marked area will be avoided in the subsequent track planning as the final route;

[0022] The posture correction module is used to obtain a track picture every time the drone reaches a specified waypoint during flight, identify and calculate the vertical distance between the drone's position and the track line, and determine whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

[0023] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the UAV route planning and posture calibration method as described in the first aspect is implemented.

[0024] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the UAV route planning and posture calibration method as described in the first aspect.

[0025] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the UAV route planning and posture calibration method as described in the first aspect.

[0026] The beneficial effects of the present invention are as follows: track identification is optimized and track identification accuracy is improved; the UAV route is drawn through the track line, and external factors such as maps and train timetables are comprehensively considered to draw the optimal UAV route while ensuring the lowest cost; the automatic calibration of the UAV trajectory in the air is achieved, thereby reducing errors during the flight process.

[0027] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0029] Figure 1 This is a flow chart of the method for unmanned aerial vehicle trajectory recognition and route planning based on computer vision according to an embodiment of the present invention.

[0030] Figure 2 This is a diagram of the image preprocessing analysis process described in an embodiment of the present invention.

[0031] Figure 3 This is a feature extraction flow chart described in an embodiment of the present invention.

[0032] Figure 4 This is a flow chart of track direction determination according to an embodiment of the present invention.

[0033] Figure 5 This is a flow chart of fork in the road determination according to an embodiment of the present invention.

[0034] Figure 6 This is a flowchart of the map-based drone detection point according to an embodiment of the present invention.

[0035] Figure 7 This is a flowchart of the map-based UAV detection point priority quantification according to an embodiment of the present invention.

[0036] Figure 8 The present invention is a flowchart of a route drawing method according to an embodiment of the present invention.

[0037] Fig. 9 This is a flow chart of a map-based UAV route optimization method according to an embodiment of the present invention.

[0038] Fig.10 This is a flowchart of the UAV posture calibration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0040] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0041] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0042] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0043] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0044] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0045] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0046] The present invention aims to solve the problems of difficult manual operation, low precision and large error of current UAV cruise, and designs a UAV route planning and posture calibration method based on computer vision. The UAV autonomous patrol flight is realized by utilizing computer vision recognition technology and route drawing, and the flight error is reduced by autonomous posture calibration in the air. The present invention optimizes the track recognition technology and improves the track recognition accuracy; the UAV route is drawn through the track line, and external factors such as maps and train timetables are comprehensively considered to draw the optimal UAV route while ensuring the lowest cost; the automatic calibration of the UAV trajectory in the air is realized to reduce the error during the flight process. The method of the present invention is widely used, not only in UAV patrol flight, but also in railway inspection, fixed-point survey and detection, UAV high-altitude recording and other aspects.

[0047] The main steps of the present invention to achieve track identification and route planning are: using the drone camera to obtain track image data and identify the track line through computer vision; drawing the initial drone route in combination with the map information of the area and the train schedule; optimizing the drawn drone route according to the map obstacle information; sending the route to the flying drone for flying along the route; and performing real-time posture calibration during the drone flight.

[0048] Example 1

[0049] In this embodiment 1, a UAV route planning and posture calibration system is first provided, including: a multi-UAV scheduling module, which is used to obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then perform fork judgment in the forward and reverse directions. According to the fork judgment result, it is determined whether to implement multi-UAV scheduling. An observation point selection module is used to obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the UAV flight range in the map, and measure the distance between each marked point and the UAV release point; according to the selection priority order of the reference object, quantify its priority, and form a reference point selection function in combination with the reference point distance, select the optimal reference point and set it as the UAV observation point. A route drawing module is used to draw the initial route according to the track line and the determined observation point; wherein, the area affecting the UAV flight is marked in combination with the map information and the UAV flight parameters. In the straight-line distance between the UAV and the observation point, if there is an obstacle that affects the UAV flight, the marked area will be avoided in the subsequent track planning as the final route. The posture correction module is used to obtain a track picture every time the drone reaches a specified waypoint during flight, identify and calculate the vertical distance between the drone's position and the track line, and determine whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

[0050] In this embodiment, the above-mentioned system is used to implement a method for UAV route planning and posture calibration, including: obtaining a reverse track line image, performing visual recognition and line extraction, obtaining a track line, and then performing fork judgment in the forward and reverse directions, and determining whether to implement multi-UAV scheduling based on the fork judgment result; obtaining map information within the observation range and performing visual recognition, marking all selectable reference objects within the UAV flight range on the map, and measuring the distance between each marked point and the UAV launch point; quantifying the priority of the reference objects according to their selection priority order, and forming a reference point selection function in combination with the reference point distance, selecting the optimal reference point and setting it as the UAV observation point; drawing a reference point according to the track line and the determined observation point. The initial route is obtained; wherein, the map information and the UAV flight parameters are combined to mark the area that affects the UAV flight. In the straight-line distance between the UAV and the observation point, if there are obstacles that affect the UAV flight, the marked area will be avoided in the subsequent track planning as the final route; during the UAV flight, each time a specified waypoint is reached, a track image is obtained to identify and calculate the vertical distance between the UAV's position and the track line, and to determine whether the UAV is within the error distance, that is, to determine whether the UAV is off course. If so, the UAV is corrected to the correct position; if not, it is determined whether the waypoint is the last waypoint. If it is not the last waypoint, the UAV continues to fly to the next waypoint. If it is the last waypoint, the UAV stops moving and reaches the destination.

[0051] like Figure 1 As shown, the method of this embodiment is a method for drone trajectory recognition and route planning based on computer vision.

[0052] Specifically, using computer vision technology to identify tracks includes:

[0053] like Figure 2As shown in the figure, image preprocessing includes: first, adjusting the image to a uniform size for subsequent processing and model input. Using filters (such as Gaussian filtering, mean filtering, and median filtering) to remove noise from the image. Using histogram equalization to improve the contrast of the image makes the track more visible. Using sharpening filters (such as Laplace filters) to enhance the edge features of the track, making it easier to identify in subsequent processing steps. Using grayscale conversion to convert the color space of the image so that it can be analyzed based on the characteristics of the image. Among them, white balance is used to eliminate the influence of ambient light; the region of interest is extracted based on the recognition result of the previous frame; and the region of interest is converted into a single-channel brightness image. White balance processing includes: using the gray world method to eliminate the influence of ambient light, and after transformation, the average values ​​of the three components R, G, and B in the image tend to the same gray value; extracting the region of interest includes: if the image is the first frame or the previous frame has no recognition result, taking the lower half of the image; if the previous frame has a recognition result, taking the vertical coordinate of the vanishing point as the upper boundary and the bottom of the image as the lower boundary, expanding outward from two linear shapes, and extracting the trapezoidal area near the recognition result; specifically, assuming that h is the image height, w is the image width, the vanishing point coordinate is (u0, v0), the coordinates of the endpoints with larger vertical coordinates in the left and right recognition results are (x0, h) and (x1, h), respectively, then the coordinates of the four vertices of the trapezoidal area are (mnx(0,x0-60), h), (5u0 / 6, v0)(7u0 / 6, v0), (min(w,x1+60), h). Converting an image into a single-channel brightness image includes: converting the image from BGR mode to HLS mode, taking the L component therein to represent the brightness information of the image; quantizing the brightness value in one level with 4 values ​​to obtain the final single-channel brightness image.

[0054] like Figure 3 As shown, feature extraction includes: using a pulse coupled neural network method to identify the track line in the video image, using a simplified pulse coupled neural network to iteratively process the single-channel brightness image to obtain a binary image where the pixels are excited, and then using the image entropy to determine the stop time of the iterative processing. The Canny algorithm is used to detect the edge of the track, which is suitable for extracting clear boundaries. The steps include: 1. Smoothing the image using a Gaussian filter to remove noise. 2. Calculating the gradient of the image to determine the edge strength and direction. 3. Applying non-maximum suppression to retain the local maximum. 4. Using double threshold processing to determine the strength of the edge, and forming the final edge map by connecting the edges. The straight line segment or curve segment where the track line is located is obtained by using a third-order Bezier curve fitting method; the local target point of the drone flight is calculated based on the straight line segment or curve segment where the track line is located.

[0055] Model training includes: first, obtaining a large number of images containing train tracks, including images of different angles, lighting conditions, weather conditions, and scene backgrounds to increase the generalization ability of the model. Divide the dataset into 70%-80% training set, 10%-15% validation set, and 10%-15% test set to evaluate the performance of the model. Process the image data through the deep learning method of convolutional neural network, and choose the existing architecture YOLO as the basis. Use the model pre-trained on a large-scale dataset and fine-tune it to adapt to specific tasks. Training process: Use a deep learning framework to build a model and load data. Pass the input image into the model and calculate the output. Calculate the difference between the model output and the true label through the loss function. Update the model parameters to minimize the loss. Repeat the above process until the set number of iterations is reached.

[0056] In this embodiment, track line selection is performed based on the train schedule and multi-drone scheduling module. In the train schedule module, the schedule should include information such as train number, departure time, arrival time, departure station, terminal station, passing station, train speed, and track number. Through the above information, track lines that do not need to be cruised can be deleted, thereby reducing unnecessary drone patrols and flights. First, it is necessary to establish drone base stations at appropriate locations along the railway. These base stations can not only serve as take-off and landing points for drones, but also include charging facilities to ensure that drones can continue to perform tasks. The distance between base stations depends on the flight radius of the drone. Usually, base stations are distributed at key nodes along the railway, such as near stations. In specific application scenarios, such as during railway maintenance, drones can perform tasks according to pre-set flight paths and receive train schedule updates from the railway system. Once it is found that a train is about to arrive, the drone will automatically send an early warning to ensure the safety of maintenance personnel. The data collected by the drone can be transmitted to the ground control center in real time through a wireless network, and maintenance personnel can view the images taken by the drone through the control center to keep abreast of the situation on the track.

[0057] like Figure 4 As shown in the figure, the multi-UAV dispatch module is designed to determine whether multiple UAVs need to be called for detection based on the acquired track line map. It is mainly divided into two modules: determining the track direction and determining the fork in the road. The module for determining the track direction will determine whether the track is a one-way track or a two-way track based on the train schedule module. If it is a one-way track, it will enter the module for determining the fork in the road; if it is a two-way track, the train schedule will be used to determine whether the incoming direction is one-way or two-way within the limited time. If it is one-way, it will enter the module for determining the fork in the road, and if it is two-way, it will prepare for multi-UAV dispatch. First, it is necessary to shoot the reverse track line map and perform visual recognition and line extraction to obtain the track line. At the same time, the line maps of both directions will be judged for the fork in the road, and finally multi-UAV dispatch will be implemented.

[0058] like Figure 5 As shown, the fork in the road judgment module will determine whether there is an intersection in the obtained railway line map and thus determine whether there is a fork in the track. If there is no fork in the road, the route drawing will be carried out directly; if there is a fork in the road, it will be determined whether the fork in the road exists within the distance range of the drone flight target (that is, whether the drone needs to pass through the fork in the road to reach the target waypoint) according to the known drone flight distance parameters. If it does not exist, the useless fork in the road track line will be cleared. If it does exist, it will be combined with the train schedule module to determine whether there will be vehicles coming from different directions within the limited time. If not, the line without vehicles is considered to be a useless fork in the road, and it will be cleared and the route drawing will continue according to the original plan. If there is, it will be determined to draw multiple drone routes, and the drone routes of the two track lines after the fork will be drawn separately, and multi-drone scheduling will be started.

[0059] like Figure 6 , Figure 7 As shown in the figure, the map-based drone observation point selection module aims to select appropriate observation points in complex environments and obtain train arrival information more stably and efficiently. This module will make judgments based on the priority of reference objects. First, obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the drone flight range on the map, and measure the distance m between each marked point and the drone launch point; secondly, quantify the priority of the reference object selection order, and combine the reference point distance to form a reference point selection function M(m) 。 Let a be the optimal distance between the detection point and the drone launch point (the optimal distance can be set by yourself). Let P~N(a,a / 100) be the reference point distance priority function, and the distribution of m satisfies the normal distribution. Let the selection priority of point m be N (the selection method and specific value of N are as follows Figure 7 As shown), the reference point selection function of point m is M(m)=100*P 2 (m)*N. The reference point with the largest function value is selected as the optimal reference point and set as the drone observation point; finally, the drone observation point is transmitted to the drone route mapping module.

[0060] After completing the above track line and observation point selection module, the initial UAV route drawing is carried out.

[0061] like Figure 8As shown, firstly, for the selected track line, points are taken along the line, the length S of the track line is calculated, the number of waypoints n required is preliminarily determined based on the length of S, and the waypoint interval S / n for the route that is approximately a straight line is calculated. Determine whether there is an inflection point along the line. If there is an inflection point, the curvature of the inflection point is determined by calculation, and then the number of waypoints required for inflection points of different curvatures is determined according to the parameters, and then the points are taken; if there is no inflection point, the points are taken according to the calculated point-taking interval. The last waypoint is the selected observation point. After the point-taking is completed, the established flight parameters are input, including the distance m that the drone needs to fly on the left or right side of the track, the drone's flight altitude h, the drone's flight speed, etc.

[0062] The judgment of the turning point of the drone is to calculate the curvature K according to the curvature calculation formula. The curve that does not reach the turning point curvature threshold is regarded as an approximate straight line, while the curve that exceeds the turning point threshold needs to further judge the curvature size. The number of turning points required is determined according to the different curvature sizes and then the points are selected.

[0063] like Fig. 9 As shown, the map-based drone obstacle area marking module is designed to mark the areas in the actual flight environment where obstacles may affect the flight of the drone based on the actual environment and with reference to the map information, so that the drone can avoid obstacles in flight and optimize the module for planning the trajectory. In planning the route, this module can combine map information with drone flight parameters (such as altitude) to mark areas that may affect the flight of the drone. In the straight-line distance between the drone and the observation point, if there are no obstacles that may affect the flight of the drone, the drone route drawing module will be run directly; otherwise, the marked area will be transmitted to the drone route drawing module, and subsequent trajectory planning will avoid this area.

[0064] like Fig.10 As shown in the figure, the drone posture calibration is designed to prevent the drone from deviating from the planned route during the flight of the drone. It is a checking mechanism to ensure that the drone can fly correctly along the route within the error range. During the flight of the drone, each time it reaches a specified waypoint, it stops to take a track picture, and calculates the vertical distance between the drone's position and the track line based on visual recognition technology to determine whether the drone is within the error distance, that is, to determine whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint (that is, the end point). If not, the drone continues to fly to the next waypoint. If so, the drone stops and reaches the end point.

[0065] The drone route planning method of this embodiment can be combined with the train timetable for railway construction monitoring. When railway workers need to conduct railway maintenance in a certain area of ​​the railway, they can choose this solution to determine whether there is a train passing during the maintenance time, so as to remind the maintenance personnel to evacuate quickly to avoid track accidents. First, take a picture of the track line map by drone, and use computer vision to identify the track line to obtain the track line map. Then enter the train timetable of the area where the track is located as shown in Table 1 below:

[0066] Table 1

[0067]

[0068] By calculating the formula, we can get the following Table 2:

[0069] Table 2

[0070]

[0071] According to the fork judgment module, it can be determined that the location of the passing station 2 is a fork in the road, numbered as track 3. According to the timetable, the train does not pass through track 3 within the reserved time, so it is possible to choose to clear the track line where track 3 is located. The track lines where tracks 1 and 2 are located are retained for route planning. This method can more accurately select track lines, and at the same time, combine the map, train timetable and other modules to optimize the route. When encountering multiple lines passing trains, a multi-drone scheduling module is also reserved. This enables more accurate and efficient route planning.

[0072] The UAV route planning method described in this embodiment can also be applied to environmental monitoring along the railway. After a natural disaster (such as a flood or earthquake) occurs, it is difficult for inspection personnel to enter and conduct field surveys due to the complex environmental conditions at the disaster site. The present invention can use a UAV to quickly assess the damage to the railway line and provide a maintenance plan. When the UAV inspects the disaster-stricken area, it can accurately identify the position of the rails. Through simple operation by the operator, the UAV can automatically find the position of the rails and plan the flight route along the rails, avoiding the safety hazards of the inspectors entering the disaster area in person. Since the flight route will be planned before the flight, the camera observation angle can be freely changed during the line inspection process. By shooting videos and pictures of environmental monitoring along the railway at multiple angles and different distances, and transmitting them to the ground base station, it is convenient for personnel to further analyze the degree of damage to the rails. The UAV can take pictures from multiple angles during the patrol process, and send the shooting data to the ground control center to determine whether there are any abnormal conditions on the railway and formulate a repair plan. The system sends maintenance instructions to the UAV, and the UAV performs corresponding operations according to the instructions, such as conducting a more detailed inspection of a specific area, recording problem points, or performing simple repair tasks. After the drone completes the task, it will feedback the execution status to the track monitoring system, and the system will adjust the maintenance plan based on the feedback information to ensure the safe operation of the track system. During the drone patrol process, the drone patrol system can directly send information to the track monitoring system, which contains the location, type and severity of track anomalies. This transfers the functions of analysis and decision-making from the ground control center to the drone itself, allowing the drone to independently decide whether further inspections or simple repair tasks are needed based on the anomalies detected in real time. For example, when a drone detects a slight crack on the track, it can send a request directly to the track monitoring system. The system decides whether maintenance personnel need to be dispatched for on-site inspections or whether a more detailed drone inspection is needed immediately based on the feedback information from the drone and the current track load. This practice of delegating decision-making power to drones can significantly reduce the processing burden of the ground control center and improve maintenance efficiency and response speed. At the same time, the autonomous decision-making ability of drones also makes track maintenance more flexible and efficient, able to adapt to the needs of different situations, and ensure the safe and stable operation of the track system.

[0073] The drone route planning method described in this embodiment can also be applied to railway tunnel inspection. When inspecting railway tunnels, it is necessary to check whether there are cracks or water leakage in the tunnel wall, and whether the lighting and ventilation equipment in the tunnel are working properly. Railway tunnels have poor lighting conditions, a tortuous and narrow space environment, and complex breathing air composition. The use of a drone carrying the present invention can avoid the entry of inspection personnel, making railway tunnel inspection safer. The drone flies according to the route planning of the railroad track of the present invention. By inputting the flight distance, the flight mileage of the drone can be controlled to avoid operating errors by the operator in a dark environment. The obstacle avoidance flight of the drone can be achieved through reasonable road planning.

[0074] The drone route planning method described in this embodiment can also be applied to the security inspection work of a large city, aiming to improve the safety and emergency response efficiency of the city. The core of this system is a number of drones equipped with high-definition cameras, infrared sensors and real-time communication equipment. Through the pre-set trajectory planning algorithm, these drones can regularly patrol key areas in the city within a specific time period, such as commercial areas, transportation hubs, large event venues, etc. In order to ensure that drones can perform tasks safely and efficiently, trajectory planning technology is used to create the optimal flight route. These paths not only take into account the maximization of coverage, but also take into account the existence of obstacles (such as high-rise buildings) and no-fly zones. In addition, the drone system adopts a dynamic adjustment mechanism, which can adjust the flight trajectory in real time according to changes in ground traffic conditions and crowd density to avoid crowded areas or sudden incidents, while maintaining effective monitoring of important targets. The data collected by the drone, including video streams and image information, is transmitted to the command center in real time via a high-speed wireless network. The command center uses artificial intelligence analysis software to automatically detect abnormal behavior or potential threats and send an alert to the nearest patrol team. For example, if the drone captures the behavior of a suspicious person or finds fire smoke, the system will immediately notify the relevant department so that quick measures can be taken. This instant feedback mechanism greatly shortens the time interval from discovery to processing, and enhances the city's ability to respond to emergencies. In addition to regular patrols, drones also play an important role during large-scale public events. They can hover over the event site, providing a bird's-eye view, helping the police better understand the situation on the scene and coordinate security forces. In an emergency, drones can also serve as temporary communication relay stations to ensure unimpeded communication between rescue teams. Overall, the application of this technology has significantly improved the city's public security level and provided citizens with a safer living environment.

[0075] In summary, the UAV route planning and posture calibration method described in this embodiment improves the accuracy of computer vision identification of track lines through model training. When drawing the initial UAV route, factors such as forks and two-way vehicles are judged in combination with the train timetable, thereby reducing unnecessary route drawing and UAV use; by combining the track map information, appropriate landmarks are selected as reference points to draw routes more efficiently. By combining the track map obstacle information, the initial UAV route is optimized to avoid visible obstacles, thereby reducing the risk of being affected by obstacles during the flight of the UAV. Real-time posture calibration is performed during the autonomous patrol flight of the UAV to ensure that the UAV flies accurately along the route and reduce the UAV flight errors caused by external factors such as wind.

[0076] On the basis of existing image processing and visual recognition, this embodiment adds a large amount of data model training, and adds track pictures of different angles, different orientations, different colors, different times, etc. to the model training database for continuous image processing and visual recognition training, thereby improving the accuracy and real-time performance of the technology. When drawing the initial drone route, factors such as forks and two-way vehicles are judged by combining the train timetable, thereby reducing unnecessary route drawing and drone use. In the train timetable module, the present invention inputs the time information of the track and train into a predetermined table, including train number, departure time, arrival time, departure station, terminal station, passing station, train speed and other information. After judging the fork module, each section of the track line is numbered, and the formula is used to calculate whether each section of the track line has a train passing through, the train passing situation within the reserved time, etc., and the useless track line (i.e., the track line that does not require drone cruising) is cleared through the required parameters. At the same time, according to the multi-drone scheduling module, the train timetable module is combined to determine whether the single and double-way tracks and forks need multiple drones to cruise, while reducing resource waste and improving efficiency, it can also improve accuracy. By combining the track map information, suitable markers are selected as reference points to draw routes more efficiently. This embodiment combines the track map information and selects observation points (that is, the end point of the patrol line) by judging markers such as bridges, tunnels, base stations, signboards, and tall trees. The method imports the map information of the area to perform map modeling and marker judgment, thereby selecting the optimal observation point. By combining the track map obstacle information, it can be determined whether there are obstacles that affect the autonomous flight of the drone near the planned route, including tall trees, tunnels, bridges, mountains, telephone poles, no-fly zones, etc. Through the obstacle optimization route module, the risk of drone flight is increased. In order to ensure the accuracy of the posture of the drone during flight and reduce the computer calculation cost and data transmission cost, the present invention uses intermittent shooting of track pictures for posture calibration, that is, after each arrival at a predetermined waypoint, a track picture of a specified orientation is taken for distance analysis, and then the position of the drone is adjusted.

[0077] The computer vision-based drone track recognition and route planning method proposed in this embodiment pre-processes images and video data through computer vision recognition methods, and then uses grayscale conversion to convert the image into color space, effectively extracts track information in the image, inputs the starting point, end point, waypoint and flight altitude, and draws the drone route accordingly. The alternative method steps are to not draw the drone route, process the captured image information in real time during the drone flight, and change the flight path according to the actual situation to control the drone in flight. There may be some other solutions proposed in the specific implementation process, which are not listed here one by one.

[0078] Example 2

[0079] This embodiment 2 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned UAV route planning and posture calibration method is implemented. The method includes:

[0080] Obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, determine whether to implement multi-UAV dispatch;

[0081] Obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; quantify the priority of the reference objects according to their selection priority order, and form a reference point selection function based on the reference point distance, select the optimal reference point and set it as the drone observation point;

[0082] The initial route is drawn based on the track line and the determined observation points. The area that affects the flight of the drone is marked by combining the map information and the drone flight parameters. If there are obstacles that affect the flight of the drone in the straight-line distance between the drone and the observation point, the marked area will be avoided in the subsequent track planning as the final route.

[0083] During the flight of the drone, each time it reaches a specified waypoint, it obtains a track picture, identifies and calculates the vertical distance between the drone's location and the track line, and determines whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

[0084] Example 3

[0085] This embodiment 3 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned UAV route planning and posture calibration method, the method comprising:

[0086] Obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, determine whether to implement multi-UAV dispatch;

[0087] Obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; quantify the priority of the reference objects according to their selection priority order, and form a reference point selection function based on the reference point distance, select the optimal reference point and set it as the drone observation point;

[0088] The initial route is drawn based on the track line and the determined observation points. The area that affects the flight of the drone is marked by combining the map information and the drone flight parameters. If there are obstacles that affect the flight of the drone in the straight-line distance between the drone and the observation point, the marked area will be avoided in the subsequent track planning as the final route.

[0089] During the flight of the drone, each time it reaches a specified waypoint, it obtains a track picture, identifies and calculates the vertical distance between the drone's location and the track line, and determines whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

[0090] Example 4

[0091] This embodiment 4 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the above-mentioned method for planning the route and calibrating the posture of the unmanned aerial vehicle, the method comprising:

[0092] Obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, determine whether to implement multi-UAV dispatch;

[0093] Obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; quantify the priority of the reference objects according to their selection priority order, and form a reference point selection function based on the reference point distance, select the optimal reference point and set it as the drone observation point;

[0094] The initial route is drawn based on the track line and the determined observation points. The area that affects the flight of the drone is marked by combining the map information and the drone flight parameters. If there are obstacles that affect the flight of the drone in the straight-line distance between the drone and the observation point, the marked area will be avoided in the subsequent track planning as the final route.

[0095] During the flight of the drone, each time it reaches a specified waypoint, it obtains a track picture, identifies and calculates the vertical distance between the drone's location and the track line, and determines whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

[0096] In summary, the UAV route planning and posture calibration method described in the embodiment of the present invention aims to achieve autonomous track patrol flight of UAVs with high precision, high efficiency, low risk and low error. It improves the accuracy of computer vision recognition of track lines by training a new model. When drawing the initial UAV route, factors such as forks and two-way vehicles are judged in combination with the train timetable, thereby reducing unnecessary route drawing and UAV use; by combining the track map information, appropriate markers are selected as reference points to draw routes more efficiently. By combining the track map obstacle information, the initial UAV route is optimized to avoid visible obstacles, thereby reducing the risk of being affected by obstacles during the flight of the UAV. Real-time posture calibration is performed during the autonomous track patrol flight of the UAV to ensure that the UAV flies accurately along the route and reduce the UAV flight errors caused by external factors such as wind.

[0097] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for UAV route planning and posture calibration, characterized in that: include: Obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, determine whether to implement multi-UAV dispatch; Obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; quantify the priority of the reference objects according to their selection priority order, and form a reference point selection function based on the reference point distance, select the optimal reference point and set it as the drone observation point; The initial route is drawn based on the track line and the determined observation points. The area that affects the flight of the drone is marked by combining the map information and the drone flight parameters. If there are obstacles that affect the flight of the drone in the straight-line distance between the drone and the observation point, the marked area will be avoided in the subsequent track planning as the final route. During the flight of the drone, each time it reaches a specified waypoint, it obtains a track picture, identifies and calculates the vertical distance between the drone's location and the track line, and determines whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

2. The method for UAV route planning and posture calibration according to claim 1, characterized in that: Determine whether to call multiple drones for detection based on the obtained track map; determine whether the track is a one-way or two-way track based on the train timetable. If it is a one-way track, perform a fork in the road judgment; if it is a two-way track, determine whether the incoming direction is one-way or two-way within the limited time based on the train timetable. If it is one-way, perform a fork in the road judgment module; if it is two-way, prepare for multi-drone dispatch.

3. The method for UAV route planning and posture calibration according to claim 1, characterized in that: According to the obtained railway route map, determine whether there is an intersection in the map and thus determine whether there is a fork in the track; if there is no fork, directly draw the route; If there is a fork in the road, determine whether the fork is within the drone flight target distance range based on the known drone flight distance parameters. If not, clear the useless fork track line. If it does, determine whether there will be vehicles coming from different directions at the fork within the limited time in combination with the train timetable module. If not, the line without vehicles is considered to be a useless fork in the road. If so, determine to draw multiple drone routes, draw the drone routes of the two track lines after the fork respectively, and start multi-drone scheduling.

4. The method for UAV route planning and posture calibration according to claim 1, characterized in that: Obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the UAV flight range on the map, and measure the distance between each marked point and the UAV launch point; secondly, quantify the priority of the reference objects according to their selection priority order, and form a reference point selection function based on the reference point distance, select the optimal reference point and set it as the UAV observation point.

5. The method for UAV route planning and posture calibration according to claim 4, characterized in that: For the selected track line, points are taken along the line, the length S of the track line is calculated, the number of waypoints n required is preliminarily determined based on the length of S, and the waypoint interval S / n for the route that is approximately a straight line is calculated; it is determined whether there are turning points along the line, and if there are turning points, the curvature of the turning points is determined by calculation, and then the number of waypoints required for turning points with different curvatures is determined according to the parameters, and then the points are taken; if there are no turning points, the points are taken according to the calculated point-taking interval.

6. The method for UAV route planning and posture calibration according to claim 5, characterized in that: The judgment of the turning point of the drone is to calculate the curvature K according to the curvature calculation formula. The curve that does not reach the turning point curvature threshold is regarded as an approximate straight line, and the curve that exceeds the turning point threshold needs to further judge the curvature size; the number of inflection points required is determined according to the different curvature sizes and then the points are selected.

7. A UAV route planning and posture calibration system, characterized in that: include: The multi-UAV dispatch module is used to obtain the reverse track line image, perform visual recognition and line extraction, obtain the track line, and then judge the forks in the forward and reverse directions. According to the fork judgment results, it is determined whether to implement multi-UAV dispatch; The observation point selection module is used to obtain map information within the observation range and perform visual recognition, mark all selectable reference objects within the flight range of the drone on the map, and measure the distance between each marked point and the drone launch point; according to the selection priority order of the reference objects, quantify their priorities, and form a reference point selection function in combination with the reference point distance, select the optimal reference point and set it as the drone observation point; The route drawing module is used to draw the initial route according to the track line and the determined observation points; the map information and the UAV flight parameters are combined to mark the area that affects the UAV flight. If there are obstacles that affect the UAV flight in the straight-line distance between the UAV and the observation point, the marked area will be avoided in the subsequent track planning as the final route; The posture correction module is used to obtain a track picture every time the drone reaches a specified waypoint during flight, identify and calculate the vertical distance between the drone's position and the track line, and determine whether the drone is within the error distance, that is, whether the drone is off course. If so, the drone corrects to the correct position; if not, it determines whether the waypoint is the last waypoint. If not, the drone continues to fly to the next waypoint. If it is the last waypoint, the drone stops moving and reaches the end.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the drone route planning and posture calibration method as described in any one of claims 1-6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the UAV route planning and posture calibration method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the UAV route planning and posture calibration method as described in any one of claims 1 to 6.

Images