Unmanned aerial vehicle railway mileage calculation method based on deep learning and application thereof

By combining deep learning and photogrammetry, the problem of low efficiency in processing UAV video data has been solved, enabling efficient and accurate calculation of railway mileage by UAV, thereby improving the efficiency and safety of railway inspection.

CN118936508AActive Publication Date: 2024-11-12BEIJING UNIV OF CIVIL ENG & ARCHITECTURE +1
View PDF 18 Cites 0 Cited by

Patent Information

Application Number
CN202411243768.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-12
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

Drone inspections generate large amounts of video data, making it difficult to quickly retrieve video information for the corresponding spatial location from the massive amount of video data, which affects the efficiency of data use. In addition, traditional manual inspections are inefficient and information is not uploaded in a timely manner, making it difficult to detect potential railway hazards in a timely manner.

Method used

A deep learning-based approach is used to train a frame extraction and segmentation model for video data. Combined with the theory of photogrammetry, the trained model is used for frame-by-frame detection. The orthogonal skeleton method is used to extract the skeleton of the railway line, and the railway mileage is calculated by combining the collinearity equation.

Benefits of technology

It enables efficient and accurate calculation of railway mileage by drones, reduces line maintenance costs, improves inspection efficiency and the timeliness of information uploading, avoids the shortcomings of manual inspection, and can promptly detect and handle line problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936508A_ABST
    Figure CN118936508A_ABST
Patent Text Reader

Abstract

The invention relates to an unmanned aerial vehicle railway mileage calculation method based on deep learning. The method comprises the following steps: obtaining geographic information data; training a deep learning model; performing frame-by-frame detection segmentation on the video; calculating ground coordinates through a collinear equation; calculating relative displacement through the calculation points; and carrying out mileage post-treatment. According to the invention, the defects of long-time and hiking completion, low working efficiency, untimely information uploading, difficulty in effective supervision of operation and need of a large amount of time cost of traditional manual inspection are overcome, and the maintenance cost of a line is reduced; line mileage information is provided, so that the problem of whole-section manual inspection when foreign matters appear on the line is avoided, and the on-site processing time of line maintenance personnel is shortened; utilizing the unmanned aerial vehicle to inspect has no skylight time requirement, long-time flight detection can be carried out, and problems can be found, processed or reported in time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of railway detection technology, and in particular to the field of accurate calculation of railway mileage in a drone inspection video scenario. Background Art

[0002] With the rapid development of China's railway construction, the railway mileage and coverage are constantly increasing. By the end of 2023, China's railway operating mileage will exceed 156,000 km. There are various types of industrial and civil buildings, natural vegetation, garbage accumulation, etc. in the railway operation environment. The coverage is large and the scene is complex. Under unfavorable conditions, disasters such as fire and foreign body intrusion may occur in the railway operation environment, affecting the normal operation of the railway, causing casualties and economic losses. In order to ensure the normal operation of the railway trunk line, a railway surrounding environmental protection zone is set up within 100m on both sides of the railway. However, for a long time, there have been phenomena such as fence destruction, illegal construction in the protection zone, and dumping of industrial and domestic garbage in the protection zone. The existence of these phenomena has greatly threatened the operation of the railway. Therefore, it is crucial to achieve efficient inspection of the railway operation environment and accurately identify the dangers and hidden dangers therein for the safety of railway operation. The traditional way is to conduct manual line inspections and railway surrounding environment inspections. However, there are many disadvantages of manual inspection. On the one hand, manual inspections must be completed on foot for a long time, with low work efficiency, untimely information upload, and difficult to effectively supervise the operation. On the other hand, the climate and environment along the railway are harsh, and the quality of operation is difficult to guarantee. If we fail to respond promptly to emergencies, safety accidents may easily occur.

[0003] Drones have the advantages of high flight flexibility, low single flight cost, large flight coverage, and operation is not restricted by train operation. If drones are used for railway inspection, it can effectively solve the shortcomings of existing methods and complement the advantages of existing methods. Drone-based inspection and testing solutions have been widely used in power inspection, road surface monitoring and other fields and have achieved good results. Using drones to inspect railway lines and operating environments will also become a trend.

[0004] However, due to the large amount of video data from drone line inspection, it is impossible to quickly find the video information of the corresponding spatial position from the massive video data during the application process, which seriously affects the efficiency of data use. Domestic research on drone video line inspection is mostly focused on the detection and tracking of video targets to extract the position of the target relative to the video scene, and there is little research on the expression of geographic information and absolute positioning of video frames.

[0005] This study uses the UAV's flight position, posture, video processing technology, and other methods combined with relevant theoretical knowledge in photogrammetry to invent an efficient and more accurate railway mileage calculation method. This method does not depend on the relative position of the UAV to the line, but only depends on some prior conditions. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method for calculating the railway mileage of a UAV based on deep learning, which specifically adopts the following technical solutions:

[0007] First, the collected video data is framed (determined by the length of the video, no less than 20 frames, and the beginning and end of the video should be included as much as possible. The number of frames extracted has a certain impact on the calculation results). After annotation, the segmentation model is trained, and then the trained model is used to detect the video frame by frame. The detection results are post-processed with photogrammetry and the skeleton method to obtain the mileage information of each frame or key frame (depending on the detection mode used).

[0008] A method for calculating railway mileage using a drone based on deep learning includes the following steps:

[0009] Step 1. Extract key information based on the SRT file information generated during flight to obtain geographic location information; this geographic location information mainly records the latitude and longitude, relative height difference, yaw angle, roll angle, pitch angle and other information of each frame image during flight, which is used to participate in the calculation of coordinates and mileage in Step 5 and Step 6. This step can be processed separately.

[0010] Step 2. Establish a conversion coordinate system; the coordinates in the flight file are longitude and latitude coordinates, which cannot be directly calculated with plane coordinates. If converted to the National 2000 coordinate system for calculation, it will cause large calculation errors and cumulative errors in the calculation results. Therefore, converting the longitude and latitude coordinates into relative coordinates with the beginning of the video as the starting point for calculation can improve the accuracy of the calculation and reduce errors.

[0011] Step 3. Train the deep learning model; in order to automatically obtain the location of the railway line in the image, train the deep learning network model to automatically identify the line.

[0012] Step 4. Detect and segment the video frame by frame; in order to obtain the position of a certain point on the railway line in the image (such as Figure 4 The punctuation point in the image is y=359 (pixel coordinate, the upper left corner of the video frame is the coordinate origin, the right is the positive x-axis, and the downward is the positive y-axis). The key point on the line used to participate in the calculation of Step 5 and Step 6, the calculation point is detected below); therefore, the line needs to be extracted. This step is to avoid the calculation error caused by the line not being in the center of the video screen during flight. The model is used to extract the position and outline of the railway line in the video frame from the image, and output the extracted line position and line mask (such as Figure 3As shown in the figure, the skeleton of the line is obtained by combining the orthogonal skeleton method (or skeleton method, which is used to extract the skeleton of the line in the mask, but is not the focus of this article). A point on the line (for example, y=359) can be obtained to obtain the point on the line for calculation. The flight video generally contains a railway line and the SRT file corresponding to the video in the first step, which is not directly related to the second step. It is sufficient to ensure that the line can be segmented in this step.

[0013] Step 5. Calculate the ground coordinates using the collinear equation; to obtain the coordinates of the calculated point (on the image) on the ground (on the ground) for the displacement calculation in Step 6. The coordinates are the actual coordinates of the calculated point on the ground at that moment. Therefore, it can be seen that connecting the coordinates of all the calculated points in the video is the actual direction of the line.

[0014] Step 6. Calculate the relative displacement by calculating the points; calculate the line distance between the two points. The relative distance between any two consecutive points in Step 5 is the actual distance of the line. The actual mileage of the line is obtained by adding up the distances between any two consecutive points of all coordinates. What needs to be noted in this step is whether the longitude and latitude-mileage data matches or not. If the match is successful, the actual mileage value will be changed according to the mileage in the database. The schematic diagram of the longitude and latitude-mileage database is shown in Table 1.

[0015] Table 1 Schematic diagram of part of the latitude and longitude-mileage database

[0016] Serial number Line In charge of engineering section Mileage Latitude longitude 1 **Wire ** Engineering Section K***+*** 123.4567891 12.34567891 2 **Wire ** Engineering Section K***+*** 123.4567892 12.34567892 3 **Wire ** Engineering Section K***+*** 123.4567893 12.34567893

[0017] Step 7. Mileage post-processing. Mileage post-processing is mainly to convert the mileage into railway mileage marks (such as DK000+000, or K000+000) for easy observation, and add these marks to the appropriate positions in the video to obtain the flight video corresponding to the railway mileage.

[0018] The technical solution of the present invention has achieved the following beneficial effects: 1. It overcomes the shortcomings of traditional manual inspections that take a long time and are completed on foot, have low work efficiency, information is not uploaded in time, operations are difficult to effectively supervise, and require a lot of time and cost, thereby reducing the maintenance cost of the line. 2. It provides line mileage information, avoids the problem of full-section manual inspections when foreign objects appear on the line, and speeds up the time for line maintenance personnel to arrive on site to deal with the problem. 3. There is no skylight time requirement for drone inspections, and flight inspections can be carried out for a long time, and problems can be discovered and handled or reported in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the overall flow chart of the method of the present invention;

[0020] Figure 2 It is a mask image of a certain frame during the test of the present invention;

[0021] Figure 3 It is the corresponding skeleton line graph obtained by using the orthogonal skeleton method when testing the present invention;

[0022] Figure 4 This is a diagram of the actual measurement results of the software interface of the present invention.

[0023] The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be pointed out that the following detailed descriptions are all exemplary and are intended to provide further explanation of the present application.

[0024] Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those generally understood by those of ordinary skill in the art to which this application belongs. It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0025] The specific embodiment of the present invention takes the existence of a line in the drone inspection flight scene as an example. The specific embodiment of the present invention is applicable to the calculation of railway mileage after the drone inspection video is collected and contains a railway scene.

[0026] See attached Figure 1 The specific algorithm flow of the specific embodiment of the present invention is as follows:

[0027] Step 1. Obtain geographic information data. Extract key information based on the SRT file information generated during flight, including the latitude and longitude of the drone, the relative flight altitude H of the aircraft, the external azimuth elements of the drone, the yaw angle κ, the roll angle ω, and the pitch angle The current frame number FPS_ID; the aircraft and camera as a whole, its yaw angle, roll angle, pitch angle and drone are consistent. Use the current frame FPS_ID as the key to save three files: longitude and latitude information position, external orientation element information outer and flight height information Fly_Height. These three files will be used to calculate the ground coordinates of the calculation point; at the same time, this step can be performed independently.

[0028] Step 2. Establish a conversion coordinate system; In the SRT file generated by the drone flight, the coordinates used to describe the drone's position are WGS84 coordinates, that is, longitude and latitude coordinates, and the mileage information of the railway can be considered as the sum of the displacements of any two points in the two-dimensional coordinates; therefore, the longitude and latitude coordinates need to be converted into plane coordinates, and the commonly used geographic coordinates, such as the National 2000 coordinate system (the longitude and latitude coordinates of the drone at a certain moment can be converted into the National 2000 coordinates after projection), but due to the influence of spherical curvature and coordinate accuracy, the final mileage information will have a large deviation. Therefore, the present invention uses a relative coordinate system with the drone at the starting point of the video as the origin, which can greatly reduce the impact of the earth's curvature. Define the geographic coordinates (WGS84) (x0, y0, z0) of the drone corresponding to the first frame image in the detection video as the origin of the projection coordinates. Define the geographic coordinates (WGS84) of the drone at a certain location as (x s , Y s , Z s ), and its corresponding object space coordinates are (X, Y). Use equation (4) for projection transformation:

[0029]

[0030] Where R is the radius of the earth, generally R = 6731 km, S is the horizontal or vertical coordinate of the drone at a certain position after being converted into plane coordinates relative to the origin, lat s is the latitude of the drone at a certain location, lon s is the longitude of the drone at a certain location, lat0 is the latitude relative to the origin of the coordinate system, and lon0 is the longitude relative to the origin of the coordinate system.

[0031] Step 3. Train the deep learning model. Extract frames from the video. The number of frames extracted can be determined according to the length of the video. Generally, it should be no less than 20 pictures, and the video frames should be distributed at each time point of the video. The extracted video frames are labeled with software such as labelme, and the segmentation model is trained (the training of the deep learning model here uses the YOLOv8 algorithm, and its training method is conventional training, which is not the focus of this article, so it will not be described), and finally the trained model model is obtained.

[0032] Step 4. Detect and segment the video frame by frame; use OpenCV to load the video, input the read video frame into the model trained in Step 3 for detection and segmentation, and after obtaining the mask, use the orthogonal skeleton method to extract the skeleton of the line, that is, the line shape of the line, and then obtain the corresponding calculation points as needed. In this example, the calculation points are obtained according to half of the video height. Figure 2As shown in the figure, the white part is the railway line, and the black part does not belong to the line. If 1 represents an object and 0 represents no object, then the two-dimensional array composed of 0 and 1 and the same size as the original video frame is the mask, which is used to represent the shape of the object; the orthogonal skeleton method mainly obtains the skeleton of the image by continuously corroding and refining the target until it cannot be corroded any further (single-layer pixel width). Finally, the image is obtained as shown in the figure. Figure 3 The skeleton of the railway line shown in FIG. ; Finally, in order to obtain a point with a fixed position during the flight of the UAV as a reference point for calculation, so that no matter how the line changes, the point is always on the line, therefore, this paper selects the point y=359 as the calculation point of the example of the present invention, as shown in FIG. Figure 4 Using calculation points to calculate the mileage of a line can avoid errors caused by the line not being in the center of the picture.

[0033] Step 5. Calculate the ground coordinates using the collinear equation; convert the pixel coordinates obtained in Step 4 into image point coordinates with the center point of the sensor as the origin. The conversion formula is shown in equation (5). Combined with the key information such as longitude and latitude, yaw angle, roll angle, pitch angle, etc. obtained in Step 1, Step 2, and Step 4, the collinear equation is used to calculate the ground coordinates (relative) corresponding to the calculated points on the image.

[0034]

[0035] Where p is each frame image; x p ,y p is the frame coordinate of the image point; x0, y0, f are the internal orientation elements of the image; X s , Y s , Z s is the exterior orientation line element of the image; X p , Y p , Z p is the ground auxiliary coordinate system coordinate of the ground point; a i 、b i 、c i (i=1,2,3) are the 9 direction cosines generated by the external azimuth elements of the image, which can form the rotation matrix R, that is:

[0036]

[0037] The specific expressions of each element in the rotation matrix are as follows:

[0038]

[0039] b1=cosωsinκ

[0040] b2=cosωcosκ

[0041] b3=-sinω

[0042]

[0043]

[0044] In the above expression, κ is the yaw angle during flight, ω is the roll angle during flight, is the pitch angle during flight.

[0045] When the ground is flat, the Z (height of the line) of any ground point in equation (5) is a constant, and the following approximate formula can be obtained:

[0046] ZZ s =-H

[0047] Step 6. Calculate the relative displacement through the calculation point; traverse the Key value obtained in Step 5, obtain the calculation point coordinates of each image according to the Key value, and calculate the Euclidean distance before the calculation point corresponding to Current_Key (current frame number) and Next_Key (next frame number), and sum it with the starting mileage to obtain the mileage of the current image frame. The calculation method is shown in formula (6); at the same time, obtain the longitude and latitude corresponding to the current image frame in position through Current_Key, match the longitude and latitude in the longitude and latitude-mileage database and update the current mileage. The matching method mainly uses the difference between the current longitude and latitude and the longitude and latitude in the database to determine whether the difference is within the given range. If so, the match is successful, otherwise the match fails.

[0048]

[0049] In the formula, mileage ck is the mileage of the current video frame, startMileage is the mileage at the beginning of the video, x ck ,y ck ,x nk ,y nk is the ground coordinates corresponding to the calculation points of the current video frame and the next video frame, ck and nk are the current video frame and the next video frame respectively, mileage q It is the mileage retrieved from the database.

[0050] Step 7. Mileage post-processing: traverse the results obtained in Step 6, convert the mileage into the required format and write it into the video frame according to the corresponding FPS_ID, finally merge the video frames into a video in sequence and output the corresponding subtitle file.

[0051] Referring to the following two tables, the specific embodiments of the present invention provide some parameters of the airborne camera of the present invention and the drone used therein.

[0052] Before flying a drone, you must first determine the no-fly zone at the local air traffic control bureau, apply for flight airspace, and conduct route planning, on-site surveys, and inspections of equipment accessories. After completing these tasks, check the equipment to ensure that the drone and the camera equipment it carries are normal, that the drone battery is fully charged, and conduct on-site empty flight tests.

[0053] Data collection: When preparing to take off, the take-off point should be at the same level as the line as much as possible. If the difference is large, it should be corrected later. When flying on site, make sure that the railway line is included in the drone video. At the same time, the flight direction is consistent with the line direction, and save the flight data file.

[0054] Data collation: Data is saved locally, and the local device captures and names the video before collecting key data.

[0055] The present invention takes the case where the direction of the UAV running in the railway scene is consistent with the direction of the line as an example. The test steps are as follows, and some parameters of the UAV used are as follows:

[0056] Aircraft Name Lenses Sensor size Sensor pixel size Perspective DJI Searle 102s 25.4 inches 0.0000039 84 degrees

[0057] Video information used in this example:

[0058]

[0059] 1. In the collected drone videos, there may be videos with high frame rates and resolutions. In order to reduce the amount of calculation and improve the efficiency of the segmentation model, some videos with long duration and high frame rates are cropped or key frames are extracted to reduce the calculation time. The SRT files generated by the flight are extracted and the output is the result with FPS_ID as the key, and the values ​​are the longitude and latitude position of each video frame, the outer azimuth element outer, and the flight height Fly_Height. Table 1 shows a schematic diagram of some flight video data.

[0060] 2. Use OpenCV to extract frames from the video every 30 frames, and finally get 45 pictures.

[0061] 3. Label the images obtained in the second step using Labelme software, and convert the annotated json format annotation file into the annotation format used by YOLO through the algorithm, and input the images and label files into the target network for iterative training. The number of iterations in this example is 200. When the number of training reaches 145 times, the early stopping mechanism is triggered, and the model training is completed in 6 minutes. The model segmentation accuracy reaches 99.5%.

[0062] 4. Segment and detect the drone video frame by frame. After obtaining the mask coordinate points of the railway line, the coordinate points of the centerline of the railway line are obtained by the orthogonal skeleton method in the skimage library. Then, according to the video y=360 or 359 or 361, the calculation points on the line are obtained, and the lat_lon dictionary type data with FPS_ID as the Key value is output, and its value is the pixel coordinate of the calculation point of each picture.

[0063] 5. Traverse the four dictionary data of position, outer, Fly_Height, and lat_lon, use the collinear equation to calculate the ground coordinates of the calculation points in the line, and output the result dictionary type data with FPS_ID as the key.

[0064] 6. Traverse the result dictionary, use formula 6 to calculate the mileage of each picture, and save the output.

[0065] Some intermediate results of experiments of specific embodiments of the present invention are as follows.

[0066] According to the SRT file generated by the flight, the key information shown in Table 2 can be extracted.

[0067] Table 2 Key information extracted from the SRT file generated by the flight

[0068]

[0069]

[0070] Parameters used for model prediction, the number of images used for training, training time, and training model accuracy and speed:

[0071] Table 3 Some parameters of the segmentation model

[0072]

[0073] In the fourth step, the coordinate positions of the calculation points are extracted using the model segmentation results and the orthogonal skeleton method.

[0074] The calculated mileage information is shown in Table 4.

[0075] Table 4 shows the calculation results of each frame

[0076]

[0077] It can be seen from the above tables that the mileage displayed at the end of the video is higher than the mileage in the video title. This is mainly because the flight direction of the drone is not always parallel to the route during the flight, which may cause the actual route mileage to be larger than the mileage obtained during the flight; this is also one of the advantages of the present invention, that is, a relatively accurate mileage can be obtained without relying on the relative position of the drone and the route.

[0078] Inevitably, the position of the calculation point may be offset or the ground sampling rate may change slightly with the position of different coordinate points, but the loss caused by this change is small, and the mileage correction can be performed through the longitude and latitude-mileage database from each distance or corresponding longitude and latitude, which greatly reduces the cumulative error. At the same time, the offset between pixel points can also be calculated according to the formula, and better and more accurate calculation results can be obtained by adjusting the ground sampling rate parameters. In summary, the use of this process can effectively perform corresponding mileage calculations on railway inspection videos collected by airborne cameras.

[0079] The railway mileage calculation method based on deep learning of the present invention utilizes deep learning to extract the area of ​​the railway line in the video, then uses the orthogonal skeleton method to obtain the direction of the center line of the line, and finally selects the calculation point as the mileage calculation position of the video frame. The mileage position of the current video frame is calculated by combining relevant information in photogrammetry, such as collinearity equations and rotation matrices. The error rate is low and the mileage coordinates are more accurate than those directly calculated based on longitude and latitude. It effectively solves the problem of insufficient utilization of video frame information. At the same time, it provides an efficient mileage calculation method to improve the efficiency of line inspection and removal of foreign objects.

[0080] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for calculating UAV railway mileage based on deep learning, characterized in that: The method comprises the following steps: Step 1. Extract key information according to the SRT file information generated during flight to obtain geographic information data: the geographic information data includes the latitude and longitude, relative height difference, yaw angle, roll angle, and pitch angle information of each frame image during flight, which is used to participate in the calculation of coordinates and mileage in Step 5 and Step 6. This step can be processed separately; Step 2. Establish a conversion coordinate system: The coordinates in the flight file are longitude and latitude coordinates. Convert the longitude and latitude coordinates to relative coordinates starting from the beginning of the video; Step 3. Train the deep learning model: By automatically obtaining the location of the railway line in the image, the deep learning network model is trained to automatically identify the line; Step 4. Detect and segment the video frame by frame: In order to avoid calculation errors caused by the line not being in the center of the video screen during flight, the deep learning model trained in Step 3 is used to extract the position and outline of the railway line in the video frame from the image, and the extracted line position and line mask are output and returned. The skeleton of the line is obtained by combining the orthogonal skeleton method, and points on the line can be obtained to participate in the calculation. Step 5. Calculate the ground coordinates using the collinear equation: To obtain the coordinates of the calculated points on the image on the ground, which are the actual coordinates of the calculated points on the ground at that moment, connect the coordinates of all the calculated points in the video to get the actual direction of the line; Step 6. Calculate relative displacement by calculating points: Calculate the line distance between two points. The relative distance between any two consecutive points in Step 5 is the actual distance of the line. The actual mileage of the line is obtained by adding up the distances between any two consecutive points of all coordinates. Step 7. Mileage post-processing: Mileage post-processing is mainly to convert the mileage into railway mileage marks for the convenience of observation, and at the same time add these marks at appropriate positions in the video to obtain a flight video corresponding to the railway mileage.

2. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 1 is characterized in that: The step 1 includes the following contents: the key information mainly includes the longitude and latitude of the UAV, the relative flight altitude H of the UAV, the yaw angle K of the UAV's external azimuth element, the roll angle ω during flight, and the pitch angle during flight. Current frame number FPS_ID; The yaw angle, roll angle, and pitch angle of the drone and camera as a whole are consistent with those of the drone.

3. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 1 is characterized in that: The step 2 includes the following contents: defining the geographic coordinates WGS84 (x0, y0, z0) of the drone corresponding to the first frame image in the detection video as the origin of the projection coordinates; defining the geographic coordinates WGS84 of the drone at a certain position as (x s , Y s , Z s ), and its corresponding object space coordinates are (X, Y). Use formula (1) for projection transformation: Where R is the radius of the earth, R = 6731 km, S is the horizontal or vertical coordinate of the drone at a certain position after being converted into plane coordinates relative to the origin, lat s is the latitude of the drone at a certain location, lon s is the longitude of the drone at a certain location, lat0 is the latitude relative to the origin of the coordinate system, and lon0 is the longitude relative to the origin of the coordinate system.

4. The method for calculating UAV railway mileage based on deep learning according to claim 1, characterized in that: The step 3 includes the following contents: extracting frames from the video, the number of frames extracted depends on the length of the video, and the video frames should be distributed at each time point of the video, annotating the extracted video frames through labelme software, and training the segmentation model to finally obtain the trained model model.

5. The method for calculating UAV railway mileage based on deep learning according to claim 1 is characterized in that: The step 4 includes the following contents: using OpenCV to load the video, detecting and segmenting the read video frame, and after obtaining the mask, extracting the skeleton of the line, that is, the line shape of the line, by combining the skeleton method, and then obtaining the corresponding calculation points as needed, and returning and saving the obtained calculation points with the current video frame number as the key and the calculation point as the value of the Json data.

6. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 5 is characterized in that: The step 5 includes the following contents: converting the pixel coordinates obtained in step 4 into image point coordinates with the center point of the sensor as the origin, and the conversion formula is as shown in formula (2); combining the latitude and longitude, yaw angle, roll angle, and pitch angle key information obtained in step 1, step 2, and step 4, and using the collinear equation to calculate the ground coordinates corresponding to the calculation point on the image; Where p is each frame image; x p ,y p is the frame coordinate of the image point; x0, y0, f are the internal orientation elements of the image; X s , Y s , Z s is the exterior orientation line element of the image; X p , Y p , Z p is the ground auxiliary coordinate system coordinate of the ground point; a i , b i 、c i , i = 1, 2, 3 are the 9 direction cosines generated by the external azimuth elements of the image, forming the rotation matrix R, that is: The specific expressions of each element in the rotation matrix are as follows: b1=cosωsinκ b2=cosωcosκ b3=-sinω In the above expression, K is the yaw angle of the external azimuth element of the UAV, ω is the roll angle during flight, is the pitch angle during flight; When the ground is flat, Z at any ground point in Formula 2 is a constant, and the following approximate formula can be obtained: Z-Z s =-H。 7. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 6, characterized in that: The step 6 includes the following contents: traverse the Key value obtained in Step 5, obtain the calculation point coordinates of each image according to the Key value, and calculate the Euclidean distance between the calculation points corresponding to the current frame sequence number Current_Key and the next frame sequence number Next_Key, and sum them with the starting mileage to obtain the mileage of the current image frame; at the same time, obtain the longitude and latitude corresponding to the current image frame through Current_Key, match the longitude and latitude in the mileage-longitude and latitude database and update the current mileage. This step should pay attention to whether the longitude and latitude-mileage data matches or not. If the match is successful, the actual mileage value will be changed according to the mileage in the database: In the formula, mileage ck is the mileage of the current video frame, startMileage is the mileage at the beginning of the video, x, y are the ground coordinates corresponding to the calculation point, ck, nk are the current video frame and the next video frame respectively, mileage q It is the mileage retrieved from the database.

8. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 7 is characterized in that: The step 7 includes the following contents: traverse the results obtained in step 6, convert the mileage into the required format and write it into the video frame according to the sequence number of the corresponding video frame, that is, the FPS_ID of the current video frame, and finally merge the video frames into a video in sequence and output the corresponding subtitle file.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method described in any one of claims 1 to 8 when executed.

Citation Information

Patent Citations

  • Railway polling and power supply contact net foreign matter removing system based on multi-rotor unmanned aerial vehicle and method for railway polling

    CN108107904A

  • Railway unmanned aerial vehicle video positioning method

    CN111414518A

  • Steel rail mileage estimation method based on information fusion

    CN111623796A

  • Method and system for line inspection based on unmanned aerial vehicle video and railway BIM

    CN113190040A

  • Railway engineering detection method and system based on unmanned aerial vehicle inspection video

    CN114217626A