A Method for Calculating Railway Mileage of UAV Based on Deep Learning and Its Application

Abstract

The present invention relates to a method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning, including: obtaining geographic information data; training a deep learning model; performing frame-by-frame detection and segmentation on a video; calculating ground coordinates using the collinearity equation; calculating relative displacement through calculation points; and post-processing the mileage. The present invention overcomes the disadvantages of traditional manual inspections, which are completed on foot for a long time, with low work efficiency, untimely information upload, difficult effective supervision of operations, and requiring a large amount of time cost, reducing the maintenance cost of the line; provides line mileage information, avoids the problem of full-section manual inspections when foreign objects appear on the line, and speeds up the time for line maintenance personnel to arrive at the scene for handling; using an unmanned aerial vehicle for inspection has no requirement for skylight time, can perform flight detection for a long time, and can timely discover problems and handle or report them.

Description

Technical Field

[0001] The present invention relates to the technical field of railway detection, especially the field of accurate calculation of railway mileage in the scenario of drone inspection videos. Background Art

[0002] With the rapid development of railway construction in China, the railway mileage and coverage have been continuously increasing. By the end of 2023, the operating mileage of China's railways exceeded 156,000 km. There are various industrial and civil buildings, natural vegetation, garbage accumulation, etc. in the railway operation environment, with a large coverage area and complex scenarios. Under adverse conditions, disasters such as fires and foreign object intrusion may occur in the railway operation environment, affecting the normal operation of the railway, causing casualties and economic losses. To ensure the normal operation of the main railway lines, a railway surrounding environmental protection area has been set within 100 m on both sides of the railway line. However, for a long time, there have been phenomena such as damaged fences, illegal buildings constructed within the protection area, and dumping of industrial and domestic waste within the protection area. The existence of these phenomena poses a great threat to the operation of the railway. Therefore, it is crucial for railway operation safety to achieve efficient inspection of the railway operation environment and accurately identify the hazards and potential risks therein. The traditional method is to conduct line inspections manually to inspect the railway surrounding environment. However, manual inspections have many drawbacks. On the one hand, manual inspections must be completed by walking for a long time, with low work efficiency, untimely information upload, and it is difficult to effectively supervise the operation. On the other hand, the climate environment along the railway line is harsh, and it is difficult to guarantee the operation quality. In case of emergencies, the response is not timely, and safety accidents are likely to occur.

[0003] Drones have the advantages of high flight flexibility, low single - flight cost, large flight coverage area, and the operation is not restricted by train operation. If drones are used for railway inspections, the drawbacks of existing methods can be effectively solved, complementing the advantages of existing methods. The detection and inspection scheme based on drones has been widely applied in fields such as power inspection and road surface monitoring and achieved good results. Using drones to inspect and detect railway lines and the operation environment will also surely become a trend.

[0004] However, due to the large amount of drone inspection video data, it is impossible to quickly find the video information corresponding to the spatial position from the massive video data during the application process, which seriously affects the data usage efficiency. Domestic research on drone video line inspection mostly focuses on the detection and tracking of video targets to extract the position of the target relative to the video scene, and there is less research on the geographical information expression and absolute positioning of video frames.

[0005] Through methods such as the flight position, attitude, and video processing technology of drones, combined with relevant theoretical knowledge in photogrammetry, the present study invented an efficient and more accurate method for calculating railway mileage. This method does not depend on the relative position of the drone to the line, but only depends on some prior conditions. Summary of the Invention

[0006] To overcome the deficiencies of the above-mentioned existing technologies, the present invention provides a method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning, and specifically adopts the following technical solutions:

[0007] First, extract frames from the collected video data (determined according to the video duration, not less than 20 frames, and try to include the beginning and end of the video as much as possible. The number of extracted frames has a certain impact on the calculation result). After annotation, perform segmentation model training, and then use the trained model to detect each frame of the video; combine the detected results with photogrammetry and the skeleton method for post-processing to obtain the mileage information of each frame or key frame (depending on the detection mode used).

[0008] A method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning includes the following steps:

[0009] Step1. Extract key information from the SRT file information generated during flight to obtain geographical location information; this geographical location information mainly records the longitude, latitude, relative height difference, heading angle, roll angle, pitch angle, etc. of each frame of the image during flight, and is used to participate in the calculation of coordinates and mileage in Step5 and Step6. This step can be processed separately.

[0010] Step2. Establish a conversion coordinate system; the coordinates in the flight file are longitude and latitude coordinates, which cannot be directly calculated with plane coordinates. If converted to the National 2000 coordinate system for calculation, it will cause large calculation errors and cumulative errors in the calculation results. Therefore, converting the longitude and latitude coordinates to relative coordinates starting from the beginning of the video for calculation can improve the calculation accuracy and reduce errors.

[0011] Step3. Train a deep learning model; in order to automatically obtain the position of the railway line in the image, train a deep learning network model for automatic recognition of the line.

[0012] Step4. Detect and segment each frame of the video; in order to obtain the position of a certain point on the railway line in the image (such as Figure 4 the punctuation in the image, and this punctuation is the key point on the line for participating in the calculation in Step5 and Step6 when y = 359 (pixel coordinates, with the upper left corner of the video frame as the coordinate origin, the right direction as the positive x-axis, and the downward direction as the positive y-axis); hereinafter referred to as the detection calculation point); therefore, it is necessary to extract the line. This step is to avoid calculation errors caused by the line not being in the center of the video screen during flight, and this model is used to extract the position and contour of the railway line in the video frame from the image and output and return the extracted line position and line mask (such as Figure 3As shown in [figure], the orthogonal skeleton method (or skeleton method, which is used to extract the skeleton of the circuit in the mask and is not the focus of this article) is combined to obtain the skeleton of the circuit. By taking points on the circuit (for example, y = 359), the points on the circuit used for calculation can be obtained. This flight video generally includes railway lines and the SRT file corresponding to the video in the first step, which has no direct connection with the second step. It is ensured that the lines can be segmented in this step.

[0013] Step5. Calculate the ground coordinates using the collinearity equation; to obtain the coordinates of the calculation points (on the image) on the ground (on the ground), which are used for the displacement calculation in Step6. This coordinate is the actual coordinate of the calculation point on the ground at this moment. Therefore, it can be known that connecting the coordinates of all calculation points in the video is the actual route of the line.

[0014] Step6. Calculate the relative displacement through the calculation points; calculate the line distance between two points. The relative distance between any two consecutive points in Step5 is the actual distance of the line. Summing up the distances between any two consecutive points of all coordinates gives the actual mileage of the line. It should be noted in this step that according to the matching of longitude and latitude - mileage data, if the matching is successful, the actual mileage value will be changed according to the mileage in the database. A partial schematic diagram of the longitude and latitude - mileage database is shown in Table 1.

[0015] Table 1 Partial schematic diagram of the longitude and latitude - mileage database

[0016] Serial number Line type Chief track maintenance section mileage Latitude longitude 1 ** Line ** Track maintenance section K***+*** 123.4567891 12.34567891 2 ** Line ** Track maintenance section K***+*** 123.4567892 12.34567892 3 ** Line ** Track maintenance section K***+*** 123.4567893 12.34567893

[0017] Step7. Post - processing of the mileage. The post - processing of the mileage is mainly to convert the mileage into the mileage mark of the railway (such as DK000 + 000, or K000 + 000) for convenient observation, and at the same time add these marks at appropriate positions in the video, so as to obtain the flight video corresponding to the railway mileage.

[0018] The technical solution of the present invention has obtained the following beneficial effects: 1. It overcomes the disadvantages of traditional manual inspection, which takes a long time and is completed on foot, with low work efficiency, untimely information upload, difficult effective supervision of operations, and requires a large amount of time cost, and reduces the maintenance cost of the line. 2. It provides line mileage information, avoids the problem of full - section manual inspection when foreign objects appear on the line, and speeds up the time for line maintenance personnel to arrive at the scene for handling. 3. Using unmanned aerial vehicle inspection has no requirement for skylight time, can perform flight detection for a long time, and can timely discover problems and handle or report them. Brief Description of the Drawings

[0019] Figure 1 It is the overall flowchart of the method of the present invention;

[0020] Figure 2 It is the mask Mask diagram of a certain frame during the test of the present invention;

[0021] Figure 3 This is the corresponding skeleton line diagram obtained by using the orthogonal skeleton method during the test of the present invention;

[0022] Figure 4 This is the measured result diagram of the software interface of the present invention.

[0023] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention. It should be noted that the following detailed descriptions are all exemplary and are intended to provide further explanations for the present application.

[0024] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to this application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Specific embodiments of the present invention take the existence of lines in the UAV inspection flight scenario as an example. Specific embodiments of the present invention are applicable to the calculation of railway mileage after UAV inspection video collection and including railway scenarios.

[0026] See the attached Figure 1 , the specific algorithm flow of the specific embodiment of the present invention is as follows:

[0027] Step1. Obtain geographic information data. Extract key information according to the SRT file information generated during flight, mainly including the latitude and longitude (Latitude, longitude) of the UAV, the relative flight height H of the aircraft, the yaw angle κ, roll angle ω, and pitch angle of the UAV's exterior orientation elements The current frame number FPS_ID; where the aircraft and the camera are regarded as a whole, and its yaw angle, roll angle, pitch angle are the same as those of the UAV. Save the latitude and longitude information position, exterior orientation element information outer, and flight height information Fly_Height in three files with the current frame FPS_ID as the key. These three files will be used to calculate the ground coordinates of the calculation points; at the same time, this step can be carried out independently.

[0028] Step 2. Establish a conversion coordinate system; in the SRT file generated by the UAV flight, the coordinates used to describe the UAV's position are WGS84 coordinates, that is, longitude and latitude coordinates, and the mileage information of the railway can be considered as the sum of displacements between any two points in the two-dimensional coordinates; therefore, it is necessary to convert the longitude and latitude coordinates into plane coordinates. Commonly used geographic coordinates, such as the National 2000 coordinate system (the longitude and latitude coordinates of the UAV at a certain moment can be converted into the National 2000 coordinates after projection), but due to the influence of spherical curvature and coordinate accuracy, the resulting mileage information will have a large deviation. Therefore, the present invention adopts a relative coordinate system with the UAV at the starting moment of the video as the origin, which can greatly reduce the influence brought by the earth's curvature. Define the geographic coordinates (WGS84) (x 0 , y 0 , z 0 ) of the UAV corresponding to the first frame image in the detection video as the origin of the projection coordinates. Define the geographic coordinates (WGS84) of the UAV at a certain position as (X s , Y s , Z s ), and its corresponding object space coordinates are (X, Y). Use Equation (4) for projection conversion:

[0029]

[0030] In the formula, R is the radius of the earth, generally taking R = 6731 km, S is the abscissa or ordinate of the UAV at a certain position after being converted into plane coordinates relative to the origin, lat s is the latitude of the UAV at a certain position, lon s is the longitude of the UAV at a certain position, lat 0 is the latitude relative to the coordinate origin, lon 0 is the longitude relative to the coordinate origin.

[0031] Step 3. Train the deep learning model. Extract frames from the video. The number of extracted frames can be determined according to the length of the video, generally not less than 20 pictures, and the video frames should be distributed at each time point of the video. Label the extracted video frames with software such as labelme and perform segmentation model training (the training of the deep learning model here adopts the YOLOv8 algorithm, and its training method is conventional training, which is not the focus of this article, so it will not be described further). Finally, obtain the trained model model.

[0032] Step4. Perform frame-by-frame detection and segmentation on the video; use OpenCV to load the video, input the read video frames into the model trained in Step3 for detection and segmentation. After obtaining the mask, combine the orthogonal skeleton method to extract the skeleton of the line, that is, the line shape of the line. Then, according to the need, obtain the corresponding calculation points. In this example, the calculation points are obtained according to half of the video height. As shown in the output of the deep learning model Figure 2 As shown, the white part is the railway line, and the black part does not belong to the line part. If 1 represents the presence of an object and 0 represents the absence of an object, then the two-dimensional array composed of 0 and 1 with the same size as the original video frame is the mask, which is used to represent the shape of the object; the orthogonal skeleton method mainly continuously erodes and refines the target until it can no longer be eroded (single-layer pixel width), and the skeleton of the image is obtained. Finally, the skeleton of the railway line as shown in Figure 3 is obtained; finally, in order to obtain a point with a fixed position during the flight of the drone as a reference point for calculation, so that no matter how the line changes, this point is always on the line. Therefore, this article selects the point where y = 359 as the calculation point of this invention example, as shown in Figure 4 . Using the calculation point to calculate the mileage of the line can avoid errors caused by the line not being in the center of the screen.

[0033] Step5. Calculate the ground coordinates using the collinearity equation; convert the pixel coordinates obtained in Step4 into image point coordinates with the center point of the sensor as the origin, and the conversion formula is as shown in Equation (5). Combine the key information such as longitude and latitude, yaw angle, roll angle, and pitch angle obtained in Step1, Step2, and Step4, and use the collinearity equation to calculate the corresponding ground coordinates (relative) of the calculation point on the image.

[0034]

[0035] In the formula, p is each frame of the image; x p , y p are the coordinates of the image point in the frame coordinate system; x 0 , y 0 , f are the interior orientation elements of the photo; X s , Y s , Z s are the exterior orientation line elements of the photo; X p , Y p , Z p are the ground coordinates of the ground point in the ground auxiliary coordinate system; a i , b i , c i (i = 1, 2, 3) are the nine direction cosines generated by the exterior orientation angle elements of the photo, which can form the rotation matrix R, that is:

[0036]

[0037] In the rotation matrix, the specific expressions of each element are as follows:

[0038]

[0039] b 1 = cosωsinκ

[0040] b 2 = cosωcosκ

[0041] b 3 = -sinω

[0042]

[0043]

[0044] In the above expressions, κ is the course deviation angle during flight, ω is the roll angle during flight, and is the pitch angle during flight.

[0045] When the ground is flat, Z (the height of the line) of any ground point in Equation (5) is a constant, and the following approximate formula can be obtained:

[0046] Z - Z s = -H

[0047] Step6. Calculate the relative displacement through the calculation points; traverse the Key values obtained in Step5, obtain the calculation point coordinates of each picture according to the Key values, and calculate the Euclidean distance between the calculation points before the Current_Key (the current frame number) and the Next_Key (the next frame number), and sum it with the starting mileage to obtain the mileage of the current image frame. The calculation method is shown in Equation (6); at the same time, obtain the latitude and longitude corresponding to the current image frame through Current_Key in position, perform latitude and longitude matching in the latitude and longitude - mileage database and update the current mileage. The matching method mainly uses calculating the difference between the current latitude and longitude and the latitude and longitude in the database, and judging whether the difference is within the given range. If so, the matching is successful; otherwise, the matching fails.

[0048]

[0049] In the formula, mileage ck is the mileage of the current video frame, startMileage is the mileage at the start of the video, x ck , y ck , x nk , y nkare the ground coordinates corresponding to the calculation points of the current video frame and the next video frame. \(c_k\) and \(n_k\) are the current video frame and the next video frame respectively, and mileage q is the mileage queried from the database.

[0050] Step 7. Mileage post-processing; traverse the results obtained in Step 6, convert the mileage into the required format, write it into the video frame according to the corresponding FPS_ID, and finally merge the video frames into a video in sequence and output the corresponding subtitle file.

[0051] Refer to the following two tables. The specific embodiments of the present invention give some parameters of the airborne camera and the used unmanned aerial vehicle of the present invention.

[0052] Before the unmanned aerial vehicle takes off, it is necessary to determine the no-fly zone in the local air traffic control department, apply for the flight airspace, and at the same time carry out route planning, on-site survey and inspection of equipment accessories; after completing these tasks, check the equipment to ensure that the unmanned aerial vehicle and the carried camera equipment are normal, ensure that the battery of the unmanned aerial vehicle is fully charged, and conduct an on-site no-load flight test.

[0053] Data collection: When preparing to take off, the position of the take-off point should be as much as possible on the same horizontal plane as the line. If the difference is large, it should be corrected later. During on-site flight, make the railway line included in the video of the unmanned aerial vehicle, and at the same time, the flight direction is consistent with the line direction, and save the flight data file.

[0054] Data sorting: The data is saved locally, and the local device intercepts and names the video and then collects key data.

[0055] The test steps of the present invention taking the case where the running direction of the unmanned aerial vehicle is consistent with the line direction in the railway scene as an example are as follows, and some parameters of the used unmanned aerial vehicle are as follows:

[0056] Aircraft name Lens Sensor size Sensor pixel size Viewing angle DJI Seer 102s 25.4 inches 0.0000039 84 degrees

[0057] The video information used in this example:

[0058]

[0059] 1. In the collected video of the unmanned aerial vehicle, the frame rate and resolution of the video may be high. To reduce the calculation amount and improve the efficiency of the segmentation model, some videos with long duration and high frame rate are cropped or key frames are extracted to reduce the calculation time. And extract the SRT file generated by the flight, and output the results with FPS_ID as the key, and the values are the longitude and latitude position, outer orientation angle element outer and flight height Fly_Height of each video frame respectively. Table 1 gives a schematic diagram of some flight video data.

[0060] 2. Frame extraction is performed on the video every 30 frames through OpenCV, and finally 45 pictures are obtained.

[0061] 3. The pictures obtained in the second step are labeled using Labelme software, and the labeled json format annotation files are converted into the annotation format used by YOLO through an algorithm. Then, the pictures and label files are input into the target network for iterative training. In this example, the number of iterations is 200 times. When the number of training times reaches 145 times, the early stopping mechanism is triggered, and the model is trained successfully. It takes 6 minutes, and the model segmentation accuracy reaches 99.5%.

[0062] 4. The drone video is segmented and detected frame by frame. After obtaining the mask Mask coordinate points of the railway line, the coordinate points of the center line of the railway line are obtained through the orthogonal skeleton method in the skimage library. Then, the calculation points on the line are obtained according to y = 360 or 359 or 361 in the video, and the lat_lon dictionary type data with FPS_ID as the key value is output, and its value is the pixel coordinates of the calculation points of each picture.

[0063] 5. Traverse the four dictionary data of position, outer, Fly_Height, and lat_lon, and use the collinearity equation to calculate the ground coordinates of the calculation points on the line, and output the result dictionary type data with FPS_ID as the key.

[0064] 6. Traverse the result dictionary, calculate the mileage of each picture using formula 6, and output and save it.

[0065] The partial intermediate results of the experiment in the specific embodiment of the present invention are as follows.

[0066] According to the SRT file generated by the flight, the key information shown in Table 2 can be extracted.

[0067] Table 2 Key information extracted from the SRT file generated by the flight

[0068]

[0069]

[0070] The parameters used during model prediction, the number of pictures used for training, the training duration, and the parameters such as the accuracy and speed of the trained model:

[0071] Table 3 Partial parameters of the segmentation model

[0072]

[0073] In the fourth step, the coordinate positions of the calculation points are extracted using the model segmentation results and the orthogonal skeleton method.

[0074] The calculated mileage information is shown in Table 4 below.

[0075] Table 4 Display of calculation results for each frame

[0076]

[0077] As can be seen from the above tables, the mileage displayed at the end of the video is higher than the mileage in the video name. This is mainly because the flight direction of the drone during flight is not always parallel to the line, which may result in the actual line mileage being larger than the mileage obtained during flight. This is also one of the advantages of the present invention, that is, a relatively accurate mileage can be obtained without relying on the relative position of the drone and the line.

[0078] Inevitably, the position of the calculation point may shift or the ground sampling rate may change slightly with the position of different coordinate points. However, the loss caused by this change is small, and mileage correction can be performed through the longitude-latitude - mileage database when passing a certain distance or corresponding longitude and latitude. This greatly reduces the cumulative error. At the same time, the offset between pixel points can also be calculated according to the formula, and better and more accurate calculation results can be obtained by adjusting the ground sampling rate parameter. In summary, the use of this process can effectively perform corresponding mileage calculations on the railway inspection video collected by the airborne camera.

[0079] The railway mileage calculation method based on deep learning of the present invention uses deep learning to extract the area of the railway line in the video, then uses the orthogonal skeleton method to obtain the trend of the center line of the line, and finally selects the calculation point as the mileage calculation position of the video frame. Combining relevant information in photogrammetry, such as collinearity equations, rotation matrices, etc., calculates the mileage position of the current video frame, with a low error rate and more accurate than directly relying on longitude and latitude calculations for mileage coordinates, effectively solving the problem of less utilization of video frame information. At the same time, it provides an efficient mileage calculation method, improving the efficiency of work such as clearing foreign objects during line inspection.

[0080] As described above, the above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. A method for calculating UAV railway mileage based on deep learning, characterized in that: The following steps are involved: Step 1. Extract key information based on the SRT file information generated during flight to obtain geographic information data: the geographic information data includes the latitude and longitude, relative height difference, yaw angle, roll angle, and pitch angle information of each frame image during flight, which is used to participate in the calculation of coordinates and mileage in Step 5 and Step 6. This step can be processed separately; Step 2. Establish a conversion coordinate system: The coordinates in the flight file are longitude and latitude coordinates. Convert the longitude and latitude coordinates to relative coordinates starting from the beginning of the video; Step 3. Train the deep learning model: By automatically obtaining the location of the railway line in the image, the deep learning network model is trained to automatically identify the line; Step 4. Detect and segment the video frame by frame: In order to avoid calculation errors caused by the line not being in the center of the video screen during flight, the deep learning model trained in Step 3 is used to extract the position and outline of the railway line in the video frame from the image, and the extracted line position and line mask are output and returned. The skeleton of the line is obtained by combining the orthogonal skeleton method, and points on the line can be obtained to participate in the calculation. Step 5. Calculate the ground coordinates using the collinear equation: To obtain the coordinates of the calculated points on the image on the ground, which are the actual coordinates of the calculated points on the ground at a certain moment, connect the coordinates of all the calculated points in the video to get the actual direction of the line; Step 6. Calculate relative displacement by calculating points: Calculate the line distance between two points. The relative distance between any two consecutive points in Step 5 is the actual distance of the line. The actual mileage of the line is obtained by adding up the distances between any two consecutive points of all coordinates. Step 7. Mileage post-processing: Mileage post-processing is mainly to convert the mileage into railway mileage marks for the convenience of observation, and at the same time add these marks at appropriate positions in the video to obtain a flight video corresponding to the railway mileage.

2. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 1 is characterized in that: The step 1 includes the following contents: the key information mainly includes the longitude and latitude of the UAV, the relative flight altitude H of the UAV, the external azimuth element yaw angle of the UAV , Roll angle during flight , Pitch angle during flight , Current frame number FPS_ID; The yaw angle, roll angle, and pitch angle of the drone and camera as a whole are consistent with those of the drone.

3. The method for calculating UAV railway mileage based on deep learning according to claim 1 is characterized in that: The step 3 includes the following contents: extracting frames from the video, the number of frames extracted depends on the length of the video, and the video frames should be distributed at each time point of the video, annotating the extracted video frames through labelme software, and training the segmentation model to finally obtain the trained model model.

4. The method for calculating UAV railway mileage based on deep learning according to claim 1, characterized in that: The step 4 includes the following contents: using OpenCV to load the video, detecting and segmenting the read video frame, and after obtaining the mask, extracting the skeleton of the line, that is, the line shape of the line, by combining the skeleton method, and then obtaining the corresponding calculation points as needed, and returning and saving the obtained calculation points with the current video frame number as the key and the calculation point as the value of the Json data.

5. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 4 is characterized in that: The step 5 includes the following contents: converting the pixel coordinates obtained in step 4 into the image point coordinates with the center point of the sensor as the origin, and the conversion formula is as shown in formula (2); combining the latitude and longitude, yaw angle, roll angle, and pitch angle key information obtained in steps 1, 2, and 4, and using the collinear equation to calculate the ground coordinates corresponding to the calculation point on the image; （2）； Where p is each frame image; , is the frame coordinate of the image point; x0, y0, f are the internal orientation elements of the image; , , is the exterior orientation line element of the image; , , The ground auxiliary coordinate system coordinates of the ground point; , , , =1,2,3 are the 9 direction cosines generated by the external azimuth elements of the image, forming the rotation matrix R, that is: ； The specific expressions of each element in the rotation matrix are as follows: ； In the above expression, is the yaw angle of the UAV's external azimuth element, is the rolling angle during flight, is the pitch angle during flight.

6. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 5 is characterized in that: The step 6 includes the following contents: traverse the Key value obtained in Step 5, obtain the calculation point coordinates of each image according to the Key value, and calculate the Euclidean distance between the calculation points corresponding to the current frame sequence number Current_Key and the next frame sequence number Next_Key, and sum them with the starting mileage to obtain the mileage of the current image frame; at the same time, obtain the longitude and latitude corresponding to the current image frame through Current_Key, match the longitude and latitude in the mileage-longitude and latitude database and update the current mileage. This step should pay attention to whether the longitude and latitude-mileage data matches or not. If the match is successful, the actual mileage value will be changed according to the mileage in the database: （3）； ； In the formula, is the mileage of the current video frame, is the mileage at the beginning of the video, is the ground coordinate corresponding to the calculation point, are the current video frame and the next video frame respectively. It is the mileage retrieved from the database.

7. The method for calculating the railway mileage of an unmanned aerial vehicle based on deep learning according to claim 6, characterized in that: The step 7 includes the following contents: traverse the results obtained in step 6, convert the mileage into the required format and write it into the video frame according to the sequence number of the corresponding video frame, that is, the FPS_ID of the current video frame, and finally merge the video frames into a video in sequence and output the corresponding subtitle file.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method described in any one of claims 1 to 7 when executed.

Images