A method for inspecting a railway overhead line support device based on an airborne pan-tilt camera

Abstract

The application provides a method for autonomous inspection of a railway catenary support device based on a UAV-mounted gimbal camera. The method comprises: constructing a target detection model of the railway catenary support device, training the target detection model using a data set of the railway catenary support device, and designing an intelligent control algorithm for controlling the gimbal and the camera to capture images of the railway catenary support device; deploying the trained target detection model and the intelligent control algorithm on a UAV, flying the UAV along the side of the railway line to inspect the railway catenary support device, and using the gimbal and the camera carried by the UAV to capture image data of the railway catenary support device; and transmitting the captured image data of the railway catenary support device to a ground server through a wireless communication network. The method can efficiently and reliably inspect the catenary support device, obtain standardized and multi-scale image data of the catenary support device during the inspection, and broaden the application scenarios of UAV inspection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of contact network inspection, and particularly relates to a railway contact network support device inspection method based on an airborne pan-tilt camera. BACKGROUND

[0002] In recent years, with the continuous growth of the mileage of high-speed railways in China, the scale of railway infrastructure is also growing synchronously, which brings severe challenges to the safety guarantee work of railway infrastructure. The contact network support device is a common infrastructure, which mainly functions to bear the load of the contact network and transmit it to the column to ensure the safety and reliability of the train power system. The reliability of the contact network support device is an important guarantee for the safe operation of the railway train, so it is necessary to conduct regular fault inspection on this kind of railway infrastructure, and to replace and repair the faulty parts in real time to ensure its safety and reliability during railway operation. The traditional manual on-site inspection is low in efficiency, high in risk coefficient and high in cost. Therefore, it is urgent to find an efficient and accurate automatic fault inspection technology for the contact network support device.

[0003] At present, the traditional manual inspection method of the contact network support device in the prior art is that the railway workers observe and inspect each contact network support device along the railway line with their naked eyes. This inspection method has the following shortcomings:

[0004] Firstly, there may be missed inspection when the distance between the support device and the inspector is far;

[0005] Secondly, there may be potential safety risks in the manual inspection in complex environments such as the wild, bridges and tunnels;

[0006] Thirdly, the manual inspection requires the train to stop on the relevant section, which greatly weakens the train operation efficiency of the entire operation line. In recent years, with the development of pattern recognition technology, especially computer vision technology, many more effective contact network support device inspection methods have emerged.

[0007] Another method for inspecting the contact network support device at night by using a special vehicle equipped with a high-definition camera in the prior art has greatly improved the efficiency compared with the manual inspection. However, this inspection method still has the following shortcomings:

[0008] Firstly, the special vehicle occupies the track resources, and can only be inspected during the time period when the train is not running;

[0009] Secondly, the image of the contact network support device obtained by the inspection is relatively single, and the high-definition multi-scale image data of the contact network support device is not obtained;

[0010] Thirdly, the use of the special vehicle for inspection not only requires a driver, an inspection worker and the mutual cooperation of each work section along the way, but also causes waste of resources.

[0011] In order to overcome these problems, in recent years, with the continuous maturity of unmanned aerial vehicle technology, it is more and more popular to use unmanned aerial vehicles carrying cameras to patrol the infrastructure along the railway. Unmanned aerial vehicle patrol needs a pilot to control the unmanned aerial vehicle to patrol. For some large facilities on key lines, convenient patrol can be carried out. However, for catenary support devices, there are a large number of them along the railway, about one every 50m. If a pilot is used to take aerial patrol one by one, the workload is too large. At the same time, because the pilot operation does not have accuracy, stability and reproducibility, standard image data of the catenary support device cannot be obtained, which will bring great challenges to subsequent fault diagnosis. SUMMARY

[0012] Embodiments of the present application provide a railway catenary support device inspection method based on an airborne gimbal camera, to effectively inspect the catenary support device on the railway site.

[0013] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions.

[0014] A railway catenary support device inspection method based on an airborne gimbal camera, comprising:

[0015] Constructing a target detection model of the railway catenary support device, training the target detection model using a data set of the railway catenary support device, and designing an intelligent control algorithm for controlling the gimbal and the camera to capture images of the railway catenary support device;

[0016] Deploying the trained target detection model and the intelligent control algorithm on an unmanned aerial vehicle, inspecting along the railway line side above using the unmanned aerial vehicle, and capturing image data of the railway catenary support device using the gimbal and the camera carried by the unmanned aerial vehicle;

[0017] Transmitting the captured image data of the railway catenary support device to a ground server through a wireless communication network.

[0018] Preferably, the construction of the target detection model of the railway catenary support device, training the target detection model using a data set of the railway catenary support device, comprises:

[0019] Flying along the railway line side above using an unmanned aerial vehicle carrying a gimbal and a camera, the flight trajectory being parallel to the track direction, labeling the image data of the catenary support device obtained by the unmanned aerial vehicle, obtaining image data of catenary support devices of different sizes and different shapes, dividing the catenary support device into right and left catenary support devices, using the image data to make a data set for catenary support device target recognition, and dividing the data set into a training set, a validation set and a test set.

[0020] selecting a target detection model, training the target detection model using image data in the training set, verifying and testing the target detection model using image data in the verification set and the test set, and saving parameters of the trained target detection model.

[0021] Preferably, the target detection model is a YOLOv7 model.

[0022] Preferably, the intelligent control algorithm comprises an anchor frame selection algorithm, a cutting algorithm based on shortest side compensation, an anchor frame center locking algorithm based on a pinhole camera model, an anchor frame magnification algorithm based on a camera pinhole model, and a catenary support device nine-square photographing method.

[0023] The anchor frame selection algorithm comprises obtaining the coordinate position of the anchor frame of the catenary support device in the image and the length and width of the anchor frame, screening and fine-tuning all detected anchor frames according to the distance between the center coordinates of all detection anchor frames and the center point coordinates of the camera frame and the area size of the anchor frame in combination with the inspection requirements, linearly combining and quantitatively sorting the two indicators, and selecting the anchor frame with the highest score and meeting the requirements as the target for subsequent inspection.

[0024] The cutting algorithm based on shortest side compensation: proportionally cutting the anchor frame where the catenary support device is located, so that the anchor frame where the catenary support device is located has the same proportion as the camera imaging frame;

[0025] The anchor frame center locking algorithm based on the pinhole camera model: calculating the rotation angle of the gimbal according to the coordinates of the anchor frame center and the camera frame center, and moving the center of the cut catenary support device anchor frame to the image frame center according to the rotation angle by controlling the gimbal;

[0026] The anchor frame magnification algorithm based on the camera pinhole model: calculating the size of the adjustment focal length based on the pinhole model of the camera, and magnifying the anchor frame to s times the full frame based on the adjustment focal length, 0

[0027] Catenary support device nine-square photographing: nine-square cutting of the frame of the magnified catenary support device anchor frame, center locking of each square, moving each square to the frame center, magnifying, and photographing to obtain nine images of the catenary support device.

[0028] Preferably, the trained target detection model and the intelligent control algorithm are deployed on a UAV, the UAV is used to patrol along the railway line side, the gimbal and camera carried by the UAV are used to capture image data of the catenary support device of the railway, and the image data comprises:

[0029] The trained target detection model is converted into a model capable of being deployed on an edge computing end, the trained target detection model and the intelligent control algorithm are deployed on an edge computing end of the unmanned aerial vehicle,

[0030] According to the distribution characteristics of the catenary support device along the railway, the best shooting flight point is marked on the map, the flight task file of the inspection is generated, the flight task file is transmitted to the unmanned aerial vehicle, and the unmanned aerial vehicle starts the target detection algorithm and the intelligent pan-tilt camera control algorithm after flying to the specified flight point;

[0031] The camera is started to acquire a video stream, the rotation angle of the pan-tilt and the adjustment size of the camera focal length are calculated, the pan-tilt is controlled to perform center locking and nine-grid shooting, the obtained image is input into the trained target detection model, the position of the catenary support device in the image is obtained according to the target detection algorithm, the best catenary support device is screened out, and the center point coordinates of the best catenary support device in the image and the length-width size of the anchor frame thereof are output, and the pan-tilt and the camera are controlled by the intelligent algorithm to capture images of the target catenary support device in multiple scales.

[0032] Preferably, the image data of the captured catenary support device is transmitted to the ground server through a wireless communication network.

[0033] The captured standardized and multi-scale image data of the catenary support device is marked in time and space, and the image data of the catenary support device with time and space information is transmitted to the ground server end through 5G or a video transmission device.

[0034] As can be seen from the technical solutions provided by the above-mentioned embodiments of the present application, the method can efficiently and reliably inspect the catenary support device and obtain standardized and multi-scale image data of the catenary support device in the inspection. The method overcomes various disadvantages of previous manual inspection and special vehicle column inspection. The method further widens the application scenario of the unmanned aerial vehicle inspection, reduces the training cost of the unmanned aerial vehicle inspection for a pilot, and reduces the difficulty of the inspection task.

[0035] Additional aspects and advantages of the present application will be described in the following description, which will become apparent from the following description, or will be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 A processing flow chart of a railway overhead line support device inspection method based on an airborne gimbal camera is provided for the embodiment of the present application.

[0038] Figure 2 A flight inspection method schematic diagram is provided for the embodiment of the present application.

[0039] Figure 3 A recognition result schematic diagram of an overhead line target detection model is provided for the embodiment of the present application.

[0040] Figure 4 An anchor frame cropping schematic diagram of an overhead line support device is provided for the embodiment of the present application.

[0041] Figure 5 A center locking schematic diagram of an overhead line support device is provided for the embodiment of the present application.

[0042] Figure 6 A camera pinhole model schematic diagram for calculating gimbal attitude angle adjustment size is provided for the embodiment of the present application.

[0043] Figure 7 An overhead line anchor frame enlargement schematic diagram is provided for the embodiment of the present application.

[0044] Figure 8 A camera pinhole model schematic diagram for calculating focal length adjustment size is provided for the embodiment of the present application.

[0045] Figure 9 A nine-square close-up photographing schematic diagram is provided for the embodiment of the present application.

[0046] Figure 10 An overhead line support device target detection output result is provided for the embodiment of the present application. DETAILED DESCRIPTION

[0047] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation of the present application.

[0048] It will be understood by those within the art that, in this disclosure, terms such as "a," "one," "the," and "said" are intended to include both singular as well as the plural or to unity plural referents to one referent unless otherwise indicated by context. It will be further understood that the terms "includes," "including," "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. Furthermore, "connected" or "coupled" as used herein can include wirelessly connected or coupled. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0049] It will be understood by those within the art that, in this disclosure, all terminology used herein, including technical and scientific terms, has the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs unless otherwise defined. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the specification and relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0050] In order to facilitate the understanding of the embodiments of the present application, the following will be further explained and described in conjunction with the drawings and several specific embodiments, and each embodiment does not constitute a limitation on the embodiments of the present application.

[0051] The embodiments of the present application propose a method for autonomous inspection of railway catenary support devices based on unmanned aerial vehicle-mounted gimbal cameras. The unmanned aerial vehicle carries a high-precision gimbal and a high-definition camera to fly above the side of the railway line to obtain image data of the catenary support devices along the line, which is used to train a target detection model for real-time detection of the catenary support devices. On this basis, a complete set of intelligent control algorithms for controlling the gimbal are proposed, which can automatically calculate and adjust the gimbal angle and camera focal length parameters to capture standardized, multi-scale and high-quality image data of the catenary support devices under the condition of detecting the catenary support devices. The two algorithms are deployed to the edge of the unmanned aerial vehicle, and the flight point task of the unmanned aerial vehicle is planned. When the unmanned aerial vehicle flies to the specified flight point, the catenary support device target detection algorithm and the intelligent algorithm for controlling the gimbal camera are automatically started, the catenary support device is detected at the same time, and the gimbal and the camera are controlled to center lock and multi-scale patrol and photograph the detected catenary support device. Finally, the photographed catenary support device is marked with time and space information and saved on the unmanned aerial vehicle or transmitted to the ground server through 5G or a video transmission link.

[0052] The processing flow of the method for autonomous inspection of railway catenary support devices based on unmanned aerial vehicle-mounted gimbal cameras proposed by the embodiments of the present application is as followsFigure 1 The processing steps include the following:

[0053] Step S1: Use a UAV equipped with a high-precision gimbal and a high-definition camera to patrol above the side of the railway line, and obtain image data of the catenary support device by the onboard gimbal camera of the UAV

[0054] Step S2: Train a target detection model capable of real-time detection of the catenary support device using the image data of the catenary support device obtained in step S1.

[0055] Step S3: Design an intelligent control algorithm capable of automatically controlling the gimbal and the camera to capture standardized and multi-scale catenary support device images according to the output of the target detection model.

[0056] Step S4: Deploy the target detection model and the intelligent control algorithm to the edge computing end of the UAV.

[0057] Step S5: Automatically generate or manually select a waypoint task for the UAV to fly to the specified waypoint.

[0058] Step S6: Automatically start the target detection model and the intelligent control algorithm at the specified waypoint to automatically capture standardized and multi-scale catenary support device image data.

[0059] Step S7: Transmit the captured standardized and multi-scale catenary support device image data to the ground server through 5G or a video transmission link.

[0060] Specifically, the above step S1 includes: Figure 2 A flight inspection method provided by an embodiment of the present application is shown in the figure. The inspection method of the embodiment of the present application uses a UAV equipped with a high-precision gimbal and a high-definition camera system to fly along the side of the railway line for flight inspection. The flight trajectory is parallel to the track direction, the horizontal distance from the line edge is 20 m, the height from the ground is 20-30 m, the flight speed is 3-4 m / s, and the UAV flies back and forth on both sides of the line to obtain relatively complete original image data of the catenary support device of the line.

[0061] Specifically, the above step S2 includes: labeling the image data of the catenary support device obtained in step S1 to obtain image data of catenary support devices of different sizes and different shapes, and dividing the catenary support devices into two categories, right side c_right and left side c_left catenary support devices. Use the image data to make a data set for catenary support device target recognition, and divide the data set into a training set, a validation set, and a test set.

[0062] The target detection model is selected, the target detection model is trained by using image data in the training set, the target detection model is verified and tested by using image data in the verification set and the test set, and parameters of the trained target detection model are saved.

[0063] The target detection model can be a YOLOv7 model. The following describes the embodiment method of the application by taking the YOLOv7 target detection model as an example.

[0064] The YOLOv7 target detection model is trained to obtain the best model parameters for real-time target detection of the overhead contact line support device.

[0065] Specifically, the step S3 includes designing an intelligent control algorithm that can automatically control the holder and the camera to capture standardized and multi-scale overhead contact line support device images according to the output of the target detection model. The position of the overhead contact line support device in the image is obtained according to the intelligent control algorithm, the adjustment size of the holder rotation angle and the camera focal length is calculated, and the holder is controlled to perform center locking and nine-grid shooting to realize multi-scale and standardized image data acquisition of the overhead contact line support device.

[0066] The basic flow of the intelligent control algorithm can be summarized as follows:

[0067] Anchor box selection algorithm: Figure 3 The recognition result schematic diagram of the overhead contact line target detection model provided by the embodiment of the application is shown in FIG. 1. The anchor box coordinate position of the overhead contact line support device in the image and the length and width of the anchor box are obtained by the YOLOv7 target detection algorithm. Figure 4 Before the shortest side compensation algorithm is executed, all detected anchor boxes need to be screened and fine-tuned according to the inspection requirements. First, the YOLOv7 algorithm detects multiple target anchor boxes of the overhead contact line support device during the inspection. Figure 10 Our purpose is to inspect a single overhead contact line support device, so we screen the two indicators of the distance between the center coordinates of all detected anchor boxes and the center point coordinates of the camera frame and the anchor box area size, linearly combine and quantitatively sort the two indicators, and select the anchor box with the highest score and meeting the requirements. Let the width and height of the i-th anchor box be W ai and H ai , the center point coordinates be (X ai , Y ai ), and the camera frame center point coordinates be (X c , Y c ). Therefore, the quantification score of the i-th anchor box is (here to balance the scales of the area and the length):

[0068]

[0069] wherein:

[0070] —S ai is the area of ​​the i-th anchor box: S ai =W ai *H ai

[0071] —L ai is the distance from the center of the i-th anchor frame to the center of the frame:

[0072] —S c and L c are the area of ​​the entire image frame and its diagonal length: S c =W c *H c ;

[0073] —m s and m d It is the adjustment weight of the area index and the distance from the center index, which is determined according to actual needs.

[0074] Calculate the quantitative scores M0, M2, ...M of all N anchor boxes detected i ...,M N-1 And sort them, and finally select the anchor box with the largest score as the target for subsequent inspection.

[0075] Cropping algorithm based on shortest side compensation: In order to facilitate the close-up photography of the contact network support device later, we need to proportionally crop the anchor frame where the contact network support device is located so that it is the same ratio as the camera imaging frame (adaptable). The purpose of cropping is to ensure that the contact network support device in the anchor frame is fully presented in the frame after enlargement without being lost. The cropping algorithm is as follows: Given the data of the anchor frame (X a ,Y a ,W a ,H a ), are the coordinates of the center point of the anchor frame, the width and height of the anchor frame respectively. The height and width of the camera frame are H c and W c , so its aspect ratio is:

[0076]

[0077] The cropping requirement is that all information in the anchor frame cannot be lost, so the anchor frame can only be expanded. Here, the algorithm for compensating the short side is adopted. First, the aspect ratio d of the anchor frame is calculated. a =H a / W a Aspect ratio of camera frame d c If it is within the acceptable error range of ±ε, the anchor box will not be cropped; if it is significantly smaller than d cThen for height H a Symmetry compensation based on anchor center is performed; if it is obviously greater than d c Then for width W a Symmetry compensation based on anchor center is performed; change anchor width-height data to: (X a , Y a , W' a , H' a ). It can be described by the following formula (3) :

[0078]

[0079] Wherein:

[0080] W' a , H' a are the width and height of the cropped anchor

[0081] W a , H a are the width and height of the uncropped anchor

[0082] d c is the aspect ratio of the camera frame

[0083] Anchor center locking algorithm based on pinhole camera model: after anchor cropping, the anchor center needs to be moved to the center of the image frame by controlling the gimbal, such as Figure 5 Here, the rotation angle of the gimbal needs to be calculated according to the coordinates of the anchor center and the camera frame center, where (X c , Y c ) is the camera frame center coordinate. At the same time, the anchor center and the camera frame center relative coordinates are converted into pixel coordinates (X ap , Y ap ), (X cp , Y cp ).

[0084] Here, the deviation of the anchor center and the camera frame center in width is controlled by the yaw angle of the gimbal, and the deviation in height is controlled by the pitch angle. The calculation method is basically the same, and the following describes the adjustment size of the pitch angle corresponding to the deviation in height: assuming that the pitch angle adjustment angle of the gimbal is pitch angle The yaw angle adjustment angle is yaw angle , unit: degree, set the tolerable pixel deviation threshold to σ, when the deviation is less than this threshold, do not adjust; at the same time, when the pixel deviation is greater than σ but not greater than the set upper limit ρ, the pixel deviation size is proportional to the gimbal adjustment angle, and the proportional coefficient is set to μ:

[0085]

[0086] Wherein:

[0087] -pitch angle adjustment angle of pitch angle

[0088] —μ represents the adjustment proportionality coefficient

[0089] -(X ap , Y ap ), (X cp , Y cp ) represent the pixel coordinates of the center of the anchor frame and the center coordinates of the camera frame

[0090] When the anchor frame center deviates from the frame center by more than p, the angle and pixel change are not proportional. To calculate the angle of pitch adjustment, consider the pinhole model of the camera. As shown in the figure, the image coordinates need to be projected onto the camera CMOS, and the projection coordinates of the anchor frame center point and the frame center point are (X acmos , Y acmos ), (X ccmos , Y ccmos ), and the focal length of the camera is f. As Figure 6 , the angle to be adjusted is:

[0091]

[0092] The calculation of the yaw angle is almost the same as that of the pitch angle, except that the width direction coordinates are used.

[0093] Anchor frame magnification algorithm based on camera pinhole model: After the previous cropping and center locking, the anchor frame is now in the image center, and the aspect ratio of the anchor frame is approximately equal to the aspect ratio of the imaging frame. Now we need to magnify the anchor frame to s(0 < s < 1) times the full frame, such as Figure 7 This requires calculating the focal length based on the pinhole model of the camera. First, record the length D a of the diagonal of the anchor frame, and the diagonal length of the camera frame is D c . The model needs to be magnified to s(0 < s < 1) times the frame, and the current focal length is f, and the adjusted focal length is f * . As Figure 8 According to the pinhole model, the following formula (6) can be obtained by the similarity theorem, and the adjusted focal length is calculated as (7):

[0094]

[0095]

[0096] Contact net support device nine-square shooting: After the previous three steps, the anchor frame of the contact net support device has been locked and magnified to the ideal state, and now the entire frame needs to be cropped into nine squares, such as Figure 9, first need to calculate the center point coordinates (x i ,y i )(i=1,2,3…,9) as follows. After obtaining the nine center coordinates of the palace, no need to cut, directly to each palace center lock will each palace to the center of the frame, enlarge, take pictures. So get nine about this catenary support device fine-grained image data.

[0097]

[0098] Specifically, the above step S4 includes: deploying the target detection model and the intelligent control algorithm to the edge computing end of the unmanned aerial vehicle.

[0099] The trained YOLOv7 model capable of real-time detection of catenary support devices is converted into a model that can be deployed on the edge computing end. At the same time, test with video: according to the target detection algorithm, obtain the position of the catenary support device in the image, calculate the adjustment size of the pan-tilt rotation angle and the camera focal length, control the pan-tilt to center lock and nine palace shooting to realize the multi-scale, standardized and automated image data acquisition of the catenary support device.

[0100] Specifically, the above step S5 includes: generating a flight waypoint task of the unmanned aerial vehicle by automatic generation or manual point selection, and flying the unmanned aerial vehicle to the specified waypoint.

[0101] According to the distribution characteristics of the catenary support device along the railway, the best shooting waypoint is marked on the map. Generate a patrol route task file in kml file format and transmit it to the unmanned aerial vehicle. After flying to the specified waypoint, the unmanned aerial vehicle starts the target detection algorithm and the intelligent pan-tilt camera control algorithm to realize the patrol of the catenary support device at the waypoint.

[0102] Specifically, the above step S6 includes: automatically starting the target detection model and the intelligent control algorithm at the waypoint to automatically capture standardized and multi-scale catenary support device image data.

[0103] First, start the camera to obtain video stream, input the obtained image into the trained catenary support device target detection model. Through the target detection model, output the best catenary support device in the field of view of the current shooting point, and output the center point coordinates of the catenary support device in the image and the length and width size of the anchor frame where the catenary support device is located. Through the center point coordinates of the target catenary support device, the intelligent algorithm controls the pan-tilt and the camera to capture multi-scale images of the target catenary support device.

[0104] Specifically, the above step S7 includes: transmitting the captured standardized and multi-scale catenary support device to the ground server through 5G or image transmission link.

[0105] The multi-scale image data of the contact network support device is temporally and spatially labeled. The image data with temporal and spatial information is transmitted to the ground server via 5G or image transmission equipment, providing a standardized, multi-scale data foundation for further contact network fault diagnosis.

[0106] This example uses video footage of the overhead contact support device collected on the Beijing-Shanghai Railway to identify and take close-up photos. The specific steps are as follows:

[0107] Step 1: An M300 drone equipped with a high-precision gimbal and high-definition camera was used to conduct a flyover inspection along the upper side of the Beijing-Shanghai Railway. The flight path was parallel to the track, 20 meters horizontally from the line edge, 20-30 meters above the ground, and at a speed of 3-4 meters per second. The drone flew back and forth along both sides of the line to obtain a relatively complete sample of image data of the catenary support system.

[0108] Step 2: Capture image data of contact network support devices of different scales and shapes, classify them into two categories, c_right and c_left, representing the right and left contact network support devices, respectively. Label them using LabelImg, select 1199 training images and 134 test images. Select the latest object detection model, here the YOLOv7 model. On the ground server system: Ubuntu

[0109] 18.04GPU: Nvidia GTX 3090 for training and saving trained model parameters.

[0110] Step 3: Obtain the position of the contact network support device in the image based on the target detection algorithm, calculate the pan-tilt rotation angle and camera focal length adjustment, and control the pan-tilt to perform center lock and nine-square grid photography to achieve multi-scale and standardized image data acquisition of the contact network support device.

[0111] Step 4: Convert the trained YOLOv7 model, which enables real-time catenary support device detection, to an Nvidia NX edge computing device capable of deployment on the M300 drone. Simultaneously, test the model using video: Using the object detection algorithm, the location of the catenary support device in the image is determined. The gimbal rotation angle and camera focal length adjustment are calculated. The gimbal is then controlled to lock the center and capture a nine-square grid image, enabling multi-scale, standardized, and automated image data capture of the catenary support device.

[0112] Step 5: According to the distribution characteristics of the catenary support device along the Beijing-Shanghai Railway, mark the best shooting flight point on the map. Generate a flight task file for inspection, and transmit it to the UAV in the form of a kml file. After the UAV flies to the specified flight point, the target detection algorithm and intelligent gimbal camera control algorithm are started to realize the inspection of the catenary support device at the flight point.

[0113] Step 6: First, start the camera to obtain video stream, and input the obtained image into the trained catenary support device target detection model. Through the target detection model, the catenary support device in the current shooting point field of view is output (such as Figure 10 ), and the best one is selected to output the center point coordinates of the catenary support device in the image and the length and width size of the anchor box where the catenary support device is located. Through the center point coordinates of the target catenary support device, the gimbal and camera are controlled by an intelligent algorithm to capture images of the target catenary support device at multiple scales.

[0114] Step 7: The multi-scale image data of the catenary support device obtained is marked in time and space. The image data of the catenary support device with time and space information is transmitted to the cloud server through 5G Figure 9 .

[0115] The experimental results show that this inspection method can efficiently and reliably inspect the catenary support device, and obtain standardized and multi-scale image data of the catenary support device during inspection. It overcomes the disadvantages of previous manual inspection and special vehicle column inspection. At the same time, it further widens the application scenario of UAV inspection, reduces the training cost of UAV pilots, and reduces the difficulty of inspection task.

[0116] In summary, the method of the embodiment of the application improves the efficiency of daily inspection and fault diagnosis of the catenary support device, and can be unaffected by the terrain and train operation state. At the same time, the image data of the catenary support device obtained by the UAV inspection is completely intelligent. After setting the inspection task, the UAV pilot can release his hands and let the UAV complete the inspection and shooting of the catenary support device automatically, which enables relevant personnel to quickly get started, saves time and training cost. Finally, because the image data obtained by the machine in an automatic manner is reproducible, the series of catenary support device image data obtained is completely standardized and multi-scale, which greatly facilitates subsequent fault diagnosis of the catenary support device.

[0117] The method can efficiently and reliably perform inspection on the overhead line support device, and standard and multi-scale image data of the overhead line support device is obtained in the inspection. The disadvantages of previous manual inspection and special vehicle column inspection are overcome. The application scenarios of the unmanned aerial vehicle inspection are further broadened, the training cost of the unmanned aerial vehicle inspection for the pilot is reduced, and the difficulty of the inspection task is reduced.

[0118] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or flows in the drawings are not necessarily required to implement the present application.

[0119] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and a necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

[0120] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts are referred to the part of the method embodiments. The above-described device and system embodiments are only schematic, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to the actual needs. Those skilled in the art can understand and implement without creative labor.

[0121] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical range disclosed in the present application can be easily thought of by those skilled in the art, and should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for inspecting a railway overhead line support device based on an airborne gimbal camera, characterized in that, include: Construct an object detection model for railway catenary support devices, train the object detection model using a dataset of railway catenary support devices, and design an intelligent control algorithm for controlling the pan / tilt head and camera to capture images of the railway catenary support devices; Deploy the trained target detection model and the intelligent control algorithm on a drone, and use the drone to patrol along the side and above the railway line, and use the drone's gimbal and camera to capture image data of the railway contact network support device; Transmitting the captured image data of the railway contact network support device to a ground server via a wireless communication network; The intelligent control algorithm includes: an anchor frame selection algorithm, a cropping algorithm based on shortest side compensation, an anchor frame center locking algorithm based on a pinhole camera model, an anchor frame magnification algorithm based on a camera pinhole model, and a nine-square grid photography method for a contact network support device; The anchor frame selection algorithm includes: obtaining the coordinate position of the anchor frame of the contact network support device in the image and the length and width of the anchor frame; based on the inspection requirements, combining the distance between the center coordinates of all detected anchor frames and the coordinates of the center point of the camera frame and the size of the anchor frame, screening and fine-tuning all detected anchor frames; performing a linear combination and quantitative sorting of these two indicators, and selecting the anchor frame with the highest score and meeting the requirements as the target for subsequent inspection; The cropping algorithm based on the shortest side compensation is as follows: the anchor frame where the contact network support device is located is proportionally cropped so that the anchor frame where the contact network support device is located has the same ratio as the camera imaging frame; The pinhole camera model-based anchor frame center locking algorithm calculates the pan / tilt rotation angle based on the coordinates of the anchor frame center and the camera frame center, and controls the pan / tilt to move the anchor frame center of the cropped contact network support device to the image frame center according to the rotation angle. The anchor frame magnification algorithm based on the camera pinhole model is as follows: the focal length is adjusted according to the camera pinhole model, and the anchor frame is magnified to s times the full frame based on the adjusted focal length. <s≤1; Photographing the catenary support device in a nine-grid: crop the enlarged frame of the anchor frame of the catenary support device into a nine-grid pattern, lock the center of each grid, move each grid to the center of the frame, enlarge it, and photograph it to obtain nine images of the catenary support device.

2. The method of claim 1, wherein, The said construction of the target detection model of the railway contact network support device and training the target detection model using the data set of the railway contact network support device include: A drone equipped with a gimbal and camera was used to conduct a flight inspection along the side of the railway line, with the flight trajectory parallel to the track. Image data of the catenary support devices captured by the drone was annotated to obtain image data of catenary support devices of different sizes and shapes. The catenary support devices were divided into right-side and left-side catenary support devices. The image data was used to create a dataset for catenary support device target recognition, which was then divided into a training set, a validation set, and a test set. Select a target detection model, train the target detection model using the image data in the training set, verify and test the target detection model using the image data in the validation set and test set, and save the parameters of the trained target detection model.

3. The method of claim 2, wherein, The target detection model is a YOLOv7 model.

4. The method of claim 1, wherein, The trained target detection model and the intelligent control algorithm are deployed on the unmanned aerial vehicle, the unmanned aerial vehicle is used to patrol along the railway line, the gimbal and the camera carried by the unmanned aerial vehicle are used to capture image data of the railway catenary support device, and the image data includes the following steps: The trained target detection model is converted into a model capable of being deployed on an edge computing terminal, the trained target detection model and the intelligent control algorithm are deployed on an edge computing terminal of the unmanned aerial vehicle, According to the distribution characteristics of the catenary support device along the railway, the best shooting waypoint is marked on the map, the route task file for the patrol is generated, the route task file is transmitted to the unmanned aerial vehicle, and the unmanned aerial vehicle starts the target detection algorithm and the intelligent gimbal camera control algorithm after flying to the specified waypoint; The camera is started to acquire a video stream, the rotation angle of the gimbal and the adjustment size of the focal length of the camera are calculated, the gimbal is controlled to perform center locking and nine-grid photographing, the obtained image is input into the trained target detection model, the position of the catenary support device in the image is obtained according to the target detection algorithm, the best catenary support device is screened out, the center point coordinates of the best catenary support device in the image and the length and width size of the anchor frame where the catenary support device is located are output, and the gimbal and the camera are controlled by the intelligent algorithm to capture images of the target catenary support device in multiple scales.

5. The method of claim 4, wherein, The captured image data of the railway catenary support device is transmitted to the ground server through a wireless communication network, and the image data includes the following steps: The captured standardized and multi-scale catenary support device image data is marked in time and space, and the catenary support device image data with time and space information is transmitted to the ground server terminal through 5G or a picture transmission device.

Images