Unmanned aerial vehicle intelligent inspection method and system based on FAST structure abnormal target recognition

Abstract

The invention discloses an unmanned aerial vehicle intelligent inspection method and system based on FAST structure abnormal target identification. The method comprises the following steps: S1, establishing an unmanned aerial vehicle control platform; s2, a flight path scheme of the unmanned aerial vehicle is set in combination with the multi-source data acquisition type and the telescope structure characteristics; s3, for the FAST complex structure abnormal target, acquiring an infrared image and a visible light image by using an unmanned aerial vehicle effective load; obtaining a visible-infrared image pair after registration; s4, marking the visible light image of the abnormal target data, and forming a multi-source FAST structure abnormal target data set by a marked abnormal target label and a visible-infrared image pair; s5, preprocessing data in the FAST structure abnormal target data set to form an amplified data set; and S6, performing intelligent identification and verification on the FAST structure abnormal target according to the multi-source fusion abnormal target detection model. The unmanned aerial vehicle technology is applied to astronomical observation equipment structure inspection for the first time, and the inspection efficiency of a radio telescope can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of astronomy, and in particular to an intelligent inspection method and system of an unmanned aerial vehicle based on FAST structural abnormality target recognition. Background Art

[0002] The Five-hundred-meter Aperture Spherical radioTelescope (FAST) is a major national scientific and technological infrastructure. It uses the natural karst depression in Guizhou as the telescope site to build the world's largest single-aperture radio telescope to achieve high-precision astronomical observations over a large sky area. Since FAST passed the national acceptance in January 2020, it has discovered more than 1,000 pulsars, and the high-quality open operation of China's Sky Eye has achieved a series of important scientific results.

[0003] FAST has a huge structure, which includes: ring beam (5,400 tons), cable net (6,670 main cables, 2,225 down cables), reflector unit (4,450 pieces), actuator anchor piers (2,225), six towers, structural foundation, dangerous rock slopes, etc. Inspection and maintenance are required during the operation of FAST. The inspection contents include: panel subunits, back frames, spherical bearings, node plates, targets and target seats, main cables and down cables, bolts and pins, sliding layers and restricting doors, etc. After more than five years of operation, the following common problems were found: panel deformation, rod instability; ring beam rust, target failure, down cable sheath damage and replacement, main cable sheath repair, main cable pin status, motion interference, tower body corrosion and connection bolt falling off, etc.

[0004] The node position structure in the reflective surface is complex, with multiple reflective surface unit back frames connected to the node upper disk, and their motion pairs are also different. It is easy to have problems such as rust, deformation of spare rods, and motion interference. Although there is a cable crane system available for maintenance, it has the disadvantages of low efficiency, high cost, and small working area for reflective surface inspection, and is not suitable for large-scale and efficient inspection work.

[0005] There are six feed towers with a tower body length of 120 to 170 meters. They are built according to the terrain and have a complex structure. Regarding the problems of tower body rust and connecting bolts falling off, manual inspections have disadvantages such as low efficiency, high risks, and being restricted by many conditions.

[0006] Currently, structural inspections are mainly carried out through manual inspection methods, which have problems such as low efficiency, no video acquisition, safety hazards, and great climate impact.

[0007] In general, the current inspection of reflectors and feed towers has the following problems:

[0008] (1) Most current inspection methods are traditional methods such as cable hanging and manual climbing, without video recording;

[0009] (2) Inspections are difficult in complex climates, such as rain, snow, severe cold, freezing weather, etc.

[0010] (3) More than half of the components are located at high places with poor accessibility, complex terrain, low inspection efficiency and high inspection risk;

[0011] (4) During inspections, only simple notes can be taken, which are difficult to review and read. Summary of the invention

[0012] In view of the problems existing in the prior art, the purpose of the present invention is to provide a FAST UAV intelligent inspection method and system, which effectively assists the staff in carrying out structural inspection work and effectively improves the detection accuracy of structural anomalies such as panel dents and metal corrosion under reasonable data collection conditions. Another purpose of the present invention is to provide a FAST UAV intelligent inspection system that implements the above-mentioned intelligent inspection method.

[0013] To achieve the above object, the FAST UAV intelligent inspection method of the present invention comprises the following steps:

[0014] S1. Establishing UAV platforms: According to the UAV inspection reflective surface path, several operating platforms for launching and retracting UAVs are set up;

[0015] S2. Combine the multi-source data collection types to set the UAV flight plan;

[0016] S3. For the FAST structural anomaly target, collect infrared images and visible light images; use the characteristic linear mapping image registration method to map the infrared image corresponding to the visible light image to the visible light image scale to obtain the registered visible-infrared image pair;

[0017] S4. Annotate the visible light images of abnormal target data, and mark them as panel structure defects, rust, screw detachment and rod detachment. The annotated abnormal target labels and visible-infrared image pairs constitute a multi-source FAST structural abnormal target data set;

[0018] S5, preprocessing the data in the abnormal target data set to form an amplified data set;

[0019] S6. Based on the data set in S5, the FAST structural abnormal targets are effectively identified and verified according to the multi-source fusion abnormal target detection model.

[0020] Furthermore, the method for setting the path of the drone inspection reflective surface is: set a number of inspection circles on the reflective surface, starting from the first circle at the center of the reflective surface, first inspect a circle of node disks along the circumference of the first circle, then expand to the second circle of circular inspection, and then expand the inspection circles outward in turn, increasing the inspection path circle by circle.

[0021] Furthermore, when the radius of the inspection circle increases to exceed the UAV's range, an operating platform for the UAV is set up on the outer FAST measurement pier, and the UAV is retracted and deployed on the operating platform.

[0022] Furthermore, based on the cable net zoning, the reflective surface of FAST was divided into five identical areas, and the drones inspected the five areas separately.

[0023] Furthermore, in S2, the UAV flight plan is set as follows: the UAV flight route is determined according to the main abnormality types of the FAST structure and the spatial distribution of the FAST structure; the UAV flight area is determined according to the sensor's collection distance, flight altitude limit, and safe flight time limit; and then the UAV inspection system structural abnormality data collection flight plan is determined.

[0024] Furthermore, in S3, the infrared image undergoes a linear transformation of grayscale values ​​and is combined with the image information after abnormal target feature extraction of the visible light image to perform geometric registration correction; the image after geometric registration correction is combined with the visible light image again for image mapping to finally obtain the registered infrared image.

[0025] Furthermore, in S5, the step of preprocessing the data in the abnormal target data set is as follows:

[0026] S5.1 According to the category of abnormal targets, the abnormal target data set is randomly grouped into training set, test set and validation set in a ratio of 8:1:1;

[0027] S5.2 performs data augmentation operations on the abnormal target data by performing image flipping, random offset of the abnormal target, image cropping, adding motion blur and white noise.

[0028] Furthermore, in S6, the multi-source fusion abnormal target detection model is specifically:

[0029] S6.1 initializes the detection model parameters based on the data set in S5;

[0030] S6.2 inputting the amplified multi-source FAST structural abnormality target training set into the detection model to train the detection model;

[0031] S6.3 Input the amplified multi-source FAST structural abnormality target validation set for validation;

[0032] After S6.4 reaches the expected result, an effective multi-source fusion abnormal target detection model is formed.

[0033] Furthermore, the training process of the multi-source fusion abnormal target detection model is specifically as follows:

[0034] 1) Set initial training parameters;

[0035] 2) Randomly collect smaller data sets from the original training set;

[0036] 3) Adjust the training rounds to conduct small sample training and obtain preliminary test results at the district and county level for classification and regression losses;

[0037] 4) Determine whether there is a problem with the data set design. If there is no problem, put all training sets into training. If there is a problem, return to the random data set collection step 2);

[0038] 5) Set Dropout to 0.5, start formal training using the initial training parameters, and set the batch size to 4;

[0039] 6) Determine whether there is an overfitting or underfitting problem based on the loss curve, and then determine whether to increase the training accuracy;

[0040] 7) Analyze the loss curve and defect detection results of model classification and regression, and then optimize the analysis results.

[0041] On the other hand, the present invention provides an unmanned aerial vehicle intelligent inspection system based on FAST structural abnormal target recognition, which is used to implement the above-mentioned FAST unmanned aerial vehicle intelligent inspection method.

[0042] Based on the inspection demand for large astronomical equipment structures, this invention applies drone technology to astronomical equipment structure inspection for the first time. By selecting appropriate image samples, four types of abnormal targets, namely panel abnormalities, metal corrosion, screw shedding, and rod fracture, can be effectively inspected. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a framework diagram of the UAV intelligent inspection method based on FAST structural abnormal target recognition;

[0044] Figure 2 A framework diagram for spatial registration of multi-source data;

[0045] Figure 3 This is a diagram of the data preprocessing framework;

[0046] Figure 4 This is a framework diagram of the multi-source fusion abnormal target detection model;

[0047] Figure 5This is a training process framework diagram for the multi-source fusion abnormal target detection model;

[0048] Figure 6 This is a schematic diagram of cable net partitioning;

[0049] Figure 7 Schematic diagram of the path planning for drone inspection of reflective surfaces Figure 1 ;

[0050] Figure 8 Schematic diagram of the path planning for drone inspection of reflective surfaces Figure 2 ;

[0051] Fig. 9 This is a schematic diagram of the operating principle of GCANet;

[0052] Fig.10 This is a schematic diagram of the Gaussian transformer encoder network;

[0053] Fig.11 is the detection result image; Figure a is the actual labeled image of the panel abnormal depression; b is the panel abnormal depression detection result image of Figure a;

[0054] Fig.12 is the detection result image of another panel, where Figure c is the actual labeled image of the abnormal depression of another panel; and d is the detection result image of the abnormal depression of the panel in Figure c. DETAILED DESCRIPTION

[0055] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0057] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0059] like Figures 1 to 12 As shown, the present invention provides an intelligent inspection method and system of UAV based on FAST structural abnormal target recognition. The optimized UAV and payload are used to perform flight inspection tests on the FAST telescope structure. Image recognition technology is used, combined with sample feature learning mechanism, deep learning algorithm and training model, to establish a set of highly reusable, scalable and innovative image defect recognition algorithms, to inspect the contamination, deformation, jamming and other faults of the telescope structure, quickly identify problems and hidden dangers of the FAST structure, provide a basis for maintenance and repair, and enable the construction unit to perform timely maintenance and construction.

[0060] The UAV intelligent inspection method based on FAST structural abnormal target recognition provided by the present invention comprises the following steps:

[0061] S1. Establishing UAV platforms: According to the UAV inspection reflective surface path, several operating platforms for launching and retracting UAVs are set up;

[0062] S2. Combine the multi-source data collection types to set the UAV flight plan;

[0063] S3. For the FAST structural anomaly target, collect infrared images and visible light images; use the characteristic linear mapping image registration method to map the infrared image corresponding to the visible light image to the visible light image scale to obtain the registered visible-infrared image pair;

[0064] S4. Annotate the visible light images of abnormal target data, and mark them as panel structure defects, rust, screw detachment and rod detachment. The annotated abnormal target labels and visible-infrared image pairs constitute a multi-source FAST structural abnormal target data set;

[0065] S5, preprocessing the data in the abnormal target data set to form an amplified data set;

[0066] S6. Based on the data set in S5, the FAST structural abnormal targets are effectively identified and verified according to the multi-source fusion abnormal target detection model.

[0067] In step S2, the UAV flight plan is as follows:

[0068] like Figure 1 The figure shows the framework diagram of the UAV flight plan. According to the actual needs of FAST structural abnormal data collection, the type of inspection abnormal target, the characteristics of inspection abnormal targets, UAV flight parameters, abnormal target detection accuracy requirements, UAV flight time and working hours requirements, collection data storage capacity and other factors are comprehensively considered to select a suitable UAV.

[0069] Attached Figure 1 Note: At the same time, for the main abnormal types of FAST structure, such as unit rod fracture, ring beam bolt detachment, unit panel holes, structural corrosion and other structural abnormalities, according to the spatial distribution characteristics of FAST structure, large span, high position, many components, complex structure, etc., combined with multi-source data collection type, comprehensive consideration of local climate conditions, drone flight altitude restrictions and aircraft safe flight time limit and other factors, the drone inspection system structure abnormal data collection flight path plan is formulated: Taking the reflection surface unit and cable net node flight collection as an example, the reflection surface node disk inspection path planning plan: It is planned to start from the center of the reflection surface and diffuse radially to the periphery, starting from the first circle, first inspect a circle of node disks along the first circle circumference, and then radially extend to the second circle of node disks circumference inspection, and then expand to the third circle, fourth circle..., and then increase the radial inspection path circle by circle. The distance for collecting reflection surface data is about 1-2 meters, the flight altitude is generally about 2 meters higher than the reflection surface, the flight area is generally above the reflection surface, and the drone collects node disk data at a downward angle of 45°. The flight safety time limit is generally 25 minutes, that is, return to the home base to replace the battery. The measurement base extending out of the reflective surface is used as the operating platform for the drone to take off and land. When the drone battery needs to be replaced or the flight needs to be adjusted, the measurement base can be used as the operating platform to retract and release the drone. After each flight inspection, the drone is returned to the operating platform along the shortest path to recover the drone and collect the reflective surface data. Multi-source data collection types include visible light video image data and infrared video image data.

[0070] S2.1 Determine the flight route of the drone: Taking the flight collection of the reflective surface unit and the cable net node as an example, the inspection path planning scheme for the reflective surface node disk is: it is planned to diffuse radially from the center of the reflective surface to the periphery, starting from the first circle, first inspect a circle of node disks along the circumference of the first circle, and then radially expand to the second circle of node disks for circumferential inspection, and then expand to the third circle, the fourth circle..., and then increase the radial inspection path circle by circle. The measurement base pier extending out of the reflective surface is used as the operating platform for the drone to take off and land. When the drone battery needs to be replaced or the flight needs to be adjusted, the measurement base pier can be used as the operating platform to retract and release the drone. After each flight inspection, return to the operating platform along the shortest path to recover the drone to realize the collection of reflective surface data.

[0071] In step S2.2 sensor collection distance, visible light and infrared camera sensors are used to collect video image data. The distance between the sensor collection drone and the current detection node is generally controlled at 1-2 meters. The flight altitude is generally about 2 meters higher than the reflective surface. The flight area is generally above the reflective surface. The drone collects node disk data at a 45° downward angle.

[0072] Step S2.3 Determine the UAV flight area: This invention is based on the development of this technology. It is not a large-scale conventional structural inspection, but based on several types of structural damage such as unit rod fractures and panel holes, the UAV intelligent inspection technology is developed and implemented. After the completion of this technology development, it will be applied to FAST structural inspection on a large scale. Structural abnormalities such as unit rod fractures, panel holes, ring beam bolts falling off, and structural corrosion are relatively common abnormal problems in telescope structures. A breakthrough in the intelligent inspection technology of the above four types of problems is achieved.

[0073] Step S2.4 determines the abnormal data collection flight plan, including data collection on structural parts of items where structural abnormalities such as unit rod breakage, panel holes, ring beam bolt detachment, structural corrosion, etc. may occur, collects video and image data, and then performs intelligent recognition on the computer algorithm software side to achieve intelligent recognition of structural abnormalities.

[0074] Step S3, such as Figure 2As shown in the figure, it is a framework diagram of multi-source data spatial registration. Multi-source data spatial registration is the basis for the application of structural anomaly detection in the FAST structure unmanned inspection system with multi-source data fusion. In order to obtain more FAST structural texture and spectral information, the original information corrected by infrared imaging will be used as much as possible to extract abnormal target information. In addition, the visual grayscale dynamic range of the original image data is narrow, and it is necessary to enhance and stretch it with a certain standard amount. After completing the above work, the data of each data source should be geometrically registered. During geometric registration correction, the image with low spatial resolution should be registered to the image with high spatial resolution, and then classified and processed, and finally corrected to the visible light image actually used, in preparation for the next step of fusion and image digital mosaic. In step S3, before detecting the abnormal target of FAST structure based on multi-source fusion, since the collected visible light image does not completely match the corresponding infrared image, it is necessary to first perform registration research on the dual light image. The feature-based registration method focuses on obtaining relatively stable features in the image, such as corners, edges, etc., and then uses these feature information for registration. The feature-based registration uses the distance function as a matching criterion to calculate and find the corresponding features. Distance criteria generally include Euclidean distance, Mahalanobis distance, and Hausdorff distance. In order to improve the accuracy of registration, a coarse-to-fine registration process is usually adopted: first, coarse registration, using the distance criterion to compare the similarity of features, and then find the corresponding features and establish a rough set of matching point pairs; second, fine registration, using some algorithms to filter the set of matching point pairs to obtain a more accurate set of matching point pairs, and using the RANSAC algorithm for fine registration.

[0075] S3.1 Collect infrared and visible light image data for areas where abnormal targets may exist in the FAST structure.

[0076] S3.2 uses the characteristic linear mapping image registration method to map the infrared image corresponding to the visible light image to the visible light image scale to obtain the registered visible-infrared image pair. Specifically, the infrared image undergoes a linear transformation of the grayscale value.

[0077] S3.3 combines the image information after the abnormal target features of the visible light image are extracted to perform geometric registration correction.

[0078] S3.4 The image after geometric registration correction is combined with the visible light image again for image mapping, and finally the registered infrared image is obtained.

[0079] In step S4, the LabelImg data annotation tool is used to annotate the visible light image of the abnormal target data, and annotate them as panel structure defects, rust, screw detachment and rod detachment respectively. The annotated abnormal target labels and visible-infrared image pairs constitute the multi-source FAST structural abnormal target data set.

[0080] Step S5, such as Figure 3 The figure shows the data preprocessing framework diagram. The multi-source fusion abnormal target detection model is the core tool for structural anomaly identification. It can provide a more efficient anomaly recognition rate. It is based on the multi-source fusion FAST defect detection network to initialize the network parameter settings. According to the expanded multi-source FAST structural abnormal target training set, the detection network model is trained to achieve the expected abnormal recognition rate. The multi-source FAST structural abnormal target data set consists of one-to-one abnormal target labels and visible-infrared image pairs. In order to improve the detection accuracy and robustness of the abnormal target detection algorithm model, the data in the abnormal target data set needs to be preprocessed. Including:

[0081] S5.1 According to the category of abnormal targets, the abnormal target data set is randomly grouped into training set, test set and validation set in a ratio of 8:1:1;

[0082] S5.2 performs data augmentation operations on the abnormal target data by performing image flipping, random offset of the abnormal target, image cropping, adding motion blur and white noise.

[0083] The entire data set is randomly grouped according to the category of abnormal targets, that is, divided into training set, test set and validation set in a ratio of 8:1:1. Due to the reasonable structural design, good use and timely maintenance of the FAST telescope, the actual number of abnormal target images collected is small, and it is necessary to perform data augmentation operations such as image flipping, random offset of abnormal targets, image cropping, adding motion blur and white noise on the abnormal target data; the actual background environment of the collection is relatively complex, and some negative samples are appropriately added to improve the detection accuracy of the detection model.

[0084] Step S6, as Figure 4 As shown in the figure, it is a framework diagram of the multi-source fusion abnormal target detection model.

[0085] Multi-source fusion abnormal target detection model, specifically:

[0086] S6.1 initializes the detection model network parameters based on the data set in S5;

[0087] S6.2 Input the amplified multi-source FAST structural abnormal target training set into the detection model to train the detection model; if the expected result is achieved, a multi-source fusion abnormal target detection network model is formed.

[0088] S6.3 Input the amplified multi-source FAST structural abnormality target validation set for validation;

[0089] S6.4 If the expectation is not met, adjust the network parameters and continue to train the detection network model until the expectation is finally met, thus forming an effective multi-source fusion abnormal target detection model.

[0090] like Figure 5 As shown in the figure, it is a training process framework diagram of the multi-source fusion abnormal target detection model. The training process of the multi-source fusion abnormal target detection model is as follows:

[0091] 1) Set initial training parameters;

[0092] 2) Randomly collect smaller data sets from the original training set;

[0093] 3) Adjust the training rounds to conduct small sample training and obtain preliminary test results at the district and county level for classification and regression losses;

[0094] 4) Determine whether there is a problem with the data set design. If there is no problem, put all training sets into training. If there is a problem, return to the random data set collection step 2);

[0095] 5) Set Dropout to 0.5, start formal training using the initial training parameters, and set the batch size to 4;

[0096] 6) Determine whether there is an overfitting or underfitting problem based on the loss curve, and then determine whether to increase the training accuracy;

[0097] 7) Analyze the loss curve and defect detection results of model classification and regression, and then optimize the analysis results.

[0098] In the present invention, comprehensive consideration is given to factors such as the inspection equipment detection application effect, equipment technical indicators, system integration, etc., and the configuration of unmanned related operation equipment is optimized according to the inspection business demand status, actual operation conditions, aviation airspace management methods, equipment procurement conditions, etc., so as to reduce costs and increase efficiency and maximize resource utilization; according to the distribution characteristics of the unmanned inspection lines of the FAST structure (upper and lower parts of the reflector surface, feed tower, etc.), various factors such as unmanned live working performance, take-off and landing flexibility, stable flight, fixed-point hovering in the air, load capacity, loading and unloading of inspection equipment are considered, and combined with the main characteristics of the inspection area, the unmanned inspection plan optimization method is studied, and reasonable classification and optimization research are carried out to optimize the inspection efficiency.

[0099] During the line inspection planning, since the center of the FAST structure is an irregular structure with a concave shape, the UAV flight path needs to be planned based on the structural characteristics. The FAST UAV structure inspection focuses on the FAST reflective panel.

[0100] The active reflector system includes cable nets, ring beams, actuators, reflector units, windbreak walls, health monitoring and other parts, which can form a 300m-diameter instantaneous parabola in the direction of the radio source, converge the electromagnetic waves radiated by celestial bodies to a focus, and cooperate with astronomical observations.

[0101] According to the cable net partition, it is divided into 5 identical areas. The cable net partition diagram is as follows Figure 6 As shown in Figure 1, each node disk in the cable net is connected to 6 main cables, 6 reflector units and 1 down-pull cable, and the displacement control of the down-pull cable by the actuator is transmitted to the cable net node, as shown in Figure 1. Figure 6-10 The main cable net surface is usually a sphere with a curvature radius of 300.4 meters, but when the FAST telescope is working, the main cable net surface with a diameter of 300 meters can be actively deformed from a neutral sphere to a rotating parabola, and the position of the parabola can be continuously moved.

[0102] FAST UAV flight patrol planning: Since the reflective surface unit is consistent with the cable net partition, both are 1 / 5 central symmetric, that is, according to the cable net partition, the reflective surface is divided into 5 identical areas, such as Figure 6 As shown, one fifth of the reflection surface, Figure 7 A flight path plan for a drone to inspect reflective surface node disks.

[0103] Taking into account the characteristics of drones such as endurance, return to the original route, alarm when approaching objects at close range, and efficient cruising at the same latitude, combined with drone performance, a special node disk inspection path planning scheme is proposed: it is planned to start from the first circle at the center of the reflective surface, first inspect a circle of the node disk along the circumference of the first circle, and then expand to the second circle, and then expand to the third circle, fourth circle..., and then increase the inspection path circle by circle. When the number of inspection circles increases, the measurement base can be used as an operating platform to retract and release the drone. The schematic diagram of the drone inspection reflective surface path planning is as follows Figure 7 , Figure 8 shown.

[0104] The deep learning network YOLOv5 is used for abnormal target detection.

[0105] (I) Feature extraction network

[0106] CSPDarknet is a convolutional neural network that aggregates and forms image features at different image granularities, and is used as the feature extraction network of YOLOv5. CSPDarknet solves the problem of repeated gradient information in network optimization of feature extraction networks in other large convolutional neural network frameworks, and integrates the gradient changes from beginning to end into the feature map, thereby reducing the number of model parameters, ensuring both inference speed and accuracy, and reducing model size.

[0107] (II) Feature Enhancement Network

[0108] The feature enhancement network passes image features to the prediction layer through a series of designed network layers. The feature enhancement network is mainly used to generate feature pyramids to enhance the model's detection of objects of different scales, so that the same object of different sizes and scales can be detected.

[0109] (III) Detection network

[0110] The detection network predicts the category and location of the target based on the image features. The detection network is mainly used for the final detection part, by applying anchor boxes on the feature map and generating a final output vector with class probability, target score and bounding box.

[0111] (IV) Loss function setting

[0112] For the FAST structural abnormal target detection task, it is necessary to set the loss function to adjust the model convergence direction. Since there are many small-sized targets in the detection task, GIoU Loss is used as supervision during the training process. Assume that the coordinates of the bounding box of the predicted target and the labeled box of the actual target are and in

[0113] B P is the bounding box coordinate of the predicted target, B g are the coordinates of the annotation box of the actual target. (x1, y1) and (x2, y2) are the coordinate points respectively.

[0114] The specific process of calculating GIoU Loss is as follows:

[0115] (1) The area of ​​the actual target annotation box and the predicted target bounding box is calculated as follows, where A g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box.

[0116]

[0117]

[0118] A g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box. (x1, y1) and (x2, y2) are the coordinate points respectively.

[0119] (2) Calculate the overlapping area I between the predicted target bounding box and the actual target annotation box.

[0120]

[0121] I is the overlap area between the predicted target bounding box and the actual target annotation box. (x1, y1) and (x2, y2) are the coordinate points of the target bounding box and the actual target annotation box, respectively.

[0122] (3) Find a bounding box B that can contain the predicted target P and the actual target annotation box Bg The minimum bounding box B c , and calculate B c Area A c .

[0123]

[0124] B P is the predicted target bounding box. B g is the actual target annotation box. c is the minimum bounding box, with area denoted by A c express.

[0125] (4) Calculate the intersection over union (IoU) of the predicted target bounding box and the actual target annotation box.

[0126]

[0127] IoU stands for Intersection over Union. g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box.

[0128] (5) Calculate the global intersection-over-union (GIoU) ​​between the predicted target bounding box and the actual target annotation box.

[0129]

[0130] GIoU stands for Global Intersection over Union. g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box.

[0131] (6) Calculate the loss function L of GIoU Loss GIoU .

[0132] L GIoU =1-GIoU

[0133] IoU stands for Intersection over Union. GIoU stands for Global Intersection over Union.

[0134] Infrared visible fusion detection algorithm

[0135] In view of the multi-source fusion problem, the present invention proposes a cross-modal multi-source fusion detection algorithm GCANet based on a cross-attention mechanism. Which step S6 does this part correspond to?

[0136] 1. Overview of methods

[0137] GCANet network overview is as follows Fig. 9As shown. They are respectively associated with feature extraction, multi-source data feature fusion and FAST structural abnormal target detection tasks. The infrared-visible light image pair is used as input. First, the feature extraction network is input. The feature extraction network consists of a deep convolutional network, which is trained in a parameter-sharing manner and extracts visible light and near-infrared depth features respectively. The extracted visible light and near-infrared depth features are input into the Gaussian transformer encoder network based on the attention module. The Gaussian transformer encoder network takes paired multi-source data features as input, enhances and fuses multi-source salient features based on a learnable two-dimensional Gaussian cross attention mechanism, and outputs high-quality feature maps containing salient local information and global distribution information of multi-source targets. The enhanced feature map is transmitted to the classification network to obtain the detection result with the target category, as well as the corresponding bounding box and confidence score.

[0138] (II) Feature Extraction Network

[0139] The purpose of the feature extraction network is to extract features from visible light and infrared images respectively by means of parameter sharing. The present invention uses the ResNet50 network as the feature extraction network.

[0140] (III) Gaussian transformer encoder network

[0141] The Gaussian Transformer Encoder Network is a transformer model that enhances the expression of target salient features by fusing deep features. In order to learn multi-source fusion feature expression with lightweight computing and memory capabilities, the present invention introduces a Gaussian cross attention mechanism module to fuse the target salient deep features in infrared and visible light features, such as Fig.10 shown.

[0142] (IV) Classification and Positioning Network

[0143] For the classification and localization network, the present invention designs a network consisting of two parallel branches, namely the classification branch and the regression branch. Different convolution blocks are also set in the two branches. There are two blocks on the classification branch and four blocks on the regression branch, each of which contains a convolution layer, a batch normalization layer and a ReLU layer. In addition, the present invention adds an implicit prediction of the object probability of each regression branch, and the final classification result will be multiplied by this implicit prediction.

[0144] (V) Loss Function

[0145] After network training, GCANet can capture local and long-range dependencies, generate clearer content, and retain most of the visual information from multi-source data features. In order to more accurately detect FAST structural anomaly targets, the present invention minimizes the loss function to train GCANet, and the total loss L is as follows:

[0146] L=L cls +λLreg

[0147] L consists of a weighted combination of classification loss and bounding box loss, where the classification loss is Focal loss and the bounding box loss is GIoU Loss.

[0148] The specific implementation of the FAST structural abnormal target detection experiment verification is as follows:

[0149] For abnormal inspection and detection evaluation, appropriate image samples are selected with reference to actual sample anomalies, and recognition accuracy and running time are used as the main evaluation indicators. Based on the panel abnormality dent database, the effectiveness of the proposed FAST structural abnormal target detection method is experimentally verified for four types of abnormal targets: panel abnormality, metal rust, screw detachment, and rod fracture.

[0150] The specific evaluation indicators of FAST structural abnormal target detection are as follows:

[0151] In order to objectively evaluate the detection performance of the model, two common evaluation indicators, recognition accuracy and single-frame detection time, are used as evaluation indicators for FAST structural abnormality target detection.

[0152] Recognition accuracy

[0153] The recognition accuracy rate indicates the proportion of correctly identified abnormal targets of a certain type to all actually labeled abnormal targets of a certain building. Fig.11 All possibilities in the detection results are shown, where the correct predictions are positive examples and the wrong predictions are negative examples. TP (true positive examples) represents the number of samples that are both predicted and actually abnormal targets of this type. FP (false positive examples) represents the number of samples that are predicted to be abnormal targets of this type but are not actually abnormal targets of this type. TN (true negative examples) represents the number of samples that are neither predicted nor actually abnormal targets of this type. FN (false negative examples) represents the number of samples that are not predicted to be abnormal targets of this type but are actually abnormal targets of this type.

[0154]

[0155] The formula for calculating the recognition accuracy is as follows:

[0156]

[0157] Single frame detection time

[0158] Single-frame detection time is an important indicator for evaluating the detection speed of the model. It indicates the time required for the model to detect each frame of the image. The shorter the detection time, the faster the detection speed of the model. The calculation formula for single-frame detection time is as follows:

[0159]

[0160] Analysis of FAST abnormal target detection results

[0161] For abnormal inspection and detection evaluation, the present invention refers to the actual sample anomalies, selects appropriate image samples, uses recognition accuracy and running time as the main evaluation indicators, and based on the panel abnormality depression database, steel structure corrosion database, screw falling database and rod falling database, experimentally verifies the effectiveness of the proposed FAST structural abnormal target detection method for four types of abnormal targets: panel abnormality, metal corrosion, screw falling and rod fracture.

[0162] Panel abnormal dent database detection results

[0163] The abnormal data of the FAST panel has the characteristics of small targets, high similarity, and large differences in different shooting angles. Based on the YOLOV5 algorithm, a method specifically used for small target detection is designed, such as mosaic enhancement of data. In addition, for the problem of large differences in different shooting angles, a targeted solution was made when designing the flight plan. When collecting data, shooting at a pitch angle of 15 degrees and minus 30 degrees has the best effect.

[0164] According to the characteristics of the panel anomaly database, the network parameters are set as follows: 150 training rounds, the training batch size is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0165] The detection results of the FAST structural abnormal target detection algorithm on the panel abnormal depression database are shown in the following table.

[0166] Table 1 Detection results of panel abnormal dent database

[0167]

[0168] There are a total of 100 verification data images used for panel dent abnormal target detection, which contain a total of 563 panel dent abnormal targets. The FAST structural abnormal target detection model correctly detected 482 panel dents, with a recognition accuracy of 85.6%. The average detection time for each image is 0.07s.

[0169] Some examples of detection image results are as follows Fig.11 and Fig.12 shown.

[0170] Fig.11and Fig.12 It is a mark identified by the pits on the reflective surface unit panel through intelligent recognition software, that is, after pits or holes appear on the reflective surface unit panel, the area is identified by the intelligent recognition algorithm software.

[0171] The FAST UAV intelligent inspection method and system provided by the present invention has the following advantages:

[0172] (1) Remote control, high safety factor, no personal safety accidents will occur;

[0173] (2) It has a wide range of applications and can be used to inspect reflectors, feed towers, surrounding mountains, etc.

[0174] (3) Less affected by terrain and weather;

[0175] (4) High detection accuracy, no visual blind spots, and can capture more details at close range;

[0176] (5) It can not only check external faults, but also detect hidden dangers such as missing pins and cracked nuts that are difficult to detect through manual inspections through refined inspections;

[0177] (6) Low labor intensity, high work efficiency, and reduced risks of manual climbing.

[0178] Based on the demand for safe and efficient inspection of large astronomical equipment structures, this invention applies drone inspection technology to astronomical equipment structure inspection for the first time, greatly improving the inspection efficiency of FAST telescope structure. This invention aims to explore the application of drone intelligent inspection system to safe and efficient inspection of FAST structure (upper and lower parts of reflector surface, feed tower, etc.), which can efficiently screen out faults such as contamination, deformation, rust, and jamming, so as to achieve early detection and early maintenance, and realize a high degree of harmony between detection and maintenance, which is of great significance for improving the reliability and robustness of FAST and safe and reliable operation.

[0179] Any process or method description in the flowchart of the present invention or described in other ways herein can be understood as a module, segment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, magnetic disks or optical disks, etc.

[0180] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without causing any contradiction.

[0181] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in the field can change, modify, replace, modify and other update operations on the above embodiments within the scope of the present invention.

Claims

1. The UAV intelligent inspection method based on FAST structural abnormal target recognition is characterized by: The following steps are involved: S1. Establishing UAV platforms: According to the UAV inspection reflective surface path, several operating platforms for launching and retracting UAVs are set up; S2. Combine the multi-source data collection types to set the UAV flight plan; S3. Collect infrared images and visible light images of FAST structural anomalies; Using the characteristic linear mapping image registration method, the infrared image corresponding to the visible light image is mapped to the visible light image scale to obtain the registered visible-infrared image pair; S4. Annotate the visible light images of abnormal target data, and annotate them as panel structure defects, rust, screw detachment and / or rod detachment. The annotated abnormal target labels and visible-infrared image pairs constitute a multi-source FAST structural abnormal target data set. S5, preprocessing the data in the abnormal target data set to form an amplified data set; S6. Based on the data set in S5, the FAST structural abnormal targets are effectively identified and verified according to the multi-source fusion abnormal target detection model.

2. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 1 is characterized in that: The method for setting the path of the drone inspection reflective surface is as follows: set up several inspection circles on the reflective surface from the inside to the outside, starting from the first circle at the center of the reflective surface, first inspect a circle of nodes along the circumference of the first circle, then expand to the second circle for inspection, and then expand the inspection circles outward in sequence, increasing the inspection path circle by circle.

3. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 2 is characterized in that: When the radius of the inspection circle increases to exceed the UAV's range, an operating platform for the UAV is set up on the outer FAST measurement pier, and the UAV is retracted and deployed on the operating platform.

4. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 1 is characterized in that: According to the cable net zoning, the reflecting surface of FAST is divided into 5 identical areas, and the drones inspect the 5 areas separately.

5. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 1 is characterized in that: In S2, the UAV flight plan is set as follows: the UAV flight route is determined according to the main abnormality types of the FAST structure and the spatial distribution of the FAST structure; the UAV flight area is determined according to the sensor's collection distance, flight altitude limit, and safe flight time limit; and then the UAV inspection system structural abnormality data collection flight plan is determined.

6. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 1 is characterized in that: In S3, the infrared image undergoes a linear transformation of grayscale values ​​and is combined with the image information after abnormal target feature extraction of the visible light image to perform geometric registration correction; the image after geometric registration correction is combined with the visible light image again for image mapping to finally obtain a registered infrared image.

7. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 1 is characterized in that: In S5, the steps of preprocessing the data in the abnormal target data set are as follows: S5.1 According to the category of abnormal targets, the abnormal target data set is randomly grouped into training set, test set and validation set in a ratio of 8:1:1; S5.2 performs data augmentation operations on the abnormal target data by performing image flipping, abnormal target random offset, image cropping, and adding motion blur and white noise.

8. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 7 is characterized in that: In S6, the multi-source fusion abnormal target detection model is specifically: S6.1 initializes the detection model parameters based on the data set in S5; S6.2 inputting the amplified multi-source FAST structural abnormality target training set into the detection model to train the detection model; S6.3 Input the amplified multi-source FAST structural abnormality target validation set for validation; After S6.4 reaches the expected result, an effective multi-source fusion abnormal target detection model is formed.

9. The UAV intelligent inspection method based on FAST structural abnormal target recognition according to claim 8 is characterized in that: The training process of the multi-source fusion abnormal target detection model is specifically as follows: 1) Set initial training parameters; 2) Randomly collect smaller data sets from the original training set; 3) Adjust the training rounds to conduct small sample training and obtain preliminary test results at the district and county level for classification and regression losses; 4) Determine whether there is a problem with the data set design. If there is no problem, put all training sets into training. If there is a problem, return to the random data set collection step 2); 5) Set Dropout to 0.5, start formal training using the initial training parameters, and set the batch size to 4; 6) Determine whether there is an overfitting or underfitting problem based on the loss curve, and then determine whether to increase the training accuracy; 7) Analyze the loss curve and defect detection results of model classification and regression, and then optimize the analysis results.

10. An unmanned aerial vehicle intelligent inspection system based on FAST structural abnormal target recognition, characterized in that: The inspection system is used to implement the FAST UAV intelligent inspection method as described in any one of claims 1-9.

Images