FAST structure damage data system based on unmanned aerial vehicle intelligent inspection

Abstract

The invention discloses a FAST structure damage data system based on unmanned aerial vehicle intelligent inspection. The FAST structure damage data system comprises a data acquisition module, an exception labeling module and a database construction module, wherein the data acquisition module is used for sorting FAST structure related image data shot in the inspection process of the unmanned aerial vehicle; the exception annotation module is used for annotating an exception target in the acquired image; the database construction module is used for constructing a corresponding database according to the abnormal classification condition, and the database comprises a panel abnormal depression database, a steel structure corrosion database, a screw falling database and a rod piece falling database; according to the invention, four engineering application data systems of FAST unmanned aerial vehicle routing inspection are established, multi-source data fusion unmanned aerial vehicle FAST structure abnormal target routing inspection data is processed and labeled based on an informatization means, and a multi-source fusion FAST abnormal target database is constructed. And a basic data model is provided for FAST structure extraction and abnormal target intelligent identification of the unmanned aerial vehicle based on the visible light image and the infrared image.

Description

Technical Field

[0001] The present invention relates to the fields of astronomical telescopes and unmanned aerial vehicles, and in particular to a FAST structural damage data system based on unmanned aerial vehicle intelligent inspection. Background Art

[0002] The Five-hundred-meter Aperture SphericalradioTelescope (FAST) is a major national scientific and technological infrastructure. It uses the natural karst depression in Guizhou as the telescope site to build the world's largest single-aperture radio telescope to achieve large-sky-area, high-precision astronomical observations.

[0003] FAST has a huge structure, which includes: ring beam (5,400 tons), cable net (6,670 main cables, 2,225 down cables), reflector unit (4,450 pieces), actuator anchor piers (2,225), six towers, structural foundation, dangerous rock slopes, etc. Inspection and maintenance are required during the operation of FAST. The inspection contents include: panel subunits, back frames, spherical bearings, node plates, targets and target seats, main cables and down cables, bolts and pins, sliding layers and restricting doors, etc. After more than five years of operation, the following common problems were found: panel deformation, rod instability; ring beam rust, target failure, down cable sheath damage and replacement, main cable sheath repair, main cable pin status, motion interference, tower body corrosion and connection bolt falling off, etc.

[0004] The node position structure in the reflective surface is complex, with multiple reflective surface unit back frames connected to the node upper disk, and their motion pairs are also different. It is easy to have problems such as rust, deformation of spare rods, and motion interference. Although there is a cable crane system available for maintenance, it has the disadvantages of low efficiency, high cost, and small working area for reflective surface inspection, and is not suitable for large-scale and efficient inspection work.

[0005] There are six feed towers with a tower body length of 120 to 170 meters. They are built according to the terrain and have a complex structure. Regarding the problems of tower body rust and connecting bolts falling off, manual inspections have disadvantages such as low efficiency, high risks, and being restricted by many conditions.

[0006] Currently, structural inspections are mainly carried out through manual inspection methods, which have problems such as low efficiency, no video acquisition, safety hazards, and great climate impact.

[0007] In general, the current inspection of reflectors and feed towers has the following problems:

[0008] (1) Most current inspection methods are traditional methods such as cable hanging and manual climbing, without video recording;

[0009] (2) Inspections are difficult in complex climates, such as rain, snow, severe cold, freezing weather, etc.

[0010] (3) More than half of the components are located at high places with poor accessibility, complex terrain, low inspection efficiency and high inspection risk;

[0011] (4) During inspections, only simple notes can be taken, which are difficult to review and read. Summary of the invention

[0012] The present invention provides a FAST structure damage data system based on unmanned aerial vehicle intelligent inspection, the inspection data system comprises a data acquisition module, an abnormality marking module and a database construction module; wherein the data acquisition module is used to sort out the FAST structure related image data taken during the unmanned aerial vehicle inspection; the abnormality marking module is used to mark abnormal targets in the collected images; the database construction module is used to construct a corresponding database according to the abnormal classification situation, the database comprises a panel abnormal dent database, a steel structure corrosion database, a screw shedding database and a rod shedding database; wherein,

[0013] The abnormal panel depression database includes depressions, pits and pit halos caused by damage to the FAST reflector unit panel, holes caused by falling objects from high altitude, and / or deformation caused by extrusion or collision of the panel;

[0014] The steel structure corrosion database includes: FAST ring beam lattice column, feed tower body, main cable head, steel plank road corrosion, paint peeling and body exposure and / or red rust coverage;

[0015] The screw-off database includes: the ring beam lattice column connection bolts are off, the feed tower connection bolts are off, the reflector unit connection bolts are off, the off bolt parts expose holes or the unit rods are pulled out;

[0016] The rod falling-off database includes: bending, breaking of the back frame rod of the reflective surface unit and / or loose connection between the rod and the ball head.

[0017] Furthermore, when collecting data on abnormal depression of the panel, original photos are taken at five shooting angles: vertical, horizontal direction plus or minus 45°, and vertical direction plus or minus 45°.

[0018] Furthermore, when collecting data on panel dents or falling object penetration anomalies, multiple panels are spliced ​​together in the laboratory and collected simultaneously and / or reflective surface panel data is collected at the FAST site.

[0019] Furthermore, the collected data is labeled using the LabelImg labeling tool, and all abnormal target data are labeled to build a database.

[0020] Further, the detailed steps of using LabelImg label annotation are as follows:

[0021] S1. Switch to the virtual environment that needs to be installed, switch to the yolov5 virtual environment;

[0022] S2. Install Labelimg and enter labelimg in the command line to open it;

[0023] S3. Open the folder where the pictures to be annotated are located;

[0024] S4. Switch to yolo mode and drag the frame to mark;

[0025] S5. Click Save, and the dataset file generated by the annotation is stored in the image folder directory;

[0026] S6. The dataset file stores the information of the frame and label, which will be automatically loaded when training the model.

[0027] Furthermore, the collected panel abnormality data are labeled using the LabelImg labeling tool, and all panel abnormality data are labeled as a panel abnormality depression database.

[0028] Furthermore, the collected steel structure corrosion data are labeled using the LabelImg labeling tool, and all corrosion data are labeled to build a steel structure corrosion database.

[0029] Furthermore, the collected screw-off data are labeled using the LabelImg labeling tool, and all screw-off data are labeled to build a screw-off database.

[0030] Furthermore, the collected bar falling-off data are labeled using the LabelImg labeling tool, and all bar falling-off or fracture data are labeled to build a bar falling-off database.

[0031] Furthermore, the deep learning-based fusion method achieves differentiated feature extraction and adaptive learning by using different network branches to obtain the FAST UAV inspection image fusion strategy.

[0032] The present invention establishes four engineering application data systems for FAST UAV inspections, processes and labels the inspection data of abnormal targets of FAST structures by UAVs through multi-source data fusion based on information technology, and constructs a multi-source fusion FAST abnormal target database, providing a basic data model for the extraction of FAST structures (upper and lower parts of the reflective surface, feed tower, etc.) and intelligent identification of abnormal targets by UAVs based on visible light images and infrared images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1This is a schematic diagram of the FAST UAV inspection system architecture;

[0034] Figure 2 This is the layout diagram of the FAST reflector panel;

[0035] Figure 3 Schematic diagram of the target location of abnormal sample anomalies for panel anomalies;

[0036] Figure 4 Schematic diagram of the distribution of aspect ratios of abnormal targets in the panel anomaly labeling samples;

[0037] Figure 5 Schematic diagram of the target location of sample anomalies marked for rust;

[0038] Figure 6 Schematic diagram of the aspect ratio distribution of abnormal targets in the rust-labeled samples;

[0039] Figure 7 Schematic diagram of the target location of the sample abnormality for screw shedding;

[0040] Figure 8 Schematic diagram of the aspect ratio distribution of abnormal targets in the sample marked for screw shedding;

[0041] Fig. 9 Schematic diagram of abnormal target location of sample for marking rod falling off;

[0042] Fig.10 Schematic diagram of the aspect ratio distribution of abnormal targets in the sample marked for rod shedding. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0045] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] The following combination Figure 1-Figure 10 The specific embodiments of the present invention are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0047] The present invention utilizes a drone to perform a flying inspection of the FAST telescope structure, utilizes image recognition technology, combines sample feature learning mechanism, deep learning algorithm and training model, studies the intelligent analysis and processing technology of FAST structure unmanned inspection data with multi-source data fusion, studies the cluster data analysis method of FAST structure unmanned inspection with multi-source data fusion, constructs a multi-source fusion FAST abnormal target database, studies the extraction and recognition technology of FAST structure abnormal targets based on visible light images and infrared images, studies the FAST structure image abnormal target recognition algorithm and configures an inspection plan, establishes a set of inspection data system, combines with image defect recognition algorithm, inspects various abnormal conditions of the FAST telescope structure, quickly and automatically identifies problems and hidden dangers of the FAST structure, and realizes efficient maintenance.

[0048] like Figure 1 As shown, the implementation process of the FAST UAV inspection system of the present invention is as follows: first, the UAV flight plan is determined according to the FAST related positions and structures that need to be inspected, and the multi-source image data of the FAST structure is obtained; then the multi-source data is spatially registered; then the multi-source abnormal target data is labeled to generate a data set; a FAST structure unmanned inspection abnormal target detection algorithm model with multi-source data fusion is created; finally, a FAST structure unmanned inspection system with multi-source data fusion is implemented, and FAST automated inspection is implemented through the system to improve inspection efficiency.

[0049] Under the above technical architecture, the present invention provides a FAST structural damage data system based on unmanned aerial vehicle intelligent inspection, including a data acquisition module, an abnormality marking module and a database construction module. By marking the abnormal target data of the FAST structure, a panel abnormal depression database, a steel structure rust database, a screw falling database and a rod falling database are established, a total of four engineering application databases, which provide a data basis for the subsequent unmanned inspection of the FAST structure by realizing multi-source data fusion. Among them, the abnormal panel depression database includes: FAST reflector unit panel damage such as depressions and pits caused by hail, holes caused by falling objects from high altitude, deformation caused by extrusion or collision, etc.; steel structure corrosion database, including: FAST ring beam lattice column, feed tower body, main cable head, steel plank road, etc., with paint peeling off and exposing the body, red rust covering, etc.; screw falling database, including: ring beam lattice column connection bolts falling off, feed tower connection bolts falling off, reflector unit connection bolts falling off, etc., the bolt parts that fall off expose holes or unit rods are pulled out; rod falling off or abnormal database, including: a total of 4450 reflector unit back frame rods are bent, broken, and the connection between the rod and the ball head is loose. The data acquisition module organizes the image data taken by the drone. In a specific embodiment, as shown in Table 1, around the FAST drone inspection, all abnormal targets are divided into four types, namely panel depression, steel structure corrosion, screw falling off, and rod falling off.

[0050] Table 1: FAST data collection statistics

[0051]

[0052]

[0053] The abnormal labeling module is used to label abnormal targets in the collected images. The FAST UAV inspection database is established based on the sorted UAV inspection abnormal target data. The LabelImg labeling tool is used to label the data in yolo format. The label corresponding to the non-abnormal image is empty. According to the labeling category, it is divided into four databases, namely the panel abnormal dent database, steel structure corrosion database, screw shedding database and rod shedding database. The following are the detailed steps for using LabelImg labeling:

[0054] 1. Switch to the virtual environment that needs to be installed, that is, switch to the yolov5 virtual environment.

[0055] 2. Install Labelimg and enter labelimg in the command line to open it.

[0056] 3. Open the folder containing the pictures that need to be annotated.

[0057] 4. Switch to yolo mode and drag the frame to mark.

[0058] 5. Click Save; the dataset file generated by the annotation is stored in the image folder directory.

[0059] 6. The dataset file stores the information of the frame and labels, which will be automatically loaded when training the model.

[0060] The database construction module builds the corresponding database according to the abnormal classification.

[0061] According to the characteristics of the panel anomaly database, the network parameters are set as follows: 150 training rounds, training batch size (batchsize) is 8, initial learning rate is 0.01, final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping to achieve data amplification and enhance the generalization ability and robustness of the model. The specific implementation of image enhancement is as follows:

[0062] Assume that the size of the original image is W×H, the coordinates of the center point of the abnormal target are (x0, y0), and the coordinates of the center point of the abnormal target after transformation are (x'0, y'0); the width and height of the abnormal target bounding box are w0 and h0 respectively, and the width and height of the abnormal target bounding box after transformation are w'0 and h'0 respectively; the image is horizontally shifted by D x , the vertical displacement is D y The translation transformation can be expressed as:

[0063]

[0064]

[0065] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box respectively. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation respectively; D x and D y To shift the image horizontally and vertically.

[0066] Flipping the image horizontally along the horizontal center line and vertically along the vertical center line can be expressed as:

[0067]

[0068] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation; D x and D y To shift the image horizontally and vertically.

[0069] Scale the image horizontally by a factor of S x , the vertical scaling factor is S y The scaling transformation can be expressed as:

[0070]

[0071] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation; D x and D y To shift the image horizontally and vertically.

[0072] The original image is re-cropped into an image group with a width of w and a height of h, which can be expressed as:

[0073]

[0074] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box respectively. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation respectively.

[0075] After the cropping transformation, the abnormal target is located in the (i, j)th block of the new image group.

[0076] The database construction module builds the database based on the output data of the anomaly annotation module. Specifically includes:

[0077] (I) Building a panel anomaly database

[0078] There are various abnormalities such as dents and holes on the FAST panel caused by falling foreign objects such as hail and stones. To address the dent problem of the FAST panel, we constructed a dataset of abnormal panel dents using image data collected by drones.

[0079] The abnormalities of the mesh surface of the FAST experimental panel mainly include mesh surface depression. Mesh surface depression (pit halo) is caused by hail, stones, etc. Mesh holes are caused by the mesh surface being hit by small stones or hail on the basis of depression. The depth of depression (pit halo) abnormality is more than 3 mm; the size of hole abnormality is about centimeter level or more.

[0080] In view of the abnormal situation of aluminum alloy panels, the factors considered in the data collection of the present invention mainly include: background, lighting, shooting distance, and shooting angle. Background: A relatively uniform simulated background is adopted, with black as the main color. Lighting conditions: Considering the influence of weather and the shadow of the FAST Sky Eye suspended observation cabin in the actual environment, it is divided into two situations: direct lighting and no direct lighting in the daytime environment. Both situations are considered in the experimental data collection. Shooting distance: Considering the flight altitude of the drone and the feasible height for actual collection, a data collection shooting distance of 1-2 meters is adopted. It is divided into 2 categories: 1.5 meters and 2 meters. Shooting angle: Considering that the camera has a certain angle during the actual flight, and the FAST mesh panel has a certain inclination. Five shooting angles are taken during the data collection process, which are divided into vertical, horizontal directions of positive and negative 45°, and vertical directions of positive and negative 45°.

[0081] A single panel in the FAST system is an isosceles triangle with three sides of 1m, 1m, and 1.2m respectively. The layout of the FAST panel is as follows: Figure 2 As shown. In a specific embodiment, the present invention collects more than 100 abnormal data samples. After collection, 460 pictures are selected as the basic data set. Among them, the number of panel anomalies exceeds 14,000 as the basic data set.

[0082] Use the LabelImg labeling tool to label the collected panel anomaly data and build a panel anomaly database.

[0083] The distribution position and aspect ratio of the abnormal targets in the image of the panel abnormal depression data annotation samples are sorted and analyzed. The distribution position of the abnormal targets: the lower left corner of the entire image is the coordinate origin (0,0), and the coordinates of the center point of the target box represent the position of the abnormal target. The positions of the center points of all target boxes are counted to characterize the position distribution of the abnormal targets in the image, such as Figure 3 As shown in the figure, the darker the color of the center of the statistical graph, the greater the density and the more points are located at the current position. It can be seen from the statistical graph results that the depressions of the FAST panel are mostly located near the center of the entire image. For the aspect ratio of abnormal targets: the (width / height) of the target box is the aspect ratio of the abnormal target, and each point in the statistical graph represents the width and height of a target box. It can be seen from the statistical graph results that the depressions of the reflective surface unit panel are mostly small-sized targets that are difficult to detect (the point density is the highest in the lower left corner of the statistical graph). Figure 3 The figure shows the abnormal target location of the panel abnormality labeling sample. Figure 4Schematic diagram of the aspect ratio distribution of abnormal targets in the panel anomaly labeled samples.

[0084] The panel anomaly database is divided into training set, validation set and test set in a ratio of 8:1:1. The distribution of sample quantity is shown in Table 2.

[0085] Table 2: Statistics of sample quantity of panel anomaly database

[0086]

[0087]

[0088] The abnormal data of the FAST panel has the characteristics of small targets, high similarity, and large differences in different shooting angles. Based on the YOLOV5 algorithm, a method specifically used for small target detection is designed, such as mosaic enhancement of data. In addition, for the problem of large differences in different shooting angles, the laboratory team has carried out targeted solutions when designing the flight plan. When collecting data, shooting at a pitch angle of 15 degrees and minus 30 degrees has the best effect.

[0089] According to the characteristics of the panel anomaly database, the network parameters are set as follows: 150 training rounds, training batch size (batchsize) is 8, initial learning rate is 0.01, final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping. After inspection, the panel concave abnormal target, the FAST structural abnormal target detection model correctly detected 482 panel concave, with an identification accuracy of 85.6%, and an average detection time of 0.07s per image.

[0090] (II) Steel structure corrosion database

[0091] FAST is a large structure. During its operation, it needs to inspect and maintain multiple steel structures. The corrosion of the ring beam is a common problem that needs to be detected during the inspection process. In order to solve the metal corrosion problem of FAST, combined with the actual image characteristics of FAST, the steel structure corrosion images were collected.

[0092] The collected steel structure corrosion data were labeled using the LabelImg labeling tool. All corrosion data were labeled as corrosion classes, and a steel structure corrosion database was constructed. A total of 3196 corrosion anomalies were labeled.

[0093] like Figure 5 The figure shows the abnormal target location of the rust-marked sample. Figure 6 Schematic diagram of the aspect ratio distribution of abnormal targets in rust-labeled samples.

[0094] The steel structure corrosion database is divided into training set, validation set and test set in a ratio of 8:1:1. The sample quantity distribution is shown in Table 3.

[0095] Table 3: Statistics of the number of samples in the steel structure corrosion database

[0096]

[0097] (III) Screw loss database

[0098] The collected screw-off data were labeled using the LabelImg labeling tool. All screw-off data were labeled as screw-off categories, and a screw-off database was built. A total of 524 screw-off anomalies were labeled.

[0099] The distribution position and aspect ratio of the abnormal targets in the image of the screw falling data annotation samples are sorted and analyzed, such as Figure 7 The figure shows the abnormal target position of the screw falling off marking sample. Figure 8 Schematic diagram of the aspect ratio distribution of abnormal targets in the sample marked for screw falling off.

[0100] The screw-off database is divided into training set, validation set and test set in a ratio of 8:1:1. The distribution of sample quantity is shown in Table 4.

[0101] Table 4: Statistics of the number of samples in the screw-falling database

[0102]

[0103]

[0104] Due to the complex structure of the ring beam, data collection was done manually and by drone. The database has the characteristics of small targets and similar defect characteristics. Based on the YOLOV5 algorithm, a method specifically designed for screw shedding was designed, such as mosaic enhancement of data and rotation image enhancement. The following is the deep learning parameter setting for the characteristics of screw shedding data. Due to the small amount of data, the number of training rounds was appropriately increased.

[0105] According to the characteristics of the screw falling database, the network parameters are set as follows: 200 training rounds, the training batch size (batchsize) is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0106] After inspection, there are 35 verification data images used for screw detachment detection, which contain a total of 61 abnormal targets of screw detachment. The FAST structural abnormal target detection model correctly detected 51 screw detachments, with an identification accuracy rate of 83.6%. The average detection time for each image is 0.23s.

[0107] (IV) Member Falling Database

[0108] The collected bar falling data were annotated using the LabelImg labeling tool. All bar falling or fracture data were annotated as bar falling classes, and a bar falling database was constructed. A total of 85 bar falling anomalies were annotated.

[0109] The distribution position and aspect ratio of abnormal targets in the image of the labeled samples of the rod falling data are sorted and analyzed, such as Fig. 9 The figure shows the abnormal target position of the sample marked by the rod falling off. Fig.10 Schematic diagram of the aspect ratio distribution of abnormal targets in the sample marked for rod shedding.

[0110] The rod fall-off database is divided into training set, validation set and test set in a ratio of 8:1:1. The sample quantity distribution is shown in Table 5.

[0111] Table 5: Statistical table of sample quantity of rod falling database

[0112]

[0113] Since the rod fracture mainly occurs in the cable net part below the FAST panel, drones cannot fly to collect data, and it is also difficult to obtain data manually. The database has a single sample and a small sample size, so this type of defect detection requires a richer sample size support. The following is the deep learning parameter setting for the characteristics of screw shedding data. Due to the small amount of data, the training rounds are appropriately increased.

[0114] According to the characteristics of the rod fracture database, the network parameters are set as follows: 200 training rounds, the training batch size (batchsize) is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0115] After inspection, there are a total of 12 verification data images used for rod fracture detection, which contain a total of 17 abnormal targets of rod fracture. The FAST structural abnormal target detection model correctly detected 14 rod fractures, with an identification accuracy rate of 82.4%. The average detection time for each image is 0.14s.

[0116] Due to the theoretical and technical limitations of hardware equipment, the image captured by a single sensor cannot effectively and comprehensively describe the imaging scene. In the actual application of the system, the present invention fuses the significant features extracted from the source image into a single image through an appropriate fusion method, which contains richer information and is more conducive to subsequent applications. Through the bidirectional coupling of local features of different modalities, the interactive fusion of features between different modalities is realized, which effectively highlights the significant features between multi-source data and improves the detection accuracy.

[0117] Multi-source data fusion is divided into three parts: feature extraction, feature fusion, and image reconstruction. Before the popularization of deep learning, traditional fusion methods manually analyzed significant features from the source images by performing relevant mathematical transformations on the source images, designed fusion rules, and reconstructed high-level images. However, these methods have great limitations, such as poor expression of the extracted features and limited fusion performance. In order to overcome these limitations of traditional methods, deep learning was introduced to achieve differentiated feature extraction and adaptive learning by using different network branches to obtain a more reasonable image fusion strategy. Fusion methods based on deep learning can be divided into fusion methods based on convolutional neural networks (CNNs) and fusion methods based on generative adversarial networks (GANs). The CNN method that introduces convolutional neural networks into image fusion usually has two forms. One is to use a carefully designed network structure and loss function to achieve end-to-end feature extraction, feature fusion, and image reconstruction tasks. The second is to combine CNN with traditional methods and use CNN to achieve some image fusion tasks in traditional methods. The GAN method relies on the adversarial game between the generator and the discriminator, and simultaneously realizes the entire image fusion process. The above method has achieved good performance in image fusion, but the enhanced image contains more prominent target features, which will achieve good results when applied to abnormal target detection in unmanned inspection of FAST structure with multi-source data fusion.

[0118] Any process or method description in the flowchart of the present invention or described in other ways herein can be understood as a module, segment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, magnetic disks or optical disks, etc.

[0119] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without causing any contradiction.

[0120] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in the field can change, modify, replace, modify and other update operations on the above embodiments within the scope of the present invention.

Claims

1. A FAST structural damage data system based on drone intelligent inspection, characterized in that: The inspection data system includes a data acquisition module, an abnormality annotation module and a database construction module; wherein the data acquisition module is used to organize the FAST structure-related image data taken during the drone inspection process; the abnormality annotation module is used to annotate abnormal targets in the collected images; the database construction module is used to construct a corresponding database according to the abnormal classification situation, and the database includes a panel abnormal dent database, a steel structure corrosion database, a screw shedding database and a rod shedding database; wherein, The abnormal panel depression database includes depressions, pits and pit halos caused by damage to the FAST reflector unit panel, holes caused by falling objects from high altitude, and / or deformation caused by extrusion or collision of the panel; The steel structure corrosion database includes: FAST ring beam lattice column, feed tower body, main cable head, steel plank road corrosion, paint peeling and body exposure and / or red rust coverage; The screw-off database includes: the ring beam lattice column connection bolts are off, the feed tower connection bolts are off, the reflector unit connection bolts are off, the off bolt parts expose holes or the unit rods are pulled out; The rod falling-off database includes: bending, breaking of the back frame rod of the reflective surface unit and / or loose connection between the rod and the ball head.

2. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection according to claim 1 is characterized in that: When collecting the data of abnormal depression of the panel, original photos are taken at 5 shooting angles, which are: vertical, horizontal direction plus or minus 45° and vertical direction plus or minus 45°.

3. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection as claimed in claim 1, characterized in that: When collecting abnormal data on panel dents or falling objects penetration, multiple panels are spliced ​​together in the laboratory and collected simultaneously and / or reflective surface panel data is collected at the FAST site.

4. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection according to claim 1 is characterized in that: Use the LabelImg labeling tool to label the collected data and label all abnormal target data to build a database.

5. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection as claimed in claim 4 is characterized in that: The detailed steps for labeling using LabelImg are as follows: S1. Switch to the virtual environment that needs to be installed, switch to the yolov5 virtual environment; S2. Install Labelimg and enter labelimg in the command line to open it; S3. Open the folder where the pictures to be annotated are located; S4. Switch to yolo mode and drag the frame to mark; S5. Click Save, and the dataset file generated by the annotation is stored in the image folder directory; S6. The dataset file stores the information of the frame and label, which will be automatically loaded when training the model.

6. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection as claimed in claim 4, characterized in that: Use the LabelImg labeling tool to label the collected panel abnormality data, and label all panel abnormality data as a panel abnormality depression database.

7. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection as claimed in claim 4 is characterized in that: The collected steel structure corrosion data are labeled using the LabelImg labeling tool, and all corrosion data are labeled to build a steel structure corrosion database.

8. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection as claimed in claim 4, characterized in that: Use the LabelImg labeling tool to label the collected screw falling-off data, and label all the screw falling-off data to build a screw falling-off database.

9. The FAST structural damage data system based on unmanned aerial vehicle intelligent inspection as claimed in claim 4, characterized in that: The collected bar falling data were labeled using the LabelImg labeling tool, and all bar falling or fracture data were labeled to build a bar falling database.

10. The FAST structural damage data system based on UAV intelligent inspection according to any one of claims 1 to 9, characterized in that: The deep learning-based fusion method achieves differentiated feature extraction and adaptive learning by using different network branches to obtain the FAST UAV inspection image fusion strategy.

Images