FAST unmanned aerial vehicle intelligent inspection structure abnormal target identification method and system

Abstract

The invention discloses an FAST unmanned aerial vehicle intelligent inspection structure abnormal target identification method and system. Comprising the following steps: S1, acquiring a FAST structure image, and extracting feature points; s2, registering a visible light image and an infrared light image of a FAST structure abnormal target by taking a distance function as a matching criterion based on feature registration; s3, preprocessing the abnormal data; a YOLOv5x model is adopted to train the model; s4, setting a network structure of a detection algorithm based on YOLOv5; s5, infrared light and visible light fusion detection; and obtaining a detection result with a target category, a corresponding bounding box and a confidence score. According to the invention, intelligent inspection of the FAST structure is realized, and the position where a defect possibly exists is discriminated. The detection accuracy of structural abnormalities such as panel depression and metal corrosion exceeds 80%, the method is the first application of a domestic unmanned aerial vehicle intelligent inspection technology on astronomical observation equipment, and breakthrough and technical innovation of a large scientific engineering FAST structure inspection method are achieved.

Description

Technical Field

[0001] The present invention relates to a method and system for identifying abnormal targets in a FAST unmanned aerial vehicle (UAV) intelligent inspection structure. Background Art

[0002] The Five-hundred-meter Aperture Spherical radioTelescope (FAST) uses the natural karst depression in Guizhou as the telescope site to build the world's largest single-aperture radio telescope to achieve large-sky-area, high-precision astronomical observations.

[0003] FAST has a huge structure, which includes: ring beam (5,400 tons), cable net (6,670 main cables, 2,225 down cables), reflector unit (4,450 pieces), actuator anchor piers (2,225), six towers, structural foundation, dangerous rock slopes, etc. Inspection and maintenance are required during the operation of FAST. The inspection contents include: panel subunits, back frames, spherical bearings, node plates, targets and target seats, main cables and down cables, bolts and pins, sliding layers and restricting doors, etc. After more than five years of operation, the following common problems were found: panel deformation, rod instability; ring beam rust, target failure, down cable sheath damage and replacement, main cable sheath repair, main cable pin status, motion interference, tower body corrosion and connection bolt falling off, etc.

[0004] The node position structure in the reflective surface is complex, with multiple reflective surface unit back frames connected to the node upper disk, and their motion pairs are also different. It is easy to have problems such as rust, deformation of spare rods, and motion interference. Although there is a cable crane system available for maintenance, it has the disadvantages of low efficiency, high cost, and small working area for reflective surface inspection, and is not suitable for large-scale and efficient inspection work.

[0005] There are six feed towers with a tower body length of 120 to 170 meters. They are built according to the terrain and have a complex structure. Regarding the problems of tower body rust and connecting bolts falling off, manual inspections have disadvantages such as low efficiency, high risks, and being restricted by many conditions.

[0006] Currently, structural inspections are mainly carried out through manual inspection methods, which have problems such as low efficiency, no video acquisition, safety hazards, and great climate impact.

[0007] In general, the current inspection of reflectors and feed towers has the following problems:

[0008] (1) Most current inspection methods are traditional methods such as cable hanging and manual climbing, without video recording;

[0009] (2) Inspections are difficult in complex climates, such as rain, snow, severe cold, freezing weather, etc.

[0010] (3) More than half of the components are located at high places with poor accessibility, complex terrain, low inspection efficiency and high inspection risk;

[0011] (4) During inspections, only simple notes can be taken, which are difficult to review and read. Summary of the invention

[0012] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method for identifying abnormal structural targets during intelligent inspection of FAST UAVs, which effectively assists staff in carrying out structural inspections and effectively improves the detection accuracy of structural abnormalities such as panel dents and metal corrosion under reasonable data collection conditions. Another purpose of the present invention is to provide a FAST UAV intelligent inspection abnormal structural target identification system that implements the above method.

[0013] To achieve the above object, the FAST UAV intelligent inspection structure abnormal target recognition method of the present invention comprises the following steps:

[0014] S1: Acquire a FAST structural image and extract feature points using a feature point extraction algorithm; the image includes a visible light image and an infrared light image;

[0015] S2: Feature-based registration uses the distance function as the matching criterion to register the visible light image and infrared image of the FAST structural anomaly target;

[0016] S3: Abnormal data preprocessing; data augmentation method based on geometric transformation to increase the number of training samples of some defect categories; YOLOv5x model is used to train it;

[0017] S4: Setting the network structure of the detection algorithm based on YOLOv5, including feature extraction network module, feature enhancement network module and detection network module;

[0018] S5: Infrared light and visible light fusion detection; The first feature extraction module extracts visible light and infrared depth features respectively; Paired multi-source data features are input into the Gaussian transformer encoder network, and a feature map containing significant local information and global distribution information of multi-source targets is output; The enhanced feature map is transmitted to the classification network to obtain the detection result with the target category, as well as the corresponding bounding box and confidence score

[0019] Furthermore, the distance function in S1 adopts a coarse-to-fine registration process: first, coarse registration, comparing the similarity of features through the distance criterion, and then finding the corresponding features to establish a rough set of matching point pairs; second, fine registration, using the RANSAC algorithm to screen the set of matching point pairs to obtain a more accurate set of matching point pairs.

[0020] Furthermore, the data augmentation method based on geometric transformation includes achieving data augmentation by performing geometric transformations such as bounding box translation, image flipping, image scaling, image rotation, and image cropping on existing training image samples.

[0021] Furthermore, the size of the original image is W×H, the coordinates of the center point of the abnormal target are (x0, y0), and the coordinates of the center point of the abnormal target after transformation are (x'0, y'0); the width and height of the abnormal target bounding box are w0 and h0 respectively, and the width and height of the abnormal target bounding box after transformation are w'0 and h'0 respectively; the image is horizontally shifted by D x , the vertical displacement is D y The translation transformation is expressed as:

[0022]

[0023]

[0024] The image is flipped horizontally along the horizontal center line and vertically along the vertical center line, expressed as:

[0025]

[0026]

[0027]

[0028] Scale the image horizontally by a factor of S x , the vertical scaling factor is S y The scaling transformation is expressed as:

[0029]

[0030]

[0031] The original image is re-cropped into an image group with a width of w and a height of h, expressed as:

[0032]

[0033]

[0034]

[0035] After the cropping transformation, the abnormal target is located in the (i, j)th block of the new image group.

[0036] Furthermore, the YOLOv5x model includes:

[0037] The second feature extraction module: aggregates at different image granularities and forms a convolutional neural network of image features;

[0038] Feature enhancement module: passes image features to the prediction layer to generate a feature pyramid;

[0039] Detection module: predicts the category and location of the object based on the image features, by applying anchor boxes on the feature map, and generates a final output vector with class probability, object score and bounding box.

[0040] Furthermore, the loss function is set to adjust the convergence direction of the YOLOv5x model, and GIoULoss is used as supervision during the training process.

[0041] Furthermore, the first feature extraction module extracts features from visible light and infrared images respectively by means of parameter sharing; the Gaussian transformer encoder network is based on a Gaussian cross attention mechanism module to fuse the significant depth features of the target in the infrared and visible light features.

[0042] Furthermore, the classification and positioning network includes a classification branch and a regression branch composed of two parallel branches, two blocks are set on the classification branch, and four blocks are set on the regression branch, each block includes a convolution layer, a batch normalization layer and a ReLU layer.

[0043] Furthermore, the loss function is minimized to train the detection algorithm GCANet, and the total loss L is as follows:

[0044] L=L cls +λL reg

[0045] L consists of a weighted combination of classification loss and bounding box loss, where the classification loss is Focal loss and the bounding box loss is GIoU Loss.

[0046] A FAST UAV intelligent inspection structure abnormal target recognition system, the system is used to implement the above-mentioned FAST UAV intelligent inspection structure abnormal target recognition method.

[0047] The FAST UAV intelligent inspection structural abnormality target recognition method and system of the present invention realizes the intelligent inspection of the FAST structure and identifies the location where defects may exist. The detection accuracy of structural abnormalities such as panel dents and metal corrosion exceeds 80%. This is the first application of domestic UAV intelligent inspection technology on astronomical observation equipment, achieving a breakthrough and technological innovation in the inspection method of the FAST structure of the large scientific project. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the implementation of Gaussian pyramid and DoG;

[0049] Figure 2 This is a schematic diagram of extreme point detection;

[0050] Figure 3 This is a comparison of the FAST structural abnormality target visible light image before and after registration;

[0051] Figure 4 This is a comparison of the infrared images of FAST structural anomaly targets before and after registration;

[0052] Figure 5 This is a schematic diagram of the GCANet network overview;

[0053] Figure 6 This is a schematic diagram of the Gaussian transformer encoder network. DETAILED DESCRIPTION

[0054] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0056] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0058] like Figures 1 to 6As shown, the present invention provides a method and system for identifying structural abnormality targets for intelligent inspection of FAST unmanned aerial vehicles. Before detecting FAST structural abnormality targets based on multi-source fusion, since the collected visible light image and the corresponding infrared image do not completely match, it is necessary to first conduct a registration study on the dual-light image. The feature-based registration method focuses on obtaining relatively stable features in the image, such as corners, edges, etc., and then uses these feature information for registration. Because processing stable features in an image is more in line with people's habits than grayscale statistical characteristics. Since the present invention only processes the extracted feature information, and does not put all the grayscale information of the image into the calculation of the registration parameters, the amount of calculation is reduced and the registration speed is accelerated.

[0059] The present invention provides a method for identifying abnormal targets in a FAST unmanned aerial vehicle intelligent inspection structure, and the process is as follows:

[0060] S1: Acquire a FAST structural image and extract feature points using a feature point extraction algorithm; the image includes a visible light image and an infrared light image.

[0061] SIFT (Scale invariant feature transform) is a scale and rotation invariant feature extraction algorithm. SIFT spot extraction mainly consists of the following three steps:

[0062] S1.1 Construction of scale space

[0063] The Gaussian convolution kernel is a linear kernel that can achieve changes in image size. The original image is transformed by the Gaussian convolution kernel function to obtain a series of images of different scales. These images constitute the scale space of the original image. The larger the scale of the image, the more blurred it looks. For an image L(x,y), its scale space L(x,y,σ) is defined as:

[0064]

[0065]

[0066] in, represents convolution, σ represents scale coordinates, and G(x,y,σ) is a Gaussian convolution kernel with variable scale. σ determines the smoothness of the image. The larger the value, the larger the image scale and the lower the image clarity. π is 3.14. L(x,y) represents the image. L(x,y,σ) represents the scale space.

[0067] The Difference of Gaussians (DoG) scale space is constructed using the scale space, and the feature points detected in this space are more stable.

[0068]

[0069] in, represents convolution, σ represents scale coordinates, G(x,y,σ) is a Gaussian convolution kernel with variable scale. L(x,y,σ) represents the scale space. π is 3.14.

[0070] DoG is an approximation of Laplace of Gaussian (LoG). Compared with LoG, DoG only needs to calculate the difference between images after Gaussian smoothing at adjacent scales, which reduces the difficulty and complexity of calculation. Figure 1 As shown. First, construct a Gaussian pyramid, downsample the original image to get each layer of images, and form a pyramid model from large to small. Then, in order to make the scale of the image continuous, Gaussian blur each layer of images at different scales to get a group of images. In addition, downsample the third to last image in this group of images, and the resulting image is used as the image of the previous layer of the pyramid. For example, Figure 1 After Gaussian blurring of the downsampled images at each layer, a group of five images is obtained. Finally, the adjacent two layers of each group of images are subtracted to obtain the Gaussian difference image (DoG).

[0071] S1.2 Feature point search and positioning

[0072] Compare the adjacent layer images of each DoG in the same group to search for feature points. Feature points are composed of extreme values ​​in the DoG scale space, so the local extreme values ​​in the DoG scale space are searched first.

[0073] For each pixel in the DoG image (except the bottom and top layers), it is necessary to compare it with the 8 adjacent pixels in the same layer and the 18 adjacent pixels in the upper and lower scales to see whether it is a maximum or minimum value. If the DoG response value of the point is larger or smaller than these 26 points, then the point is a local extreme point. Figure 2 shown.

[0074] S1.3 Allocation of main directions of feature points

[0075] In order to make SIFT feature points robust to image rotation, the main direction of the feature points is given by calculating the gradient distribution of the pixels in the neighborhood of the feature points. The gradient of the pixel point can be expressed as:

[0076]

[0077] Among them, the gradient amplitude m and direction θ are expressed as:

[0078]

[0079] L(x,y) represents the image. m(x,y) represents the gradient magnitude.

[0080] S2: Feature-based registration uses the distance function as the matching criterion to register the visible light image and infrared image of the FAST structural anomaly target.

[0081] Feature-based registration uses the distance function as a matching criterion to calculate and find corresponding features. Distance criteria include Euclidean distance, Mahalanobis distance, and Hausdorff distance. In order to improve the accuracy of registration, a coarse-to-fine registration process is adopted: first, coarse registration, using the distance criterion to compare the similarity of features, and then find the corresponding features to establish a rough set of matching point pairs; second, fine registration, using related algorithms to filter the set of matching point pairs to obtain a more accurate set of matching point pairs. The present invention uses the RANSAC algorithm for fine registration. Comparison chart of FAST structural abnormality target visible light image before and after registration, as shown in the figure. Figure 3 As shown in the figure; FAST structural abnormality target infrared image comparison before and after registration, as shown in the figure Figure 4 As shown,

[0082] S3: Abnormal data preprocessing; data augmentation method based on geometric transformation to increase the number of training samples of some defect categories; and training it using the YOLOv5x model.

[0083] In the FAST structural anomaly data detection based on deep learning, since most of the abnormal samples are small-sized targets and the number of samples of different types is not balanced, it is difficult to obtain a large number of FAST structural anomaly data image samples with real annotations for model training for some types of abnormal targets. In addition, when actually collecting FAST structural anomaly images, the number of specific types of abnormal targets is much smaller than the number of other types of defects. Therefore, in order to solve the problem of insufficient number of training samples and lack of diversity, a data augmentation method based on geometric transformation is used to expand the number of training samples of some defect categories to improve the detection accuracy of FAST structural anomaly targets.

[0084] The data augmentation method based on geometric transformation and pixel transformation realizes data augmentation by performing geometric transformations such as bounding box translation, image flipping, image scaling, image rotation, and image cropping on existing training image samples, thereby enhancing the generalization ability and robustness of the model.

[0085] Assume that the size of the original image is W×H, the coordinates of the center point of the abnormal target are (x0, y0), and the coordinates of the center point of the abnormal target after transformation are (x'0, y'0); the width and height of the abnormal target bounding box are w0 and h0 respectively, and the width and height of the abnormal target bounding box after transformation are w'0 and h'0 respectively; the image is horizontally shifted by D x , the vertical displacement is D y The translation transformation can be expressed as:

[0086]

[0087]

[0088] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box respectively. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation respectively; D x and D y To shift the image horizontally and vertically.

[0089] Flipping the image horizontally along the horizontal center line and vertically along the vertical center line can be expressed as:

[0090]

[0091]

[0092]

[0093] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation; D x and D y To shift the image horizontally and vertically.

[0094] Scale the image horizontally by a factor of S x , the vertical scaling factor is S y The scaling transformation can be expressed as:

[0095]

[0096]

[0097] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation; D x and D y To shift the image horizontally and vertically.

[0098] The original image is re-cropped into an image group with a width of w and a height of h, which can be expressed as:

[0099]

[0100]

[0101]

[0102] (x0, y0) are the coordinates of the center point of the abnormal target. (x'0, y'0) are the coordinates of the center point of the abnormal target after transformation; w0 and h0 are the width and height of the abnormal target bounding box respectively. w'0 and h'0 are the width and height of the abnormal target bounding box after transformation respectively.

[0103] After the cropping transformation, the abnormal target is located in the (i, j)th block of the new image group.

[0104] S4, the network structure of the detection algorithm is set based on YOLOv5. The present invention uses the YOLOv5x model for training. The YOLOv5 model is mainly composed of three main components.

[0105] (I) Feature extraction network module

[0106] By aggregating different images at fine-grained levels and forming a convolutional neural network of image features, it is used as the second feature extraction module of YOLOv5. It solves the problem of repeated network optimization gradient information in the second feature extraction module of other large convolutional neural network frameworks, and integrates the gradient changes from beginning to end into the feature map, thereby reducing the number of model parameters, ensuring both the speed and accuracy of inference and reducing the model size. By combining aggregated pixel focused attention and convolution to simulate biological visual systems, especially for the visual perception of the fovea. This method enables each pixel to achieve global perception and strengthens the information mixing and natural visual perception capabilities of the model.

[0107] (II) Feature Enhancement Network Module

[0108] Improve the performance of the model by enhancing the feature representation. Traditional feature processing methods often cannot handle the differences between different samples well, resulting in poor performance of the model when faced with new data. The feature enhancement module enables the model to better adapt to various scenarios by generating personalized feature representations for each sample. The feature enhancement module passes image features to the prediction layer through a series of designed network layers. The feature enhancement module is mainly used to generate feature pyramids to enhance the model's detection of targets of different scales, so that it can detect the same target of different sizes and scales, significantly improving the model's expressiveness and robustness.

[0109] (III) Detection network module

[0110] The detection module predicts the category and location of the target based on the image features. The detection module is mainly used for the final detection part, which improves the accuracy and efficiency of the system by applying anchor boxes on the feature map and generating the final output vector with class probability, target score and bounding box.

[0111] (IV) Loss function setting

[0112] GIoU Loss (Generalized Intersection over Union Loss) is a loss function used for target detection and segmentation tasks, which aims to improve the shortcomings of traditional IoU Loss (Intersection over Union Loss). For the FAST structural abnormality target detection task, it is necessary to set the loss function to adjust the model convergence direction. Since there are many small-sized targets in the detection task, GIoU Loss is used as supervision during the training process. Assume that the coordinates of the bounding box of the predicted target and the annotation box of the actual target are and in

[0113] B P are the bounding box coordinates of the predicted target, B g are the coordinates of the annotation box of the actual target. (x1,y1) and (x2,y2) are the coordinate points respectively.

[0114] The specific process of calculating GIoU Loss is as follows:

[0115] (1) Calculate the area of ​​the actual target annotation box and the predicted target bounding box as shown in Formula 17 and Formula 18, where A g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box.

[0116]

[0117]

[0118] A g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box. (x1,y1) and (x2,y2) are the coordinate points respectively.

[0119] (2) Calculate the overlapping area I between the predicted target bounding box and the actual target annotation box:

[0120]

[0121]

[0122]

[0123] I is the overlap area between the predicted target bounding box and the actual target annotation box. (x1, y1) and (x2, y2) are the coordinate points of the target bounding box and the actual target annotation box, respectively.

[0124] (3) Find a bounding box B that can contain the predicted target P and the actual target annotation box B g The minimum bounding box B c , and calculate B c Area A c :

[0125]

[0126]

[0127]

[0128] B P is the predicted target bounding box. B g is the actual target annotation box. c is the minimum bounding box, with area denoted by A c express.

[0129] (4) Calculate the intersection over union (IoU) of the predicted target bounding box and the actual target annotation box:

[0130]

[0131] IoU stands for Intersection over Union. g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box.

[0132] (5) Calculate the global intersection-over-union (GIoU) ​​of the predicted target bounding box and the actual target annotation box:

[0133]

[0134] GIoU stands for Global Intersection over Union. g is the area of ​​the actual target annotation box, A p is the area of ​​the predicted target bounding box.

[0135] (6) Calculate the loss function L of GIoU Loss GIoU :

[0136] L GIoU =1-GIoU, (27)

[0137] IoU stands for Intersection over Union. GIoU stands for Global Intersection over Union.

[0138] 5. Infrared light and visible light fusion detection.

[0139] To address the problem of multi-source fusion, the present invention adopts a cross-modal multi-source fusion detection algorithm GCANet based on a cross-attention mechanism.

[0140] The network architecture of GCANet according to the present invention is as follows Figure 5 As shown in the figure. The network structure is based on three consecutive modules, which are respectively associated with feature extraction, multi-source data feature fusion and FAST structural abnormal target detection tasks. It takes infrared-visible light image pairs as input. The first module is the feature extraction module, which consists of a deep convolutional network, trained in a parameter-sharing manner, and extracts visible light and near-infrared deep features respectively. The second module is a Gaussian transformer encoder network based on the attention module, which takes paired multi-source data features as input, enhances and fuses multi-source salient features based on a learnable two-dimensional Gaussian cross attention mechanism, and outputs high-quality feature maps containing salient local information and global distribution information of multi-source targets. The enhanced feature map is transmitted to the last classification and localization network to obtain the detection result with the target category, as well as the corresponding bounding box and confidence score.

[0141] GCANet specifically includes:

[0142] 1) The first feature extraction module:

[0143] The purpose of the first feature extraction module is to extract features from visible light and infrared images respectively by means of parameter sharing. The ResNet50 network is used as the feature extraction module. The feature point extraction principle of ResNet50 is based on a deep residual network. In order to solve the problem of gradient disappearance or explosion caused by the increase of network depth, the residual learning method is introduced, and the residual block is constructed by introducing jump connections. The jump connection directly transfers the input to the output, so that the neural network can input more accurate modeling by learning the residual.

[0144] 2) Gaussian transformer encoder network:

[0145] The Gaussian transformer encoder network is a transformer model that enhances the expression of target salient features by fusing deep features. In order to learn multi-source fusion feature expression with lightweight computing and memory capabilities, the present invention introduces a Gaussian cross attention mechanism module to fuse the target salient deep features in infrared and visible light features. The schematic diagram of the Gaussian transformer encoder network is shown in Figure 6 shown.

[0146] 3) Classification and positioning network

[0147] For the classification and localization network, it includes a network consisting of two parallel branches, namely the classification branch and the regression branch, and different convolution blocks are set in the two branches. There are two blocks on the classification branch and four blocks on the regression branch. Each block contains a convolution layer, a batch normalization layer, and a ReLU layer. In addition, an implicit prediction of the object probability of each regression branch is added, and the final classification result will be multiplied by this implicit prediction.

[0148] 4) Loss Function

[0149] The detection algorithm GCANet is trained through the network to capture local and long-range dependencies, generate clearer content, and retain most of the visual information from multi-source data features. In order to more accurately detect FAST structural anomaly targets, the loss function is minimized to train the detection algorithm GCANet. The total loss L is as follows:

[0150] L=L cls +λL reg , (28)

[0151] L consists of a weighted combination of classification loss and bounding box loss, where the classification loss is Focal loss and the bounding box loss is GIoU Loss.

[0152] S6. Experimental verification of FAST structural abnormal target detection

[0153] In addition, the present invention can also set up a FAST structural abnormal target detection experimental verification step. For abnormal inspection detection evaluation, refer to the actual sample abnormalities, select appropriate image samples, use recognition accuracy and running time as the main evaluation indicators, and experimentally verify the effectiveness of the proposed FAST structural abnormal target detection method for four types of abnormal targets, namely, panel abnormality, metal corrosion, screw loss, and rod breakage, based on the panel abnormality dent database, steel structure corrosion database, screw loss database, and rod loss database. Specifically including:

[0154] (I) FAST structural abnormal target detection evaluation indicators

[0155] In order to objectively evaluate the detection performance of the model, two common evaluation indicators, recognition accuracy and single-frame detection time, are used as evaluation indicators for FAST structural abnormality target detection.

[0156] 1) Recognition accuracy

[0157] The recognition accuracy rate indicates the proportion of correctly identified abnormal targets of a certain type to all actually labeled abnormal targets of a certain building. Table 1 shows all the possibilities in the detection results, where the correct predictions are positive examples, the wrong predictions are negative examples, TP (true positive examples) represents the number of samples that are both predicted and actually abnormal targets of this type, FP (false positive examples) represents the number of samples that are predicted to be abnormal targets of this type but are not actually abnormal targets of this type, TN (true negative examples) represents the number of samples that are neither predicted nor actually abnormal targets of this type, and FN (false negative examples) represents the number of samples that are not predicted to be abnormal targets of this type but are actually abnormal targets of this type.

[0158] Table 1: Possible classification of test results

[0159]

[0160] The formula for calculating the recognition accuracy is as follows:

[0161]

[0162] 2) Single frame detection time

[0163] Single-frame detection time is an important indicator for evaluating the detection speed of the model. It indicates the time required for the model to detect each frame of the image. The shorter the detection time, the faster the detection speed of the model. The calculation formula for single-frame detection time is as follows:

[0164]

[0165] (II) Analysis of FAST abnormal target detection results

[0166] For abnormal inspection detection evaluation, appropriate image samples are selected with reference to actual sample anomalies, and recognition accuracy and running time are used as the main evaluation indicators. Based on the panel abnormality dent database, steel structure corrosion database, screw falling database and rod falling database, the effectiveness of the proposed FAST structural abnormal target detection method is experimentally verified for four types of abnormal targets: panel abnormality, metal corrosion, screw falling and rod fracture.

[0167] 1) Panel abnormal dent database detection results

[0168] The abnormal data of the FAST panel has the characteristics of small targets, high similarity, and large differences in different shooting angles. Based on the YOLOV5 algorithm, a method specifically used for small target detection is designed, such as mosaic enhancement of data. In addition, for the problem of large differences in different shooting angles, a targeted solution was made when designing the flight plan. When collecting data, shooting at a pitch angle of 15 degrees and minus 30 degrees has the best effect.

[0169] According to the characteristics of the panel anomaly database, the network parameters are set as follows: 150 training rounds, the training batch size is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0170] The detection results of the FAST structural abnormal target detection algorithm on the panel abnormal depression database are shown in Table 2.

[0171] Table 2: Detection results of abnormal panel dent database

[0172]

[0173]

[0174] There are a total of 100 verification data images used for panel dent abnormal target detection, which contain a total of 563 panel dent abnormal targets. The FAST structural abnormal target detection model correctly detected 482 panel dents, with a recognition accuracy of 85.6%. The average detection time for each image is 0.07s.

[0175] 2) Steel structure corrosion database detection results

[0176] Due to the complex structure of the ring beam, the database has the characteristics of small targets and similar defect characteristics. Based on the YOLOV5 algorithm, a method specifically designed for metal corrosion detection is designed, such as mosaic enhancement of data and flip image enhancement.

[0177] According to the characteristics of the metal corrosion database, the network parameters are set as follows: 150 training rounds, the training batch size (batchsize) is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0178] The detection results of the FAST structural abnormal target detection algorithm on the steel structure corrosion database are shown in Table 3.

[0179] Table 3: Detection results of steel structure corrosion database

[0180]

[0181]

[0182] There are 50 verification data images used for steel structure corrosion detection, which contain 84 abnormal steel structure corrosion targets. The FAST structural abnormal target detection model correctly detected 70 steel structure corrosions, with an identification accuracy rate of 83.3%. The average detection time for each image is 0.23s.

[0183] 3) Screw loss database detection results

[0184] Due to the complex structure of the ring beam, data collection was done manually and by drone. The database has the characteristics of small targets and similar defect characteristics. Based on the YOLOV5 algorithm, a method specifically designed for screw shedding was designed, such as mosaic enhancement of data and rotation image enhancement. The following is the deep learning parameter setting for the characteristics of screw shedding data. Due to the small amount of data, the number of training rounds was appropriately increased.

[0185] According to the characteristics of the screw falling database, the network parameters are set as follows: 200 training rounds, the training batch size (batchsize) is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0186] The detection results of the FAST structural abnormal target detection algorithm on the screw falling-off database are shown in Table 4.

[0187] Table 4: Detection results of screw-off database

[0188]

[0189] There are 35 verification data images used for screw detachment detection, which contain 61 abnormal screw detachment targets. The FAST structural abnormal target detection model correctly detected 51 screw detachments, with an identification accuracy of 83.6%. The average detection time for each image is 0.23 seconds.

[0190] 4) Detection results of rod fracture database

[0191] Since the rod fracture mainly occurs in the cable net part below the FAST panel, the database sample is single and the sample size is small, so this type of defect detection requires more abundant sample size support. The following is the deep learning parameter setting for the characteristics of screw falling data. Due to the small amount of data, the training rounds are appropriately increased.

[0192] According to the characteristics of the rod fracture database, the network parameters are set as follows: 200 training rounds, the training batch size (batchsize) is 8, the initial learning rate is 0.01, the final learning rate is 0.001, the training process uses the SGD (Stochastic gradient descent) optimization algorithm to iteratively update the weights, the momentum of the model training is 0.9, the IoU training threshold is 0.2, and the image enhancement uses mosaic enhancement, image HSV-hue enhancement, image HSV-saturation enhancement, image HSV value enhancement, image translation, image scaling, and image flipping.

[0193] The detection results of the FAST structural abnormal target detection algorithm on the bar fracture database are shown in Table 5.

[0194] Table 5: Detection results of the bar fracture database

[0195]

[0196] There are a total of 12 verification data images used for rod fracture detection, which contain a total of 17 abnormal targets of rod fracture. The FAST structural abnormal target detection model correctly detected 14 rod fractures, with an identification accuracy rate of 82.4%. The average detection time for each image is 0.14 seconds.

[0197] Any process or method description in the flowchart of the present invention or described in other ways herein can be understood as a module, segment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, magnetic disks or optical disks, etc.

[0198] In the description of this specification, the description with reference to the terms "embodiment", "example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art can combine or combine different embodiments or examples described in this specification and the features therein without causing any contradiction.

[0199] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in the field can change, modify, replace, modify and other update operations on the above embodiments within the scope of the present invention.

Claims

1. The FAST UAV intelligent inspection structure abnormal target recognition method is characterized by: The method comprises the following steps: S1: Acquire a FAST structural image and extract feature points using a feature point extraction algorithm; the image includes a visible light image and an infrared light image; S2: Feature-based registration uses the distance function as the matching criterion to register the visible light image and infrared image of the FAST structural anomaly target; S3: Abnormal data preprocessing; data augmentation method based on geometric transformation to increase the number of training samples of some defect categories; YOLOv5x model is used to train it; S4: Setting the network structure of the detection algorithm based on YOLOv5, including feature extraction network module, feature enhancement network module and detection network module; S5: Infrared light and visible light fusion detection; the first feature extraction module extracts visible light and infrared depth features respectively; paired multi-source data features are input into the Gaussian transformer encoder network, and a feature map containing significant local information and global distribution information of multi-source targets is output; the enhanced feature map is transmitted to the classification network to obtain the detection result with the target category, as well as the corresponding bounding box and confidence score.

2. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 1 is characterized in that: The distance function in step S2 adopts a coarse-to-fine registration process: first, coarse registration, comparing the similarity of features through the distance criterion, and then finding the corresponding features to establish a rough set of matching point pairs; second, fine registration, using the RANSAC algorithm to screen the set of matching point pairs to obtain a more accurate set of matching point pairs.

3. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 1 is characterized in that: In step S3, the data augmentation method based on geometric transformation includes implementing data augmentation by performing geometric transformations such as bounding box translation, image flipping, image scaling, image rotation, and / or image cropping on existing training image samples.

4. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 3 is characterized in that: The size of the original image is W×H, the coordinates of the center point of the abnormal target are (x0, y0), and the coordinates of the center point of the abnormal target after transformation are (x'0, y'0); the width and height of the abnormal target bounding box are w0 and h0 respectively, and the width and height of the abnormal target bounding box after transformation are w'0 and h'0 respectively; the image is horizontally shifted by D x , the vertical displacement is D y The translation transformation is expressed as: The image is flipped horizontally along the horizontal center line and vertically along the vertical center line, expressed as: Scale the image horizontally by a factor of S x , the vertical scaling factor is S y The scaling transformation is expressed as: The original image is re-cropped into an image group with a width of w and a height of h, expressed as: After the cropping transformation, the abnormal target is located in the (i, j)th block of the new image group.

5. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 1 is characterized in that: The YOLOv5x model includes: The second feature extraction module: aggregates at different image granularities and forms a convolutional neural network of image features; Feature enhancement module: passes image features to the prediction layer to generate a feature pyramid; Detection module: predicts the category and location of the object based on the image features, by applying anchor boxes on the feature map, and generates a final output vector with class probability, object score and bounding box.

6. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 5 is characterized in that: The loss function is set to adjust the convergence direction of the YOLOv5x model, and GIoULoss is used as supervision during the training process.

7. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 1 is characterized in that: The first feature extraction module extracts features from visible light and infrared images respectively by means of parameter sharing; the Gaussian transformer encoder network is based on a Gaussian cross attention mechanism module to fuse the significant depth features of the target in the infrared and visible light features.

8. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 1 is characterized in that: The classification and positioning network includes a classification branch and a regression branch composed of two parallel branches. Two blocks are arranged on the classification branch, and four blocks are arranged on the regression branch. Each block includes a convolution layer, a batch normalization layer and a ReLU layer.

9. The FAST UAV intelligent inspection structure abnormal target recognition method according to claim 1 is characterized in that: The loss function is minimized to train the detection algorithm GCANet, and the total loss L is as follows: L=L cls +λL reg L consists of a weighted combination of classification loss and bounding box loss, where the classification loss is Focalloss and the bounding box loss is GIoU Loss.

10. A FAST UAV intelligent inspection structure abnormal target recognition system, characterized in that: The system is used to implement the FAST UAV intelligent inspection structure abnormal target identification method according to any one of claims 1-9.

Images