An unmanned aerial vehicle image recognition method based on a neural network model

Abstract

This invention relates to a UAV image recognition method based on a neural network model. First, it acquires the original crack image captured by the UAV. Then, it uses an improved YOLO deep convolutional neural network to recognize the crack in the original image. The improved YOLO deep convolutional neural network's loss function is a weighted sum of four metrics: Classification Loss, Objectness Loss, Bounding Box Regression Loss, and Offset Angle Loss, to obtain the crack location in the real-time crack image. A crack post-processing algorithm is then used to post-process the crack location in the real-time crack image, resulting in a crack segmentation image and crack measurement results. Finally, a crack mapping algorithm converts the crack measurement results into the actual crack size, using elliptical shapes instead of traditional rectangular boxes to label the cracks. Five-dimensional parameters are used to accurately quantify the morphological features, providing a quantitative description of the crack morphology and improving recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method for UAV image recognition based on a neural network model. Background Technology

[0002] Image recognition-based inspections primarily utilize image acquisition equipment to obtain the appearance of dams, corridors, spillway facilities, slopes, and river channels. By employing image recognition technology, warnings are issued for abnormal conditions such as water accumulation and drainage in dam foundation corridors, cracks and seepage on the downstream side of the dam body, damage to the dam surface deformation observation prism, the condition of the upstream and downstream bank slopes of the dam, and floating objects on the reservoir surface. The aim is to replace or reduce manual inspection work by conducting inspections of major equipment and facilities or some important parts.

[0003] Cracks on the surface of hydraulic structures, especially dams, are a significant factor affecting their safe operation. The causes of these cracks are complex: some are due to loads exceeding the ultimate bearing capacity, leading to insufficient tensile strength; others are caused by excessive structural deformation during operation (such as temperature deformation cracks, uneven settlement cracks, etc.); still others are caused by alkali-aggregate reaction in concrete or corrosion of the reinforcing steel. Cracks can severely impact the integrity and safety of the structure, and may even lead to structural failure and disasters. Monitoring surface cracks on dams and other hydraulic structures is essential for timely detection of problems and prevention of accidents. Therefore, research on crack monitoring methods based on image recognition technology is of great significance, as it not only enables long-term continuous monitoring and safety analysis but also provides crucial data support for dam maintenance and performance analysis.

[0004] According to Chinese Invention Announcement No. CN119580105B, a method and system for identifying bridge cracks are proposed. The invention describes that "the invention first acquires image data of the target bridge, preprocesses the image data to obtain a processed image, performs foreground segmentation on the processed image to obtain a segmented image, then extracts a first feature from the segmented image to obtain a first feature image, then extracts a second feature from the segmented image to obtain a second feature image, then fuses the first feature image and the second feature image to obtain a fused image, finally acquires a bridge training image, inputs the bridge training image into a preset crack identification model for training, and inputs the fused image into the trained preset crack identification model for crack identification to obtain the crack identification result. The invention first segments the image, which can improve the accuracy of foreground segmentation, and then performs feature extraction and fusion on the image, which can effectively avoid noise interference on the image, with low distortion and effective preservation of detailed information in the image, thereby improving the accuracy of subsequent model crack identification."

[0005] Although the bridge crack identification method and system improves image segmentation accuracy and suppresses noise interference by using foreground segmentation, dual feature extraction and fusion, and pre-set model training, and retains detailed information to improve crack identification accuracy, it does not involve morphological annotation optimization, dedicated post-processing procedures, and physical size mapping, which affects the identification accuracy. Therefore, a UAV image recognition method based on a neural network model is proposed to solve the above problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a UAV image recognition method based on a neural network model, which has advantages such as high recognition accuracy. It solves the problem that although image segmentation accuracy is improved and noise interference is suppressed by foreground segmentation, dual feature extraction and fusion, and preset model training, and detailed information is preserved to improve the accuracy of crack recognition, the method does not involve morphological annotation optimization, dedicated post-processing procedures, and physical mapping of dimensions, which affects the recognition accuracy.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for UAV image recognition based on a neural network model, comprising the following specific steps: S1. Obtain the raw crack image captured by the drone; S2. The original crack image is identified based on the improved YOLO deep convolutional neural network. The loss function of the improved YOLO deep convolutional neural network is a weighted sum of four measurement functions: Classification Loss, Objectness Loss, Bounding Box Regression Loss, and Offset Angle Loss, which is used to obtain the crack location in the real-time crack image. S3. Based on the crack post-processing algorithm, the crack location of the real-time crack image is post-processed to obtain the crack segmentation effect map and crack measurement results; S4. Based on the crack mapping algorithm, the crack measurement results are converted into the actual crack size.

[0008] Furthermore, in step S1, the drone is equipped with a high-definition camera to take fixed-point photos of the dam to obtain original images of the dam surface, including original normal images and original crack images.

[0009] Furthermore, in the improved YOLO deep convolutional neural network described in step S2, the Classification Loss adopts the BCEcls loss (binary cross-entropy loss), and the expression for the BCEcls loss is: In the formula, For prediction, and All are weighted.

[0010] Furthermore, in step S2, the improved YOLO deep convolutional neural network uses the BECLogits loss function for Objectness Loss.

[0011] Furthermore, in step S2, the improved YOLO deep convolutional neural network's Bounding Box Regression Loss considers three important geometric factors: overlap area, center point distance, and length and width. Compared to object imaging in ImageNet, crack imaging is distributed in the form of thin, elongated lines. To address this characteristic, the original algorithm's box annotations were replaced with elliptical shapes that more closely resemble crack lines. Based on the general equation of an ellipse: In the formula, Let these be the coordinates of the center of the ellipse. For the length of the major axis, For the minor axis length, For the deflection angle, Let be the coordinates of any point on the ellipse. Based on the general equation of the ellipse, the improved loss function is set as follows: In the formula, Let these be the coordinates of the center of the ellipse. For the length of the major axis, For the minor axis length, All values ​​are after Sigmoid transformation. , , All are weighted coefficients. , , , .

[0012] Furthermore, the expression for the offset angle loss function in the improved YOLO deep convolutional neural network described in step S2 is as follows: In the formula, For the crack trend, is a coefficient.

[0013] Furthermore, the crack post-processing algorithm described in step S3 includes the following specific steps: S3.1: Convert the identified high-confidence crack regions into grayscale images and use a segmentation algorithm to obtain the segmentation map within the region; S3.2: Use a morphological erosion and dilation algorithm to filter out foreground noise in the segmentation image; S3.3: Calculate the crack extension angle on each sub-image block using the HOG operator, statistically analyze the crack edge trend angle of each image block where cracks exist, and calculate the overall crack trend angle of the current large image. S3.4: Project the crack along its inclination angle, merge all crack detection boxes in this direction, and remove false alarm boxes caused by background texture interference by using a threshold. Finally, obtain the crack segmentation effect map and the crack pixel-level measurement results.

[0014] Furthermore, the crack mapping algorithm in step S4 includes the following specific steps: S4.1: Establish the projection relationship between all planar photographs and the 3D model, and find the position of each original image in the 3D model of the dam; S4.2: Based on the convolution back projection method, the pixel-level measurement results of the crack are transformed into the true size of the crack, and the true length, width and location information of the crack are obtained.

[0015] The present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the above-described method for UAV image recognition based on a neural network model.

[0016] Compared with the prior art, the technical solution of this application has the following beneficial effects: 1. This UAV image recognition method based on a neural network model uses elliptical shapes instead of traditional rectangular boxes to label cracks. It accurately quantifies morphological features through five-dimensional parameters, introduces a geometric loss function to balance the accuracy of position, size, and angle, and optimizes the classification effect with BCEcls binary classification cross-entropy. This method breaks through the limitations of rectangular boxes, quantifies the crack morphology, and improves the recognition accuracy. 2. This UAV image recognition method based on a neural network model filters noise through morphological operations, extracts the crack extension angle using the HOG operator, and eliminates background interference through projection merging, thus obtaining crack segmentation effect images and crack pixel-level measurement results. 3. This UAV image recognition method based on a neural network model directly outputs the true length, width, and spatial location of cracks, improving the crack recognition effect. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention; Figure 2 This is a diagram showing the crack detection effect of the present invention; Figure 3 This is a flowchart of the crack post-processing algorithm of the present invention; Figure 4 This is an image showing the effect of crack merging against a simple texture background, as presented in this invention. Figure 5 This is an image showing the cracks merged against a complex texture background, as described in this invention. Figure 6 This is a comparison of the segmentation effects of the traditional CV algorithm and the crack post-processing algorithm of this invention; Figure 7 This is a diagram showing the true dimensions of the defect obtained through the back projection relationship in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1-7 The UAV image recognition method based on a neural network model in this embodiment is characterized by the following specific steps: S1. Obtain the raw crack image captured by the drone; S2. Based on the improved YOLO deep convolutional neural network, the original crack image is identified to obtain the crack location in the real-time crack image; S3. Based on the crack post-processing algorithm, the crack location of the real-time crack image is post-processed to obtain the crack segmentation effect map and crack measurement results; S4. Based on the crack mapping algorithm, the crack measurement results are converted into the actual crack size.

[0020] Most existing crack detection algorithms require significant computational power for training, and many still lack real-time performance when facing complex scenarios. YOLO, developed based on the lightweight Darknet-53 network, is renowned for its speed, and its C-language platform ensures good portability and real-time performance. YOLO utilizes many residual modules to reduce the risk of gradient explosion and improve the learning ability of the neural network. Furthermore, YOLO has three different scale outputs, enhancing the neural network's ability to identify cracks at different scales. Therefore, an improved YOLO deep convolutional neural network is proposed.

[0021] The convolutional feature vectors extracted by the YOLO deep convolutional neural network are passed through a Soft Max activation layer to obtain the probability prediction vectors for each category corresponding to the region. The loss functions for object detection tasks generally consist of Classification Loss, Objectness Loss, and Bounding Box Regression Loss. YOLO uses the BECLogits loss function to calculate Objectness Loss, and the Cross-Entropy Loss function (BCEclsloss) for Classification Loss. A good bounding box regression function should consider three important geometric factors: overlap area, center distance, and aspect ratio. Considering the discontinuities in pixel grayscale and spatial distribution in structural defects, YOLO sets the loss function to the GIOU_Loss measure.

[0022] Because crack image defect detection is unique compared to the recognition of other objects, crack images have a smaller area in the training image and less obvious features. General methods are greatly affected by background texture. Therefore, based on the special characteristics of cracks, the loss function is improved to elliptical output for training on crack data. Specifically, the loss function is rewritten as a weighted sum of four measures: Classification Loss, Objectness Loss, Bounding Box Regression Loss, and Offset Angle Loss.

[0023] (1) The Classification Loss function uses BCEcls loss for binary classification and cross-entropy loss.

[0024] ; To address the imbalance in the number of targets of different categories in the training dataset, the BCE loss function was modified, as shown in the formula above, by adding a weighted weight to the interaction information entropy of each target category. and This balances the training loss of classes with fewer samples with larger samples, thus solving the problem of imbalanced training sets.

[0025] (2) Objectness Loss uses the BECLogits loss function, which combines sigmoid and BCELoss. It is more numerically stable than using either of them alone. The calculation process is similar to BCELoss, but with the addition of a sigmoid layer.

[0026] (3) Bounding Box Regression Loss: The general bounding box regression function considers three important geometric factors: overlap area, center point distance, and length and width. Compared with the target imaging in ImageNet, crack imaging is distributed in the form of thin lines. In view of this characteristic, a special target position loss function is proposed, that is, the box label in the original algorithm is replaced with an elliptical shape distribution that is closer to the crack line form. The main considerations are: the major and minor axes of the crack line, the center point distance of the training crack data, and an additional offset angle loss function for the training crack data. For crack data, the crack trend is also an important attribute. The box label in the original algorithm is replaced with an elliptical shape distribution that is closer to the crack line form. According to the general equation of the ellipse: In the formula, Let these be the coordinates of the center of the ellipse. For the length of the major axis, For the minor axis length, For the deflection angle, Let be the coordinates of any point on the ellipse. Based on the general equation of the ellipse, the improved loss function is set as follows: ; In the formula, Let these be the coordinates of the center of the ellipse. For the length of the major axis, For the minor axis length, All values ​​are after Sigmoid transformation. , , All are weighted coefficients. , , , .

[0027] The improved loss function mainly considers the influence of the major and minor axes and the center coordinates of the circumscribed ellipse of the crack: it uses a general Euclidean distance to calculate the error between the predicted result and the actual result, and then multiplies it by a coefficient to adjust the influence of the center coordinates on the overall loss value. For the major and minor axes, it uses the square of the difference between the predicted result and the actual result to calculate the error, and then multiplies it by a coefficient to adjust the influence of the major and minor axes on the overall loss value.

[0028] (4) Offset angle loss function: ; The crack trend θ has a significant impact on crack identification and detection. Therefore, a separate loss function is set for the offset angle to improve identification accuracy. By using a cosine function, the loss function approaches 0 when the predicted value is close to the actual value. Then, a coefficient k is multiplied to adjust the influence of the offset angle. Since the correctness of the crack trend directly affects the prediction result, even if the major and minor axes and center point of the predicted result are similar to those of the training data, a large deviation in the offset angle will result in a large deviation from the actual value. Therefore, it is necessary to multiply by a coefficient k to adjust the influence of the offset angle for more accurate learning and ultimately improve the accuracy of the model.

[0029] The improved YOLO deep convolutional neural network identifies the original crack images. By refining the model and collecting more local training data, the recognition rate of the model is improved. Under optimal conditions, the recognition rate of cracks of 0.1 mm and above can be better than 99.0%, and the width detection accuracy is better than ±0.2 mm.

[0030] Since actual cracks exist continuously in the detected image rather than in an isolated image patch, after the entire large image is traversed and identified, the detected target boxes are merged to form the final detection boxes. Therefore, a crack post-processing algorithm module is proposed to filter out unnecessary background texture interference.

[0031] The main process is as follows: First, the identified high-confidence crack regions are converted into grayscale images, and a segmentation algorithm is used to obtain the segmentation map within the region. Then, a morphological erosion and dilation algorithm is used to filter out foreground noise in the segmentation map. Next, the crack extension angle on each sub-image block is calculated using the HOG operator. The crack edge trend angle of each image block containing a crack is statistically analyzed to calculate the overall crack trend inclination angle of the current large image. Finally, the crack trend inclination angle is projected, and all crack detection boxes in this direction are merged. False alarm boxes caused by background texture interference are removed by thresholding, and the bounding box containing the crack is finally obtained.

[0032] To test the effect of crack merging on removing false alarms and improving work efficiency, we randomly selected dozens of images. These images all contained water stains and structural lines. Some of these complex textures, which are very similar to cracks, were identified as cracks, reducing the efficiency of subsequent segmentation by the model. By comparing the effects before and after merging, it was shown that the merged crack bounding box could completely frame the crack area, which could partially simulate the work of an inspection expert.

[0033] For images with identified defects, a series of processes such as enhancement, denoising, image segmentation, and edge detection are performed to obtain crack segmentation effect images and crack pixel-level measurement results. An optimized segmentation algorithm based on deep learning is adopted, which has a high crack pixel-level overlap rate and can give correct segmentation results under interference such as small cracks, complex backgrounds, and human markings.

[0034] To obtain the true length, width, and location information of the cracks, the projection relationship between all planar photos and the 3D model was first established, the position of each photo in the dam 3D model was found, and a crack mapping algorithm was developed based on the back projection relationship. The pixel-level measurement results of the cracks were converted into the true size of the cracks based on the convolution back projection method, thus obtaining the true length, width, and location information of the cracks.

[0035] For images with identified defects, a series of processes such as enhancement, denoising, image segmentation, and edge detection are performed to obtain crack segmentation results and pixel-level measurement results of cracks. Compared with traditional CV algorithms, the crack pixel-level overlap rate is high, and it can provide correct segmentation results under interference such as fine cracks, complex backgrounds, and human markings.

[0036] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a disk, optical disk, read-only memory, or random access memory.

[0037] The working principle of the above embodiments is as follows: This UAV image recognition method based on a neural network model replaces the traditional rectangular bounding box for crack labeling. It accurately quantifies morphological features through five-dimensional parameters, introduces a geometric loss function to balance the accuracy of position, size, and angle, and optimizes the classification effect with BCEcls binary classification and cross-entropy optimization. This method overcomes the limitations of rectangular bounding boxes, quantifies the crack morphology, and improves recognition accuracy. It filters noise through morphological operations, extracts the crack extension angle using the HOG operator, and eliminates background interference through projection merging. It obtains crack segmentation effect images and pixel-level measurement results of cracks, directly outputting the true length, width, and spatial location of cracks, thus improving the crack recognition effect.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for UAV image recognition based on a neural network model, characterized in that, The specific steps include the following: S1. Obtain the raw crack image captured by the drone; S2. The original crack image is identified based on the improved YOLO deep convolutional neural network. The loss function of the improved YOLO deep convolutional neural network is a weighted sum of four measurement functions: Classification Loss, Objectness Loss, Bounding Box Regression Loss, and Offset Angle Loss, which is used to obtain the crack location in the real-time crack image. S3. Based on the crack post-processing algorithm, the crack location of the real-time crack image is post-processed to obtain the crack segmentation effect map and crack measurement results; S4. Based on the crack mapping algorithm, the crack measurement results are converted into the actual crack size.

2. The UAV image recognition method based on a neural network model according to claim 1, characterized in that, In step S1, the drone is equipped with a high-definition camera to take fixed-point photos of the dam and obtain original images of the dam surface, including original normal images and original crack images.

3. The UAV image recognition method based on a neural network model according to claim 1, characterized in that, In step S2, the improved YOLO deep convolutional neural network uses the BCEcls loss (binary cross-entropy loss) for classification loss. The expression for the BCEcls loss is as follows: In the formula, For prediction, and All are weighted.

4. The UAV image recognition method based on a neural network model according to claim 1, characterized in that, In the improved YOLO deep convolutional neural network described in step S2, the Objectness Loss function adopts the BECLogits loss function.

5. The UAV image recognition method based on a neural network model according to claim 1, characterized in that, Step S2 describes an improved YOLO deep convolutional neural network where the Bounding Box Regression Loss considers three important geometric factors: overlap area, center point distance, and length and width. Compared to object imaging in ImageNet, crack imaging is distributed in the form of thin, elongated lines. To address this characteristic, the original algorithm replaced the rectangular box annotations with elliptical shapes that more closely resemble crack lines. This is based on the general equation of an ellipse. In the formula, Let these be the coordinates of the center of the ellipse. For the length of the major axis, For the minor axis length, For the deflection angle, Let be the coordinates of any point on the ellipse. Based on the general equation of the ellipse, the improved loss function is set as follows: In the formula, Let these be the coordinates of the center of the ellipse. For the length of the major axis, For the minor axis length, All values ​​are after Sigmoid transformation. , , All are weighted coefficients. , , , .

6. The UAV image recognition method based on a neural network model according to claim 1, characterized in that, The expression for the offset angle loss function in the improved YOLO deep convolutional neural network described in step S2 is as follows: In the formula, For the crack trend, is a coefficient.

7. The UAV image recognition method based on a neural network model according to claim 1, characterized in that, The crack post-processing algorithm described in step S3 includes the following specific steps: S3.1: Convert the identified high-confidence crack regions into grayscale images and use a segmentation algorithm to obtain the segmentation map within the region; S3.2: Use a morphological erosion and dilation algorithm to filter out foreground noise in the segmentation image; S3.3: Calculate the crack extension angle on each sub-image block using the HOG operator, statistically analyze the crack edge trend angle of each image block where cracks exist, and calculate the overall crack trend angle of the current large image. S3.4: Project the crack along its inclination angle, merge all crack detection boxes in this direction, and remove false alarm boxes caused by background texture interference by using a threshold. Finally, obtain the crack segmentation effect map and the crack pixel-level measurement results.

8. The UAV image recognition method based on a neural network model according to claim 2, characterized in that, The crack mapping algorithm described in step S4 includes the following specific steps: S4.1: Establish the projection relationship between all planar photographs and the 3D model, and find the position of each original image in the 3D model of the dam; S4.2: Based on the convolution back projection method, the pixel-level measurement results of the crack are transformed into the true size of the crack, and the true length, width and location information of the crack are obtained.

9. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the UAV image recognition method based on a neural network model according to any one of claims 1 to 8.

Images