Intelligent inspection and identification method for hidden danger of highway slope in loess area

Abstract

The invention relates to a loess area expressway slope hidden danger intelligent inspection and identification method, an algorithm model provided by the invention fuses an image segmentation model SAM, a semantic segmentation model U-net and a target detection model, and first, a semantic segmentation network is used to extract a coarsely divided slope hidden danger Mask; in order to further improve the accuracy and integrity of the contour and consider the terrain, height, shadow and other information of the side slope hidden danger, a side slope hidden danger target frame is generated by using a target detection model, the side slope hidden danger target frame is used as a prompt of an SAM module, and a prompt encoder is used for encoding. Besides, the coarsely divided side slope hidden danger Mask is used as the image input of the SAM, and an SAM image decoder is used, so that the subdivided side slope hidden danger Mask is obtained, and finally the identification precision standard reaching condition is evaluated according to the data label.

Description

Technical Field

[0001] The present invention belongs to the technical field of highway slope monitoring, and particularly relates to an intelligent inspection and identification method for hidden dangers of highway slopes in loess areas. Background Technique

[0002] The casualties, damage to transportation facilities, collapse of houses and buildings caused by slope disasters in loess areas seriously endanger the economic and social development. Although manual inspection can penetrate into the areas with disease hidden dangers and systematically conduct all-round monitoring and evaluation of geological disaster points, the cost of manual inspection is relatively high and there are certain risks. The intelligent inspection and identification technology can better solve the above problems.

[0003] UAV inspection mainly relies on advanced sensors and image processing technologies to perform high-precision visualization and data collection of objects. UAVs can carry various sensors, such as infrared thermometers, high-definition cameras, radars, etc., to achieve inspections and monitoring of high-altitude and relatively long-distance objects such as buildings, bridges, and power lines. These sensors can capture a large amount of data, providing rich information for subsequent intelligent identification. The intelligent identification technology is responsible for analyzing and processing the collected data. Using machine learning algorithms, the input data is compared with the pre-stored data to identify new and unique objects or information. This technology can automatically identify parameters such as the shape, size, and position of objects, and determine whether there are defects or damages in the objects. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the existing slope inspection technology, and provide an intelligent inspection and identification method for hidden dangers of highway slopes in loess areas, so as to improve the efficiency and automation degree of highway slope inspection in loess areas.

[0005] The technical solution is as follows: An intelligent inspection and identification method for hidden dangers of highway slopes in loess areas, including: UAV inspection, pixel-by-pixel local identification, slope panoramic identification, and intelligent detection; The pixel-by-pixel local identification relies on an intelligent identification algorithm to detect disease points; The slope panoramic identification is realized through UAV inspection and intelligent 3D reconstruction technology. The 3D base map is constructed by using the full-automatic modeling and decoration function. At the same time, a 2D base map is constructed based on the real-scene 3D model. Through data collection and processing, a two-three-dimensional integrated base map is constructed; based on the base map, the scattered large-range UAV inspection images, ground fine inspection images, identification results and the real-scene 3D model are integrated; The intelligent detection passes through the image segmentation module SAM (Segment Anything Model), and uses the object detection network and the semantic segmentation network U-net to obtain the slope hidden danger mask.

[0006] Furthermore, the intelligent detection specifically includes the following steps: The first step: Obtain DOM (Digital Orthophoto Map) from multi-directional satellite images through preprocessing, and then input the cropped DOM into the semantic segmentation network of deep learning to extract the rough slope hazard rough Mask; The second step: To further improve the accuracy and integrity of the contour, and considering the terrain, height, and shadow information where the slope hazard is located, input the rough slope hazard Mask into the U-net encoder to obtain refined image features, and input them into SAM (Segment Anything Model); The third step: Convolve the rough slope hazard Mask, combine the object detection bounding box as Prompt information, and input them into the SAM decoder together to obtain the refined slope hazard Mask.

[0007] Furthermore, SAM (Segment Anything Model) borrows the Prompt idea in NLP tasks. In order to more accurately identify highway slope hazards, Prompt specifically refers to the object detection anchor box information, which is expressed as follows: ,

[0008] Among them, represents the original image, represents the object detection network. In this method, yolov5 is used for the disease target detection task, represents the anchor box position information; represents the image encoder, which is used to extract image features , and the U-net network encoder is selected in this method; represents the Prompt encoder, which encodes the anchor box information, and finally inputs the image features and the encoded prompt information into the semantic segmentation model .

[0009] Furthermore, the intelligent detection performs disease point detection, and intelligently identifies and accurately locates slope diseases according to the visible light pixel point features and settlement amount parameters. The disease types include four disease types: cracks, collapses, protective damage, and sinkholes.

[0010] Furthermore, the UAV inspection conducts data collection to obtain visible light images, settlement amounts, and deformation amount parameters of the target area, so as to construct a slope disease sample library.

[0011] Furthermore, four evaluation metrics are used: Accuracy, Recall, Precision, and F1-score.

[0012] Technical effects brought by the present invention: SAM (Segment Anything Model) borrows the Prompt idea in NLP tasks and completes the rapid segmentation of any target by providing Prompt hints for the image segmentation task. The hints can be foreground / background point sets, rough boxes or masks, any form of text, or any information indicating what needs to be segmented in the image. To more accurately identify potential hazards on highway slopes, U-net performs excellently in many semantic segmentation scenarios. The network has characteristics such as easy training and high generality. Its feature fusion method adopts a skip connection structure, fully connecting the encoder features and the decoder features. By merging the feature maps output by all convolutional modules with the corresponding decoded structure feature maps, deep semantic features are retained, and upsampling is performed step by step to obtain the semantic segmentation result. The method provided by the present invention combines intelligent detection with the algorithm of the semantic segmentation network U-net and the SAM framework, which can bring the following increments: (1) Higher accuracy: The deep learning model has excellent classification and segmentation capabilities, can more accurately identify the location of slope hazards, and reduce false alarms and missed detections.

[0013] (2) Spatial perception: The SAM model helps to understand the spatial relationships in the image, making the algorithm more sensitive to the layout and relative positions of slope hazards.

[0014] (3) Adaptability: The deep learning algorithm can adapt to different types of terrains and the shapes of slope hazards, making it perform excellently in applications in different scenarios.

[0015] (4) Multi-source data fusion: By combining DOM data, the algorithm can utilize visual information and altitude information simultaneously, improving the ability to understand slope hazards. This multi-source data fusion helps to more accurately identify slope hazards.

[0016] (5) Reduced manual intervention: The deep learning model and automated processing reduce the need for manual intervention, thus improving the processing speed and efficiency. Brief Description of the Drawings

[0017] Figure 1 is the flowchart of the method of the present invention.

[0018] Figure 2 is the structural diagram of the intelligent inspection and recognition algorithm.

[0019] Figure 3 is the panoramic recognition result of the slope (taking the Kangning slope as an example).

[0020] Figure 4 It is the crack result of local pixel-by-pixel recognition of the slope.

[0021] Figure 5 It is the crack and collapse results of local pixel-by-pixel recognition of the slope.

[0022] Figure 6 It is the result of the protection damage recognized by local pixel-by-pixel of the slope. Detailed implementation manners

[0023] To enable those skilled in the art to better understand the technical solution of the present invention, the intelligent inspection and recognition of highway slopes in loess areas of the present invention will be described in detail below in conjunction with embodiments. The following embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0024] The present invention relates to a method for intelligent inspection and recognition of hidden dangers of highway slopes in loess areas. The recognition of hidden dangers of highway slopes usually refers to the recognition of hidden dangers of slopes such as collapses, sinkholes, protection damage, and cracks from images or videos, and has wide applications in fields such as highway slope maintenance, geographic information systems, and geological disaster surveys. Traditional methods usually involve edge detection, segmentation, and shape analysis techniques, but in recent years, deep learning techniques have brought significant improvements. The present invention first collects slope image data and ground parameter information through drone inspection, then performs data preprocessing, and then performs model training. The model proposed by the present invention integrates the image segmentation model SAM, the semantic segmentation model U-net, and the object detection model. First, the semantic segmentation network is used to extract the roughly segmented hidden danger mask of the slope. To further improve the accuracy and integrity of the contour and consider information such as the terrain, height, and shadow where the hidden danger of the slope is located, the object detection model is used to generate the target box of the hidden danger of the slope, which is used as the prompt of the SAM module and encoded using the prompt encoder. In addition, the roughly segmented hidden danger mask of the slope is used as the image input of SAM, and the SAM image decoder is used to obtain the finely segmented hidden danger mask of the slope. Finally, the recognition accuracy compliance is evaluated according to the data label. This method greatly improves the slope inspection efficiency by performing refined hidden danger recognition on highway slopes and combining drone inspection technology, providing convenience for slope disaster prevention and control work. Embodiment

[0025] 1. Construction of the sample library First, use the DJI Mavic 3E drone to collect highway slope image data. The drone parameters are set according to the target terrain change. The inspection range of the present invention is set as follows: Longitude (110°14’ ~ 114°33’E); Latitude (34°34’ ~ 40°44’N); Altitude (1km ± 0.2km above sea level).

[0026] The parameters of the image data are as follows: Resolution of a single image: 5280×3950; Color representation: RGB; Horizontal resolution: 96 dpi; Vertical resolution: 95 dpi.

[0027] Preprocess the collected slope image data. Combine the collected ground parameter indicators such as settlement amount and deformation amount to conduct geoscientific analysis on the slope to determine the main disease types. Determine the target category according to the determined disease type, perform instance annotation at the pixel level on the image data, and randomly divide it into a training set and a validation set. Thus, a complete sample library of highway slope diseases in the loess area is constructed.

[0028] 2. Image preprocessing Before performing intelligent recognition, it is necessary to preprocess the relevant data. The preprocessing operations are mainly divided into the following steps: (1) Image preprocessing: Preprocess the collected images, crop the size to 512×512, and denoise the images to improve the stability and accuracy of the model.

[0029] (2) Annotation data processing: Convert the annotated slope defect information into a suitable JSON format for model training and testing.

[0030] (3) Data augmentation: In order to increase the diversity of data and the generalization ability of the model, data augmentation techniques such as rotation, mirroring, and cropping are adopted.

[0031] (4) Data division: Divide the labeled data into a training set, a validation set, and a test set.

[0032] 3. Model training Use the obtained dataset to train the model. The actual training parameters can be adjusted according to the server computing power and image size, and multiple experiments can be carried out until the detection accuracy reaches the optimal. Then test the trained model on the test set to prove the effectiveness of the model. The model is trained based on the Ubuntu18.04 system, the cuda version is 11.1, the python3.8 programming language is used, and the GPU (NVIDIA GeForce 3060, single-card video memory 16G) is used for acceleration during the training process.

[0033] 1) Evaluation indicators To measure visual models, common evaluation metrics are needed. All evaluation metrics are derived from the confusion matrix, which is a table commonly used in the fields of machine learning and statistics to evaluate classification results. It is usually an N×N matrix. The confusion matrix (also called the error matrix) is a matrix drawn with the statistical information of the number of categories predicted by the model as the horizontal axis and the statistical information of the number of true labels as the vertical axis. It represents the classification results of the statistical classification model. The confusion matrix for computer vision tasks is as follows: Table 1 Confusion Matrix

[0034] The following four evaluation metrics are used in this method: Accuracy: ,

[0035] Recall: Based on the actual samples as the judgment basis, among the samples that are actually positive examples, the proportion of positive examples that are correctly predicted to the total actual positive example samples. The larger the recall, the higher the sensitivity of the model. It is defined as follows ,

[0036] Precision: Based on the prediction results as the judgment basis, the proportion of correctly predicted samples among the samples predicted as positive examples. The larger the value, the stronger the performance of the model in judging changing pixels. It is defined as follows: ,

[0037] Usually, the best results cannot be achieved simultaneously for recall and precision. Therefore, the F1score metric for binary classification tasks is introduced. The higher the value, the higher the accuracy. F1 is an evaluation metric that comprehensively considers the precision and recall of the classification model and is defined as the harmonic mean of precision and recall. It is defined as follows: ,

[0038] 4. 3D Reconstruction Construct a 2D base map based on the real-scene 3D model. Through data collection and processing, construct a fused 2D and 3D base map. Based on the base map, fuse the scattered large-scale UAV inspection images, ground fine inspection images, disease recognition results with the real-scene 3D model to achieve intuitive spatial visualization management of inspection images.

[0039] Compared with the prior art, the intelligent inspection and identification method for slopes in loess areas proposed by the present invention combines unmanned aerial vehicle (UAV) inspection with intelligent technologies to achieve full-process automation and intelligence in slope inspection in loess areas. Through intelligent UAV inspection, image data of slope diseases and ground information parameters in loess areas are collected to construct an image database of slope diseases. Based on AI recognition of images, typical diseases and potential safety hazards in the inspection images can be accurately identified and marked, reducing the amount of manual interpretation during the inspection process and improving the inspection efficiency and accuracy. Through the registration and fusion of multi-period images and real-scene three-dimensional models, the development trend of slope diseases is compared and analyzed, as well as historical backtracking is carried out. Highly automated inspection and intelligent identification of slope hidden dangers in loess areas are achieved.

[0040] The above examples of the present invention have been described in detail in combination with the embodiments. However, the present invention is not limited to the above examples. Various changes can be made without departing from the spirit of the present invention within the scope of knowledge possessed by those of ordinary skill in the art, and these should also be regarded as the protection scope of the present invention.

Claims

1. A method for intelligent inspection and identification of hidden dangers on highway slopes in Loess Plateau, characterized in that: include: Drone inspection, pixel-by-pixel local recognition, slope panoramic recognition, and intelligent detection, including: The pixel-by-pixel local recognition relies on intelligent recognition algorithms to detect disease points; The panoramic slope recognition is achieved through drone inspection and intelligent 3D reconstruction technology. The 3D base map is constructed by using the fully automatic modeling and modification function. At the same time, a 2D base map is constructed based on the real-scene 3D model. Through data collection and processing, a 2D and 3D fused base map is constructed. Based on the base map, scattered drone large-scale inspection images, ground fine inspection images, recognition results and real-scene 3D models are integrated. The intelligent detection obtains slope hazard mask by using the image segmentation module SAM, the target detection network and the semantic segmentation network U-net.

2. The intelligent inspection and identification method for slope hazards of highways in loess regions according to claim 1 is characterized in that: The intelligent detection specifically comprises the following steps: 1) Obtain DOM from multi-directional satellite images through preprocessing, and then input the cropped DOM into the deep learning semantic segmentation network to extract the coarse slope hazard mask; 2) In order to further improve the accuracy and completeness of the contour and consider the terrain, height, and shadow information of the slope hazards, the coarse slope hazard Mask is input into the U-net encoder to obtain refined image features and then input into the SAM decoder; 3) The coarse slope hazard Mask is convolved, combined with the target detection frame as the prompt information, and input into the SAM decoder to obtain the fine slope hazard Mask.

3. The intelligent inspection and identification method for slope hazards of highways in loess regions according to claim 2 is characterized in that: SAM is combined with the Prompt encoder to identify highway slope hazards. Prompt specifically refers to the target detection anchor box information, which is expressed as follows: ， in, represents the original image, represents the target detection network, and uses yolov5 for disease target detection tasks. Indicates the anchor box position information; Represents an image encoder, used to extract image features , select U-net network encoder; Represents the Prompt encoder, which encodes the anchor box information and finally encodes the image features Input the semantic segmentation model with the encoded prompt information middle.

4. The intelligent inspection and identification method for slope hazards of highways in loess regions according to claim 3 is characterized in that: The intelligent detection performs disease point detection, and intelligently identifies and accurately locates slope diseases based on visible light pixel point features and settlement parameters. The disease types include four types: cracks, collapses, protective damage, and sinkholes.

5. The intelligent inspection and identification method for slope hazards of highways in loess regions according to claim 4 is characterized in that: The drone inspection collects data to obtain visible light images, settlement and deformation parameters of the target area, thereby forming a slope disease sample library.

6. The intelligent inspection and identification method for slope hazards of highways in Loess Region according to claim 2 is characterized in that: Four evaluation indicators are used: accuracy, recall, precision, and F1 score.

Images