A small sample-based earth-rock dam defect image recognition method
By combining image classification and transfer learning techniques with a small-sample-based image recognition method for earth-rock dam defects, the problems of large sample requirements and low recognition efficiency in existing technologies for earth-rock dam defect image recognition are solved, and efficient and accurate earth-rock dam defect recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING HYDRAULIC RES INST
- Filing Date
- 2023-02-13
- Publication Date
- 2026-06-19
AI Technical Summary
Existing automatic image recognition methods for dam defects require a large number of samples, are only applicable to defects such as cracks in concrete dams, and cannot be applied to image recognition of defects in earth-rock dams. Furthermore, they rely on experienced dam construction technicians for recognition, resulting in a large workload and poor timeliness.
A method for identifying defects in earth-rock dams based on small samples is adopted. By classifying defects in earth-rock dams from the perspective of images, the TensorFlow framework and Faster R-CNN InceptionResNet pre-trained models are used, combined with transfer learning techniques. Only a small number of samples are needed to train the recognition model, which is then integrated into inspection APP and video monitoring software.
It enables efficient and accurate identification of defects in earth-rock dams, reduces the difficulty and workload of sample collection, has a wide range of applications, improves identification speed and accuracy, and can detect dam hazards in a timely manner.
Smart Images
Figure CN116309342B_ABST
Abstract
Description
Technical Field
[0001] This invention patent belongs to the field of image recognition technology, specifically relating to a method for image recognition of defects in earth-rock dams based on small samples. Background Technology
[0002] Reservoirs and dams are crucial infrastructure for the national economy, vital engineering measures for regulating the spatial and temporal distribution and optimizing water resource allocation, and essential components of the national flood control, drought relief, and disaster reduction system and river flood control engineering system. The safety of reservoirs and dams is not only related to flood control, water supply, and food security, but also to economic security, ecological security, and national security. Of my country's more than 90,000 existing reservoirs, over 90% are earth-rock dams, most of which were built between the 1950s and 1970s. Due to the limitations of economic and technological conditions at the time, these reservoirs and dams generally suffer from low flood control standards and poor engineering quality. Coupled with insufficient investment in operation and management over the years, aging and disrepair have led to a large number of dilapidated and unsafe reservoirs, posing significant safety hazards.
[0003] To monitor the safety of earth-rock dams, monitoring facilities are deployed for daily observation, reflecting the dam's safety status. However, these facilities can only monitor specific points and cannot comprehensively cover the entire reservoir dam, resulting in missed detections and blind spots. Inspection tours, on the other hand, can provide a comprehensive and intuitive way to quickly diagnose the dam's safety status. For most small and medium-sized reservoirs without monitoring facilities or with damaged facilities, dam inspection tours are particularly important for ensuring the safe operation of the dam. With the reform of the water management system and the streamlining of reservoir management personnel, many reservoirs have outsourced inspection work to property management companies. The inspectors are often non-professionals, only able to patrol along designated routes and take photos with their mobile phones. Most reservoirs have installed cameras on the dam to periodically photograph the dam, reservoir surface, and other important areas. However, identifying defects in earth-rock dams from these images requires a large number of experienced dam engineers to examine each image individually, which is labor-intensive and time-consuming. Therefore, if automatic image recognition of dam defects could be achieved, this problem of relying on a large number of experienced dam engineers to examine each image individually could be solved.
[0004] Current research on automatic image recognition of dam defects mainly involves extracting various features (such as Gabor and LBP) from images and using convolutional neural networks to identify dam defects. However, these methods require a large number of samples (generally thousands or even tens of thousands), making data collection difficult. Furthermore, they often only identify defects such as cracks in concrete dams, resulting in a narrow scope of application and poor practicality. Therefore, the problem of image recognition of defects in earth-rock dams remains unsolved. Summary of the Invention
[0005] The purpose of this invention is to address the problem that existing automatic image recognition methods for dam defects require a large number of samples and are only applicable to defects such as cracks in concrete dams, but not to defects in earth-rock dams. This invention provides a method for recognizing defects in earth-rock dams based on a small sample size.
[0006] Technical content: A method for image recognition of defects in earth-rock dams based on small samples, comprising the following steps:
[0007] Step 1: Classify the defects of earth-rock dams from an image perspective to obtain the defect types;
[0008] Step 2: Select a pre-trained model and establish a defect recognition model based on the defect types obtained in Step 1;
[0009] Step 3: Select defect image samples of earth-rock dams and process them to make them suitable for training and evaluation of the defect recognition model;
[0010] Step 4: Use the samples processed in Step 3 to train and evaluate the defect identification model established in Step 2;
[0011] Step 5: Integrate the defect identification model obtained in Step 4 into the inspection APP software and video monitoring software to perform defect image recognition of earth-rock dam.
[0012] Furthermore, in step one, the specific method for classifying defects in earth-rock dams is as follows: based on the "Technical Specification for Safety Monitoring of Earth-Rock Dams", the defect types of earth-rock dams are analyzed, and from the perspective of images, considering the similarity within defect categories, defects within the same defect type that can be visually identified are reclassified; considering the differences between defect categories, different defect types that cannot be visually identified are merged.
[0013] Furthermore, in step two, the TensorFlow framework is used, and the Faster R-CNN InceptionResNet pre-trained model is selected. The object detection method of the Faster R-CNN Inception ResNet pre-trained model is Faster R-CNN, the feature extraction network is ResNet, and the Inception module is used. The training dataset is COCO.
[0014] Furthermore, in step two, the defect identification model includes a full defect model containing all defect types and several single defect models containing only one defect type.
[0015] Furthermore, in step three, the sample processing procedure includes sample labeling, sample checking, and sample file conversion:
[0016] ①The sample annotation refers to generating defect label information based on defects on image samples and storing the defect label information in an annotation file;
[0017] ②The sample inspection refers to checking whether there are any missing or incorrect defect labels on the images. For sample files with problems, they are deleted or re-labeled.
[0018] ③ The sample file conversion refers to converting the image files and annotation files that have undergone sample annotation and sample checking into TFRecord files; before performing the sample file conversion, the entire sample that has undergone sample annotation and sample checking is used as the training sample, and a portion of the samples are randomly selected as the evaluation sample.
[0019] Furthermore, the sample annotation method is a B / S architecture-based sample annotation method developed based on the labelImg function and annotation file format; the sample annotation method includes the following steps:
[0020] ① When an image is used for a full defect model, all defects in the image are labeled; when used for a single defect model, only the defects that the model can identify are labeled.
[0021] ② Draw the image on the front end: Mark the defects in the image with a rectangle, obtain the coordinates of the top left and bottom right corners of the rectangle, select the defect type, and generate label information;
[0022] ③ The label information is transmitted to the backend in JSON format, and the backend saves all the label information of the image to the corresponding .xml annotation file in labelImg annotation file format.
[0023] Furthermore, in step four, when training the defect identification model, transfer learning technology is applied to retain the feature extractor in the pre-trained model and only retrain its classifier; the full defect model training uses all samples, while the single defect model training uses only the samples of the corresponding defect.
[0024] Furthermore, in step four:
[0025] 1) Defect identification model training, the specific training process is as follows:
[0026] The defect identification model is loaded using the `load_model` method of the `keras.Model` class, and the model training parameters are read from the model configuration file. The model is configured using the `compile` method. The defect identification model is trained using the `fit` method with the TFRecord files of training and evaluation samples to obtain the model's checkpoint file. The model file is saved using the `save` method. During model training, the TensorBoard's event accumulator is used to read the calculation process data from the model training record file and display the relevant parameters and graphs.
[0027] 2) Evaluation of the defect identification model, specifically the training process:
[0028] The trained defect recognition model is evaluated using the `evaluate` method of the `keras.Model` class. Data is read from the model evaluation log file using the TensorBoard event accumulator, and the model performance parameters and graphs are displayed.
[0029] Furthermore, in step five, after the inspection image or video surveillance image is identified by the defect identification model, the defect type is determined by the comprehensive defect identification method.
[0030] The defect comprehensive identification method includes the following process:
[0031] Step 1) Select each defect one by one as the current defect according to the order of the defect identification model and the order of defects identified by the defect identification model;
[0032] Step 2) Identify all defects in the subsequent defect identification model that are of the same type and location as the current defect, store the confidence level of these defects and the AP of this type of defect in the defect identification model, delete the defect from its respective defect identification model, and avoid selecting the defect again when processing defects in the defect identification model;
[0033] Step 3) Update the confidence of the current defect based on the confidence level given when the defect is identified by each defect identification model stored in Step 2) and the AP of this type of defect in each defect identification model;
[0034] Step 4) If the confidence level of the current defect is greater than 80%, determine that the defect exists in the inspection image or video surveillance image; otherwise, the defect does not exist.
[0035] Furthermore, in step 2), whether the locations of the defects are the same is determined by the degree of overlap. The degree of overlap is calculated as shown in formula (1). When the degree of overlap is greater than 0.95, the defect is considered to be a defect. j and defects l For the same defect:
[0036]
[0037] Where x j1 and x j2 For defects j Position left and right coordinates, y j1 and y j2 For defects j Position, vertical coordinates, x l1 and x l2 For defects l Position left and right coordinates, y l1 and yl2 For defects l The vertical coordinates of the location, where w and h are the width and height of the image captured from the inspection image or video surveillance image;
[0038] In step 3), the confidence level is calculated as shown in formula (2). The confidence level threshold is set to 80%. When the confidence level of the defect is greater than 80%, it is determined that the defect exists in the image.
[0039]
[0040] Where n is the total number of models that identified the defect, and the confidence level is... p For defect identification model p The confidence level for identifying the defect.
[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0042] This invention classifies earth-rock dam defects from an image perspective. By collecting only a small number of typical defect images and retraining the classifier of the constructed defect recognition model, a defect recognition model with high accuracy and wide applicability can be obtained. By integrating the defect recognition model into inspection APP software on mobile phones and video monitoring software on hard disk recorders, defects in earth-rock dams can be identified from inspection images and monitoring images. This allows for the early detection of dam hazards, timely alerts to reservoir inspection personnel and reservoir management personnel, and provides technical support for the safe operation of reservoir dams. Attached Figure Description
[0043] Figure 1 This is the technical approach of the small-sample-based image recognition method for earth-rock dam defects in this invention;
[0044] Figure 2 This is a schematic diagram of the basic structure of the object detection method Faster R-CNN;
[0045] Figure 3 This is a schematic diagram illustrating the principle of ResNet as the feature extraction network.
[0046] Figure 4 This is a schematic diagram of the Inception module structure;
[0047] Figure 5 Flowchart for sample annotation;
[0048] Figure 6 This is a flowchart of the defect comprehensive identification method.
[0049] in, Figure 5 In the text, N represents "No", indicating that all defects have been marked, and Y represents "Yes", indicating that there are defects that need to be marked. Figure 6In the diagram, N represents "no", indicating that the inequality is not true, and Y represents "yes", indicating that the inequality is true. Detailed Implementation
[0050] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific implementation examples.
[0051] like Figure 1 As shown, this invention discloses a method for image recognition of defects in earth-rock dams based on small samples, including defect classification, model building, sample processing, model training and evaluation, and defect recognition steps. The specific steps are as follows:
[0052] Step 1: Defect Classification
[0053] Defect classification refers to determining classification principles and classifying defects based on the analysis of defects in earth-rock dams.
[0054] 1) This invention analyzes defect types based on the "Technical Specification for Safety Monitoring of Earth-Rock Dams" (SL 551-2012).
[0055] The "Technical Specification for Safety Monitoring of Earth-Rock Dams" (SL 551-2012) comprehensively describes the potential defects in the dam body, foundation and dam area, water conveyance tunnels, spillways, gates and hoists, and near-dam slopes of earth-rock dams. These defects are categorized in several ways: some are based on their formation mechanism, such as cracks and seepage; some on their structural morphology, such as uplift and collapse; some on their severity, such as seepage and piping; and some on their extent, such as seepage and diffuse seepage. While most defects can be identified using these classification methods, some may be missed or misidentified. This is mainly because, from an image perspective, some defect types appear overly similar, while others exhibit significant differences within the same defect type.
[0056] 2) Classification Principles
[0057] To address the aforementioned problems, this invention, based on the above defect analysis, classifies earth-rock dam defects from an image perspective to avoid omissions and misidentifications. The principles for classifying earth-rock dam defects from an image perspective include:
[0058] ① Considering intra-class similarity of defects, defects within the same defect type that can be visually identified as different are reclassified;
[0059] Specifically, this refers to reclassifying defects that belong to the same defect type as described in the "Technical Specification for Safety Monitoring of Earth and Rock Dams" but differ significantly from each other in images. For example, cracks can be divided into fine cracks and wide cracks. A fine crack is just a line, while a wide crack is a large opening. The two differ significantly in images, so separating them makes their respective characteristics more prominent and focused.
[0060] ②Considering the differences between defect categories, different defect types that cannot be visually identified are merged;
[0061] Specifically, this refers to merging several different defect types described in the "Technical Specification for Safety Monitoring of Earth and Rockfill Dams" that have minor differences in appearance when viewed from images into one category; for example, seepage and leakage are two types of defects that only differ in severity and have minor differences in appearance when viewed from images, so merging them together avoids the existence of two highly similar defect types.
[0062] 3) Defect Types
[0063] This invention combines traditional earth-rock dam defect classification methods with the "Technical Specification for Safety Monitoring of Earth-Rock Dams" (SL551-2012) and the above classification principles. From the perspective of images, it classifies visually identifiable defects, and the obtained defect types are shown in Table 1.
[0064] Table 1 Types of Defects in Earth-Rock Dams
[0065]
[0066] Step 2: Model Building
[0067] Identifying defects in earth-rock dams from images typically requires collecting tens of thousands of defect image samples. After manual processing and annotation, the defect recognition model needs to be trained before it can be used for defect identification in earth-rock dams. Collecting tens of thousands of defect image samples for each type is very difficult, and the processing and annotation are time-consuming and labor-intensive.
[0068] This invention employs transfer learning technology, which can train a high-performance earth-rock dam identification model with only a small number of samples. Transfer learning involves transferring knowledge from a model already trained on a large-scale dataset to another model, preserving the feature extractor, and retraining the classifier on a new dataset with similar features. Since the feature extractor does not need to be retrained, the number of parameters that need to be trained is greatly reduced, requiring only a small number of samples.
[0069] 1) Select a pre-trained model
[0070] This invention employs the TensorFlow framework, and its pre-trained models can be downloaded from the open-source TensorFlow project hosted on GitHub. The TensorFlow pre-trained model library provides several models already trained on datasets such as COCO, KITTI, Open Images, and AVAv2.1. After testing and comparison, this invention selects the Faster R-CNN InceptionResNet pre-trained model, which can be trained to obtain a high-performance model for identifying defects in earth-rock dams.
[0071] The object detection method of the Faster R-CNN Inception ResNet pre-trained model is Faster R-CNN, the feature extraction network is ResNet, and the Inception module is used. The training dataset is COCO.
[0072] Faster R-CNN is a region-based object detection method. Building upon R-CNN and Fast R-CNN, it integrates feature extraction, candidate region generation, bounding box refinement, and classification to improve the overall performance of object recognition. Its basic structure is shown in the appendix. Figure 2 The feature extraction module consists of a set of basic convolutional layers, activation layers, and pooling layers, specifically designed to extract convolutional feature maps from the image, forming a crucial foundation for the recognition process. The Region Proposal Network (RPN) shares the convolutional feature maps from the feature extraction module to generate region candidate boxes, improving the generation speed compared to the Selective Search method. The RoI pooling layer extracts the corresponding convolutional feature maps from the input convolutional feature maps and region candidate boxes, feeding the results into a fully connected layer to determine the target's category. The classifier uses the convolutional feature maps of the candidate regions to classify targets within those regions, while simultaneously refining the bounding boxes to obtain the target's precise location.
[0073] ResNet (Residual Neural Network) adds a skip connection between the input and output, giving the neural network the ability to back up and solving the gradient and gradient explosion problems in deep neural networks. See the appendix for the principles of ResNet. Figure 3 ResSet uses the Inception module to increase network complexity; the structure of the Inception module is attached. Figure 4 The Inception module takes X as input, passes through four sub-networks to obtain four network outputs, and then concatenates them along the channel axis to form the output of the Inception module.
[0074] COCO (Common Objects in Context) is a large-scale dataset for object detection that includes natural images and images of common objects in daily life. The backgrounds of these images are relatively complex, and the objects are small in size. Pre-trained models trained on this dataset perform well.
[0075] 2) Establish a defect identification model
[0076] Using the Tensorflow framework, a defect identification model was built based on the Faster R-CNN Inception ResNet pre-trained model and the defect types obtained in step one. The defect identification model includes one full-defect model and 27 single-defect models. The full-defect model includes all defects and is used to identify which earth-rock dam defects exist in the image and their locations. The single-defect models only include a specific type of defect and are used to identify whether a specific type of earth-rock dam defect exists in the image and its location. Combining the two improves the accuracy of defect identification.
[0077] The specific steps for building a defect identification model based on a pre-trained model are as follows:
[0078] ① Determine the model identification, model name, and defect name. For a full defect model, defect names are separated by commas.
[0079] ② Create a model folder and its subfolders to store: predefined label files, .csv files, .tfrecord files; model evaluation files; image files and .xml annotation files for training samples, evaluation samples, and test samples; pre-trained model files; post-trained model files; and training configuration files and process files.
[0080] ③ Download the pre-trained model from the TensorFlow open-source project hosted on GitHub, generate a predefined label file based on the defect name, and store it in the corresponding subfolder.
[0081] Step 3: Sample Processing
[0082] The collected images of earth-rock dam defects are stored in the image subfolder of the model folder. These images must undergo sample processing before they can be used for model training and evaluation. The sample processing includes sample annotation, sample checking, and sample file conversion (including image files and annotation files). This method requires at least 150 image samples for each type of defect.
[0083] In this embodiment, for each of the 27 single-defect models, 150 image samples containing the corresponding defects are prepared. When annotating the samples, only the defects corresponding to that model are annotated. For each single-defect model, 150 image samples are also prepared, but when annotating the samples, all defects in the images need to be annotated.
[0084] 1) The sample annotation refers to generating defect label information based on defects on image samples and storing the defect label information in an annotation file;
[0085] Defects in an image can be marked with boxes, typically using the labelImg tool. The labelImg tool is a client / server architecture and can only be used by one person at a time.
[0086] Based on the labelImg function and annotation file format, this invention develops a sample annotation method based on a B / S architecture, which can be used by multiple users simultaneously. The process is shown in the appendix. Figure 5 Specifically, it includes the following steps:
[0087] ① When an image is used for a full defect model, all defects in the image are labeled; when used for a single defect model, only the defects that the model can identify are labeled.
[0088] ② Draw an image on the front end: Press the left mouse button at the defect in the image, move the mouse, and release the left mouse button to draw a rectangle. Select the defect type and generate label information, which includes the defect name and rectangle coordinates.
[0089] ③ The label information is transmitted to the backend in JSON format. On the backend, all the label information of the image is saved to the corresponding .xml annotation file in labelImg annotation file format. The content of the .xml annotation file includes the folder where the image file is located, the image file name, the image size, the defect name, the coordinates of the rectangle, etc.
[0090] 2) The sample inspection refers to checking whether there are any missing or incorrect defect labels on the images. For sample files with problems, they are deleted or re-labeled.
[0091] Before model training, the samples must be checked to prevent them from being invalid or incorrectly labeled, which could affect the model training results. Sample checks include compliance checks and accuracy checks.
[0092] ① Compliance Check: Check if each image file has a corresponding annotation file, whether the label names in the annotation files are defined in the predefined label files, and whether each annotation file has a corresponding image file. Problematic image files and annotation files can be deleted directly, or additional annotations can be added to the sample annotations.
[0093] ② Accuracy check: Display images and defect labels one by one, and manually judge whether the defect names are correct and the defect locations are appropriate. For sample files with problems, they can be deleted directly or re-labeled in the sample annotation.
[0094] 3) Sample file conversion refers to converting image files and annotation files that have undergone sample annotation and verification into TFRecord files;
[0095] Training models within the TensorFlow framework using TFRecord format files as input data is the fastest and most efficient method. Therefore, image and annotation files need to be converted to TFRecord format. Before converting the sample files, 20% of the training samples are randomly selected as evaluation samples, and then converted separately.
[0096] The sample file conversion steps are as follows:
[0097] ① First, combine multiple .xml annotation files into a single .csv file: Read the filename, image width, image height, image depth, defect name, and defect bounding box coordinates from each .xml annotation file, and save them to the .csv file;
[0098] ② Then convert the .csv file and image file to .tfrecord file: Read the label information of the sample from the .csv file and the image data from the image file using named tuples to form a TensorFlow training sample, and store it in the .tfrecord file in the form of a record.
[0099] Step 4: Model Training and Evaluation
[0100] The processed samples were used to train one full-defect model and 27 single-defect models. The full-defect model was trained using image samples labeled with all defects in the image, while the single-defect models were trained using image samples labeled with only the corresponding defects.
[0101] 1) Parameter settings
[0102] Before training the model, the model training parameters are configured based on the sample data and stored in the model configuration file. The training model parameters include the defect label mapping file, the pre-trained model detection point file, the training batch size, the number of training steps, the training sample input file, the evaluation sample data, the evaluation sample input file, and the maximum number of evaluations.
[0103] 2) Model Training
[0104] Model training uses the `load_model` method of the `keras.Model` class to load the defect recognition model, reads the model training parameters from the model configuration file, configures the model using the `compile` method, applies the TFRecord files of training and evaluation samples, trains the defect recognition model using the `fit` method, obtains the model's checkpoint file, and saves the model file using the `save` method.
[0105] Model training can be performed on multiple models sequentially at once. Because model training takes a long time, starting a new model training session will be prevented until the current model has completed its training.
[0106] During model training, the TensorBoard event accumulator is used to read computation process data from the model training record file and display relevant parameter graphs, including learning rate, classification loss, localization loss, total loss, and single-step training time, which facilitates targeted adjustment of training parameters.
[0107] 3) Model Evaluation
[0108] The model evaluation uses the `evaluate` method of the `keras.Model` class to evaluate the trained model. The TensorBoard event accumulator reads data from the model evaluation log file and displays the model performance parameters and graphs, including classification loss, localization loss, AP (Average Precision) and mAP (Mean Average Precision) for each defect class, and evaluation sample images. Based on these parameters, the model performance can be determined.
[0109] Step 5: Defect Identification
[0110] After training, the defect identification model can be integrated into inspection APP software and video monitoring software, and use the predict method of the keras.Model class in the TensorFlow framework or other framework methods to identify defects in inspection images or video monitoring images.
[0111] After inspection images or video surveillance images are identified by a defect identification model (including one full defect model and 27 single defect models), a comprehensive defect identification method is used to finally determine the defect type. These two steps can determine whether there are defects in the earth-rock dam in the inspection images and video surveillance images, what type of defects exist, and the confidence level of the existence of that type of defect. The higher the confidence level, the greater the probability that the defect exists.
[0112] The specific identification process of the inspection images or video surveillance images in step five is as follows:
[0113] 1) The full defect model and 27 single defect models in the defect recognition model are used to identify defects in the image respectively; the defect recognition model can identify 0 to n defects from the image, and the information of each defect includes type, coordinates and confidence level, etc.
[0114] 2) Then, the defect comprehensive identification method is used to determine the (comprehensive) confidence level of the identified defects. The defect identification results are more accurate, and the two models also play a role in mutual verification.
[0115] The defect comprehensive identification method updates the confidence level of each defect based on its information and the AP of each type of defect in each model. When the confidence level is greater than a confidence threshold, the defect is determined to exist in the image. The defect comprehensive identification process is shown in the appendix. Figure 6 The defect comprehensive identification method includes the following steps:
[0116] Step 1) Select each defect one by one as the current defect according to the order of the defect identification model and the order of defects identified by the defect identification model;
[0117] Step 2) Identify all defects in the subsequent defect identification model that are of the same type and location as the current defect, store the confidence level of these defects and the AP of this type of defect in the defect identification model, delete the defect from its respective defect identification model, and avoid selecting the defect again when processing defects in the defect identification model.
[0118] In step 2), whether the locations of the defects are the same is determined by the degree of overlap. The degree of overlap is calculated as shown in formula (1). When the degree of overlap is greater than 0.95, the defect is considered to be a defect. j and defects l For the same defect;
[0119]
[0120] Where x j1 and x j2 For defects j Position left and right coordinates, y j1 and y j2 For defects j Position, vertical coordinates, x l1 and x l2 For defects l Position left and right coordinates, y l1 and y l2 For defects l The vertical coordinates of the location, where w and h are the width and height of the image captured from the inspection image or video surveillance image;
[0121] An image may contain multiple defects of the same or different types. When each model identifies a defect, it provides the defect's location on the image (a rectangular coordinate), defect type, and confidence level (the higher the confidence level, the greater the probability that the defect exists). In order to calculate the (comprehensive) confidence level, the overlap degree is needed to determine whether the defects identified by different models are the same defect. The (comprehensive) confidence level is calculated for the same defect.
[0122] Step 3) Update the confidence of the current defect based on the confidence level given when the defect is identified by each defect identification model stored in Step 2) and the AP of this type of defect in each defect identification model;
[0123] In step 3), the confidence level is calculated as shown in formula (2). The confidence level threshold is set to 80%. When the confidence level of the defect is greater than 80%, it is determined that the defect exists in the image.
[0124]
[0125] Where n is the total number of models that identified the defect, and the confidence level is... p For defect identification model p The confidence level for identifying the defect.
[0126] When each model identifies a defect, it provides a confidence level for that defect. To improve identification accuracy, multiple models are used for identification. For the same defect identified by these models, the AP (average precision) of each model for identifying that type of defect is considered, and the (comprehensive) confidence level (equivalent to the average value) of the defect is calculated.
[0127] Step 4) If the confidence level of the current defect is greater than 80%, determine that the defect exists in the inspection image or video surveillance image; otherwise, the defect does not exist.
[0128] The method of this invention for identifying defects in earth-rock dams has the following advantages:
[0129] 1) Compared with manual identification methods, it has a high degree of intelligence, does not require experienced dam construction technicians, requires less workload, and has a fast identification speed;
[0130] 2) Compared with existing automatic identification methods, it requires fewer samples, has a wider range of applications, and is more practical.
[0131] 3) Traditional methods for classifying defects in earth-rock dams take into account the defect generation mechanism, structural morphology, severity, and size. Based on this, this invention mainly classifies earth-rock dam defects from an image perspective, reducing the omission or misidentification of defects in earth-rock dam defect identification.
[0132] 4) This invention applies transfer learning technology, retains the feature extractor in the pre-trained model unchanged, and only retrains its classifier. The training samples are small, and the difficulty of sample collection is greatly reduced, so it can be used to identify various earth-rock dam defects.
[0133] 5) The full defect model and single defect model are used to identify defects in earth-rock dams in the image. Then, the presence and type of defects are judged based on the confidence level, which has high recognition accuracy.
Claims
1. A method for image recognition of defects in earth-rock dams based on small samples, characterized in that, Includes the following steps: Step 1: Classify the defects of earth-rock dams from an image perspective to obtain the defect types; Step 2: Select a pre-trained model and establish a defect identification model based on the defect types obtained in Step 1; in Step 2, the defect identification model includes a full defect model containing all defect types and several single defect models containing only one defect type. Step 3: Select defect image samples of earth-rock dams and process them to make them suitable for training and evaluation of the defect recognition model; Step 4: Use the samples processed in Step 3 to train and evaluate the defect identification model established in Step 2; Step 5: Integrate the defect identification model obtained in Step 4 into the inspection APP software and video monitoring software to perform defect image recognition of earth-rock dams; In step five, after the inspection image or video surveillance image is identified by the defect identification model, the defect type is determined by the comprehensive defect identification method. The defect comprehensive identification method includes the following process: Step 1) Select each defect one by one as the current defect according to the order of the defect identification model and the order of defects identified by the defect identification model; Step 2) Find all defects in the subsequent defect identification model that are the same type and location as the current defect, store the confidence level of these defects and the AP of this type of defect in the defect identification model, delete the defect from its corresponding defect identification model, and avoid selecting the defect again when processing defects in the defect identification model. Defect AP refers to the average accuracy of the defect. Step 3) Update the confidence of the current defect based on the confidence level given when the defect is identified by each defect identification model stored in Step 2) and the AP of this type of defect in each defect identification model; Step 4) When the confidence level of the current defect is greater than 80%, confirm that the defect exists in the inspection image or video surveillance image; Otherwise, the defect does not exist; In step 2), whether the locations of the defects are the same is determined by the degree of overlap. The degree of overlap is calculated as shown in formula (1). When the degree of overlap is greater than 0.95, the defect is considered to be a defect. j and defects l For the same defect: (1) Where x j1 and x j2 For defects j Position left and right coordinates, y j1 and y j2 For defects j Position coordinates, x l1 and x l2 For defects l Position left and right coordinates, y l1 and y l2 For defects l The vertical coordinates of the location, where w and h are the width and height of the image captured from the inspection image or video surveillance image; In step 3), the confidence level is calculated as shown in formula (2). The confidence level threshold is set to 80%. When the confidence level of the defect is greater than 80%, it is determined that the defect exists in the image. (2) Where n is the total number of models that identified the defect, and the confidence level is... p For defect identification model p The confidence level for identifying the defect.
2. The method for image recognition of earth-rock dam defects based on small samples according to claim 1, characterized in that, In step one, the specific method for classifying defects in earth-rock dams is as follows: based on the "Technical Specification for Safety Monitoring of Earth-Rock Dams", the defect types of earth-rock dams are analyzed, and from the perspective of images, the similarity within the defect category is considered, and defects within the same defect type that can be visually identified as different are reclassified. Considering the differences between defect categories, different defect types that cannot be visually identified are merged.
3. The method for image recognition of earth-rock dam defects based on small samples according to claim 1, characterized in that, In step two, the TensorFlow framework is used, and the Faster R-CNN Inception ResNet pre-trained model is selected. The object detection method of the Faster R-CNN Inception ResNet pre-trained model is Faster R-CNN, the feature extraction network is ResNet, and the Inception module is used. The training dataset is COCO.
4. The method for image recognition of earth-rock dam defects based on small samples according to claim 3, characterized in that, Step three, the sample processing procedure includes sample labeling, sample checking, and sample file conversion: ①The sample annotation refers to generating defect label information based on defects on image samples and storing the defect label information in an annotation file; ②The sample inspection refers to checking whether there are any missing or incorrect defect labels on the images. For sample files with problems, they are deleted or re-labeled. ③ The sample file conversion refers to converting the image files and annotation files that have undergone sample annotation and sample checking into TFRecord files; before performing the sample file conversion, the entire sample that has undergone sample annotation and sample checking is used as the training sample, and a portion of the samples are randomly selected as the evaluation sample.
5. The method for image recognition of earth-rock dam defects based on small samples according to claim 4, characterized in that, The sample annotation method is a B / S architecture-based sample annotation method developed based on the labelImg function and annotation file format; the sample annotation method includes the following steps: ① When an image is used for a full defect model, all defects in the image are labeled; when used for a single defect model, only the defects that the model can identify are labeled. ② Draw the image on the front end: Mark the defects in the image with a rectangle, obtain the coordinates of the top left and bottom right corners of the rectangle, select the defect type, and generate label information; ③ The label information is transmitted to the backend in JSON format, and the backend saves all the label information of the image to the corresponding .xml annotation file in labelImg annotation file format.
6. The method for image recognition of earth-rock dam defects based on small samples according to claim 5, characterized in that, In step four, when training the defect identification model, transfer learning technology is applied to retain the feature extractor in the pre-trained model and only retrain its classifier. The full-defect model is trained using all samples, while the single-defect model is trained using only samples of the corresponding defect.
7. The method for image recognition of earth-rock dam defects based on small samples according to claim 6, characterized in that, In step four: 1) Defect identification model training, the specific training process is as follows: The defect identification model is loaded using the `load_model` method of the `keras.Model` class, and the model training parameters are read from the model configuration file. The model is configured using the `compile` method. The defect identification model is trained using the `fit` method with the TFRecord files of the training and evaluation samples to obtain the model's checkpoint file. The model file is saved using the `save` method. During model training, TensorBoard's event accumulator is used to read computation process data from the model training log file and display relevant parameters and graphs. 2) Evaluation of the defect identification model, specifically the training process: The trained defect recognition model is evaluated using the `evaluate` method of the `keras.Model` class. Data is read from the model evaluation log file using the TensorBoard event accumulator, and the model performance parameters and graphs are displayed.