Quantitative method and device for detecting damage and exposed reinforcement defects in concrete structure bridges

Abstract

The present invention discloses a method and device for quantitative detection of concrete structure bridge damage and exposed steel bar defects, and relates to the technical field of bridge defect detection quantification and image processing. The method includes constructing a sample library of concrete damage and exposed steel bar defects; constructing a multi-scale multi-branch concrete defect detection network based on a target detection algorithm; introducing a loss function combining IoU and IoG in the constructed multi-scale multi-branch concrete defect detection network; performing target object defect detection, and adopting a region-based target detection evaluation method to perform detection frame clustering through a clustering algorithm; segmenting the detected defect position, and calculating the pixel size of the defect; and completing the conversion of the pixel size to the actual size of the real world through camera calibration technology and three-dimensional measurement technology. The present application can achieve segmentation quantification with a high accuracy rate on the basis of ensuring accurate detection, and realize intelligent detection of concrete structure bridge damage and exposed steel bar defects.

Description

Technical Field

[0001] The present application relates to the technical field of bridge defect detection quantification and image processing, and specifically to a method and device for detecting and quantifying damage and exposed reinforcement defects in concrete structure bridges. Background Art

[0002] As an important transportation infrastructure, bridges are of great significance to social and economic development. Structural damage and exposed reinforcement diseases of concrete bridges are common surface diseases, which have important reference value in evaluating the service performance of bridges. Up to now, the number of highway bridges has exceeded 1.3 million, of which nearly 8,000 are extra-large bridges, and concrete bridges account for a considerable proportion. Under the current background, the demand for bridge surface disease detection is large, the task is heavy and the precision is required, which brings significant challenges in terms of detection equipment, detection methods and detection accuracy. With the rapid development of computer vision and artificial intelligence technology, machine vision is gradually becoming one of the emerging means of bridge disease detection due to its significant advantages such as non-contact, non-destructive, large range, high precision and long distance.

[0003] At present, many achievements have been made in the research on concrete damage and exposed reinforcement disease detection based on machine vision, but there are still difficulties and challenges in the following aspects:

[0004] (1) Construction of a high-quality bridge disease sample library: At present, the original image data of bridge diseases are usually scattered and temporarily stored in only a few road bridge maintenance units, making it difficult to obtain sufficient original disease sample data. In addition, the data collected from various regular inspection systems usually come from different cameras or mobile phones. Due to differences in equipment, shooting distance, shooting angle and on-site lighting conditions, the original data samples have low resolution, low light / low light, low contrast and a small proportion of main structures (the background has a large proportion, such as the sky, bare land, trees, etc.). Therefore, these original samples need to go through a full process of screening, enhancement processing and re-screening. At present, these processes mainly rely on manual operations, which are inefficient. It is urgent to develop a simple sample rough screening automatic processing program;

[0005] (2) Complexity of concrete structure damage and exposed rebar characteristics: Concrete structure damage and exposed rebar diseases exhibit multi-scale (different sizes) morphological characteristics. Local damage and exposed rebar also have characteristics such as connectivity and fragmentation. The current mainstream object detection and segmentation networks are difficult to effectively extract these multi-scale, fragmented complex features, resulting in low algorithm detection accuracy.

[0006] (3) Limitations of detection and segmentation methods: Currently, the detection and quantification of concrete damage and exposed rebar defects usually adopt a single target detection or segmentation method. However, a single detection method has the problem of limited accuracy and inability to quantify, while a single segmentation method has the problem of large computational complexity and low segmentation accuracy. Summary of the invention

[0007] The present application provides a method and device for quantitative detection of damage and exposed rebar defects in concrete structure bridges, which can achieve segmentation quantification with a high accuracy rate while ensuring accurate detection, thereby realizing intelligent detection of damage and exposed rebar defects in concrete structure bridges.

[0008] In a first aspect, an embodiment of the present application provides a method for detecting and quantifying damage and exposed steel bar defects in a concrete structure bridge, the method comprising:

[0009] Acquire original images of concrete structure damage and exposed reinforcement through multiple data sources, and build a sample library of concrete damage and exposed reinforcement diseases;

[0010] Based on the target detection algorithm, multi-branch parallel dilated convolution is used as the backbone network, multi-branch auxiliary is used as the neck network, and the head information network is introduced to construct a multi-scale and multi-branch concrete disease detection network;

[0011] A loss function combining IoU and IoG is introduced into the constructed multi-scale and multi-branch concrete defect detection network;

[0012] The target object's defect detection is realized through a multi-scale and multi-branch concrete defect detection network, and a region-based target detection evaluation method is adopted to cluster the detection frames through a clustering algorithm;

[0013] Segment the detected disease location based on the segmentation model and calculate the pixel size of the disease;

[0014] For the calculated pixel size, the conversion from pixel size to actual size in the real world is completed through camera calibration technology and three-dimensional measurement technology.

[0015] In combination with the first aspect, in one implementation, the method of acquiring original images of concrete structure damage and exposed reinforcement through multiple data sources and constructing a sample library of concrete damage and exposed reinforcement disease specifically includes:

[0016] Acquire images of concrete structure damage and exposed reinforcement from multiple data sources to establish a set of original images of the damage, wherein the data sources include inspection data of a bridge disease inspection system, active shooting data, literature data, and network data;

[0017] The images in the original disease image set are judged to see whether the image resolution meets the requirements, whether the image brightness meets the requirements, whether the image contrast meets the requirements, and whether the proportion of the main structure meets the requirements, and the images that meet the requirements are placed in the created original disease sample library;

[0018] Performing transformation processing on the images in the disease original image set, and constructing a disease extended sample library based on the transformed images, wherein the transformation processing includes rotation, mirroring, and color change;

[0019] The images in the original disease sample library and the disease extended sample library are annotated to establish a disease sample library of concrete damage and exposed reinforcement.

[0020] In combination with the first aspect, in one implementation,

[0021] The target detection algorithm is based on the multi-branch parallel hole convolution as the backbone network, the multi-branch auxiliary as the neck network, and the head information network is introduced to construct a multi-scale multi-branch concrete disease detection network, which specifically includes:

[0022] Based on the YOLO, RT-DETR or Faster-RCNN target detection algorithm, the preset multi-branch parallel dilated convolution is used as the backbone network, the preset multi-branch auxiliary is used as the neck network, and the head information network is introduced;

[0023] Combining multi-branch parallel atrous convolution and multi-branch auxiliary, a multi-scale multi-branch concrete disease detection network for concrete damage and exposed reinforcement target detection is created;

[0024] The multi-scale multi-branch concrete disease detection network includes three parts: backbone network, neck network and head network;

[0025] For the multi-scale and multi-branch concrete defect detection network, the input image to be detected first passes through the backbone network, which consists of four stages, each of which corresponds to a feature extraction network of multi-branch parallel dilated convolutions with different numbers of branches. The multi-branch auxiliary is designed as a neck network. The SAF module in the neck network is used to extract multi-scale features from the backbone network, perform preliminary auxiliary fusion in the shallow layer of the neck network, and collect gradient information, thereby guiding the head network to obtain diversified output information across three resolutions. Finally, the head network predicts the object bounding box and its corresponding category based on the feature map of each scale to calculate its loss.

[0026] In combination with the first aspect, in one implementation, using the preset multi-branch parallel dilated convolution as a backbone network specifically includes:

[0027] For the expansion rate , the convolution kernel size is The dilated convolution, receptive field The calculation method is:

[0028] ;

[0029] For the multi-branch parallel dilated convolution with four branches, a kernel size of 1 is added before the dilated convolution of each branch. , , , Ordinary convolution of

[0030] The kernel size is The number of channels is reduced by ordinary convolution, and after extracting features of different scales by using dilated convolutions with different expansion rates, channel splicing is performed. Before channel splicing, the features obtained by the four branches are:

[0031]

[0032]

[0033]

[0034]

[0035] in, represents the input features, , , , represents the features extracted for the four branches respectively, represents the convolution operation, , , , The kernel sizes are , , , Ordinary convolution, , , , The kernel sizes are , the expansion rate is , , , Atrous convolution;

[0036] Add a step-by-step fusion method before channel splicing, specifically:

[0037]

[0038] in, represents the output features, Represents a channel concatenation operation.

[0039] In combination with the first aspect, in one implementation, the using the preset multi-branch auxiliary as the neck network specifically includes:

[0040] The shallow feature-assisted fusion SAF network structure is constructed and applied, and the output result after applying the shallow feature-assisted fusion SAF network structure is:

[0041]

[0042] in, , Represents feature maps at different resolutions, Indicates that brothers are shallow, represents the upsampling operation, express Downsampling convolutions with batch normalization layers, represents the SiLU activation function, Indicates the number of control channels convolution, represents across shallow high-resolution layers, Indicates the fusion of features extracted by the multi-branch feature extraction network;

[0043] A high-dimensional feature-assisted fusion AAF network structure is constructed and applied, and the output result after applying the high-dimensional feature-assisted fusion AAF network structure is:

[0044]

[0045] in, Indicates the current network feature layer, represents a shallow low-resolution layer, Represents the previous network feature layer.

[0046] In combination with the first aspect, in one implementation, a loss function combining IoU and IoG is introduced into the constructed multi-scale multi-branch concrete disease detection network, wherein the specific construction process of the loss function combining IoU and IoG includes:

[0047] To perform IoU calculation, given two bounding boxes box1 and box2, define the coordinates of the upper left corner of the bounding box box1 as , the coordinates of the lower right corner are , the coordinates of the upper left corner of the bounding box box2 are , the coordinates of the lower right corner are ,in, Indicates the width of the bounding box box1, Indicates the height of the bounding box box1, Indicates the width of the bounding box box2, Indicates the height of the bounding box box2;

[0048] Calculate the coordinates of the upper left corner of the intersection of bounding boxes box1 and box2 , lower right corner coordinates ;

[0049] Calculate the area of ​​the intersection of bounding boxes box1 and box2 :

[0050]

[0051] Calculate the area of ​​the union of bounding boxes box1 and box2 :

[0052]

[0053] Calculate the intersection-union ratio :

[0054]

[0055] in, Indicates the value corresponding to IoU;

[0056] Perform IoG calculation, given the real bounding box gt_box and the predicted bounding box pred_box, and define the upper left corner coordinate of the real bounding box gt_box as , the coordinates of the lower right corner are , the coordinates of the upper left corner of the predicted bounding box pred_box are , the coordinates of the lower right corner are ;

[0057] The upper left corner coordinates of the intersection of the real bounding box gt_box and the predicted bounding box pred_box are calculated as , the coordinates of the lower right corner of the intersection of the real bounding box gt_box and the predicted bounding box pred_box are ;

[0058] The width of the intersection of the true bounding box gt_box and the predicted bounding box pred_box is calculated as :

[0059] ;

[0060] The height of the intersection of the real bounding box gt_box and the predicted bounding box pred_box is calculated as :

[0061] ;

[0062] Calculate the area of ​​the intersection of the true bounding box gt_box and the predicted bounding box pred_box :

[0063] ;

[0064] Calculate the area of ​​the true bounding box gt_box :

[0065]

[0066] Calculate the intersection-union ratio :

[0067]

[0068] in, Indicates the value corresponding to IoG;

[0069] Design smoothing functions to control the response speed of the loss function in different situations and its sensitivity to abnormal situations:

[0070]

[0071] in, represents a smooth function, Indicates the value corresponding to IoU or IoG, Represents the threshold parameter of the smoothing function, and its value range is ;

[0072] Construct a joint loss function combining IoU and IoG, which includes the loss between the real bounding boxes and the loss between the predicted bounding boxes. Specifically:

[0073]

[0074]

[0075] in, represents the loss between the ground-truth bounding boxes, Indicates For the ground-truth bounding box, Indicates For the predicted bounding box, represents the loss between predicted bounding boxes, Represents the IoG between the real bounding box and the predicted bounding box, represents the IoU between predicted bounding boxes, and represents the threshold parameter of the smoothing function, and Indicates the number of target detection boxes.

[0076] In combination with the first aspect, in one implementation, the region-based target detection evaluation method is used to cluster detection frames using a clustering algorithm, specifically including:

[0077] When there is only one target detection frame, expand the original target detection frame by several pixels as the clustering target frame;

[0078] When there are at least 3 object detection boxes, the centroid list is obtained by calculating the center points of the bounding boxes, and the silhouette coefficient is used to determine the number of clusters;

[0079] When there are only two target detection boxes, the number of clusters is determined based on the position and size of the detection boxes and the image size;

[0080] Among them, after completing the clustering of the prediction box and the label box, a cross prediction box method is created based on the IoU intersection-over-Union judgment method to achieve subtle detection and judgment of concrete damage and exposed reinforcement defects.

[0081] In combination with the first aspect, in one implementation, determining the number of clusters based on the position and size of the detection frame and the image size specifically includes:

[0082] Calculate the distance and direction of the center of mass, and establish a coordinate system with the center of mass on the left. , vertical axis , the centroid connection line is calculated from the centroid relative to Axis angle , the length of the centroid connecting line is ,when When the absolute value of is less than or equal to 1, calculate the lateral component , ,when When the absolute value of is greater than 1, calculate the longitudinal component , ;

[0083] Get the target frame and image size. For the two target detection frames, get the pixel coordinates of the four corners of each frame and define the The axis direction is long, along The axis direction is high, and the pixel length of the first target detection box on the left is calculated to be , pixel width is , the pixel length of the second target detection box on the right is calculated to be , pixel width is , the image size is along The size of the axis direction is ,along The size of the axis direction is ;

[0084] for When the absolute value of is less than or equal to 1, the calculated lateral component ,like , then the number of clusters is 1, if and , then the number of clusters is 1, in other cases, the number of clusters is 2;

[0085] for When the absolute value of is greater than 1, the calculated longitudinal component ,like , then the number of clusters is 1, if and , then the number of clusters is 1, in other cases, the number of clusters is 2;

[0086] in, and represents the parameter, and and The values ​​of are as follows:

[0087]

[0088] .

[0089] In combination with the first aspect, in one implementation, based on the IoU intersection-over-union determination method, a cross prediction frame method is created to achieve fine detection and determination of concrete damage and exposed reinforcement defects, specifically including:

[0090] Set the threshold of IoU and the threshold of cross prediction box, and calculate the IoU and cross prediction box IoP values. The calculation method of the cross prediction box value is:

[0091]

[0092] in, Represents the value corresponding to the cross prediction box, Represents the intersection area of ​​the predicted box and the true box, Represents the area of ​​the prediction box;

[0093] Based on the calculation results, when the calculated IoU is greater than or equal to the set IoU threshold, it indicates that the detection box matches the target box and the target is detected. When the calculated IoU is less than the set IoU threshold, the detection box is separated, contained or intersected with the target box. When the calculated cross prediction box value is greater than the set cross prediction box threshold, it indicates that the target is detected. When the calculated cross prediction box value is less than the set cross prediction box threshold, it indicates that the target is not detected.

[0094] Calculate the indicator using precision and recall As indicators, specifically:

[0095]

[0096]

[0097] in, , , They represent the accurately detected objects, the falsely detected objects, and the missed detected objects respectively;

[0098] For accuracy , perform region-based evaluation Calculation of recall , perform region-based evaluation The calculation is as follows:

[0099]

[0100]

[0101] in, , , They represent the accurately detected objects, falsely detected objects, and missed detected objects after clustering, respectively.

[0102] In combination with the first aspect, in one implementation, segmenting the detected disease location based on the segmentation model and calculating the pixel size of the disease specifically includes:

[0103] Defect detection is realized based on the constructed multi-scale multi-branch concrete defect detection network, and detection frame clustering is performed through a clustering algorithm, the location information of the clustered defect detection frame is obtained, and the obtained location information of the defect detection frame is saved, wherein the location information of the defect detection frame includes the pixel coordinates of the upper left corner and the pixel coordinates of the lower right corner;

[0104] The acquired disease detection frame position information is used as the input of the segmentation model to guide the segmentation model to complete the disease segmentation within the disease detection clustering frame;

[0105] Based on the segmented mask contour boundary, the pixel area of ​​damaged and exposed reinforcement defects is calculated.

[0106] In combination with the first aspect, in one implementation, the calculated pixel size is converted to an actual size in the real world by using a camera calibration technology and a three-dimensional measurement technology, specifically including:

[0107] Perform camera calibration to remove radial distortion and tangential distortion while obtaining the intrinsic and extrinsic parameter matrices;

[0108] Convert the image coordinate system, camera coordinate system and world coordinate system. The image coordinate system takes the upper left corner of the image as the origin and the horizontal right corner as the The positive direction of the axis is vertically downward. The coordinate system of the positive direction of the axis sets the coordinates of a point in the image to ; The camera coordinate system takes the optical center of the camera as its origin. The axis points to the right of the camera, The axis points to the top of the camera, and the axis points to the viewing direction of the camera. The coordinates of a point in the camera coordinate system are set to ;The world coordinate system is a global reference system used to describe the position and orientation of objects in the real world;

[0109] Set the coordinates of an object in the world coordinate system to , the image coordinates Normalize it and get :

[0110]

[0111]

[0112] in, represents the focal length of the camera, Represents the optical center coordinates, that is, the origin of the image coordinate system;

[0113] Convert the image coordinate system to the camera coordinate system. Specifically:

[0114]

[0115] in, Represents the coordinates in the camera coordinate system, Represents the external parameter matrix of the camera, specifically the position matrix of the camera in space;

[0116] The coordinates in the camera coordinate system Convert to world coordinate system , specifically:

[0117]

[0118] in, represents the transpose of the rotation matrix, Represents the external parameter matrix of the camera, specifically the direction matrix of the camera in space;

[0119] After completing the transformation from image coordinates to world coordinates, the actual area of ​​the real world is calculated.

[0120] In combination with the first aspect, in one implementation, after completing the conversion from image coordinates to world coordinates, the actual area of ​​the real world is calculated, wherein the specific steps of calculating the actual area of ​​the real world include:

[0121] Set the actual area of ​​the target in the real world to , perform the cross product calculation of the vector and represent each point as a vector, where the point Represented as a vector ;

[0122] Perform cumulative cross product and calculation for adjacent points and Point , calculation point and Point The cross product vector , sum the cross product vectors of all adjacent points to get a total cross product vector ;

[0123] Based on the total cross product vector obtained , calculate the actual area of ​​the target in the real world:

[0124] .

[0125] In combination with the first aspect, in one implementation, when a camera is used to photograph a diseased area, the actual area in the real world is calculated, specifically including:

[0126] Calculate the ground sampling spacing, specifically:

[0127]

[0128] in, represents the ground sampling spacing, Indicates the camera shooting distance. Indicates the pixel size, Indicates the focal length of the camera;

[0129] Calculate the actual area of ​​the target in the real world, specifically:

[0130]

[0131] in, represents the actual area of ​​the target in the real world, Represents the total number of pixel areas of the object.

[0132] In a second aspect, an embodiment of the present application provides a quantification device for detecting damage and exposed reinforcement defects in a concrete structure bridge, including:

[0133] A construction module is used to obtain original images of concrete structure damage and exposed reinforcement through multiple data sources, and to construct a sample library of concrete damage and exposed reinforcement diseases;

[0134] Create a module for constructing a multi-scale and multi-branch concrete disease detection network based on the target detection algorithm, using multi-branch parallel dilated convolution as the backbone network, multi-branch auxiliary as the neck network, and introducing the head information network;

[0135] A module is introduced to introduce a loss function combining IoU and IoG in the constructed multi-scale and multi-branch concrete disease detection network;

[0136] The detection module is used to realize the disease detection of the target object through a multi-scale and multi-branch concrete disease detection network, and adopts a region-based target detection evaluation method to perform detection frame clustering through a clustering algorithm;

[0137] A calculation module, which is used to segment the detected disease location based on the segmentation model and calculate the pixel size of the disease;

[0138] The conversion module is used to convert the calculated pixel size into the actual size in the real world through camera calibration technology and three-dimensional measurement technology.

[0139] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0140] It has strong adaptability and robustness, and can achieve segmentation quantification with high accuracy on the basis of ensuring accurate detection, alleviate the problem of insufficient number and low quality of original samples of bridge defects, and effectively solve the problems of insufficient automation and intelligence, poor adaptability and low detection efficiency in bridge structure defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] Figure 1 A schematic diagram of the process of the quantitative method for detecting damage and exposed reinforcement defects in concrete structure bridges in this application;

[0142] Figure 2 This is the overall architecture diagram of MMFPN;

[0143] Figure 3This is the structure diagram of multi-branch parallel dilated convolution MBPD;

[0144] Figure 4 This is the overall structure diagram of multi-branch parallel dilated convolution;

[0145] Figure 5 This is the network structure diagram of shallow feature assisted fusion SAF;

[0146] Figure 6 This is the network structure diagram of high-dimensional feature assisted fusion AAF;

[0147] Figure 7 Schematic diagram of the positional relationship between the detection box and the target box. DETAILED DESCRIPTION

[0148] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0149] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0150] On the first aspect, an embodiment of the present application provides a quantitative method for detecting damage and exposed rebar defects in concrete structure bridges, which realizes intelligent detection of damage and exposed rebar defects in concrete structure bridges based on a two-stage method. This method, based on the establishment of a high-quality sample library, aims to achieve efficient detection and quantification of multi-scale, connected, and fragmented damage and exposed rebar defects in complex backgrounds, while narrowing the gap between the quantification of image coordinates and real-world coordinates.

[0151] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the quantitative method for detecting damage and exposed reinforcement defects in concrete structure bridges in this application. Figure 1 As shown in the figure, the quantitative methods for detecting damage and exposed reinforcement defects in concrete structure bridges include:

[0152] S1: Acquire original images of concrete structure damage and exposed reinforcement through multiple data sources, and build a sample library of concrete damage and exposed reinforcement;

[0153] S2: Based on the target detection algorithm, multi-branch parallel dilated convolution is used as the backbone network, multi-branch auxiliary as the neck network, and the head information network is introduced to construct a multi-scale and multi-branch concrete disease detection network;

[0154] Specifically, based on target detection algorithms such as YOLO, RT-DETR, and Faster-RCNN, a multi-scale and multi-branch concrete disease detection network (Multi-scale and Multi-Branch Feature Pyramid Networks, MMFPN) is designed, which uses multi-branch parallel dilated convolution (MBPD) as the backbone network, multi-branch auxiliary (MAFPN) as the neck network, and introduces the head information network Head. Finally, the head information network Head can be guided to obtain diversified output information across three resolutions.

[0155] S3: A loss function combining IoU and IoG is introduced into the constructed multi-scale and multi-branch concrete defect detection network; IoU, namely Intersection over Union, is an algorithm for calculating the overlapping ratio of different images; IoG, namely Interactive Object Segmentation, is an interactive object segmentation algorithm.

[0156] S4: The target object’s disease detection is realized through a multi-scale and multi-branch concrete disease detection network, and a region-based target detection evaluation method is adopted to cluster the detection frames through a clustering algorithm;

[0157] That is, after MMFPN detects the target object to be detected, a region-based target detection evaluation method is proposed. This target detection evaluation method combines the clustering algorithm of K-means clustering and improves the prediction box IoU to realize clustering evaluation of overlapping targets and fragmented small targets, effectively improving the detection effect and better meeting the needs of bridge detection.

[0158] S5: Segment the detected disease location based on the segmentation model, and calculate the pixel size of the disease;

[0159] Specifically, based on the detection of the disease location, the pixel size of the diseased part is segmented and calculated by combining segmentation models such as SAM (Segment Anything Model, image segmentation task, model) and SAM's lightweight version Efficient SAM.

[0160] S6: For the calculated pixel size, the conversion from the pixel size to the actual size in the real world is completed through camera calibration technology and three-dimensional measurement technology.

[0161] It should be noted that the above steps S2 to S4 represent the first stage, and steps S5 and S6 represent the second stage. After completing the conversion analysis from pixel size to actual size in the real world, the detection and quantification of concrete structure damage and exposed reinforcement disease can be achieved.

[0162] Furthermore, in one embodiment, original images of concrete structure damage and exposed reinforcement are obtained through multiple data sources to construct a sample library of concrete damage and exposed reinforcement defects, specifically including:

[0163] S101: Acquire images of concrete structure damage and exposed reinforcement from multiple data sources to establish a set of original images of the damage, wherein the data sources include inspection data of a bridge disease inspection system, active shooting data, literature data, and network data;

[0164] For example, historical inspection image data of bridge defects can be crawled from the bridge defect inspection system, the latest defect data can be collected based on existing projects using mobile phones, cameras, drones and other equipment, public data can be collected from papers and the Internet, and data from multiple parties can be collected and organized to establish a set of original defect images.

[0165] S102: Determine whether the image resolution, image brightness, image contrast, and main structure ratio of the images in the disease original image set meet the requirements, and put the images that meet the requirements into the created original disease sample library;

[0166] Specifically, a sample coarse screening module can be designed, which is used to determine whether the image resolution meets the requirements, whether the image brightness meets the requirements, whether the image contrast meets the requirements, and whether the proportion of the main structure meets the requirements. Images that meet the requirements will enter the original disease sample library.

[0167] For data whose resolution does not meet the requirements, image super-resolution processing is performed, and the data enters the sample coarse screening module again. Those that pass enter the image super-resolution extended disease sample library, and those that fail are discarded; for data whose brightness does not meet the requirements, low-light image enhancement is performed, and the data enters the sample coarse screening module again. Those that pass enter the low-light image enhancement extended disease sample library, and those that fail are discarded; for data whose contrast does not meet the requirements, contrast enhancement is performed, and the data enters the sample coarse screening module again. Those that pass enter the contrast enhancement extended disease sample library, and those that fail are discarded; for data whose main structure accounts for a small proportion and does not meet the requirements, large model segmentation models including but not limited to SAM are used for segmentation and cropping, and the data enters the sample coarse screening module again. Those that pass enter the large model segmentation and cropping extended disease sample library, and those that fail are discarded.

[0168] S103: performing transformation processing on the images in the disease original image set, and constructing a disease extended sample library based on the transformed images, wherein the transformation processing includes rotation, mirroring, and color change;

[0169] S104: Annotate the images in the original disease sample library and the disease extension sample library, so as to establish a concrete damage and exposed steel bar disease sample library. Specifically, the original disease sample library and the disease extension sample library are annotated with data under a unified rule, and finally a high-quality concrete damage and exposed steel bar disease sample library is established.

[0170] Furthermore, in one embodiment, based on the target detection algorithm, multi-branch parallel dilated convolution is used as the backbone network, multi-branch auxiliary is used as the neck network, and the head information network is introduced to construct a multi-scale multi-branch concrete disease detection network, which specifically includes:

[0171] S201: Based on the YOLO, RT-DETR or Faster-RCNN target detection algorithm, the preset multi-branch parallel dilated convolution is used as the backbone network, the preset multi-branch auxiliary is used as the neck network, and the head information network is introduced;

[0172] S202: Combining multi-branch parallel atrous convolution and multi-branch assistance, a multi-scale multi-branch concrete disease detection network for concrete damage and exposed reinforcement target detection is created;

[0173] Specifically, MBPD and MAFPN are combined to design a concrete damage and exposed reinforcement target detection network MMFPN which is suitable for concrete damage, various missed detection scales, and complex characteristics such as disease connectivity, overlap, and fragmentation.

[0174] The overall architecture of MMFPN is shown in the figure below. Figure 2 As shown, the overall architecture of the single-stage object detector is decomposed into three main parts, namely, the multi-scale and multi-branch concrete disease detection network includes three parts: the backbone network, the neck network and the head network.

[0175] For the multi-scale multi-branch concrete disease detection network, the input image to be detected first passes through the backbone network, which consists of four stages, such as Figure 2The P2, P3, P4, and P5 marked in each stage correspond to the feature extraction network of multi-branch parallel dilated convolution with different numbers of branches. The multi-branch auxiliary design is the neck network. The SAF module in the neck network is used to extract multi-scale features from the backbone network, and perform preliminary auxiliary fusion in the shallow layer of the neck network, as well as collect gradient information, thereby guiding the head network to obtain diversified output information across three resolutions. Finally, the head network predicts the object bounding box and its corresponding category based on the feature map of each scale to calculate its loss. Multi-branch parallel dilated convolution can also be expressed as MBPD-N, where N is used to represent the number of branches.

[0176] Specifically, the multi-branch auxiliary design is used as the neck network. In the first bottom-up path, the SAF (Superficial Assisted Fusion) module is responsible for extracting multi-scale features from the backbone network and performing preliminary auxiliary fusion in the shallow layer of the neck. At the same time, AAF (Advanced Assisted Fusion) collects gradient information from each layer through denser connections in the second top-down path, and finally guides the head network to obtain diversified output information across three resolutions. Both of the above structures use the MBPD-N module for feature extraction, which uses dynamically sized convolution kernels to achieve adaptive receptive fields. Finally, the detection head predicts the object bounding box and its corresponding category based on the feature map of each scale to calculate its loss. Figure 2 Input represents input, upsample represents upsampling, downsample represents downsampling, copy represents copy, reg loss, cls loss, and dflloss represent losses.

[0177] Furthermore, in one embodiment, a preset multi-branch parallel dilated convolution is used as a backbone network, specifically including:

[0178] S20111: For expansion rate , the convolution kernel size is The dilated convolution, receptive field The calculation method is:

[0179] ;

[0180] S20112: For the multi-branch parallel dilated convolution with four branches, a kernel size of 1 is added before the dilated convolution of each branch. , , , Ordinary convolution of

[0181] Specifically, when the multi-branch parallel dilated convolution uses four branches (in this case, the multi-branch parallel dilated convolution is denoted as MBPD-4), the convolution kernel sizes are , the expansion rate is , , , The dilated convolutions produce receptive fields of sizes respectively. , , , , the dilated convolution increases the receptive field capacity by the expansion rate without increasing the number of parameters, which is very useful for capturing a wider range of contextual information; correspondingly, MBPD-3 adopts three branches, with convolution kernel sizes of , the expansion rate is , , The dilated convolutions produce receptive fields of sizes respectively. , , ; The structures of MBPD-2 and MBPD-1 are similar.

[0182] Considering that the hole convolution does not sample the hole part, the extracted information is not continuous. Therefore, in order to extract the feature information of the hole part, the multi-branch parallel hole convolution MBPD structure proposed in this application is as follows: Figure 3 As shown in Figure 2, the overall structure of multi-branch parallel dilated convolution is as follows: Figure 4 shown. Figure 3 The conv in it means convolution.

[0183] MBPD-4 adds a kernel size of , , , Ordinary convolution, Figure 3 The points in the grid box in the lower middle represent the features extracted by the hole convolution, and the rest are the features extracted by the ordinary convolution. After adding the ordinary convolution, the information of the hole part can be extracted by the ordinary convolution, while enhancing the continuity of information, obtaining receptive fields of different sizes, and thus extracting the features of objects of different scales.

[0184] It should be noted that in convolutional neural networks, the size of the convolution kernel determines the size of the convolution receptive field, and receptive fields of different sizes are conducive to the detection of objects of different scales. The convolution kernel can produce The receptive field of the size is suitable for detecting small-sized concrete damage and exposed steel bar targets at a long distance, but not suitable for detecting large and distant targets. For medium-sized concrete damage and exposed steel bar targets, use The receptive field of a larger size is more suitable, but for small or distant targets, The receptive field of different sizes includes background information other than the target, and the receptive field is too large. The multi-branch parallel dilated convolution structure developed and designed designs receptive fields of different sizes according to targets of different sizes to obtain richer target features and achieve the purpose of improving target detection accuracy.

[0185] S20113: Using a kernel size of The number of channels is reduced by ordinary convolution, and after extracting features of different scales by using dilated convolutions with different expansion rates, channel splicing is performed. Before channel splicing, the features obtained by the four branches are:

[0186]

[0187]

[0188]

[0189]

[0190] in, represents the input features, , , , represents the features extracted for the four branches respectively, represents the convolution operation, , , , The kernel sizes are , , , Ordinary convolution, , , , The kernel sizes are , the expansion rate is , , , Atrous convolution;

[0191] That is, in order to reduce the amount of calculation, we first use a kernel size of Ordinary convolution reduces the number of channels, and dilated convolutions with different expansion rates are used to extract features of different scales, and then channel splicing is performed.

[0192] S20114: Add a step-by-step fusion method before channel splicing. Specifically:

[0193]

[0194] in, represents the output features, represents the channel splicing operation. That is, from the expansion rate The features extracted by the hole convolution are added pixel by pixel. , and The features of the branch include not only the features extracted by the dilated convolution with a larger dilation rate, but also the small target features extracted by the dilated convolution with a smaller dilation rate, thereby improving the network's extraction effect on features of different scales.

[0195] Furthermore, in one embodiment, the preset multi-branch auxiliary is used as the neck network, specifically including:

[0196] S20121: Construct and apply the shallow feature assisted fusion SAF network structure, and the output result after applying the shallow feature assisted fusion SAF network structure is:

[0197]

[0198] in, , Represents feature maps at different resolutions, Indicates that brothers are shallow, represents the upsampling operation, express Downsampling convolutions with batch normalization layers, represents the SiLU activation function, Indicates the number of control channels convolution, represents across shallow high-resolution layers, Indicates the fusion of features extracted by the multi-branch feature extraction network;

[0199] Specifically, we construct a network structure of shallow feature assisted fusion SAF. The main goal of SAF is to combine deep information with features of the same level in the backbone network and high-resolution shallow features, aiming to retain rich local details to enhance the spatial representation of the network. Figure 5 shown.

[0200] S20122: Construct and apply a high-dimensional feature-assisted fusion AAF network structure, and the output result after applying the high-dimensional feature-assisted fusion AAF network structure is:

[0201]

[0202] in, Indicates the current network feature layer, represents a shallow low-resolution layer, Represents the previous network feature layer.

[0203] The network structure of high-dimensional feature assisted fusion AAF is as follows Figure 6 To further enhance the interactive utilization of feature layer information, the AAF module is used in the deeper layers of MAFPN to perform multi-scale information fusion. Specifically, Figure 6 It is shown in AAF connections in the , shallow low-resolution layer , Brothers Shallow and the previous layer At this point, the final output layer P4 can fuse information from four different layers at the same time, thus significantly improving the performance of medium-sized targets. AAF also uses Convolution controls the channels to adjust the impact of each layer on the result. Experiments show that when the strategy in SAF is adopted, that is, the number of channels in the three shallow layers is set to half of the number of channels in the deeper layers, the performance will decrease slightly. Drawing on the traditional single-path architecture assisted by multiple branches, assuming that the initial guidance information has been embedded in the shallow layers of MAFPN, this application balances the number of channels in each layer to ensure that the model obtains different outputs.

[0204] Furthermore, in one embodiment, a loss function combining IoU and IoG is introduced into the constructed multi-scale multi-branch concrete disease detection network, wherein the specific construction process of the loss function combining IoU and IoG includes:

[0205] S301: Perform IoU calculation. Given two bounding boxes box1 and box2, define the coordinates of the upper left corner of the bounding box box1 as , the coordinates of the lower right corner are , the coordinates of the upper left corner of the bounding box box2 are , the coordinates of the lower right corner are ,in, Indicates the width of the bounding box box1, Indicates the height of the bounding box box1, Indicates the width of the bounding box box2, Indicates the height of the bounding box box2;

[0206] S302: Calculate the coordinates of the upper left corner of the intersection of bounding boxes box1 and box2 , lower right corner coordinates ;

[0207] S303: Calculate the area of ​​the intersection of bounding boxes box1 and box2 :

[0208]

[0209] S304: Calculate the area of ​​the union of bounding boxes box1 and box2 :

[0210]

[0211] S305: Calculate the intersection-and-union ratio :

[0212]

[0213] in, Indicates the value corresponding to IoU;

[0214] S306: Perform IoG calculation, given the real bounding box gt_box and the predicted bounding box pred_box, and define the upper left corner coordinate of the real bounding box gt_box as , the coordinates of the lower right corner are , the coordinates of the upper left corner of the predicted bounding box pred_box are , the coordinates of the lower right corner are ;

[0215] S307: The coordinates of the upper left corner of the intersection of the real bounding box gt_box and the predicted bounding box pred_box are calculated as , the coordinates of the lower right corner of the intersection of the real bounding box gt_box and the predicted bounding box pred_box are ;

[0216] S308: Calculate the width of the intersection of the real bounding box gt_box and the predicted bounding box pred_box. :

[0217] ;

[0218] S309: Calculate the height of the intersection of the real bounding box gt_box and the predicted bounding box pred_box. :

[0219] ;

[0220] S3010: Calculate the area of ​​the intersection of the real bounding box gt_box and the predicted bounding box pred_box :

[0221] ;

[0222] S3011: Calculate the area of ​​the true bounding box gt_box :

[0223]

[0224] S3012: Calculate the intersection and union ratio :

[0225]

[0226] in, Indicates the value corresponding to IoG;

[0227] S3013: Design smoothing functions to control the response speed of the loss function in different situations and its sensitivity to abnormal situations, thereby improving the robustness and stability of the model:

[0228]

[0229] in, represents a smooth function, Indicates the value corresponding to IoU or IoG, Represents the threshold parameter of the smoothing function, and its value range is ;

[0230] S3014: Construct a joint loss function combining IoU and IoG, denoted as combUG_loss. The joint loss function includes the loss between the real bounding boxes and the loss between the predicted bounding boxes. Specifically:

[0231]

[0232]

[0233] in, represents the loss between the ground-truth bounding boxes, Indicates For the ground-truth bounding box, Indicates For the predicted bounding box, represents the loss between predicted bounding boxes, Represents the IoG between the real bounding box and the predicted bounding box, represents the IoU between predicted bounding boxes, and represents the threshold parameter of the smoothing function, and Indicates the number of target detection boxes.

[0234] It should be noted that the threshold parameters of the smoothing function are usually determined based on the characteristics of the specific problem and the data set. These parameters affect the performance of the smoothing function within different input value ranges and can be determined in the following ways:

[0235] Method 1: Empirical value tuning. In practical applications, threshold parameters are usually determined through experiments and tuning. You can try different parameter values ​​and select the best parameter based on the performance of the model on the validation set. Usually, you choose to try within a certain range, such as values ​​between 0.5 and 0.95.

[0236] Method 2: Cross-validation technology: Use cross-validation technology to divide the data set into a training set and a validation set. The performance of the model under different threshold parameters is evaluated on the validation set, and the parameters with the best performance are selected.

[0237] Method 3: Problem-specific adjustment: According to the characteristics of the specific problem, such as the scale and shape of the target, the threshold parameters are adjusted to achieve the best effect.

[0238] Furthermore, in one embodiment, a region-based target detection evaluation method is adopted to cluster the detection frames using a clustering algorithm, specifically including:

[0239] When there is only one target detection frame, expand the original target detection frame by several pixels as the clustering target frame;

[0240] When there are at least 3 object detection boxes, the centroid list is obtained by calculating the center points of the bounding boxes, and the silhouette coefficient is used to determine the number of clusters;

[0241] When there are only two target detection boxes, the number of clusters is determined based on the position and size of the detection boxes and the image size;

[0242] Among them, after completing the clustering of the prediction box and the label box, a cross prediction box method is created based on the IoU intersection-over-Union judgment method to achieve subtle detection and judgment of concrete damage and exposed reinforcement defects.

[0243] Specifically, the region-based target detection evaluation method is combined with the clustering algorithm to improve the IoU calculation method of the prediction box and label box to realize clustering evaluation of overlapping targets and fragmented small targets.

[0244] When there is only one target detection box, expand the original target detection box by several pixels (such as 5 or 10) as the cluster target box, with the coordinates of the upper left corner (cluster_xmin, cluster_ymin) and the lower right corner (cluster_xmax, cluster_ymax). At the same time, determine whether cluster_xmin, cluster_ymin, cluster_xmax, and cluster_ymax exceed the image boundary. If so, appropriate processing is required.

[0245] When there are at least 3 target detection boxes, the K-means algorithm is taken as an example in this application: first, the centroid list is obtained by calculating the center point of the bounding box, and the silhouette coefficient (Silhouette Score) is used to determine the optimal number of clusters. The higher the silhouette coefficient, the better the clustering effect. In order to optimize the selection, for a specific number of clusters (such as 3, 4, 5), the silhouette coefficient is multiplied by a weight factor, which will guide the number of clusters to shift toward the preferred option, so it is necessary to conduct experimental testing in combination with the data distribution characteristics.

[0246] When there are only two target detection frames, a comprehensive clustering number determination method based on the position, size, and image size of the detection frame is designed to determine whether the number of clusters is 2 or 1 to avoid the problem of over-clustering.

[0247] Furthermore, based on the position and size of the detection box and the image size, the number of clusters is determined, including:

[0248] S401: Calculate the distance and direction of the centroid, establish a coordinate system with the centroid on the left, and the horizontal axis , vertical axis , the centroid connection line is calculated from the centroid relative to Axis angle , the length of the centroid connecting line is ,when When the absolute value of is less than or equal to 1, calculate the lateral component , ,when When the absolute value of is greater than 1, calculate the longitudinal component , ;

[0249] S402: Obtain the target frame and image size. For the two target detection frames, obtain the pixel coordinates of the four corners of each frame and define the The axis direction is long, along The axis direction is high, and the pixel length of the first target detection box on the left is calculated to be , pixel width is , the pixel length of the second target detection box on the right is calculated to be , pixel width is , the image size is along The size of the axis direction is ,along The size of the axis direction is ;

[0250] S403: For When the absolute value of is less than or equal to 1, the calculated lateral component ,like , then the number of clusters is 1, if and , then the number of clusters is 1, in other cases, the number of clusters is 2;

[0251] S404: For When the absolute value of is greater than 1, the calculated longitudinal component ,like , then the number of clusters is 1, if and , then the number of clusters is 1, in other cases, the number of clusters is 2;

[0252] in, and represents the parameter, and and The values ​​of are as follows:

[0253]

[0254] .

[0255] Furthermore, in one embodiment, based on the IoU intersection-over-union (IoU) determination method, a cross prediction frame method is created to achieve fine detection and determination of concrete damage and exposed reinforcement defects, specifically including:

[0256] S411: setting the threshold of IoU and the threshold of cross prediction box, and calculating the IoU and cross prediction box IoP values, wherein the cross prediction box value is calculated as follows:

[0257]

[0258] in, Represents the value corresponding to the cross prediction box, Represents the intersection area of ​​the predicted box and the true box, Represents the area of ​​the prediction box;

[0259] For the cross prediction box, the full name is Intersection over Prediction Box, referred to as IOP. The threshold of IoU and the threshold of the cross prediction box can be modified according to the project situation;

[0260] S412: Based on the calculation result, when the calculated IoU is greater than or equal to the set IoU threshold, it indicates that the detection box matches the target box and the target is detected. When the calculated IoU is less than the set IoU threshold, the detection box and the target box are separated, contained or intersected. When the calculated cross prediction box value is greater than the set cross prediction box threshold, it indicates that the target is detected. When the calculated cross prediction box value is less than the set cross prediction box threshold, it indicates that the target is not detected.

[0261] See also Figure 7 As shown in , it is an illustration of the positional relationship between the detection frame and the target frame. Figure 7 As shown in (1), when the IoU is greater than or equal to the set IoU threshold (the IoU threshold is usually 0.5 or 0.7), the detection box matches the target box and the target is detected. When the calculated IoU is less than the set IoU threshold, there are multiple situations. Figure 7 As shown in (2) to (4), the detection box and the target box are separated, contained, and intersected, respectively. Obviously, (3) and (4) have achieved detection and met the requirements from the perspective of concrete damage and exposed rebar disease detection. However, due to the traditional IoU calculation method, the IoU calculated between the fragmented and small-sized damage and exposed rebar disease clustering boxes is very low and cannot reach the established threshold. Therefore, the concept of IOP is proposed for the detection and evaluation of fragmented and small-sized disease targets: when the calculated cross-prediction box value is greater than the set cross-prediction box threshold (the cross-prediction box threshold can be greater than 0.7), it indicates that the target is detected at this time, such as Figure 7 In the cases of (3) and (4), when the calculated cross prediction box value is less than the set cross prediction box threshold, it indicates that the target is not detected. When the calculated cross prediction box value is equal to 0, the situation is as follows Figure 7 As shown in (2).

[0262] S413: Calculate the index using precision and recall As indicators, specifically:

[0263]

[0264]

[0265] in, , , They represent the accurately detected objects, the falsely detected objects, and the missed detected objects respectively;

[0266] S414: For accuracy , perform region-based evaluation Calculation of recall , perform region-based evaluation The calculation is as follows:

[0267]

[0268]

[0269] in, , , They represent the accurately detected objects, falsely detected objects, and missed detected objects after clustering, respectively.

[0270] Furthermore, in one embodiment, the detected disease location is segmented based on the segmentation model, and the pixel size of the disease is calculated, which specifically includes:

[0271] S501: Implementing defect detection based on the constructed multi-scale multi-branch concrete defect detection network and clustering the detection frames using a clustering algorithm, obtaining location information of the clustered defect detection frames, and saving the obtained location information of the defect detection frames, wherein the location information of the defect detection frames includes the pixel coordinates of the upper left corner and the pixel coordinates of the lower right corner;

[0272] S502: Using the acquired disease detection frame position information as input to the segmentation model, and guiding the segmentation model to complete the segmentation of the disease within the disease detection clustering frame;

[0273] S503: Based on the segmented mask contour boundary, the pixel area of ​​the damaged and exposed reinforcement defects is calculated.

[0274] Furthermore, in one embodiment, for the calculated pixel size, the conversion of the pixel size to the actual size in the real world is completed through camera calibration technology and three-dimensional measurement technology, which specifically includes:

[0275] S601: Perform camera calibration to obtain internal and external parameter matrices while removing radial distortion and tangential distortion; internal parameters include focal length, optical center, etc., and external parameters include the position and direction of the camera in space;

[0276] S602: Convert the image coordinate system, the camera coordinate system, and the world coordinate system. The image coordinate system takes the upper left corner of the image as the origin and the horizontal right corner as the The positive direction of the axis is vertically downward. The coordinate system of the positive direction of the axis sets the coordinates of a point in the image to ; The camera coordinate system takes the optical center of the camera as its origin. The axis points to the right of the camera, The axis points to the top of the camera, and the axis points to the viewing direction of the camera. The coordinates of a point in the camera coordinate system are set to ;The world coordinate system is a global reference system used to describe the position and orientation of objects in the real world;

[0277] S603: Set the coordinates of an object in the world coordinate system to , the image coordinates Normalize it and get :

[0278]

[0279]

[0280] in, represents the focal length of the camera, Represents the optical center coordinates, that is, the origin of the image coordinate system;

[0281] S604: Convert the image coordinate system to the camera coordinate system. Specifically:

[0282]

[0283] in, Represents the coordinates in the camera coordinate system, Represents the external parameter matrix of the camera, specifically the position matrix of the camera in space;

[0284] S605: The coordinates in the camera coordinate system Convert to world coordinate system , specifically:

[0285]

[0286] in, represents the transpose of the rotation matrix, Represents the external parameter matrix of the camera, specifically the direction matrix of the camera in space;

[0287] S606: After completing the transformation from image coordinates to world coordinates, calculate the actual area of ​​the real world.

[0288] Furthermore, after completing the conversion from the image coordinates to the world coordinates, the actual area of ​​the real world is calculated, wherein the specific steps of calculating the actual area of ​​the real world include:

[0289] S6061: Set the actual area of ​​the target in the real world to , perform the cross product calculation of the vector and represent each point as a vector, where the point Represented as a vector ;

[0290] S6062: Perform cumulative cross product and calculation for adjacent points and Point , calculation point and Point The cross product vector , sum the cross product vectors of all adjacent points to get a total cross product vector ;

[0291] S6063: Based on the total cross product vector obtained , calculate the actual area of ​​the target in the real world:

[0292] .

[0293] It should be noted that in practical applications, in most cases the camera is facing the diseased area, and the camera distortion is small at this time. Therefore, when the camera is facing the diseased area to shoot, the actual area in the real world is calculated, including:

[0294] S60611: Calculate the ground sampling spacing, specifically:

[0295]

[0296] in, Indicates the ground sampling spacing, which refers to the actual ground distance represented by each pixel in the aerial photo. Indicates the camera shooting distance. Indicates the pixel size, Indicates the focal length of the camera;

[0297] S60612: Calculate the actual area of ​​the target in the real world, specifically:

[0298]

[0299] in, represents the actual area of ​​the target in the real world, Represents the total number of pixel areas of the object.

[0300] In a second aspect, an embodiment of the present application also provides a quantification device for detecting damage and exposed reinforcement defects in concrete structure bridges, including: a construction module, a creation module, an introduction module, a detection module, a calculation module, and a conversion module.

[0301] The construction module is used to obtain the original images of concrete structure damage and exposed reinforcement through multiple data sources, and build a sample library of concrete damage and exposed reinforcement; the creation module is used to build a multi-scale multi-branch concrete disease detection network based on the target detection algorithm, and uses multi-branch parallel void convolution as the backbone network, multi-branch auxiliary as the neck network, and introduces the head information network; the introduction module is used to introduce the loss function combining IoU and IoG in the constructed multi-scale multi-branch concrete disease detection network; the detection module is used to realize the disease detection of the target object through the multi-scale multi-branch concrete disease detection network, and adopts the region-based target detection evaluation method, and clusters the detection boxes through the clustering algorithm; the calculation module is used to segment the detected disease position based on the segmentation model, and calculate the pixel size of the disease; the conversion module is used to complete the conversion of the calculated pixel size to the actual size of the real world through camera calibration technology and three-dimensional measurement technology.

[0302] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0303] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0304] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0305] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0306] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0307] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0308] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A quantitative method for detecting damage and exposed reinforcement defects in concrete structure bridges, characterized in that: The quantitative method for detecting damage and exposed reinforcement defects of concrete structure bridges includes: Acquire original images of concrete structure damage and exposed reinforcement through multiple data sources, and build a sample library of concrete damage and exposed reinforcement diseases; Based on the target detection algorithm, multi-branch parallel dilated convolution is used as the backbone network, multi-branch auxiliary is used as the neck network, and the head information network is introduced to construct a multi-scale and multi-branch concrete disease detection network; A loss function combining IoU and IoG is introduced into the constructed multi-scale and multi-branch concrete defect detection network; The target object's defect detection is realized through a multi-scale and multi-branch concrete defect detection network, and a region-based target detection evaluation method is adopted to cluster the detection frames through a clustering algorithm; Segment the detected disease location based on the segmentation model and calculate the pixel size of the disease; For the calculated pixel size, the conversion from pixel size to actual size in the real world is completed through camera calibration technology and three-dimensional measurement technology.

2. A method for detecting and quantifying damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 1, characterized in that: The method of obtaining original images of concrete structure damage and exposed reinforcement through multiple data sources and constructing a sample library of concrete damage and exposed reinforcement disease specifically includes: Acquire images of concrete structure damage and exposed reinforcement from multiple data sources to establish a set of original images of the damage, wherein the data sources include inspection data of a bridge disease inspection system, active shooting data, literature data, and network data; The images in the original disease image set are judged to see whether the image resolution meets the requirements, whether the image brightness meets the requirements, whether the image contrast meets the requirements, and whether the proportion of the main structure meets the requirements, and the images that meet the requirements are placed in the created original disease sample library; Performing transformation processing on the images in the disease original image set, and constructing a disease extended sample library based on the transformed images, wherein the transformation processing includes rotation, mirroring, and color change; The images in the original disease sample library and the disease extended sample library are annotated to establish a disease sample library of concrete damage and exposed reinforcement.

3. A method for detecting and quantifying damage and exposed reinforcement defects of concrete structure bridges as claimed in claim 1, characterized in that: The target detection algorithm is based on the multi-branch parallel hole convolution as the backbone network, the multi-branch auxiliary as the neck network, and the head information network is introduced to construct a multi-scale multi-branch concrete disease detection network, which specifically includes: Based on the YOLO, RT-DETR or Faster-RCNN target detection algorithm, the preset multi-branch parallel dilated convolution is used as the backbone network, the preset multi-branch auxiliary is used as the neck network, and the head information network is introduced; Combining multi-branch parallel atrous convolution and multi-branch auxiliary, a multi-scale multi-branch concrete disease detection network for concrete damage and exposed reinforcement target detection is created; The multi-scale multi-branch concrete disease detection network includes three parts: backbone network, neck network and head network; For the multi-scale and multi-branch concrete defect detection network, the input image to be detected first passes through the backbone network, which consists of four stages, each of which corresponds to a feature extraction network of multi-branch parallel dilated convolutions with different numbers of branches. The multi-branch auxiliary is designed as a neck network. The SAF module in the neck network is used to extract multi-scale features from the backbone network, perform preliminary auxiliary fusion in the shallow layer of the neck network, and collect gradient information, thereby guiding the head network to obtain diversified output information across three resolutions. Finally, the head network predicts the object bounding box and its corresponding category based on the feature map of each scale to calculate its loss.

4. A method for detecting and quantifying damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 3, characterized in that: The preset multi-branch parallel dilated convolution is used as the backbone network, specifically including: For the expansion rate , the convolution kernel size is The dilated convolution, receptive field The calculation method is: ； For the multi-branch parallel dilated convolution with four branches, a kernel size of 1 is added before the dilated convolution of each branch. , , , Ordinary convolution of The kernel size is The number of channels is reduced by ordinary convolution, and after extracting features of different scales by using dilated convolutions with different expansion rates, channel splicing is performed. Before channel splicing, the features obtained by the four branches are: in, represents the input features, , , , represents the features extracted for the four branches respectively, represents the convolution operation, , , , The kernel sizes are , , , Ordinary convolution, , , , The kernel sizes are , the expansion rate is , , , Atrous convolution; Add a step-by-step fusion method before channel splicing, specifically: in, represents the output features, Represents a channel concatenation operation.

5. A method for detecting and quantifying damage and exposed reinforcement defects of concrete structure bridges as claimed in claim 3, characterized in that: The method of using the preset multi-branch auxiliary as the neck network specifically includes: The shallow feature-assisted fusion SAF network structure is constructed and applied, and the output result after applying the shallow feature-assisted fusion SAF network structure is: in, , Represents feature maps at different resolutions, Indicates that brothers are shallow, represents the upsampling operation, express Downsampling convolution with batch normalization layer, represents the SiLU activation function, Indicates the number of control channels convolution, represents across shallow high-resolution layers, Indicates the fusion of features extracted by the multi-branch feature extraction network; A high-dimensional feature-assisted fusion AAF network structure is constructed and applied, and the output result after applying the high-dimensional feature-assisted fusion AAF network structure is: in, Indicates the current network feature layer, represents a shallow low-resolution layer, Represents the previous network feature layer.

6. A method for quantitatively detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 1, characterized in that: The loss function combining IoU and IoG is introduced into the constructed multi-scale multi-branch concrete disease detection network, wherein the specific construction process of the loss function combining IoU and IoG includes: To perform IoU calculation, given two bounding boxes box1 and box2, define the coordinates of the upper left corner of the bounding box box1 as , the coordinates of the lower right corner are , the coordinates of the upper left corner of the bounding box box2 are , the coordinates of the lower right corner are ,in, Indicates the width of the bounding box box1, Indicates the height of the bounding box box1, Indicates the width of the bounding box box2, Indicates the height of the bounding box box2; Calculate the coordinates of the upper left corner of the intersection of bounding boxes box1 and box2 , lower right corner coordinates ; Calculate the area of ​​the intersection of bounding boxes box1 and box2 : Calculate the area of ​​the union of bounding boxes box1 and box2 : Calculate the intersection-union ratio : in, Indicates the value corresponding to IoU; Perform IoG calculation, given the real bounding box gt_box and the predicted bounding box pred_box, and define the upper left corner coordinate of the real bounding box gt_box as , the coordinates of the lower right corner are , the coordinates of the upper left corner of the predicted bounding box pred_box are , the coordinates of the lower right corner are ; The upper left corner coordinates of the intersection of the real bounding box gt_box and the predicted bounding box pred_box are calculated as , the coordinates of the lower right corner of the intersection of the real bounding box gt_box and the predicted bounding box pred_box are ; The width of the intersection of the true bounding box gt_box and the predicted bounding box pred_box is calculated as : ； The height of the intersection of the real bounding box gt_box and the predicted bounding box pred_box is calculated as : ； Calculate the area of ​​the intersection of the true bounding box gt_box and the predicted bounding box pred_box : ； Calculate the area of ​​the true bounding box gt_box : Calculate the intersection-union ratio : in, Indicates the value corresponding to IoG; Design smoothing functions to control the response speed of the loss function in different situations and its sensitivity to abnormal situations: in, represents a smooth function, Indicates the value corresponding to IoU or IoG, Represents the threshold parameter of the smoothing function, and its value range is ; Construct a joint loss function combining IoU and IoG, which includes the loss between the real bounding boxes and the loss between the predicted bounding boxes. Specifically: in, represents the loss between the ground-truth bounding boxes, Indicates For the ground-truth bounding box, Indicates For the predicted bounding box, represents the loss between predicted bounding boxes, Represents the IoG between the real bounding box and the predicted bounding box, represents the IoU between predicted bounding boxes, and represents the threshold parameter of the smoothing function, and Indicates the number of target detection boxes.

7. A method for quantitatively detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 1, characterized in that: The target detection evaluation method based on regions is adopted, and the detection frame clustering is performed through a clustering algorithm, which specifically includes: When there is only one target detection frame, expand the original target detection frame by several pixels as the clustering target frame; When there are at least 3 object detection boxes, the centroid list is obtained by calculating the center points of the bounding boxes, and the silhouette coefficient is used to determine the number of clusters; When there are only two target detection boxes, the number of clusters is determined based on the position and size of the detection boxes and the image size; Among them, after completing the clustering of the prediction box and the label box, a cross prediction box method is created based on the IoU intersection-over-Union judgment method to achieve subtle detection and judgment of concrete damage and exposed reinforcement defects.

8. A quantitative method for detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 7, characterized in that: The method of determining the number of clusters based on the position and size of the detection frame and the image size specifically includes: Calculate the distance and direction of the center of mass, and establish a coordinate system with the center of mass on the left. , vertical axis , the centroid connection line is calculated from the centroid relative to Axis angle , the length of the centroid connecting line is ,when When the absolute value of is less than or equal to 1, calculate the lateral component , ,when When the absolute value of is greater than 1, calculate the longitudinal component , ; Get the target frame and image size. For the two target detection frames, get the pixel coordinates of the four corners of each frame and define the The axis direction is long, along The axis direction is high, and the pixel length of the first target detection box on the left is calculated to be , pixel width is , the pixel length of the second target detection box on the right is calculated to be , pixel width is , the image size is along The size of the axis direction is ,along The size of the axis direction is ; for When the absolute value of is less than or equal to 1, the calculated lateral component ,like , then the number of clusters is 1, if and , then the number of clusters is 1, in other cases, the number of clusters is 2; for When the absolute value of is greater than 1, the calculated longitudinal component ,like , then the number of clusters is 1, if and , then the number of clusters is 1, in other cases, the number of clusters is 2; in, and represents the parameter, and and The values ​​of are as follows: 。 9. A method for quantitatively detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 7, characterized in that: Based on the IoU intersection-over-union determination method, a cross prediction frame method is created to achieve fine detection and determination of concrete damage and exposed reinforcement defects, specifically including: Set the threshold of IoU and the threshold of cross prediction box, and calculate the IoU and cross prediction box IoP values. The calculation method of the cross prediction box value is: in, Represents the value corresponding to the cross prediction box, Represents the intersection area of ​​the predicted box and the true box, Represents the area of ​​the prediction box; Based on the calculation results, when the calculated IoU is greater than or equal to the set IoU threshold, it indicates that the detection box matches the target box and the target is detected. When the calculated IoU is less than the set IoU threshold, the detection box is separated, contained or intersected with the target box. When the calculated cross prediction box value is greater than the set cross prediction box threshold, it indicates that the target is detected. When the calculated cross prediction box value is less than the set cross prediction box threshold, it indicates that the target is not detected. Calculate the indicator using precision and recall As indicators, specifically: in, , , They represent the accurately detected objects, the falsely detected objects, and the missed detected objects respectively; For accuracy , perform region-based evaluation Calculation of recall , perform region-based evaluation The calculation is as follows: in, , , They represent the accurately detected objects, falsely detected objects, and missed detected objects after clustering, respectively.

10. A method for detecting and quantifying damage and exposed reinforcement defects of concrete structure bridges as claimed in claim 1, characterized in that: The method of segmenting the detected disease location based on the segmentation model and calculating the pixel size of the disease specifically includes: Defect detection is realized based on the constructed multi-scale multi-branch concrete defect detection network, and detection frame clustering is performed through a clustering algorithm, the location information of the clustered defect detection frame is obtained, and the obtained location information of the defect detection frame is saved, wherein the location information of the defect detection frame includes the pixel coordinates of the upper left corner and the pixel coordinates of the lower right corner; The acquired disease detection frame position information is used as the input of the segmentation model to guide the segmentation model to complete the disease segmentation within the disease detection clustering frame; Based on the segmented mask contour boundary, the pixel area of ​​damaged and exposed reinforcement defects is calculated.

11. A method for quantitatively detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 1, characterized in that: The calculated pixel size is converted to the actual size in the real world through camera calibration technology and three-dimensional measurement technology, which specifically includes: Perform camera calibration to remove radial distortion and tangential distortion while obtaining the intrinsic and extrinsic parameter matrices; Convert the image coordinate system, camera coordinate system and world coordinate system. The image coordinate system takes the upper left corner of the image as the origin and the horizontal right corner as the The positive direction of the axis is vertically downward. The coordinate system of the positive direction of the axis sets the coordinates of a point in the image to ; The camera coordinate system takes the optical center of the camera as its origin. The axis points to the right of the camera, The axis points to the top of the camera, and the axis points to the viewing direction of the camera. The coordinates of a point in the camera coordinate system are set to ;The world coordinate system is a global reference system used to describe the position and orientation of objects in the real world; Set the coordinates of an object in the world coordinate system to , the image coordinates Normalize it and get : in, represents the focal length of the camera, Represents the optical center coordinates, that is, the origin of the image coordinate system; Convert the image coordinate system to the camera coordinate system. Specifically: in, Represents the coordinates in the camera coordinate system, Represents the external parameter matrix of the camera, specifically the position matrix of the camera in space; The coordinates in the camera coordinate system Convert to world coordinate system , specifically: in, represents the transpose of the rotation matrix, Represents the external parameter matrix of the camera, specifically the direction matrix of the camera in space; After completing the transformation from image coordinates to world coordinates, the actual area of ​​the real world is calculated.

12. A method for quantitatively detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 11, characterized in that: After completing the conversion from the image coordinates to the world coordinates, the actual area of ​​the real world is calculated, wherein the specific steps of calculating the actual area of ​​the real world include: Set the actual area of ​​the target in the real world to , perform the cross product calculation of the vector and represent each point as a vector, where the point Represented as a vector ; Perform cumulative cross product and calculation for adjacent points and Point , calculation point and Point The cross product vector , sum the cross product vectors of all adjacent points to get a total cross product vector ; Based on the total cross product vector obtained , calculate the actual area of ​​the target in the real world: 。 13. A quantitative method for detecting damage and exposed reinforcement defects in concrete structure bridges as claimed in claim 11, characterized in that: When the camera is facing the diseased area, the actual area in the real world is calculated, including: Calculate the ground sampling spacing, specifically: in, represents the ground sampling spacing, Indicates the camera shooting distance. Indicates the pixel size, Indicates the focal length of the camera; Calculate the actual area of ​​the target in the real world, specifically: in, represents the actual area of ​​the target in the real world, Represents the total number of pixel areas of the object.

14. A quantification device for detecting damage and exposed reinforcement defects in concrete structure bridges, characterized in that: include: A construction module is used to obtain original images of concrete structure damage and exposed reinforcement through multiple data sources, and to construct a sample library of concrete damage and exposed reinforcement diseases; Create a module for constructing a multi-scale and multi-branch concrete disease detection network based on the target detection algorithm, using multi-branch parallel dilated convolution as the backbone network, multi-branch auxiliary as the neck network, and introducing the head information network; A module is introduced to introduce a loss function combining IoU and IoG in the constructed multi-scale and multi-branch concrete disease detection network; The detection module is used to realize the disease detection of the target object through a multi-scale and multi-branch concrete disease detection network, and adopts a region-based target detection evaluation method to cluster the detection frames through a clustering algorithm; A calculation module, which is used to segment the detected disease location based on the segmentation model and calculate the pixel size of the disease; The conversion module is used to convert the calculated pixel size into the actual size in the real world through camera calibration technology and three-dimensional measurement technology.

Images