A concrete crack rapid identification method based on faster-rIR7-EC

Abstract

The application provides a concrete crack rapid identification method based on Faster-rIR7-EC, and relates to the technical field of concrete damage detection. The method improves the traditional inverted residual structure, proposes an InvertedResidual-ECA structure combining the inverted residual structure with an ECA attention mechanism, constructs a Faster-rcnn feature extraction layer through seven InvertedResidual-ECA structures and CBAM and the like machine vision algorithms, completes Faster-rIR7-EC crack identification network building, and outputs the position and type of the concrete crack in the image. The Faster-rIR7-EC network model automatically, quickly and accurately identifies the concrete crack in the image. Compared with the general Faster-rcnn network of computer vision, the application avoids the disadvantages of parameter redundancy, long training time, high hardware memory occupation and the like of the general network, has the characteristics and advantages of small network model parameter quantity, fast training convergence speed and high concrete crack identification accuracy, constitutes a special model for intelligently and efficiently providing cracks from concrete image big data, and has significant engineering application potential.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual damage identification of concrete, and particularly relates to a concrete crack rapid identification method based on Faster-rIR7-EC. BACKGROUND

[0002] Concrete is widely used in infrastructure construction such as houses and bridges as the most common building material, but cracks will occur during the operation of the infrastructure, affecting its safety. In order to avoid the risk of structure collapse caused by crack expansion, it is necessary to detect the damage of concrete cracks.

[0003] With the development of unmanned aerial vehicle technology and wireless transmission technology, it is possible to efficiently obtain massive data of mass concrete, which provides a basis for the development of intelligent identification. Compared with traditional structure damage detection methods, crack detection using machine vision technology avoids the problems of time-consuming and laborious manual detection and subjectivity that cannot be frequently detected. However, general networks often have a large size in order to meet the identification needs of various fields. When a deep learning network is used for concrete crack identification, there are only tens of crack types, and the general network has a large size, a long training time and high hardware requirements. It is necessary to develop a lightweight network to build a concrete crack special deep learning target identification algorithm with fast convergence speed and high mAP (mean average precision). SUMMARY

[0004] To solve the above problems, the present application provides a concrete crack rapid identification method based on Faster-rIR7-EC.

[0005] To achieve the above purpose, the present application provides the following technical solutions.

[0006] A concrete crack rapid identification method based on Faster-rIR7-EC, comprising the following steps:

[0007] A Faster-rIR7-EC network model is built; the Faster-rIR7-EC network model comprises an IR7-EC feature extraction layer, a region generation network RPN, an ROI Head and a post-processing layer connected in sequence;

[0008] The IR7-EC feature extraction layer comprises a convolution layer, 7 Inverted Residual-ECA structures and a CBAM attention mechanism connected in sequence; the Inverted Residual-ECA structure is a connection of an inverted residual structure and an ECA attention mechanism; the ROI Head comprises an ROI pooling, a full connection layer and a postprocess detection;

[0009] Collecting a crack image to be identified, extracting features of the crack image in the channel and spatial level through an IR7-EC feature extraction layer, and obtaining a feature extraction image containing crack information;

[0010] Inputting the feature extraction image into a region generation network RPN, generating a candidate box on the feature extraction image, and determining cracks and backgrounds contained in the candidate box; projecting the candidate box generated by the RPN onto the feature map output by the IR7-EC to obtain a corresponding feature matrix;

[0011] Passing each feature matrix through an ROI Head to output the position and type of the concrete crack in the feature map;

[0012] Projecting the position and type of the concrete crack in the feature map back to the actual crack image through a post-processing layer to obtain the actual position and type of the concrete crack.

[0013] Preferably, the Faster-rIR7-EC network model is further trained, including the following steps:

[0014] Collecting a crack image to establish a crack type identification image dataset, wherein the crack image dataset includes a crack identification image training set and a validation set;

[0015] Normalizing the crack image dataset: traversing all crack images, finding the image with the largest height and width, taking the largest width and height as a template, aligning the upper left corner of all other images with the template, and supplementing 0 to the right and lower side of the insufficient position until the size is consistent with the template, so that the input image resolution is uniform and consistent with the template;

[0016] Inputting the normalized image dataset into the Faster-rIR7-EC network model to obtain an identification result;

[0017] Comparing the identification result with the true result of the training set, bringing the loss into an optimization function, updating the network parameters, and fitting the network until the network is fitted; wherein the loss includes RPN loss and ROI Head loss.

[0018] Preferably, the IR7-EC feature extraction layer is built, specifically including the following steps:

[0019] Establishing a 10-layer network framework connected in turn;

[0020] The first layer includes a convolution layer with a convolution kernel size of 3x3, a Batch normalization layer, and a Hardswish activation function, with a step of 2, an input channel of 3, and an output channel number of 16;

[0021] The second layer to the eighth layer are Inverted Residual-ECA structures, including three parts:

[0022] The first part includes a convolution layer with a convolution kernel of 1x1 size, a Batch normalization layer, and a ReLU6 activation function, the second part includes a convolution layer with a convolution kernel of 3x3 size, a Batch normalization layer, and a ReLU6 activation function, the third part is an ECA attention mechanism, and the structure includes an average pooling layer, a convolution layer with a convolution kernel of 1x1 size, and a Sigmoid function, and the fourth part includes a convolution layer with a convolution kernel of 1x1 size, a Batch normalization layer, and a linear activation function;

[0023] The 9th layer includes a convolution layer with a convolution kernel of 3x3 size, a Batch normalization layer, and a Hardswish activation function, a step of 1, an input channel of 96, and an output channel number of 96;

[0024] The 10th layer is a CBAM attention mechanism, including two parts:

[0025] The first part is a channel attention mechanism, including an average pooling layer, a maximum pooling layer, a full connection layer 1, a ReLU6 activation function, a full connection layer 2, and a Sigmoid function, and the second part is a spatial attention mechanism, including an average pooling layer, a maximum pooling layer, a convolution layer with a convolution kernel of 7x7 size, and a Sigmoid function.

[0026] Preferably, the feature extraction process of the IR7-EC feature extraction layer includes the following steps:

[0027] The normalized image sample data is input into the 1st layer, and sequentially passes through a convolution layer, a Batch normalization layer, and a Hardswish activation function to preliminarily extract features, normalize data, and perform nonlinear transformation on the image sample data;

[0028] The features preliminarily extracted by the 1st layer are input into the 2nd layer to the 8th layer, and sequentially pass through 7 InvertedResidual-ECA structures. After the input data is processed by the first part of the InvertedResidual-ECA structure, the channel dimension of the input data is expanded. The data with the expanded channel dimension is input into the second part for deep feature extraction. The data after deep feature extraction is input into the third part. The ECA attention mechanism is used to weight and highlight the part of the network channel with higher network accuracy, to obtain an enhanced concrete crack feature extraction graph. The enhanced concrete crack feature extraction graph is input into the fourth part for reducing the data channel dimension. According to similar operations, the data sequentially passes through all the InvertedResidual-ECA structures;

[0029] The data obtained after passing through the 2nd layer to the 8th layer is input into the 9th layer for feature extraction;

[0030] The data after feature extraction by the 9th layer is input into the 10th layer CBAM attention mechanism for further feature extraction in the channel and spatial layers, obtaining a feature map containing more crack information.

[0031] Preferably, the normalization processing of the Batch normalization layer is as follows:

[0032]

[0033]

[0034]

[0035]

[0036] In the formula, x i is the feature map input into the Batch normalization, y i is the feature map output after the Batch normalization, m is the number of feature maps input into the layer in the current training batch, γ and β are variables that change with network gradient update;

[0037] When building the IR7-EC feature extraction layer, the following formula is used to perform nonlinear processing on the data passing through the ReLU6 activation function in each layer:

[0038] f(x i ) = min(max(x i , 0), 6)

[0039] In the formula, x i is the feature map before inputting into the ReLU6 activation function, and f(x i ) is the feature map output by the ReLU6 activation function.

[0040] When building the IR7-EC feature extraction layer, the following formula is used to perform nonlinear processing on the data passing through the Hardswish activation function in each layer:

[0041]

[0042] In the formula, x is the feature map before inputting into the Hardswish activation function, and f(x) is the feature map output by the Hardswish activation function.

[0043] When constructing the IR7-EC feature extraction layer, the data from each layer through the ECA attention mechanism are used for cross-channel interaction to obtain an enhanced concrete crack feature extraction map:

[0044]

[0045] E s (F)=σ(f k*k [AvgPool(F)])

[0046] In the formula, |t| odd Represents the most recent odd number t; C represents the number of channels of data input to the ECA attention mechanism; γ and b are two hyperparameters; E s (F) represents the ECA attention mechanism, σ is the sigmoid operation, and f k*k [·] indicates a k*k convolution operation, F is the input feature map, and AvgPool() is average pooling;

[0047] When constructing the IR7-EC feature extraction layer, average pooling and max pooling are used to aggregate the spatial information of the feature maps, compress the spatial dimension of the input feature maps, and sum and merge them element-wise to generate channel attention maps:

[0048] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0049] In the formula, M c MLP() represents channel attention and consists of a fully connected layer 1 + a ReLU6 activation function + a fully connected layer 2. σ is the sigmoid operation, F is the input feature map, AvgPool() is average pooling, and MaxPool() is max pooling. s This represents the spatial attention mechanism, where σ is the sigmoid operation;

[0050] When constructing the IR7-EC feature extraction layer, the input feature map is compressed in the spatial attention module using the following formula with average pooling and max pooling methods to obtain a feature extraction map containing more crack information:

[0051] M s (F)=σ(f 7*7 [AvgPool(F), MaxPool(F)])

[0052] In the formula, M s This represents the spatial attention mechanism, where σ is the sigmoid operation and f is the spatial attention mechanism. 7*7[·] represents a 7*7 convolution operation, F is the input feature map, AvgPool() is the average pooling, and MaxPool() is the maximum pooling.

[0053] Preferably, the RPN comprises an Anchor generator and an RPNhead.

[0054] Preferably, it further comprises:

[0055] The anchor frame is generated by the Anchor generator, comprising the following steps:

[0056] Based on the 42*42 resolution feature map output by the IR7-EC feature extraction layer, 1764 groups of anchor frames of different scales are generated by the Anchor generator, and each pixel point on the feature map is projected onto the original crack image before inputting into the network. Each of the 1764 feature map projection positions on the original image is assigned a group of anchor frames generated by the Anchor generator;

[0057] The rules for dividing positive and negative samples of the crack image based on the anchor frames generated by the Anchor generator are as follows, wherein the crack is positive and the no crack is negative:

[0058] In an image of a crack, a series of ground truth bounding boxes GT Box and anchor frames Anchor are included; wherein the anchor frame is determined as positive according to the following rules: the anchor frame with IOU≥0.7 with the GT Box; when the IOU of all anchor frames intersecting the GT Box is less than 0.7, the anchor frame with the maximum IOU with the GT Box is directly determined as positive;

[0059] The negative determination rule is: the anchor frame with IOU<0.3 with the GT Box;

[0060] Randomly select 256 positive and negative samples as label real values for calculating the target loss in the RPNhead, wherein the IOU calculation formula is as follows:

[0061]

[0062] Wherein, area(A) is the area of the anchor frame, and area(G) is the area of the GT Box frame.

[0063] Preferably, the RPNhead comprises a 3*3 convolution layer, two parallel 1*1 convolution layers and a ReLU activation function.

[0064] The training steps of the RPNhead comprise:

[0065] The 42x42 resolution feature map output by IR7-EC is passed through a 3x3 convolution layer, and then through two parallel 1x1 convolution layers and ReLU activation functions, to output the target scores of all pixels of the feature map corresponding to all anchor boxes and the boundary box regression parameters as follows:

[0066] cls = [crack probability]

[0067] t i = [t x , t y , t w , t h ]

[0068] where cls is the crack probability predicted by RPNhead, t i represents the boundary box regression parameter of the i-th anchor box predicted by RPNhead;

[0069] The anchor box is adjusted by the boundary box regression parameter to obtain the candidate box proposals, and the formula is as follows:

[0070] x = w a t x +x a

[0071] y = h a t y +y a

[0072] w = w a exp(t w )

[0073] h = h a exp(t h )

[0074] where x, y, w, h are the center coordinates (x, y) and the width and height of the boundary box of proposals, x a , y a , w a , h a are the center coordinates and the width and height of the anchor box, and t x , t y , t w , t h are the boundary box regression parameters predicted by RPNhead;

[0075] The proposals are screened as follows:

[0076] Based on the target score of each proposal in the crack image, the top 2000 proposals with the highest crack probability are selected, those that do not meet the criteria are deleted, and then proposals with smaller areas are deleted.

[0077] Find the proposal with the largest coordinate among all proposals, calculate the intersection and union of other proposals with it, use the non-maximum suppression algorithm to filter all proposals, and finally project the proposals onto the feature map output by IR7-EC to obtain the corresponding feature matrix.

[0078] The RPNhead loss calculation method is as follows:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] t x =(xx) a ) / w a , t y =(yy) a ) / h a

[0087] t w =ln(w / w) a ), t h =ln(h / h) a )

[0088] Among them, Loss({p i},{t i}) represents the loss of RPN, including classification loss and bounding box regression loss, p i This represents the probability that the i-th anchor is predicted as the target, when the anchor is a positive sample. When the value is 1, it is a negative sample. t is 0 i This represents the bounding box regression parameters for the predicted i-th anchor. Boundary box regression parameters of the i-th anchor corresponding to the GT Box, N cls N represents the number of all samples in a mini-batch reg t represents the number of sample positions i Boundary box regression parameters of the i-th anchor frame predicted by the RPN head, x * , y * , w * , h * The center coordinates (x, y) and the boundary box width and height of the GT Box, and λ is used to balance the classification loss and the boundary box regression loss.

[0089] Preferably, it further comprises:

[0090] All proposals are extracted by a pooling operation feature through ROI pooling, and the size of all proposals is converted into a 7*7 size feature map;

[0091] The full connection layer structure of the ROI Head is two serial full connection layers (FC1, FC2), which are used to flatten the feature map and then pass into two parallel full connection layers (FC3, FC4) through two full connection layers, for predicting the crack class score and boundary box regression parameters of each proposal. The boundary box regression parameters output by FC4 are used to adjust the proposals, and the full connection layer loss is calculated through similar steps as RPN as follows:

[0092] Loss(p, u, t u , v) = L cls (p, u) + λ[u≥1]L loc (t u , v)

[0093] L cls (p, u) = -log p u

[0094]

[0095] Wherein, Loss(p, u, t u , v) is the loss function of the ROI Head, including the classification loss and the boundary box regression loss, p is the softmax probability distribution predicted by the classifier p = (p o , …, p k ), k is the number of crack categories plus 1, u corresponds to the target real category label, t u corresponds to the boundary box regression parameter of the corresponding category u predicted by the boundary box regressor FC4, and v corresponds to the real target GT Box boundary box regression parameter v x , vy , v w , v h ;

[0096] The post-process detections are used to post-process the prediction data of the network, including:

[0097] The final bounding box (bbox) coordinates are calculated according to the proposals and the regression parameters predicted by the FC4;

[0098] The prediction class results are subjected to a softmax process to obtain a crack probability;

[0099] All background information is removed;

[0100] Low-probability targets and small-size targets are removed;

[0101] The prediction results in the ROI Head are screened by the hms method;

[0102] The final bbox coordinate calculation formula is as follows:

[0103]

[0104]

[0105]

[0106]

[0107] wherein x, y, w, h are the center coordinates (x, y) and the bounding box width and height of the proposals output by the RPN, are the bounding box regression parameters predicted by the fully connected layer FC4, x p , y p , w p , h p are the crack position box center coordinates and the width and height predicted by the ROI Head;

[0108] Preferably, the network parameters are updated as follows:

[0109] f (θ) = Loss

[0110]

[0111] m t = β1·m t-1 +(1-β1)·g t

[0112]

[0113]

[0114]

[0115]

[0116] wherein, Loss is the loss function of the network RPN or ROI Head, theta is the parameter to be updated in the model, g t is the gradient obtained by derivation of the loss function f(theta) to theta, beta1 is the first moment decay coefficient, beta2 is the second moment decay coefficient, m t is the expectation of the gradient g t , v t is the expectation of , is the bias correction of m t , v t is the bias correction of t-1 , theta t is the parameter before network updating, theta i is the parameter after network updating, and alpha is the learning rate.

[0117] The application provides a concrete crack rapid identification method based on a Faster-rIR7-EC, which has the following beneficial effects:

[0118] Compared with the current Faster-rcnn-based computer vision target identification network, the Faster-rIR7-EC network model has smaller parameter quantity, shorter training time, and meanwhile maintains a higher concrete crack identification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0119] Figure 1 FIG. 1 is a flowchart of the Faster-rIR7-EC network used for identifying concrete cracks in the embodiment of the application;

[0120] Figure 2 FIG. 2 is a schematic diagram of seven kinds of concrete crack images and spliced images in the embodiment of the application;

[0121] Figure 3 FIG. 3 is a Faster-rIR7-EC structure diagram in the embodiment of the application;

[0122] Figure 4 FIG. 4 is an IR7-EC structure diagram in the Faster-rIR7-EC in the embodiment of the application;

[0123] Figure 5 FIG. 5 is a crack identification result diagram of the Faster-rIR7-EC network in the embodiment of the application. Detailed Implementation

[0124] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0125] Example 1

[0126] This invention provides a method for rapid identification of concrete cracks based on Faster-rIR7-EC, such as... Figures 1-5 As shown in Table 1, the computer platform and environment configuration implemented in this patent are as follows.

[0127] Table 1 Computer Platform and Environment Configuration

[0128]

[0129]

[0130] See Figure 1 This invention provides a method for rapid identification of concrete cracks based on Faster-rIR7-EC, specifically including the following steps:

[0131] Step 1: Collect crack images and establish a crack type recognition image dataset, which includes a crack recognition image training set and a validation set.

[0132] Step 2: Preprocess the dataset.

[0133] Step 3: Input the preprocessed crack image data into the IR7-EC feature extraction layer to extract features.

[0134] Step 4: Input the feature map output by IR7-EC into RPN (Region Proposal Network) to generate candidate boxes and determine the contents of the candidate boxes (cracks, background). Project the candidate boxes generated by RPN onto the feature map output by IR7-EC to obtain the corresponding feature matrix (calculate RPN loss during training).

[0135] Step 5: Output the location and type of concrete cracks in the feature map through ROI Head (including ROI pooling, fully connected layers and postprocess detections) for each feature matrix (ROIHead loss is calculated during the training phase).

[0136] Step 6: Through post-processing, the location and type of concrete cracks in the feature map are projected back to the actual crack image to obtain the actual location and type of concrete cracks.

[0137] Step 7: During the training phase, the loss is fed into the optimization function to update the network parameters until the network converges.

[0138] Step 8: Input the concrete crack image to be detected into the trained Faster-rIR7-EC network to obtain the location and type of cracks in the concrete crack image.

[0139] Specifically, after acquiring crack images in step 1, the concrete crack image training and validation sets are manually labeled, including: transverse cracks, vertical cracks, oblique cracks, mesh cracks, irregular cracks, holes, and background images. The training and validation sets contain the above seven image types and their corresponding pattern labels. These seven types of images are then randomly stitched together into a nine-square grid, as shown below. Figure 2 As shown, there are a total of 674 images with a resolution of 681×681. The training set has 500 images, and the validation set has 174 images.

[0140] Specifically, the preprocessing operation in step 2 includes image normalization: traversing all crack images, finding the image with the largest height and width, using the largest width and height as a template, aligning the upper left corner of all other images with the template, and padding the right and lower sides with zeros if the size is insufficient until they are the same as the template, so as to achieve a unified input image resolution consistent with the template.

[0141] Specifically, the steps for constructing the IR7-EC feature extraction layer in step 3 include:

[0142] A 10-layer network framework is established with sequential connections. Specific parameters are shown in Table 2, and a schematic diagram is provided. Figure 4 ;

[0143] The first layer consists of a 3×3 convolutional layer, a batch normalization layer, and a Hardswish activation function, with a stride of 2, 3 input channels, and 16 output channels.

[0144] The 2nd layer to the 8th layer is an inverted residual-ECA structure, which specifically includes three parts, the first part includes a convolution layer with a convolution kernel of 1*1 size, a Batch normalization normalization layer and a ReLU6 activation function, the second part includes a convolution layer with a convolution kernel of 3*3 size, a Batch normalization normalization layer and a ReLU6 activation function, and the third part is an ECA attention mechanism, which includes an average pooling layer, a convolution layer with a convolution kernel of 1*1 size and a Sigmoid function, and the fourth part structure includes a convolution layer with a convolution kernel of 1*1 size, a Batch normalization normalization layer and a linear activation function;

[0145] The 9th layer includes a convolution layer with a convolution kernel of 3*3 size, a Batch normalization normalization layer and a Hardswish activation function, a step of 1, an input channel of 96 and an output channel number of 96;

[0146] The 10th layer is a CBAM attention mechanism, which is divided into two parts, the first part is a channel attention mechanism, which includes an average pooling layer, a maximum pooling layer, a full connection layer 1, a ReLU6 activation function, a full connection layer 2 and a Sigmoid function, and the second part is a spatial attention mechanism, which includes an average pooling layer, a maximum pooling layer, a convolution layer with a convolution kernel of 7*7 size and a Sigmoid function.

[0147] Table 2 IR7-EC feature extraction layer structure and internal specific parameters

[0148]

[0149]

[0150] Specifically, the step of inputting the image sample data into the IR7-EC feature extraction layer for training includes:

[0151] The pre-processed image sample data is input into the 1st layer, and is sequentially subjected to a convolution layer, a Batch normalization normalization layer and a Hardswish activation function, so as to preliminarily extract features, normalize data and perform nonlinear transformation on the image sample data.

[0152] The features extracted by the first layer are input into the second layer to the eighth layer, sequentially passing through seven InvertedResidual-ECA structures. After the input data is processed by the first part of the InvertedResidual-ECA structure, the channel dimension of the input data is expanded. The data with the expanded channel dimension is input into the second part for deep feature extraction. The data after deep feature extraction is input into the third part. The ECA attention mechanism is used to weight and highlight the part of the network channel with higher network accuracy, so as to obtain an enhanced concrete crack feature extraction graph. The enhanced concrete crack feature extraction graph is input into the fourth part for reducing the data channel dimension. The data is sequentially passed through all the InvertedResidual-ECA structures according to similar operations.

[0153] The data obtained after the second layer to the eighth layer is input into the ninth layer for feature extraction.

[0154] The data after feature extraction by the ninth layer is input into the tenth layer CBAM attention mechanism for further feature extraction in the channel and spatial layers, so as to obtain a feature graph containing more crack information.

[0155] Specifically, when building the IR7-EC feature extraction layer, the data passing through the Batch normalization normalization layer in each layer is normalized by the following formula:

[0156]

[0157]

[0158]

[0159]

[0160] In the formula, x i is the feature graph input into the Batch normalization, y i is the feature graph output after the Batch normalization, m is the number of feature graphs input into the layer in the current training batch, and γ and β are variables that change with the network gradient update.

[0161] Specifically, when building the IR7-EC feature extraction layer, the data passing through the ReLU6 activation function in each layer is nonlinearly processed by the following formula:

[0162] f(x i )=min(max(x i ,0),6)

[0163] In the formula, x is the feature map before inputting the ReLU6 activation function, and f(x) is the feature map output by the ReLU6 activation function. i i In the formula, x is the feature map before inputting the ReLU6 activation function, and f(x) is the feature map output by the ReLU6 activation function.

[0164] Specifically, when building the IR7-EC feature extraction layer, the data passing through the Hardswish activation function in each layer is used for nonlinear processing:

[0165]

[0166] In the formula, x is the feature map before inputting the ReLU6 activation function, and f(x) is the feature map output by the ReLU6 activation function.

[0167] Specifically, when building the IR7-EC feature extraction layer, the data passing through the ECA attention mechanism in each layer is used for cross-channel interaction to obtain enhanced concrete crack feature extraction maps:

[0168]

[0169] E s (F)=σ(f k*k [AvgPool(F)])

[0170] In the formula, |t| odd represents the nearest odd number t. C represents the number of channels of the data input into the ECA attention mechanism, and γ and b are two hyperparameters. In this patent, γ is set to 2 and b is set to 1. E s (F) is the ECA attention mechanism, σ is the sigmoid operation, f k*k [·] represents a k*k convolution operation, F is the input feature map, and AvgPool() is the average pooling.

[0171] Specifically, when building the IR7-EC feature extraction layer, the average pooling and maximum pooling in the formula are used to aggregate the spatial information of the feature map, compress the spatial dimension of the input feature map, and element-wise sum to merge to produce a channel attention map:

[0172] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0173] In the formula, M c represents channel attention, MLP() is composed of a fully connected layer 1 + a ReLU6 activation function + a fully connected layer 2, σ is the sigmoid operation, F is the input feature map, AvgPool() is the average pooling, MaxPool() is the maximum pooling, and M s ​Spatial attention mechanism, and sigma is a sigmoid operation.

[0174] Specifically, when building the IR7-EC feature extraction layer, the average pooling and maximum pooling methods are used to compress the input feature map in the spatial attention module by the following formula to obtain a feature extraction map containing more crack information:

[0175] M s (F) = sigma(f 7*7 [AvgPool(F), MaxPool(F)])

[0176] In the formula, M s represents a spatial attention mechanism, sigma is a sigmoid operation, f 7*7 [·] represents a 7*7 convolution operation, F is the input feature map, AvgPool() is average pooling, and MaxPool() is maximum pooling.

[0177] Specifically, the RPN (Region Proposal Network) structure in step 4 includes an anchor generator and an RPN head.

[0178] Specifically, based on the 42*42 resolution feature map output by IR7-EC, 1764 (42*42) groups of anchor boxes of different scales are generated by Anchor generator, and each pixel point (each channel has 1764) on the feature map is projected onto the original crack image before inputting into the network. Each of the 1764 feature map projection positions on the original image will be assigned a group of anchor boxes (anchors) generated by Anchor generator.

[0179] Specifically, the rules for dividing positive and negative samples of crack images based on anchor boxes generated by Anchor generator are as follows:

[0180] In a crack image, a series of ground truth boxes (GT boxes) and anchor boxes are included, and the rules for determining positive anchor boxes are as follows: ① Anchor boxes with IOU≥0.7 with GT boxes, ② When the IOU of all anchor boxes intersecting with GT boxes is less than 0.7, the anchor box with the maximum IOU with GT boxes is directly determined as positive;

[0181] Negative determination rule: anchor boxes with IOU <0.3 with GT boxes;

[0182] Randomly select 256 positive and negative samples as label true values for calculating the target loss in RPNhead, wherein the IOU calculation formula is as follows:

[0183]

[0184] Wherein area(A) is the area of the anchor, and area(G) is the area of the GT Box.

[0185] Specifically, the RPNhead structure comprises a 3x3 convolution layer and two parallel 1x1 convolution layers and ReLU activation functions.

[0186] Specifically, the RPNhead structure training step comprises: passing the 42x42 resolution feature map output by the IR7-EC through a 3x3 convolution layer, and then through two parallel 1x1 convolution layers and ReLU activation functions, to output the target scores and boundary box regression parameters of all anchor boxes corresponding to all pixel points of the feature map as follows:

[0187] cls=[crack probability]

[0188] t i ===[t x , t y , t w , t h ]

[0189] Wherein cls is the crack probability predicted by RPNhead, and t i represents the boundary box regression parameter of the i-th anchor box predicted by RPNhead.

[0190] Specifically, the obtained anchor box is adjusted by the boundary box regression parameter to obtain the candidate box (proposals), and the formula is as follows:

[0191] x=w a t x +x a

[0192] y=h a t y +y a

[0193] w=w a exp(t w )

[0194] h=h a exp(t h )

[0195] where x, y, w, h are the center coordinates (x, y) and the bounding box width height of proposals, x a , y a , w a , h a are the center coordinates and the width height of anchor boxes, t x , t y , t w , t h are the bounding box regression parameters predicted by the RPNhead.

[0196] Specifically, the proposals are screened. First, according to the target score of each proposal in the crack image, the top 2000 proposals with the largest crack probability are selected, and those that do not meet the conditions are deleted. Then, the proposals with small areas are deleted. Finally, the proposal corresponding to the largest coordinate value in all proposals is found, and the intersection over union (IOU) is calculated for other proposals. The non-maximum suppression algorithm (NMS) is used to screen all proposals. Finally, the proposals are projected onto the feature map output by IR7-EC to obtain the corresponding feature matrix.

[0197] Specifically, the RPNhead structure loss calculation method is as follows:

[0198]

[0199]

[0200]

[0201]

[0202]

[0203]

[0204]

[0205] t x =(x-x a ) / w a , t y =(y-y a ) / h a

[0206] t w =ln(w / w a ), t h =ln(h / h a )

[0207] Among them, Loss({p i}, {t i}) represents the loss of RPN, including classification loss and bounding box regression loss, p i This represents the probability that the i-th anchor is predicted as the target, when the anchor is a positive sample. When the value is 1, it is a negative sample. t is 0 i This represents the bounding box regression parameters for the predicted i-th anchor. N represents the bounding box regression parameters of the ground truth box corresponding to the i-th anchor. cls N represents the total number of samples in a mini-batch, which is 96 in this patent. reg t represents the number of sample locations. i Let x represent the bounding box regression parameters of the i-th anchor box predicted by RPNhead. * y * w * h * Let (x, y) be the center coordinates of the GT Box and the width and height of the bounding box. λ is used to balance the classification loss and the bounding box regression loss, and is taken as 10 in this patent.

[0208] Specifically, in step 5, ROI pooling extracts features from all the obtained proposals through pooling operations, and the size of all proposals is transformed into a 7×7 feature map.

[0209] Specifically, in step 5, the fully connected layer structure consists of two cascaded fully connected layers (FC1, FC2). After flattening the feature map, it is passed through the two fully connected layers and then fed into two parallel fully connected layers (FC3, FC4) to predict the crack category score and bounding box regression parameters for each proposal. The bounding box regression parameters output by FC4 are used to adjust the proposals. The loss of the fully connected layer is calculated through steps similar to RPN as follows:

[0210] Loss(p, u, t) u ,v)=L cls (p, u) + λ[u≥1]L loc (t u v)

[0211] L cls (p, u) = -log p u

[0212]

[0213] Where Loss(p, u, t) u, v) is the loss function of ROI Head, including classification loss and bounding box regression loss, p is the softmax probability distribution predicted by the classifier p = (p o ,..., p k ), k is the number of crack categories plus 1, u corresponds to the target real category label, t u corresponds to the bounding box regression parameter of the corresponding category u predicted by the bounding box regressor (FC4), and v corresponds to the real target GT Box bounding box regression parameter v x , v y , v w , v h , the calculation method is referred to the RPN loss calculation.

[0214] Specifically, the predicted data of the network is post-processed by postprocess detections in step 5, including (1) calculating the final bounding box (bbox) coordinates according to the proposals and the regression parameters predicted by FC4, (2) performing softmax processing on the predicted category results to obtain crack probability, (3) removing all background information, (4) removing low-probability targets and small-size targets, and (5) referring to the nms method in RPNhead to screen the prediction results in ROI Head.

[0215] Specifically, the final bbox coordinate calculation formula is as follows:

[0216]

[0217]

[0218]

[0219]

[0220] Where x, y, w, h are the center coordinates (x, y) and the width and height of the bounding box of the proposals output by RPN, is the bounding box regression parameter predicted by the fully connected layer FC4, x p , y p , w p , h p is the center coordinates and width and height of the crack position box predicted by roihead.

[0221] Specifically, the internal parameters of the network are optimized by using the following Adam algorithm in step 7:

[0222] f(θ) = Loss

[0223]

[0224] m t = β1·m t-1 + (1- β1)·g t

[0225]

[0226]

[0227]

[0228]

[0229] wherein, Loss is the loss function of network RPN or ROI Head, θ is the parameter to be updated in the model, g t is the gradient obtained by derivation of the loss function f(θ) to θ, β1 is the first moment decay coefficient, β2 is the second moment decay coefficient, m t is the expectation of the gradient g t , v t is the expectation of , m t is the bias correction of , v t is the bias correction of , θ t-1 is the parameter before network update, θ t is the parameter after network update, and α is the learning rate.

[0230] The real shot concrete image is input into the trained Faster-rIR-7EC concrete crack recognition algorithm as a test set, the algorithm structure is shown in Figure 3 , and finally the concrete crack image recognition result is output, as shown in Figure 5 is the crack recognition result image obtained by using the Faster-rIR-7EC concrete crack recognition algorithm.

[0231] Referring to Figure 1 , in the embodiment, the Faster-rIR-7EC algorithm proposed in the patent and the general neural network as the Faster-rcnn feature extraction layer, including vgg16, resnet34 and Mobilenet_v3_large which also contains a large number of inverted residual structures, are trained respectively, and after 20 cycles of training, the above networks are used to recognize cracks of the concrete crack image test set, the model size, training time, mAP (mean Average Precision), mAR (mean Average Recall) and FPS (number of images processed per second) of each model are shown in Table 3.

[0232] Table 3 Model size, training time, mAP, mAR and FPS of each model

[0233]

[0234] wherein the precision is the proportion of correct judgments in all positive samples, the higher the precision, the lower the possibility of false positives of the network. The mAP is the average of the precision of all categories of samples. The precision calculation formula is as follows:

[0235]

[0236] The recall (True Positive Rate) is the proportion of all positive samples predicted correctly to all actual positive samples, the higher the recall, the lower the possibility of false negatives of the network. The mAR is the average of the recall of all categories of samples. The recall calculation formula is as follows:

[0237]

[0238] wherein TP, FP and FN are explained as follows: TP: the number of detection boxes with IOU>0.5; FP: the number of detection boxes with IOU≤0.5; FN: the number of GTs (ground truths) that are not detected.

[0239] From table 3, the proposed Faster-rIR7-EC concrete crack recognition algorithm has smaller model size, shorter model training time and higher FPS compared with general network VGG16_bn, resnet34 and mobilenet_large as Faster-rcnn feature extraction layer. At the same time, it still has mAP and mAR not inferior to general network with high memory occupation and long training time under the condition of predicting the intersection over union greater than 0.5 (IOU=0.5), 0.75 (IOU=0.75) of the GTbox. In order to further prove the universality of the model, the corresponding mAP is calculated under the condition of the intersection over union greater than 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, and then the average is obtained. The mAP under the condition of IoU=0.50:0.05:0.95 is 65.8%, which is only next to 66% of the mobilenet_large model, higher than other general networks, but the training time of the proposed Faster-rIR7-EC is 1 / 9 of VGG_bn and 1 / 3 of mobilenet_large, and the FPS is also far more than all general networks in table 3, which proves that the proposed Faster-rIR7-EC is fast and accurate in concrete crack recognition problem.

[0240] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for rapid identification of concrete cracks based on Faster-rIR7-EC, characterized in that, Includes the following steps: Construct a Faster-rIR7-EC network model; the Faster-rIR7-EC network model includes an IR7-EC feature extraction layer, a region generation network RPN, an ROI head, and a post-processing layer connected in sequence. The IR7-EC feature extraction layer comprises a 10-layer network framework connected sequentially. Layer 1 consists of a 3×3 convolutional layer, a batch normalization layer, and a Hardswish activation function connected sequentially. Layers 2 through 8 form an Inverted Residual-ECA structure, which includes four parts: the first part consists of a 1×1 convolutional layer, a batch normalization layer, and a ReLU6 activation function connected sequentially; the second part consists of a 3×3 convolutional layer, a batch normalization layer, and a ReLU6 activation function connected sequentially; the third part is an ECA attention module; the fourth part consists of a 1×1 convolutional layer, a batch normalization layer, and a linear activation function connected sequentially; layer 9 consists of a 3×3 convolutional layer, a batch normalization layer, and a Hardswish activation function; and layer 10 is a CBAM attention module. The ROI Head includes ROI pooling, a fully connected layer, and postprocess detections. The crack image to be identified is acquired, and the crack image is extracted at the channel and spatial levels through the IR7-EC feature extraction layer to obtain a feature extraction map containing crack information. The feature extraction map is input into the Region Generation Network (RPN), which generates candidate boxes on the feature extraction map and determines the cracks and background contained within the candidate boxes. The candidate boxes generated by the RPN are then projected onto the feature map output by the IR7-EC to obtain the corresponding feature matrix. By passing each feature matrix through an ROI Head, the location and type of concrete cracks in the feature map are output. The post-processing layer projects the location and type of concrete cracks in the feature map back to the actual crack image to obtain the actual location and type of concrete cracks. When constructing the IR7-EC feature extraction layer, the data from each layer passing through the ECA attention module are used for cross-channel interaction to obtain an enhanced concrete crack feature extraction map: In the formula, Represents the most recent odd number t ; C The number of channels representing the data input to the ECA attention module. γ and b There are two hyperparameters; For the output of the ECA attention module, For sigmoid operations, Indicates to proceed k*k Convolution operation, Given the input feature map, For average pooling; When constructing the IR7-EC feature extraction layer, average pooling and max pooling are used to aggregate the spatial information of the feature maps, compress the spatial dimension of the input feature maps, and sum and merge them element by element to generate channel attention maps: In the formula, This represents the output of the channel attention module. It consists of a fully connected layer 1, a ReLU6 activation function, and a fully connected layer 2. For sigmoid operations, Given the input feature map, For average pooling, For max pooling, For sigmoid operations; When constructing the IR7-EC feature extraction layer, the input feature map is compressed in the spatial attention module using the following formula with average pooling and max pooling methods to obtain a feature extraction map containing more crack information: In the formula, This represents the output of the spatial attention module. For sigmoid operations, This indicates that a 7x7 convolution operation is being performed. Given the input feature map, For average pooling, This is for max pooling.

2. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 1, characterized in that, Also includes: Training the Faster-rIR7-EC network model includes the following steps: Collect crack images and establish a crack type recognition image dataset, which includes a crack recognition image training set and a validation set; Normalize the crack image dataset: Traverse all crack images, find the image with the largest height and width, use the largest width and height as the template, align the top left corner of all other images with the template, and pad the right and bottom sides with 0s if the size is insufficient until they are the same as the template, so that the input image resolution is uniform and consistent with the template. The normalized image dataset is input into the Faster-rIR7-EC network model to obtain the recognition results; The identification results are compared with the real results in the training set, and the loss is fed into the optimization function to update the network parameters until the network fits; the loss includes RPN loss and ROI Head loss.

3. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 1, characterized in that, The feature extraction process of the IR7-EC feature extraction layer includes the following steps: The normalized image sample data is input into the first layer, and then passes through the convolutional layer, the batch normalization layer, and the Hardswish activation function to perform preliminary feature extraction, data normalization, and nonlinear transformation on the image sample data. The features initially extracted by the first layer are input into the second to eighth layers, and then sequentially through seven Inverted Residual-ECA structures. After processing by the first part of the Inverted Residual-ECA structure, the input data channel dimension is increased. The data with increased channel dimension is input into the second part for deep feature extraction. The data with deep feature extraction is input into the third part, and the ECA attention module is used to weight the network channels to highlight the channels that improve the network accuracy, resulting in an enhanced concrete crack feature extraction map. The enhanced concrete crack feature extraction map is input into the fourth part to reduce the data channel dimension. The data is then passed through all Inverted Residual-ECA structures in a similar manner. The data obtained from layers 2 to 8 are input into layer 9 for feature extraction. The data extracted through the 9th layer is input into the 10th layer CBAM attention module to further extract features at the channel and spatial levels, resulting in a feature map containing more crack information.

4. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 3, characterized in that, The normalization process of the Batch normalization layer is shown in the following formula: In the formula, The feature map is used as input for batch normalization. To output the feature map after batch normalization, This represents the number of feature maps input to this layer in the current training batch. and These are variables that change as the network gradient is updated. When constructing the IR7-EC feature extraction layer, the data in each layer that have passed the ReLU6 activation function are non-linearly processed using the following formula: In the formula, The feature map before the input ReLU6 activation function. To output the feature map of the ReLU6 activation function; When constructing the IR7-EC feature extraction layer, the data from each layer that has passed the Hardswish activation function are used for nonlinear processing, as shown in the following formula: In the formula, This is the feature map before inputting the Hardswish activation function. This is the feature map for the output Hardswish activation function.

5. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 1, characterized in that, The RPN includes an anchor generator and an RPNhead.

6. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 5, characterized in that, Also includes: The process of generating anchor frames using the anchor generator includes the following steps: Based on the 42×42 resolution feature map output by the IR7-EC feature extraction layer, 1764 sets of anchor boxes with different proportions are generated by the Anchor generator. Each pixel on the feature map is projected onto the original crack image before inputting into the network. A set of anchor boxes generated by the Anchor generator is assigned to each of the 1764 feature map projection positions on the original image. The rules for dividing crack images into positive and negative samples based on anchor boxes generated by the anchor generator are as follows: cracks are considered positive, and the absence of cracks is considered negative (background). A crack image includes a series of ground truth bounding boxes (GT Boxes) and anchor boxes. The rule for determining an anchor box as positive is: its Intersection over Union (IOU) with the ground truth box. The anchor frame; when the IOU of all anchor frames intersecting the GT Box is less than 0.7, the anchor frame with the largest IOU with the GT Box is directly determined as Positive; The negative discrimination rule is: IOU with GT Box. The anchor frame; 256 positive and negative samples were randomly selected as the true labels to calculate the target loss in RPNhead. The IOU calculation formula is as follows: in, The area of ​​the anchor frame. Let be the area of ​​the GT Box frame.

7. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 6, characterized in that, The RPNhead includes a 3×3 convolutional layer, two parallel 1×1 convolutional layers, and a ReLU activation function; The training steps for the RPNhead include: The 42×42 resolution feature map output by IR7-EC is passed through a 3×3 convolutional layer, then through two parallel 1×1 convolutional layers and a ReLU activation function. The target scores and bounding box regression parameters for all anchor boxes corresponding to all pixels in the output feature map are as follows: in, The crack probability predicted by RPNhead. Represents the first prediction by RPNhead The bounding box regression parameters of each anchor box; The anchor boxes are adjusted using bounding box regression parameters to obtain candidate box proposals, as shown in the following formula: in, The center coordinates of the proposals And the bounding box width and height, The center coordinates and width and height of the anchor frame. The bounding box regression parameters predicted by RPNhead; Filter proposals: Based on the target score of each proposal in the crack image, the top 2000 proposals with the highest crack probability are selected, those that do not meet the criteria are deleted, and then proposals with smaller areas are deleted. Find the proposal with the largest coordinate among all proposals, calculate the intersection and union of other proposals with it, use the non-maximum suppression algorithm to filter all proposals, and finally project the proposals onto the feature map output by IR7-EC to obtain the corresponding feature matrix. The RPNhead loss calculation method is as follows: in, The loss for RPN includes classification loss and bounding box regression loss. Indicates the first i The probability that each anchor is the target is calculated when the anchor is a positive sample. When the value is 1, it is a negative sample. =0, Indicates the predicted first i The bounding box regression parameters of each anchor. Indicates the first i The bounding box regression parameters of the ground truth boxes corresponding to each anchor. This represents the total number of samples in a mini-batch. Indicates the number of sample locations. Represents the first prediction by RPNhead Boundary box regression parameters of each anchor box Center coordinates of GT Box And the bounding box width and height, Used to balance classification loss and bounding box regression loss.

8. The method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 7, characterized in that, Also includes: All proposals were extracted using ROI pooling, and all proposals were converted into 7×7 feature maps. The fully connected layer structure of the ROI Head consists of two cascaded fully connected layers (FC1, FC2). After flattening the feature map, it is passed through the two fully connected layers and then fed into two parallel fully connected layers (FC3, FC4) to predict the crack category score and bounding box regression parameters for each proposal. The bounding box regression parameters output by FC4 are used to adjust the proposals. The fully connected layer loss is calculated through a similar process to RPN as follows: in, The loss function for the ROI Head includes classification loss and bounding box regression loss. It is the softmax probability distribution predicted by the classifier. , k Increment the number of crack categories by 1. Corresponding to the target's true category label, The corresponding category predicted by the bounding box regressor FC4 u The bounding box regression parameters, v Corresponding GTBox bounding box regression parameters ; The postprocess detections are used to postprocess the network's prediction data, including: The final bounding box (bbox) coordinates are calculated based on the proposals and the regression parameters predicted by FC4. The crack probability is obtained by performing softmax processing on the predicted category results; Remove all background information; Remove low-probability targets and small-sized targets; The prediction results in the ROI Head are filtered using the NMS method; The final formula for calculating the bounding box coordinates is as follows: in, The center coordinates of the proposals output by RPN And the bounding box width and height, These are the bounding box regression parameters predicted by the fully connected layer FC4. The center coordinates and width and height of the crack location bounding box predicted for the ROI Head.

9. A method for rapid identification of concrete cracks based on Faster-rIR7-EC according to claim 2, characterized in that, The updated network parameters are shown in the following formula: oss in, It is the loss function of the network RPN or ROI Head. These are the parameters in the model that need to be updated. It is a loss function right The gradient obtained by differentiation, It is the first-order moment attenuation coefficient. It is the second-order moment attenuation coefficient. It is the gradient Expectations yes Expectations yes Bias correction, yes bias correction, These are the parameters before the network update. These are the updated parameters from the network. It is the learning rate.

Images