Concrete member crack measurement method based on convolutional neural network

Abstract

The invention discloses a concrete member crack measurement method based on a convolutional neural network, and relates to the field of concrete crack detection, and the method comprises the steps: carrying out the marking, size clustering, image enhancement, coordinate transformation and other preprocessing operations of a concrete member image, and constructing a detection model architecture; wherein the backbone network adopts a pre-trained ResNet-50 network to extract multi-scale features of the image; the box neck and the box head are used for generating local crack box output; the grid neck and the grid head are used to generate a global fracture grid output. In a model training process, local box output and global grid output are optimized by adopting a joint loss function, and prior category information of crack size clustering is introduced to guide learning of a detection model. In the reasoning stage, an image to be detected is input into the trained detection model, box output and grid output are obtained, and a final crack detection result is obtained through post-processing such as non-maximum suppression. And aiming at low crack detection precision under a complex background, the detection precision is improved.

Description

Technical Field

[0001] The present application relates to the field of concrete crack detection, and in particular to a method for measuring cracks in concrete components based on a convolutional neural network. Background Art

[0002] With the rapid development of my country's economic construction, concrete has become one of the most important building materials in the field of civil engineering and construction. However, due to the characteristics of concrete materials and the influence of construction technology, cracks will inevitably appear in concrete components during use. The existence of cracks will not only affect the appearance of concrete components, but also reduce the durability and safety of components and shorten the service life of components. Therefore, crack detection and evaluation of concrete components is an important means to ensure the quality and safety of the project.

[0003] Traditional concrete crack detection mainly relies on manual visual observation and manual measurement, which has disadvantages such as low efficiency and strong subjectivity. In order to overcome the limitations of manual detection, researchers began to explore the use of image processing and computer vision technology to realize automatic crack detection and measurement. Among them, machine learning methods have received widespread attention. By building a classifier or regression model, it can automatically identify the crack area in the image and estimate the width and length of the crack. However, traditional machine learning methods are usually based on manually designed feature extraction and shallow learning models. It is difficult to effectively process crack images in complex environments and has the problem of poor measurement robustness.

[0004] In recent years, deep learning technology represented by convolutional neural networks has made great progress and demonstrated powerful feature learning and representation capabilities in image classification, target detection and other fields. Introducing deep learning into the field of concrete crack detection and measurement is expected to overcome the shortcomings of traditional methods and improve the robustness and accuracy of crack measurement. However, directly applying existing general detection models to crack measurement tasks still faces challenges. First, concrete crack images often have complex backgrounds and illumination changes, which puts higher requirements on the generalization ability of the detection model; second, the diversity of crack sizes and shapes increases the difficulty of detection and positioning; in addition, the lack of prior knowledge of cracks also limits the interpretability and reliability of the detection model. Summary of the invention

[0005] In response to the problem of poor robustness of concrete component crack measurement in the prior art, the present application provides a concrete component crack measurement method based on a convolutional neural network. By annotating, size clustering, image enhancement and coordinate transformation of concrete component images, a detection model including a backbone network, a box neck, a box head, a grid neck and a grid head is constructed, and a joint loss function is used for model training, etc., thereby improving the robustness of crack detection under complex backgrounds and lighting changes.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present application provides a method for measuring cracks in a concrete component based on a convolutional neural network: acquiring image data of a concrete component, and marking the position and size information of the crack to obtain a frame label and a grid label; combining the image data, the frame label and the grid label to form a data set; preprocessing the data set; the preprocessing includes size clustering of the frame label to divide the crack size into multiple categories; enhancing the image data; coordinate transformation of the frame label and the grid label; constructing a detection model architecture including a backbone network, a box neck, a box head, a grid neck and a grid head; wherein the backbone network adopts a pre-trained ResNet-50 network; the box neck and the box head are used to generate a box output, and the grid neck and the grid head are used to generate a grid output; inputting the preprocessed data set into the detection model, optimizing the model through a joint loss function, and training the model using the frame label after size clustering and the enhanced image data; inputting the image of the concrete component to be measured into the trained detection model to obtain the box output and the grid output generated by the detection model; performing non-maximum suppression processing on the box output and the grid output to obtain a crack detection result; and calculating the length, width and area of ​​the crack according to the crack position and size information in the crack detection result.

[0008] Furthermore, image data of concrete components are obtained, and the position and size information of cracks are marked to obtain frame labels and grid labels; the image data, frame labels and grid labels are combined to form a data set, including: collecting image data of the surface of concrete components; marking the position of cracks on the image data, and using a minimum circumscribed rectangular frame to select the crack area, the position coordinates and size information of the rectangular frame are used as frame labels, and the frame labels represent the position and size of the cracks in the image; using a sliding window to divide the image data into multiple square grids, and assigning a label value between 0 and 1 to each grid according to the proportion of crack pixels in each grid, the label value reflects the proportion of cracks in the grid, and the label values ​​of all grids constitute grid labels, and the grid labels reflect the distribution of cracks in the image.

[0009] Furthermore, the size of the frame labels is clustered to divide the crack sizes into multiple categories, including: extracting the crack size information in the frame labels, obtaining the width and height data of all cracks, and forming a size data set; using the minimum-maximum normalization method to linearly transform the width and height data to the range of 0 to 1; setting the number of clusters k, randomly selecting a data point as the first cluster center in the normalized size data, calculating the shortest distance from each data point to the selected cluster center, and selecting the next cluster center according to the ratio of the square of the distance to the sum of the squares of the distances, and repeating the steps of selecting cluster centers until k initial cluster centers are selected; calculating the Mahalanobis distance from each data point in the normalized size data set to the k initial cluster centers, and clustering each data point according to the nearest Mahalanobis distance. The points are divided into corresponding clusters to obtain k initial clusters; the size data in each initial cluster is fitted by the least squares method to obtain the new center point coordinates of each cluster; the sum of the Euclidean distances between the new center point of each cluster and the original center point in the two dimensions of width and height is calculated as the change of the cluster center. If the change is less than the threshold, the clustering result is considered to have converged and the clustering process is terminated. Otherwise, the new center point is used as the current cluster center and the iterative optimization center point is returned; the k clustering results that finally converge are used as k categories of crack size, and the width and height values ​​of the center point of each category are used as the size interval boundaries of the corresponding category. The cracks with the same size in the box label are classified into the same group, and cracks of different sizes are grouped.

[0010] Furthermore, the image data is enhanced, including: geometrically transforming the image data, using random horizontal flipping and random vertical flipping to mirror-reverse the image data left and right and up and down with a preset probability to obtain the image data after the geometric transformation; scaling the geometrically transformed image data, using random scaling to scale the geometrically transformed image data within a preset scale range, and using random cropping to randomly select a fixed-size area on the scaled image as new image data to obtain the image data after the scale transformation; superimposing random occlusion on the crack area of ​​the scaled image data, using a square with a color similar to the background at a random position in the crack area to obtain the image data after adding the random occlusion; mixing the original image data with the image data after adding the random occlusion according to a preset ratio, and mixing the crack position and size information in the frame label and the grid label accordingly to obtain the enhanced image data, frame label and grid label.

[0011] Specifically, a square with a similar background color refers to a rectangular area that is similar or close to the background color of the image. It is usually used to block or cover a specific area in an image to simulate the occlusion in a real scene. The square with a similar background color can be generated in the following way: statistically calculate the color distribution of the background area of ​​the image, calculate the average or mode of the background color, and use it as the color of the square. Randomly sample a small block from the background area of ​​the image and use its color as the color of the square. According to a preset color range, randomly generate a color value close to the background color as the color of the square. In this application, according to label information or detection results, determine the area where cracks exist in the image. The crack area can be the area where one or more crack instances are located. According to the background color around the crack area, generate a square with a similar background color. The size of the square can be determined according to the size of the crack area and the preset occlusion ratio. The color of the square can be obtained by the above-mentioned statistical, sampling or random generation method. Randomly select a position in the crack area as the center of the square. Ensure that the square is completely covered in the crack area and does not exceed the boundary of the crack area. Superimpose the generated square with a similar background color on the crack area. The area covered by the squares will occlude the cracks, simulating the occlusion in real scenes. The superimposed image data will contain the crack areas after random occlusion. For the crack areas superimposed with squares of similar background colors, the crack position and size information in the box label and grid label are updated accordingly. If the square completely occludes the crack, the corresponding label information can be marked as "occluded" or removed. If the square partially occludes the crack, the crack position and size information in the label can be adjusted accordingly. By superimposing squares of similar background colors on the crack area, the occlusion in real scenes can be simulated, increasing the diversity and robustness of the image data. This helps to improve the model's ability to detect cracks under occlusion, making the model more adaptable to complex scenes in practical applications.

[0012] Furthermore, coordinate transformation of the frame label and the grid label is performed, including: normalizing the frame label coordinates, dividing the crack position coordinates in the frame label by the width and height of the image data to obtain a normalized frame label; mapping the grid label coordinates, mapping the position coordinates of each grid in the grid label to a coordinate space consistent with the size of the image data according to the width and step size of the window when dividing the grid, to obtain a mapped grid label; encoding the normalized frame label and the mapped grid label coordinates, using log-square encoding, encoding the crack center coordinates, width and height in the normalized frame label as a preset offset and scaling factor, encoding the grid center coordinates, width and height in the mapped grid label as a preset offset and scaling factor, to obtain an encoded frame label and grid label; adding a category label obtained by crack size clustering to the encoded frame label, adding the size category index to which the crack area belongs to the corresponding encoded frame label, to obtain an encoded frame label with a size category.

[0013] Furthermore, a detection model architecture including a backbone network, a box neck, a box head, a grid neck and a grid head is constructed; wherein the backbone network adopts a pre-trained ResNet-50 network; the box neck and the box head are used to generate a box output, and the grid neck and the grid head are used to generate a grid output, including: the backbone network receives enhanced image data, and adopts a pre-trained ResNet-50 network to extract multi-scale features of the image to obtain a multi-scale feature map; the feature pyramid network FPN is connected to the multi-scale feature map extracted by the backbone network, and the multi-scale feature map is upsampled and fused through the FPN to obtain a fused feature map; the box neck and the grid neck are respectively connected after the fused feature map, and the box neck and the grid neck respectively use convolutional neural networks to extract features and transform dimensions of the fused feature map to obtain transformed feature maps for predicting the box output and the grid output respectively; the box head is connected after the transformed feature map output by the box neck, and the grid head is connected after the transformed feature map output by the grid neck, and the box head and the grid The head uses a fully connected layer and a sigmoid activation function respectively, maps the transformed feature map of the box neck output to a box prediction output that matches the encoded box label with the size category, and maps the transformed feature map of the grid neck output to a grid prediction output that matches the encoded grid label after the transformation; decodes the box prediction result output by the box head, obtains the position coordinates, width, height and size category of the detected crack area after decoding, and maps the decoded width and height back to the original size space according to the category interval boundary determined by size clustering to obtain the box output, which contains the crack position, size and category information; performs threshold processing on the grid prediction result output by the grid head, determines the grid with a prediction value greater than the threshold in the grid prediction result as the grid containing the crack, decodes the center coordinates, width and height of the grid containing the crack, maps the decoded center coordinates, width and height of the grid to the image coordinate space, and obtains the grid output, which contains the crack distribution information.

[0014] Furthermore, the preprocessed data set is input into the detection model, the model is optimized by the joint loss function, and the model is trained using the size-clustered box labels and the enhanced image data, including: the preprocessed image data, the encoded box labels with size categories, and the encoded grid labels after shape transformation are used as training data sets, and input into the constructed detection model; the parameters of the detection model are initialized, and the weights of the convolutional layers and fully connected layers of the backbone network, box neck, box head, grid neck and grid head are initialized by the Xavier initialization method, and the bias is initialized to 0; a joint loss function is constructed, and the joint loss function includes a box loss function and a grid loss function; the box loss function adopts Smooth The L1 loss function is used to optimize the error between the box prediction result output by the box head and the encoded box label with size category; the grid loss function adopts the binary cross entropy loss function, which is used to optimize the error between the grid prediction result output by the grid head and the encoded grid label after shape transformation; according to the set batch size and number of iterations, the detection model is iteratively trained using the mini-batch gradient descent algorithm. In each iteration, a batch of image data, box labels and grid labels are randomly selected from the training data set and input into the detection model. The box prediction output and grid prediction output are calculated by forward propagation. The loss between the predicted output and the label is calculated using the joint loss function. The gradient of the loss function to the model parameters is calculated through the back propagation algorithm. The loss function is minimized to obtain the trained detection model.

[0015] Furthermore, the box loss function adopts the Smooth L1 loss function, which is expressed as follows:

[0016]

[0017] Where N is the number of images in the training dataset; M is the number of boxes detected in each image; I ij represents the indicator function of the jth box in the i-th image. When the box matches the ground-truth box label, I ij =1, otherwise I ij =0; Represents the predicted output of the jth box in the ith image, including the center coordinates width high and category confidence Represents the ground truth box label of the jth box match in the i-th image, including the center coordinate label Width Tag Height Label and class confidence labels represents the Smooth L1 function; α represents the weight coefficient, which is used to balance the ratio of width and height loss to center coordinate loss; β represents the weight coefficient, which is used to balance the ratio of category confidence loss to position loss; represents the binary cross entropy function,

[0018] Furthermore, the grid loss function adopts the binary cross entropy loss function, which is expressed as follows: Where H represents the number of grid rows after the image is divided into grids; W represents the number of grid columns after the image is divided into grids; y ij represents the true label of the grid in the i-th row and j-th column, which takes a value of 0 or 1, indicating whether the corresponding grid contains a crack; Represents the predicted output of the grid in the i-th row and j-th column, with a value between 0 and 1, indicating the confidence that the corresponding grid contains cracks.

[0019] Further, the image of the concrete component to be tested is input into the trained detection model to obtain the box output and grid output generated by the detection model; the box output and the grid output are subjected to non-maximum suppression processing to obtain the crack detection result, including: inputting the image of the concrete component to be tested into the trained detection model, and obtaining the box output and the grid output by forward propagation calculation; wherein the box output includes the predicted position, size and category confidence of the crack box, and the grid output includes the predicted position and confidence of the crack grid; according to a preset crack size threshold, the box output is size-filtered to filter the boxes with a size smaller than the threshold to obtain the size-filtered box output; according to a preset intersection-over-union ratio threshold and a confidence threshold, the size-filtered box output is subjected to non-maximum suppression processing to remove the boxes with an overlap higher than the threshold and a confidence lower than the threshold to obtain the crack box detection result; according to a preset confidence threshold, the grid output is binarized to mark the grid with a confidence higher than the threshold as containing cracks to obtain the crack grid detection result; the crack box detection result is combined with the crack root detection result to obtain the crack detection result of the concrete component to be tested, and the position, size and category of the crack are output.

[0020] Compared with the prior art, the advantages of this application are:

[0021] This application can expand the diversity of training data and enhance the adaptability of the detection model to complex backgrounds, illumination changes and other factors by preprocessing the training data set through size clustering, image enhancement, coordinate transformation, etc. Among them, size clustering can divide cracks of different sizes into multiple categories, so that the detection model can adapt to cracks of different scales; image enhancement simulates the changes of images in complex environments through operations such as geometric transformation, scale transformation and random occlusion, and improves the generalization ability of the detection model; coordinate transformation encodes label information into a unified coordinate space, which is conducive to extracting consistent feature representations. Therefore, the preprocessed data set can comprehensively improve the robustness and measurement accuracy of the detection model in complex environments.

[0022] The detection model constructed in this application includes two branches: box output and grid output, which extract the characteristic information of cracks from local and global perspectives respectively. Among them, the box output is generated by the box neck and box head, which can accurately locate and characterize the local crack area and obtain detailed crack position and size information; the grid output is generated by the grid neck and grid head, which can characterize the distribution of cracks in the entire image range and obtain global crack shape and structure information. By optimizing the local box output and the global grid output through the joint loss function, the local and global crack features can be fully integrated to make up for the shortcomings of single feature representation, thereby achieving high-precision crack detection and measurement.

[0023] This application introduces the prior category information obtained by clustering crack sizes into the box loss function. By dividing the crack sizes into multiple categories, the detection model is provided with guidance of size prior knowledge. During the model training process, the size category information participates in the optimization of the loss function together with the position and size prediction, so that the detection model can learn the prior distribution law of the crack size. This training method that integrates prior knowledge can enhance the interpretability of the detection model, so that it can give clear category attributes when predicting the crack size, which is convenient for judging and grading the severity of cracks in engineering applications.

[0024] After obtaining the box output and grid output generated by the detection model in the inference stage, the application fuses and post-processes the two outputs. First, the box output is size filtered and non-maximum suppressed to remove redundant boxes with small size and high overlap, and accurate crack box detection results are obtained; then, the grid output is binarized to obtain a complete crack grid distribution result. Finally, the box detection results and grid detection results are combined to combine the advantages of the two detection results to obtain crack detection results that take into account both accuracy and completeness. This method of combining box detection and grid detection can effectively improve the performance of crack detection in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0026] Figure 1 is an exemplary flow chart of a method for measuring cracks in a concrete member based on a convolutional neural network according to some embodiments of the present application;

[0027] Figure 2 is an exemplary flow chart of obtaining a label frame and a grid label according to some embodiments of the present application;

[0028] Figure 3 is an exemplary flow chart of performing size clustering processing on image data according to some embodiments of the present application;

[0029] Figure 4 is an exemplary flow chart of performing data enhancement processing on image data according to some embodiments of the present application;

[0030] Figure 5 It is an exemplary flow chart of the grid structure shown in some embodiments of the present application. DETAILED DESCRIPTION

[0031] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 The present invention is an exemplary flow chart of a method for measuring cracks in a concrete component based on a convolutional neural network according to some embodiments of the present application, comprising: acquiring image data of a concrete component, and marking the position and size information of the cracks to obtain a frame label and a grid label; combining the image data, the frame label and the grid label to form a data set; preprocessing the data set; the preprocessing includes size clustering of the frame labels to classify the crack sizes into multiple categories; enhancing the image data; performing coordinate transformation on the frame labels and the grid labels; constructing a detection model architecture including a backbone network, a box neck, a box head, a grid neck and a grid head; wherein the backbone The network adopts a pre-trained ResNet-50 network; the box neck and box head are used to generate box output, and the grid neck and grid head are used to generate grid output; the preprocessed data set is input into the detection model, and the model is optimized by a joint loss function, and the model is trained using the box labels after size clustering and the enhanced image data; the image of the concrete component to be tested is input into the trained detection model to obtain the box output and grid output generated by the detection model; the box output and grid output are subjected to non-maximum suppression processing to obtain the crack detection results; according to the crack position and size information in the crack detection results, the length, width and area of ​​the crack are calculated.

[0033] Data preparation stage: Image data acquisition: Select the concrete component to be inspected, and use a high-resolution digital camera or industrial camera to shoot the component surface to obtain a clear digital image. Control the lighting conditions during shooting to avoid the effects of shadows, reflections and other factors on image quality. The shooting angle should be as close to the component surface as possible, and the distance between the camera and the component surface should be kept consistent to reduce the geometric distortion of the image. According to the size and complexity of the component, collect multiple local images to ensure that the image covers all areas of the component.

[0034] Figure 2 This is an exemplary flow chart for obtaining label boxes and grid labels according to some embodiments of the present application. Crack annotation: Use image annotation software, such as LabelImg, LabelMe, etc., to load the collected concrete component image. Experienced engineers or technicians will annotate the crack areas in the image. Use the mouse to draw a rectangular box on the image to select the minimum circumscribed rectangular area of ​​each crack. Record the position coordinates (x, y) and size information (width, height) of each rectangular box, and generate a corresponding label file, such as XML, JSON and other formats. The position coordinates and size information constitute a box label, which indicates the exact position and size of the crack in the image.

[0035] Grid label generation: Preprocess the collected image data and adjust the image size and resolution to make it suitable for subsequent grid division. Use a sliding window method to divide the image into multiple square grids of fixed size. The size of the grid is set according to the typical size of the crack and the detection accuracy requirements. For each grid, count the number of crack pixels and calculate the proportion of crack pixels to the total pixels of the grid. Map the proportion of crack pixels to the interval [0, 1] to obtain the label value of the grid. The closer the label value is to 1, the higher the proportion of cracks in the grid; the closer the label value is to 0, the grid basically contains no cracks. Generate a grid label matrix of the same size as the image. Each element in the matrix corresponds to the label value of a grid, reflecting the distribution of cracks in the image.

[0036] Dataset construction: The collected concrete component images, corresponding crack box labels and grid labels are combined into samples of the dataset. Each sample includes image data, box label files and grid label matrices to form a complete training data unit. A certain number of concrete component images are collected to construct a dataset of appropriate size. The size of the dataset is determined according to the model complexity and performance requirements. The dataset is divided into training set, validation set and test set according to a certain ratio for model training, tuning and evaluation.

[0037] Figure 3This is an exemplary flow chart of size clustering processing of image data according to some embodiments of the present application. In the data preprocessing stage: size clustering of frame labels is performed to divide the crack sizes into multiple categories. Crack size data extraction: the width and height information of each crack frame is read from the frame label file to form a crack size data set. Each sample of the crack size data set includes two features: width and height, which represent the size of the crack. Data normalization: the minimum-maximum normalization method is used to normalize the crack size data set. For the width feature, the minimum width min in the data set is found. width and the maximum width max width , normalize the width value width of each sample: For the height feature, find the minimum height min in the data set height and the maximum height max height , normalize the height value height of each sample:

[0038] The normalized width and height values ​​are both in the range of [0, 1], eliminating the dimension effect of the size data.

[0039] Initialization of cluster centers: Set the number of clusters k: According to the distribution characteristics of crack size and application requirements, determine the appropriate number of clusters k, which can usually be selected through domain knowledge or data visualization. For example, you can observe the distribution of data based on the histogram of crack width and height, and select the number of clusters that can better reflect the characteristics of the data. Select the first cluster center: Randomly select a data point from the normalized size data set as the first cluster center, denoted as c1. Random selection can be achieved by generating a random integer index between 0 and the data set size - 1.

[0040] Calculate the Mahalanobis distance: For each data point x in the dataset i , calculate its Mahalanobis distance d(x i ,c l ). The calculation formula of Mahalanobis distance is: Where S is the covariance matrix of the data set, T represents the transpose, and -1 represents the inverse of the matrix. The Mahalanobis distance takes into account the correlation between features and can better measure the similarity between data points. Select the next cluster center: Calculate the shortest Mahalanobis distance min_d(x i ). Calculate the shortest Mahalanobis distance sum of all data points to the selected cluster center, denoted as D. For each data point x i , calculate the probability of being selected as the next cluster center According to the probability P(x i), use the roulette wheel method to select the next cluster center, that is, the data point with a larger probability is more likely to be selected. Repeat the selection of cluster centers: Repeat until k initial cluster centers {c1, c2, ..., c k This probability-based selection method can make the cluster centers more evenly distributed and cover different areas of the data set.

[0041] Clustering process: Calculate the Mahalanobis distance: For each data point in the normalized size dataset x i , calculate it to k initial cluster centers {c1,c2,......,c k}. Get an N*k distance matrix, where N is the number of data points and k is the number of cluster centers. Assign data points to clusters: For each data point x i , find its minimum Mahalanobis distance to the k cluster centers, and assign it to the corresponding cluster to form k initial clusters {c1,c2,......,c k}, each cluster contains a part of the data points. Update cluster center: For each initial cluster C j , use the least squares method to fit the data points in the cluster and get the new cluster center c' j The least squares method finds the optimal center point coordinates by minimizing the sum of squared distances from data points within the cluster to the cluster center.

[0042] Calculate the change in cluster center: For each cluster C j , calculate the new cluster center c' j With the original cluster center c j The Euclidean distance in the width and height dimensions gives the distance vector d j . Calculate the L1 norm of the distance vectors of all clusters, that is, sum them up, as the total change δ of the cluster center. Determine whether it has converged: If the total change delta is less than the preset threshold ε, the clustering result is considered to have converged and the clustering process is terminated. Otherwise, the new cluster center {c'1, c'2, ..., c' k} as the current cluster center and return to continue iterative optimization.

[0043] Crack size classification: Determine the size category: The k clustering results {C1, C2, ..., C k} as k categories of crack size. Each cluster corresponds to a size category, and the data points in the cluster belong to the same size category. Determine the size interval boundary: For each size category C j , with its cluster center c' j The width and height values ​​of the class are used as the size interval boundaries of the class. The width interval is [c' j_width -δwidth ,c' j_width +δ width ], the height interval is [c' j_height -δ height ,c' j_height +δ height ], where δ width and δ height are preset width and height tolerance values.

[0044] Group crack frames: Traverse each crack frame in the frame label and extract its width width and height height. For each crack frame, determine whether its width and height fall within the interval of a certain size category. If the width and height of the crack frame are within the interval of the same size category, the crack frame is classified into the corresponding size group. If the width and height of the crack frame belong to different size categories, it is classified into the size group with a larger area according to its area. Output the grouping result For each size category, output the index or identifier of the crack frame belonging to that category. Get k crack size groups, each of which contains crack frames belonging to the same size category.

[0045] Figure 4 This is an exemplary flow chart of performing data enhancement processing on image data according to some embodiments of the present application, and the image data is enhanced, including geometric transformation, scale transformation and random occlusion. Geometric transformation: Randomly flip the original image data horizontally. Set the probability p of horizontal flipping h , generate a random number r between 0 and 1 h If r h Less than p h , the image is flipped horizontally, that is, the image is mirrored left and right; otherwise, the image remains unchanged. The original image data is randomly flipped vertically. Set the probability of vertical flip p v , generate a random number r between 0 and 1 v If r v Less than p v , the image is flipped vertically, that is, the image is mirrored up and down; otherwise, the image is kept unchanged. The horizontally flipped and vertically flipped images are used as the image data after geometric transformation.

[0046] Scale transformation: Randomly scale the image data after geometric transformation. Set the scale range of the scale

[0047] [s min ,s max], generate a random scaling factor s within the range. Multiply the width and height of the image data by the scaling factor s to get the scaled image size. Use bilinear interpolation or nearest neighbor interpolation to scale the image data to the new size. Perform random cropping on the scaled image. Set the cropping target size to a fixed size (w crop ,h crop ). A rectangular area is randomly selected on the scaled image, with the coordinates of the upper left corner being (x, y) and the coordinates of the lower right corner being (x+w crop ,y+h crop ), where x and y are randomly generated. The selected rectangular area is used as the new image data to obtain the scale-transformed image data.

[0048] Random occlusion: Get crack area information: Extract the location and size information of each crack area in the image based on the box label or grid label. For the box label, directly get the bounding box coordinates (x, y, w,

[0049] h), indicating the coordinates of the upper left corner, width and height of the crack. For the grid label, the bounding box coordinates (x, y, w, h) of the corresponding crack area are calculated according to the position coordinates and size of the grid. Randomly select the occlusion position: For each crack area, randomly select a position inside it as the center point of the occlusion. Generate two random numbers r x and r y , respectively represent the offset of the occlusion center point relative to the upper left corner of the crack area. Calculate the coordinates of the occlusion center point (x occ ,y occ ), where x occ =x+r x ×w,y occ =y+r y ×h, ensuring that the occlusion center point is inside the crack area.

[0050] Generate occlusion blocks: Set the size of the occlusion blocks to a fixed size (w occ ,h occ ), which can be adjusted according to actual needs. Create a blank image of the same size as the occlusion to represent the occlusion square. Fill the occlusion square image with a color close to the background color, which can be obtained by randomly perturbing the background color. Convert the occlusion square image to the same data type and number of channels as the original image data. Overlay the occlusion square: Calculate the overlay position (x start ,y start ,x end ,y end ),in:

[0051]

[0052] Width and height represent the width and height of the original image data respectively. The occluded square image is superimposed on the corresponding position of the original image data, which is implemented using numpy's slicing operation:

[0053] img_data[y start :y end ,x start :x end ]=occ_block, where img_data represents the original image data and occ_block represents the occluded block image. After processing all crack areas, the image data with random occlusion added is obtained.

[0054] Data mixing: Mix the original image data with the image data after adding random occlusion. Set the mixing ratio to p mix , generate a random number r between 0 and 1 mix If r mix Less than p mix , use the image data after adding random occlusion; otherwise, use the original image data. Perform corresponding processing on the box label and grid label. If the image data after adding random occlusion is used, adjust the crack position and size information in the box label and grid label accordingly according to the position and size of the occlusion. If the original image data is used, keep the box label and grid label unchanged. The mixed image data, box label and grid label are used as the enhanced data.

[0055] The coordinate transformation of the box label and the grid label includes coordinate normalization, coordinate mapping and coordinate encoding. Coordinate normalization of box label: For each box label, extract its crack position coordinates (x, y, w, h), which represent the upper left corner coordinates and width and height of the crack box respectively. Get the width and height of the image data (img w ,img h ). Calculate the normalized crack position coordinates (x norm ,y norm ,w norm ,h norm ),in, The normalized crack position coordinates are used to replace the original crack position coordinates to obtain the normalized box label. Coordinate mapping of grid labels: Get the width and step size of the window when dividing the grid (grid w ,grid stride ). For each grid label, extract its grid position coordinates (grid x ,grid y), which represents the coordinates of the upper left corner of the grid. Calculate the actual position coordinates of the grid in the image data coordinate space (mapped x ,mapped y ), where mapped x =grid x ×grid stride , mapped y =grid y ×grid stride . Replace the original grid position coordinates with the actual position coordinates to obtain the mapped grid labels.

[0056] Coordinate encoding: For the normalized box labels and mapped grid labels, extract their position coordinates (x, y, w, h). Set the preset offset (t x ,t y ,t w ,t h ) and scaling factor (s x ,s y ,s w ,s h ). Using log-square encoding, calculate the encoded position coordinates (encoded x ,encoded y ,encoded w ,encoded h ),in:

[0057] The encoded position coordinates are used to replace the original position coordinates to obtain the encoded box label and grid label.

[0058] Add size category label: Get crack size category information: Get k size categories through crack size clustering algorithm, each category is represented by its center point coordinates (center w ,center h ) and size interval boundaries [min w ,max w ],[min h ,max h ] to represent. Store the size category information in a list or dictionary, where each element contains the category index, center point coordinates, and size interval boundaries, for example:

[0059] size_classes=[{'index': 0, 'center_w': 0.1, 'center_h': 0.2, 'min_w': 0.05, 'max_w': 0.15, 'min_h': 0.1, 'max_ h': 0.3}, {'index': 1, 'center_w': 0.3, 'center_h': 0.4, 'min_w': 0.2, 'max_w': 0.4, 'min_h': 0.3, 'max_h': 0.5}].

[0060] Traverse the encoded box labels: For each encoded box label, extract its crack position coordinates (x, y, w, h). Initialize a variable size_class to store the size category index to which the crack belongs. The initial value is -1, indicating unclassified. Determine the size category to which the crack belongs: traverse each category in the size category list size_classes. For each category, determine whether the width w of the crack falls within the width interval [min w ,max w ], whether the height h falls within the height range [min h ,max h ]. If the width and height of the crack both fall within the range of a certain category, the index of the category is assigned to size_class, indicating that the crack belongs to this size category. If the width and height of the crack fall within different category intervals, the category whose center point is closer to the width and height of the crack is selected as the size category to which the crack belongs. If the width and height of the crack do not fall within the range of any category, size_class is kept as -1, indicating that the crack does not belong to any predefined size category. Add size category label: Add the value of size_class as an additional label to the encoded box label. The encoded box label with size category is obtained in the format of (encoded x ,encoded y ,encoded w ,encoded h ). After processing all encoded box labels, return a list of encoded box labels with size categories.

[0061] Figure 5It is an exemplary flow chart of the grid structure shown in some embodiments of the present application. The model construction stage: backbone network feature extraction: adjust the enhanced image data to the input size required by the ResNet-50 network, usually 224x224 or 512x512. Input the adjusted image data into the first convolution layer (conv1) of the ResNet-50 network for initial feature extraction. The image data passes through the four stages (stage1-stage4) of ResNet-50 in sequence, and each stage contains multiple residual blocks. The first stage (stage1): extract low-level features through a 7x7 convolution layer and a maximum pooling layer. The second stage (stage2): extract intermediate features through multiple residual blocks, each of which contains two 3x3 convolution layers. The third stage (stage3): extract high-level features through multiple residual blocks, each of which contains three 3x3 convolution layers. The fourth stage (stage4): extract higher-level features through multiple residual blocks, each of which contains three 3x3 convolution layers. Get multi-scale feature maps from different stages of ResNet-50: conv2: output feature map of the first stage, with a size of 1 / 4 of the original image. conv3: output feature map of the second stage, with a size of 1 / 8 of the original image. conv4: output feature map of the third stage, with a size of 1 / 16 of the original image. conv5: output feature map of the fourth stage, with a size of 1 / 32 of the original image. The obtained multi-scale feature maps are used as the input of FPN.

[0062] Feature Pyramid Network (FPN): The multi-scale feature maps extracted by the backbone network are arranged in order from large to small scale, and are denoted as {C2, C3, C4, C5}. The number of channels of the feature maps of each scale is adjusted through a 1x1 convolution layer to make them consistent, and the adjusted feature maps {P2, P3, P4, P5} are obtained. A 3x3 convolution layer is performed on P5 to obtain P6. A maximum pooling layer with a step size of 2 is performed on P6 to obtain P7. P4 is upsampled to make its spatial size the same as P3, denoted as P4_upsampled. P4_upsampled is added to P3 at the element level to obtain the fused feature map P3'. A 3x3 convolution layer is applied to P3' to extract the fused feature representation and obtain the final P3 feature map. P3 is upsampled to make its spatial size the same as P2, denoted as P3_upsampled. P3_upsampled is added to P2 at the element level to obtain the fused feature map P2'. Apply a 3x3 convolution layer to P2', extract the fused feature representation, and get the final P2 feature map. Through the above steps, FPN obtains a series of feature maps {P2, P3, P4, P5, P6, P7} with different scales and semantic information, where: P2 is 1 / 4 of the original image in size and contains low-level features and high-level semantic information. P3 is 1 / 8 of the original image in size and contains mid-level features and high-level semantic information. P4 is 1 / 16 of the original image in size and contains high-level features and high-level semantic information. P5 is 1 / 32 of the original image in size and contains higher-level features and semantic information. P6 and P7 are obtained by convolution and pooling of P5, which are smaller in size and contain more abstract and global features.

[0063] Box Neck: The fused feature map output by FPN is used as the input of the box neck, denoted as P box . box A series of convolutional layers are applied to extract features relevant to box prediction. First convolutional layer: use 3x3 convolutional kernel, stride 1, padding 1, output channels 256, activation function ReLU. Second convolutional layer: use 3x3 convolutional kernel, stride 1, padding 1, output channels 256, activation function ReLU. Third convolutional layer: use 3x3 convolutional kernel, stride 1, padding 1, output channels 256, activation function ReLU. Fourth convolutional layer: use 3x3 convolutional kernel, stride 1, padding 1, output channels 256, activation function ReLU. The transformed feature map used to predict the box output is obtained, denoted as F box .

[0064] Grid Neck: The fused feature map output by FPN is used as the input of the grid neck, denoted as P_grid. A series of convolutional layers are applied to P_grid to extract features related to grid prediction. The first convolutional layer: uses a 3x3 convolutional kernel, a stride of 1, a padding of 1, an output channel number of 256, and an activation function of ReLU. The second convolutional layer: uses a 3x3 convolutional kernel, a stride of 1, a padding of 1, an output channel number of 256, and an activation function of ReLU. The third convolutional layer: uses a 3x3 convolutional kernel, a stride of 1, a padding of 1, an output channel number of 256, and an activation function of ReLU. The fourth convolutional layer: uses a 3x3 convolutional kernel, a stride of 1, a padding of 1, an output channel number of 256, and an activation function of ReLU. The transformed feature map used to predict the grid output is obtained, denoted as F grid .

[0065] Box Head: The transformed feature map F output by the box neck box As the input of the box head. box A fully connected layer is applied to map the feature map into a box prediction output that matches the encoded box label with size category. The input dimension of the fully connected layer is F box The number of channels of the output is (4+1+num_size_classes), where 4 represents the location coordinates of the box (x, y, w, h), 1 represents the confidence of the box, and num_size_classes represents the number of size classes. A sigmoid activation function is applied to the output of the fully connected layer to map the predicted values ​​to the range [0, 1]. The box prediction output is obtained, which contains the location coordinates, confidence, and size class of each box.

[0066] GridHead: The transformed feature map F output by the grid neck grid As the input of the grid head. grid A fully connected layer is applied to map the feature map to a grid prediction output that matches the transformed encoded grid label. The input dimension of the fully connected layer is the number of channels of Fgrid, and the output dimension is (grid size ,grid size ), where grid size Represents the size of the grid. Apply the sigmoid activation function to the output of the fully connected layer and map the predicted value to the range of [0, 1], indicating the probability that each grid contains a crack. The grid prediction output is obtained, indicating whether each grid contains a crack.

[0067] Box output decoding: Get the box prediction results output by the box head, including the position coordinates, confidence and size category of each box. Decode the box prediction results: Position coordinate decoding: Map the predicted position coordinates (x, y, w, h) back to the original image size. Assuming the predicted value is (x', y', w', h') and the original image size is (W, H), the decoded position coordinates are: x = x'×W; y = y'×H; w = w'×W; h = h'×H; Confidence threshold processing: According to the preset confidence threshold (such as 0.5), filter out boxes with confidence lower than the threshold. Size category decoding: According to the predicted size category index, find the corresponding size category label. According to the category interval boundary determined by size clustering, map the decoded width and height back to the original size space: Assuming the category interval boundary is

[0068] [s0,s1,s2,......,s n ], where s0 and s n Represent the minimum and maximum sizes respectively. For the decoded width w and height h, find the category interval [s i ,s i+1 ]. Map w and h to the interval [s i ,s i+1 ] Actual size within:

[0069] Get the box output, including the crack location coordinates (x, y, w mapped ,h mapped ), confidence, and size category.

[0070] Grid output decoding: Get the grid prediction results output by the grid head, indicating the probability of each grid containing cracks. Threshold processing of the grid prediction results: According to the preset threshold (such as 0.5), the grids with prediction values ​​greater than the threshold are determined as grids containing cracks. Get the index (i, j) of the grid containing cracks, where i and j represent the horizontal and vertical indices of the grid respectively. Decode the grid containing cracks: Calculate the center coordinates of the grid: x center =(i+0.5)×grid size ;y center =(j+0.5)×grid size ; where grid size Indicates the size of the grid. Calculate the width and height of the grid: width = grid size ; height = grid size ; Map the decoded grid center coordinates, width and height to the image coordinate space: Assume the image size is (W, H) and the number of grids is (grid width ,grid height ). The center coordinates after mapping are:

[0071] The mapped width and height are:

[0072] The grid output contains the crack distribution information, that is, the center coordinates (x mapped ,y mapped ) and width mapped and height mapped .

[0073] Model training phase: Image preprocessing: Preprocess the original image, including resizing, normalization and other operations. Resizing: Scale the image to a uniform size (such as 512×512) so that it can be input into the detection model. Normalization: Normalize the image pixel values ​​to the range of [0, 1] or [-1, 1], which helps to accelerate model convergence and improve performance. Box label encoding: Encode the original box label and convert it into a format suitable for model training. For each image, extract its corresponding crack position box label, including the center coordinates, width and height of the crack. Normalize the width and height of the crack and divide it by the width and height of the image to get the relative size. According to the width and height of the crack, it is divided into different size categories, such as small, medium, large, etc. Encode the location, size and category information of the crack into a vector as a box label. Grid label encoding: Encode the original crack distribution label and convert it into a format suitable for model training. Divide the image into fixed-size grids (such as 16×16), and each grid corresponds to a label. For each mesh, if it contains cracks, it is labeled as 1, otherwise it is labeled as 0. The mesh labels are reshaped to the same shape as the mesh head output.

[0074] Construct training dataset: Combine preprocessed image data, encoded box labels, and encoded grid labels into a training dataset. Each sample includes image data, box labels, and grid labels, forming a tuple of (image, box_label, grid_label). Batch partitioning: Divide the training dataset into several batches, each containing a certain number of samples. The batch size is selected based on the available computing resources and model complexity, such as 32, 64, 128, etc. In each batch, a certain number of samples are randomly selected to form a mini-batch of data.

[0075] Model initialization: Xavier initialization: Use the Xavier initialization method to initialize the weights of the convolutional layer and the fully connected layer of the detection model. The Xavier initialization method aims to make the output variance of each layer the same as the input variance, which helps the propagation and convergence of the gradient. in ,fan out ), the formula for Xavier initialization is:

[0076] Bias initialization: Initialize the bias of the convolutional layer and the fully connected layer of the detection model to 0. Initializing the bias to 0 helps the convergence and stability of the model. Layer-by-layer initialization: Initialize each layer of the backbone network, box neck, box head, grid neck, and grid head of the detection model separately. Use the Xavier initialization method to initialize the weight of each layer, and initialize the bias to 0.

[0077] Construct a joint loss function, including a box loss function and a grid loss function. The box loss function uses an improved Smooth L1 loss function to optimize the error between the box prediction result output by the box head and the encoded box label with the size category. The grid loss function uses a binary cross entropy loss function to optimize the error between the grid prediction result output by the grid head and the encoded grid label after shape transformation. Box loss function:

[0078] The box loss function adopts the improved Smooth L1 loss function, and its formula is:

[0079]

[0080] Where N is the number of images in the training dataset; M is the number of boxes detected in each image; I ij represents the indicator function of the jth box in the i-th image. When the box matches the ground-truth box label, I ij =1, otherwise I ij =0; Represents the predicted output of the jth box in the ith image, including the center coordinates width high and category confidence

[0081] Represents the ground truth box label of the jth box match in the i-th image, including the center coordinate label Width Tag Height Label and class confidence labels

[0082] Represents the Smooth L1 function;

[0083] α represents the weight coefficient, which is used to balance the ratio of width and height loss to center coordinate loss;

[0084] β represents the weight coefficient, which is used to balance the ratio of category confidence loss to position loss;

[0085] represents the binary cross entropy function,

[0086]

[0087] The box loss function measures the difference between the predicted center coordinates, width and height and the true label through the Smooth L1 function, and measures the difference between the predicted category confidence and the true label through the binary cross entropy function. The weight coefficients α and β are introduced to balance the proportion of different loss terms. Logarithmic encoding is used for width and height to make the loss function more robust to scale changes.

[0088] Grid loss function: The grid loss function adopts the binary cross entropy loss function, and its formula is:

[0089]

[0090] Where, H represents the number of grid rows after the image is divided into grids; W represents the number of grid columns after the image is divided into grids; y ij represents the true label of the grid in the i-th row and j-th column, which takes a value of 0 or 1, indicating whether the corresponding grid contains a crack; Represents the predicted output of the grid in the i-th row and j-th column, with a value of 0 to 1, indicating the confidence that the corresponding grid contains cracks. The grid loss function measures the difference between the predicted grid confidence and the true label through the binary cross entropy function, which is used to optimize the grid prediction results output by the grid head.

[0091] Training iterations: Set hyperparameters: Batch size (batch_size): The number of samples used in each iteration, such as 32, 64, 128, etc. Number of iterations (num_epochs): The number of times the model iterates over the entire training dataset. Learning rate (learning_rate): The step size for updating model parameters in the optimization algorithm, which controls the speed of parameter update.

[0092] Mini-batch gradient descent: The training dataset is divided into multiple mini-batches, each containing batch_size samples. In each iteration, a mini-batch of image data, box labels, and grid labels is randomly selected as input. Forward propagation: The input data passes through the various components of the detection model, including the backbone network, FPN, box neck, box head, grid neck, and grid head. The backbone network extracts the feature representation of the image. FPN fuses features of different scales to improve the detection ability of objects of different sizes. The box neck and box head are used to predict the location, size, and category of the object. The grid neck and grid head are used to predict the distribution of cracks. Finally, the box prediction output and the grid prediction output are obtained. Loss calculation: The loss between the predicted output and the true label is calculated using the joint loss function. The joint loss function includes two parts: box loss and grid loss. The box loss uses the improved Smooth L1 loss function to measure the difference between the predicted location, size, and category and the true label. The grid loss uses the binary cross entropy loss function to measure the difference between the predicted grid confidence and the true label. Backward propagation: The gradient of the loss function to the model parameters is calculated through the back propagation algorithm. According to the chain rule, the gradient of the loss function is passed to each component and layer of the model. The gradient of each parameter is calculated, indicating the sensitivity of the loss function to that parameter.

[0093] Parameter update: Use an optimization algorithm (such as Adam, SGD, etc.) to update the model parameters according to the calculated gradient. The optimization algorithm adjusts the value of the parameter according to the direction and size of the gradient to minimize the loss function. Commonly used optimization algorithms include stochastic gradient descent (SGD), Adam, RMSprop, etc. Iterative process: Repeat steps 2-6 and iterate the training data set multiple times until the set number of iterations is reached or the early stopping condition is met. In each iteration, the model is trained with different small batches of data to update the model parameters. As the iterations proceed, the model gradually learns the characteristic representation and detection rules of cracks.

[0094] Model preservation: Regular preservation: During the training process, save the model parameters and optimizer status every certain number of iterations (such as each epoch). The saved content includes the model's weights, biases and other parameters, as well as the optimizer's momentum, learning rate and other status. Regular preservation helps with subsequent model evaluation, reasoning and training resumption. Optimal model preservation: During the training process, evaluate the performance of the model on the validation set at the same time. Record the model parameters with the best performance on the validation set as the final training result. Performance evaluation indicators can be accuracy, precision, recall, F1 score, etc., selected according to the needs of the task.

[0095] Model inference phase: Image acquisition: Get the image of the concrete component to be tested, which can be obtained by shooting with a camera, reading from a file, etc. The image should be clear and complete, and can reflect the surface condition of the concrete component. Image preprocessing: Preprocess the acquired image to make it meet the input requirements of the detection model. Resizing: Scale the image to the same size as the training phase, usually a fixed size, such as 512×512 or 640×640. Normalization: Normalize the pixel values ​​of the image to the range of [0, 1] or [-1, 1], consistent with the training phase. Channel order adjustment: Adjust the channel order of the image according to the requirements of the detection model, such as converting the BGR format to the RGB format. Image batch processing: If multiple images need to be processed in batches, multiple images can be grouped into a batch. Multiple preprocessed images are grouped into a 4-dimensional tensor with a shape of (batch_size, height, width, channels). batch_size represents the number of images in the batch, height and width represent the height and width of the image, and channels represents the number of channels of the image.

[0096] Forward propagation: Backbone network: Input the preprocessed image into the backbone network, such as ResNet, VGG, etc. The backbone network extracts the feature representation of the image through a series of convolutional layers and pooling layers. Different backbone networks have different structures and depths, and the appropriate backbone network can be selected according to the requirements of the task. Feature Pyramid Network (FPN): Input the feature map extracted by the backbone network into FPN to fuse features of different scales. FPN fuses feature maps of different levels through lateral connections and upsampling operations. The fused feature map has richer semantic information and a wider receptive field, which is conducive to detecting cracks of different sizes. Box neck and box head: Input the feature map output by FPN into the box neck and box head to predict the location, size and category of the crack. The box neck further extracts features through a series of convolutional layers and pooling layers to reduce the spatial size of the feature map. The box head predicts the location, size and category confidence of the crack through a fully connected layer or a convolutional layer. The output of the box head includes the center coordinates, width and height of the crack box, and the confidence of each category.

[0097] Grid neck and grid head: The feature map output by FPN is input into the grid neck and grid head to predict the distribution of cracks. The grid neck further extracts features through a series of convolutional layers and pooling layers to reduce the spatial size of the feature map. The grid head predicts whether each grid contains a crack and the confidence of the crack through a fully connected layer or a convolutional layer. The output of the grid head is a two-dimensional tensor of the same size as the input image, representing the crack confidence of each grid. Output results: The result of the forward propagation includes two outputs: box output and grid output. The box output is a two-dimensional tensor, each row corresponds to a predicted crack box, containing the location, size and category confidence information of the crack. The grid output is a two-dimensional tensor of the same size as the input image, representing the crack confidence of each grid.

[0098] Box output processing: Size filtering: Size filter the box output according to the preset crack size threshold. Traverse each predicted crack box and calculate its width and height. If the width or height of the crack box is less than the size threshold, filter it out. Keep the crack boxes with sizes greater than or equal to the threshold to get the size-filtered box output. Non-maximum suppression (NMS): Perform NMS processing on the size-filtered box output to remove boxes with high overlap and low confidence. Sort the crack boxes according to the category confidence, and give priority to the boxes with high confidence.

[0099] Traverse the sorted crack boxes, and for each box: calculate its intersection over union (IoU) with other boxes. If the IoU is greater than the preset IoU threshold, and the confidence of the current box is lower than the confidence of other boxes, filter out the current box. Keep the unfiltered crack boxes to obtain the crack box detection results after NMS processing. Format the detection results: format the crack box detection results after NMS processing. Extract the location information (center coordinates and width and height), size information, and category information of each crack box. Organize the detection results into a standardized format, such as a list or dictionary, for subsequent processing and output.

[0100] Grid output processing: Binarization: Binarize the grid output according to the preset confidence threshold. Traverse each position of the grid output and compare its confidence value with the threshold. If the confidence value is higher than the threshold, mark the position as containing cracks (set to 1). If the confidence value is lower than or equal to the threshold, mark the position as not containing cracks (set to 0). Crack distribution visualization: Visualize the grid output after binarization to intuitively display the distribution of cracks in the image. The grid output can be superimposed on the original image, and different colors or transparency indicate the confidence of the crack. The visualization results help to quickly locate the crack area and evaluate the distribution and severity of the crack. Detection result combination: Correspondence between crack box and grid: Combine the crack box detection result with the crack grid detection result, and use the grid information to verify the validity of the box. For each detected crack box, calculate its grid position. Determine whether the corresponding grid position is marked as containing cracks in the crack grid detection result.

[0101] Filtering of crack boxes: If the grid where the crack box is located is marked as containing cracks, the crack box is retained. If the grid where the crack box is located is marked as not containing cracks, the crack box is filtered out. Verifying the validity of the box through grid information can reduce false detection and improve detection accuracy. Generating the final detection result: The retained crack box is used as the final crack detection result. Each crack box contains the location, size and category information of the crack.

[0102] Output of detection results: Position information: For each crack box detected, output its position information. Position information can be expressed as the center coordinates (x, y) and width and height (width, height) of the crack box. The center coordinates indicate the center position of the crack box, and the width and height indicate the size of the crack box. Size information: Calculate the actual size of the crack based on the position of the crack box and the size of the image. The relative coordinates and width and height of the crack box can be converted into actual pixel values ​​or physical dimensions. Size information can help assess the severity and impact range of the crack. Category information: Output the category information of each crack box, indicating the type of crack. Category information can be a predefined category label, such as horizontal cracks, longitudinal cracks, oblique cracks, etc. According to the category of the crack, corresponding processing and repair measures can be taken. Visualization and saving: Visualize the detection results and draw the crack box and category label on the original image. The visualization results intuitively show the location, size and type of the crack, which is convenient for manual verification and analysis. Save the detection results as files or database records for subsequent statistics, reporting and decision support.

[0103] Crack measurement stage: Extract crack detection results: Extract the position coordinates and size information of the crack box and crack grid from the crack detection results. The information of the crack box includes the center coordinates (x, y) and width and height (width, height) of the box. The information of the crack grid includes the position coordinates and mark (whether it contains cracks) of the grid. Calculate the length and width of the crack box: For each crack box in the detection results, calculate the length and width of the crack based on the width and height information of the box. The length of the crack can be approximated as the height (height) of the box, indicating the extension length of the crack in the vertical direction. The width of the crack can be approximated as the width (width) of the box, indicating the width of the crack in the horizontal direction. Associate the calculated crack length and width with the position coordinates of the crack box to obtain the length and width measurement results of each crack box.

[0104] Calculate the area of ​​the crack grid: According to the crack grid detection results, count the number of grids marked as containing cracks. The area of ​​a single grid can be calculated based on the image size and the number of grids. Multiply the number of grids containing cracks by the area of ​​a single grid to get the total area of ​​the crack grid. The area of ​​the crack grid represents the distribution range and impact area of ​​the crack in the image. Weighted average measurement results: Take a weighted average of the length, width, and area of ​​the crack box to get the final crack measurement result. When weighted averaging, the confidence weights of the crack box and the crack grid need to be considered. The confidence weight indicates the degree of credibility of the detection result. The higher the confidence, the greater the weight. The formula for weighted average can be expressed as:

[0105]

[0106] in, and Respectively represent the length and width of the i-th crack box, Indicates the confidence of the crack box. area Represents the area of ​​the crack grid, grid confidence Represents the confidence of the crack grid. Through weighted averaging, the measurement results of the crack box and the crack grid are comprehensively considered to obtain more accurate and reliable estimates of crack length, width and area.

[0107] Output crack measurement results: The weighted average of the crack length, width and area is output as the final crack measurement result. The measurement results can be associated with the location coordinates of the crack to generate a crack measurement report or visual display. The crack measurement results can be used to assess the severity of the cracks and guide subsequent repair and reinforcement measures.

Claims

1. A method for measuring cracks in concrete components based on convolutional neural networks, characterized in that: Obtain image data of concrete components, annotate the location and size information of cracks, and obtain frame labels and grid labels; Combine image data, box labels, and grid labels to form a dataset; Preprocess the data set; Preprocessing Including,size clustering of box labels to classify crack sizes into multiple categories; Perform enhancement processing on image data; Transform the coordinates of the box label and the grid label; Construct a detection model architecture including a backbone network, a box neck, a box head, a grid neck, and a grid head; the backbone network uses a pre-trained ResNet-50 network; the box neck and the box head are used to generate box outputs, and the grid neck and the grid head are used to generate grid outputs; The preprocessed dataset is fed into the detection model, the model is optimized through the joint loss function, and the model is trained using the size-clustered box labels and the enhanced image data; The image of the concrete component to be tested is input into the trained detection model to obtain the box output and grid output generated by the detection model; the box output and grid output are subjected to non-maximum suppression processing to obtain the crack detection result; According to the crack position and size information in the crack detection results, the length, width and area of ​​the crack are calculated.

2. The method for measuring cracks in concrete components based on a convolutional neural network according to claim 1, characterized in that: Obtain image data of concrete components, annotate the location and size information of cracks, and obtain frame labels and grid labels; The image data, box labels, and grid labels are combined to form a dataset, including: Collect image data of the surface of concrete components; The position of the crack is marked on the image data, and the crack area is selected with a minimum circumscribed rectangular frame. The position coordinates and size information of the rectangular frame are used as a frame label. The frame label indicates the position and size of the crack in the image. A sliding window is used to divide the image data into multiple square grids. A label value between 0 and 1 is assigned to each grid according to the proportion of crack pixels in each grid. The label value reflects the proportion of cracks in the grid. The label values ​​of all grids constitute the grid label, which reflects the distribution of cracks in the image.

3. The method for measuring cracks in concrete components based on convolutional neural networks according to claim 2, characterized in that: Size clustering is performed on the box labels to classify crack sizes into multiple categories, including: Extract the crack size information in the frame label, obtain the width and height data of all cracks, and form a size data set; The minimum-maximum normalization method is used to linearly transform the width and height data to the range of 0 to 1; Set the number of clusters k, randomly select a data point as the first cluster center in the normalized size data, calculate the shortest distance from each data point to the selected cluster center, and select the next cluster center according to the ratio of the square of the distance to the sum of the squares of the distances, and repeat the steps of selecting cluster centers until k initial cluster centers are selected; Calculate the Mahalanobis distance from each data point in the normalized size data set to the k initial cluster centers, and divide each data point into the corresponding cluster according to the nearest Mahalanobis distance to obtain k initial clusters; The size data in each initial cluster is fitted using the least squares method to obtain the new center point coordinates of each cluster; Calculate the sum of the Euclidean distances between the new center point of each cluster and the original center point in the width and height dimensions as the change in the cluster center. If the change is less than the threshold, the clustering result is considered to have converged and the clustering process is terminated. Otherwise, the new center point is used as the current cluster center and the iterative optimization center point is returned. The k clustering results that finally converge are used as k categories of crack sizes. The width and height values ​​of the center point of each category are used as the size interval boundaries of the corresponding category. The cracks with sizes in the box labels falling in the same interval are classified into the same group, and cracks of different sizes are grouped.

4. The method for measuring cracks in concrete components based on convolutional neural networks according to claim 3 is characterized in that: Perform enhancement processing on image data, including: Performing geometric transformation on the image data, using random horizontal flipping and random vertical flipping to mirror-reverse the image data left-right and up-down with a preset probability, to obtain geometrically transformed image data; The image data after geometric transformation is scaled, and the image data after geometric transformation is scaled proportionally within a preset scale range by random scaling, and a fixed-size area is randomly selected on the scaled image as new image data by random cropping to obtain the image data after scale transformation; Random occlusion is superimposed on the crack area of ​​the scale-transformed image data, and a square with a color similar to the background is superimposed at a random position in the crack area to obtain image data after random occlusion is added; The original image data and the image data after adding random occlusion are mixed according to a preset ratio, and the crack position and size information in the frame label and the grid label are mixed accordingly to obtain enhanced image data, frame label and grid label.

5. The method for measuring cracks in concrete components based on convolutional neural networks according to claim 4 is characterized in that: Transform the coordinates of the box labels and grid labels, including: Normalize the coordinates of the frame label, divide the crack position coordinates in the frame label by the width and height of the image data to obtain a normalized frame label; The grid labels are mapped to coordinates. According to the width and step size of the window when dividing the grid, the position coordinates of each grid in the grid label are mapped to a coordinate space consistent with the size of the image data to obtain the mapped grid labels. Coordinate encoding is performed on the normalized frame label and the mapped grid label. The log-square encoding method is used to encode the crack center coordinates, width and height in the normalized frame label as a preset offset and scaling factor, and the grid center coordinates, width and height in the mapped grid label as a preset offset and scaling factor to obtain the encoded frame label and grid label. The category label obtained by clustering the crack size is added to the encoded frame label, and the size category index to which the crack area belongs is added to the corresponding encoded frame label to obtain the encoded frame label with the size category.

6. The method for measuring cracks in concrete components based on a convolutional neural network according to any one of claims 2 to 5, characterized in that: Construct a detection model architecture including a backbone network, a box neck, a box head, a grid neck, and a grid head. The backbone network uses a pre-trained ResNet-50 network. The box neck and the box head are used to generate box outputs, and the grid neck and the grid head are used to generate grid outputs, including: The backbone network receives the enhanced image data and uses the pre-trained ResNet-50 network to extract the multi-scale features of the image to obtain a multi-scale feature map; Connect the feature pyramid network FPN to the multi-scale feature map extracted by the backbone network, upsample and fuse the multi-scale feature map through FPN to obtain the fused feature map; The box neck and the grid neck are respectively connected to the fused feature map, and the box neck and the grid neck respectively use convolutional neural networks to extract features and transform dimensions of the fused feature map to obtain transformed feature maps for predicting box output and grid output respectively; The box head is connected to the transformed feature map output by the box neck, and the grid head is connected to the transformed feature map output by the grid neck. The box head and the grid head respectively use a fully connected layer and a sigmoid activation function to map the transformed feature map output by the box neck to a box prediction output matching the encoded box label with the size category, and to map the transformed feature map output by the grid neck to a grid prediction output matching the encoded grid label after the transformation. Decode the box prediction result output by the box head to obtain the position coordinates, width, height and size category of the detected crack area, and map the decoded width and height back to the original size space according to the category interval boundary determined by size clustering to obtain the box output, which contains the crack position, size and category information; The grid prediction results output by the grid head are threshold processed, and the grids with prediction values ​​greater than the threshold in the grid prediction results are determined as grids containing cracks. The center coordinates, width and height of the grid containing cracks are decoded, and the decoded center coordinates, width and height of the grid are mapped to the image coordinate space to obtain the grid output, which contains the crack distribution information.

7. The method for measuring cracks in concrete components based on convolutional neural networks according to claim 6, characterized in that: The preprocessed dataset is fed into the detection model, and the model is optimized through a joint loss function. The model is trained using the size-clustered box labels and the augmented image data, including: The preprocessed image data, the encoded box labels with size categories, and the encoded grid labels after shape transformation are used as training datasets and input into the constructed detection model; Initialize the parameters of the detection model. Use the Xavier initialization method to initialize the weights of the convolutional layers and fully connected layers of the backbone network, box neck, box head, grid neck, and grid head. Initialize the bias to 0. Construct a joint loss function, which includes a box loss function and a grid loss function; the box loss function uses the Smooth L1 loss function to optimize the error between the box prediction result output by the box head and the encoded box label with size category; the grid loss function uses the binary cross entropy loss function to optimize the error between the grid prediction result output by the grid head and the encoded grid label after shape transformation; According to the set batch size and number of iterations, the detection model is iteratively trained using the mini-batch gradient descent algorithm. In each iteration, a batch of image data, box labels, and grid labels are randomly selected from the training data set and input into the detection model. The box prediction output and grid prediction output are calculated by forward propagation. The loss between the prediction output and the label is calculated using the joint loss function. The gradient of the loss function with respect to the model parameters is calculated through the back-propagation algorithm. The loss function is minimized to obtain the trained detection model.

8. The method for measuring cracks in concrete components based on convolutional neural networks according to claim 7, characterized in that: The box loss function uses the Smooth L1 loss function, which is expressed as follows: Where N is the number of images in the training dataset; M is the number of boxes detected in each image; I ij represents the indicator function of the jth box in the i-th image. When the box matches the ground-truth box label, I ij =1, otherwise Represents the predicted output of the jth box in the ith image, including the center coordinates width high and category confidence Represents the ground truth box label of the jth box match in the i-th image, including the center coordinate label Width Tag Height Label and class confidence labels Among them, smooth L1 (x) represents the Smooth L1 function; α represents the weight coefficient, which is used to balance the ratio of width and height loss to center coordinate loss; β represents the weight coefficient, which is used to balance the ratio of category confidence loss to position loss; Represents the binary cross entropy function:

9. The method for measuring cracks in concrete components based on convolutional neural networks according to claim 7, characterized in that: The grid loss function uses the binary cross entropy loss function, which is expressed as follows: Where H represents the number of grid rows after the image is divided into grids; W represents the number of grid columns after the image is divided into grids; y ij represents the true label of the grid in the i-th row and j-th column, which takes a value of 0 or 1, indicating whether the corresponding grid contains a crack; Represents the predicted output of the grid in the i-th row and j-th column, with a value between 0 and 1, indicating the confidence that the corresponding grid contains cracks.

10. The method for measuring cracks in concrete components based on convolutional neural network according to claim 9, characterized in that: Input the image of the concrete component to be tested into the trained detection model, and obtain the box output and mesh output generated by the detection model; Perform non-maximum suppression on the box output and grid output to obtain crack detection results, including: The image of the concrete component to be tested is input into the trained detection model, and the box output and the grid output are obtained by forward propagation calculation; wherein the box output includes the predicted position, size and category confidence of the crack box, and the grid output includes the predicted position and confidence of the crack grid; According to a preset crack size threshold, the box output is size-filtered, and boxes with sizes smaller than the threshold are filtered to obtain a size-filtered box output; According to the preset intersection-over-union ratio threshold and confidence threshold, non-maximum suppression processing is performed on the box output after size filtering to remove boxes with overlap higher than the threshold and confidence lower than the threshold, and the crack box detection result is obtained; According to the preset confidence threshold, the grid output is binarized, and the grids with confidence higher than the threshold are marked as containing cracks, so as to obtain the crack grid detection result; The crack box detection results and the crack Wanggen detection results are combined to obtain the crack detection results of the concrete component to be tested, and the location, size and category of the cracks are output.

Images