Bridge crack identification method and system, electronic equipment and storage medium

Abstract

The invention discloses a bridge crack identification method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a bridge image; a bridge image is input into the trained bridge crack recognition network, and the trained bridge crack recognition network is used for performing feature extraction on the input bridge image by adopting a C2fPonv module, performing feature fusion on the extracted features by adopting a multi-scale attention module EMA, recognizing the fused features to obtain a plurality of prediction frames, and identifying the prediction frames; screening the prediction frames by adopting non-maximum suppression to obtain a bridge crack recognition result; a loss function adopted in the network training process is used for inhibiting the gradient when the difference between a prediction frame and a target frame is smaller than a set threshold value, so that the network focuses on identification of a difficult-to-predict sample.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge crack identification, and in particular to a bridge crack identification method, system, electronic equipment and storage medium. Background Art

[0002] As the operation time of bridges increases, their structural surfaces are prone to damage such as broken bars and cracks. If not handled in time, it will affect the service life and bearing capacity of the bridge. In recent years, with the continuous development of machine vision, bridge defect detection technology has made great progress. Real-time and accurate positioning of crack feature points from bridge images is the core requirement for ensuring the integrity of bridges.

[0003] The inventors found that the technical problems existing in the prior art are:

[0004] 1. When faced with small target defects, the model’s positioning and detection accuracy performance is poor, and may even miss the defect directly.

[0005] 2. The existing visual processing algorithms have a slow processing speed and cannot be installed on edge computing devices for real-time detection.

[0006] 3. However, in actual on-site bridge detection tasks, bridge crack images often have a lot of noise interference outdoors, which sometimes leads to false detections and a decrease in detection accuracy. Summary of the invention

[0007] In order to address the deficiencies in the prior art, the present invention provides a bridge crack recognition method, system, electronic device and storage medium. The present invention converts the feature point positioning task in a two-dimensional image into a target detection task, and takes the center point position of the target detection frame as the detection target, which not only effectively meets the recognition requirements of the feature point category and position for bridge quality detection, but also combines the crack positioning detection task, so that the network can meet the crack recognition requirements with minimal computing occupancy.

[0008] In one aspect, a bridge crack identification method is provided, comprising:

[0009] Acquire a bridge image; input the bridge image into the trained bridge crack recognition network, wherein the trained bridge crack recognition network is used to extract features from the input bridge image using the C2f_Ponv module, fuse the extracted features using the multi-scale attention module EMA, identify the fused features, obtain several prediction frames, and screen the prediction frames using non-maximum suppression to obtain bridge crack recognition results; the loss function used in the network training process is used to suppress the gradient when the difference between the prediction frame and the target frame is less than a set threshold, so that the network focuses on the recognition of difficult-to-predict samples.

[0010] In another aspect, a bridge crack identification system is provided, comprising:

[0011] The recognition module is configured to: obtain a bridge image; input the bridge image into a trained bridge crack recognition network, wherein the trained bridge crack recognition network is used to extract features from the input bridge image using a C2f_Ponv module, fuse the extracted features using a multi-scale attention module EMA, recognize the fused features, obtain several prediction frames, and screen the prediction frames using non-maximum suppression to obtain bridge crack recognition results; the loss function used in the network training process is used to suppress the gradient when the difference between the prediction frame and the target frame is less than a set threshold, so that the network focuses on the recognition of difficult-to-predict samples.

[0012] On the other hand, there is also provided an electronic device, comprising:

[0013] a memory for non-transitory storage of computer-readable instructions; and

[0014] a processor for executing the computer readable instructions,

[0015] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0016] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0017] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0018] The above technical solution has the following advantages or beneficial effects:

[0019] The present invention determines the location of the intersection of cracks through the improved YOLOV8 network structure. The present invention improves the positioning and detection accuracy performance of the model and reduces missed detections through the multi-scale attention module EMA.

[0020] The present invention achieves improvement in processing speed by replacing C2f with C2f_Ponv.

[0021] An efficient multi-scale attention module (EMA) is introduced into the network, which can help the network better highlight important features, enhance the recognition ability of the network model in a strong noise environment, and improve the detection accuracy.

[0022] During the network training process, a bounding box loss function is designed to make the network focus on samples that are difficult to locate, rather than the loss fluctuations caused by labeling errors, thereby reducing the network's sensitivity to labeling errors.

[0023] In the network post-processing module NMS (non-maximum suppression), EDIoU is proposed to replace the IoU judgment item in NMS, which improves the algorithm's ability to screen pre-selected boxes with larger length and width, and reduces missed detection and false detection rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a diagram of the improved YOLOV8 network structure of Example 1;

[0026] Figure 2 Schematic diagram of the working principle of the PConv structure of Example 1;

[0027] FIG3( a ) is a schematic diagram of the internal structure of C2f according to Example 1;

[0028] FIG3( b ) is a schematic diagram of the internal structure of C2f_Ponv of Example 1;

[0029] Figure 4 This is a schematic diagram of the internal structure of the multi-scale attention module EMA of Example 1;

[0030] Figure 5 This is a schematic diagram of the internal structure of the first Head module of Example 1;

[0031] Figure 6 This is a schematic diagram of the internal structure of the SPPF module of Example 1;

[0032] Figure 7 This is a schematic diagram of the improved YOLOV8 network recognition results of Example 1. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0034] Embodiment 1

[0035] This embodiment provides a bridge crack identification method;

[0036] A bridge crack identification method, comprising:

[0037] Acquire a bridge image; input the bridge image into the trained bridge crack recognition network, wherein the trained bridge crack recognition network is used to extract features from the input bridge image using the C2f_Ponv module, fuse the extracted features using the multi-scale attention module EMA, identify the fused features, obtain several prediction frames, and screen the prediction frames using non-maximum suppression to obtain bridge crack recognition results; the loss function used in the network training process is used to suppress the gradient when the difference between the prediction frame and the target frame is less than a set threshold, so that the network focuses on the recognition of difficult-to-predict samples.

[0038] Furthermore, the trained bridge crack recognition network includes:

[0039] The backbone network Backbone, neck network Neck and head network Head are connected in sequence;

[0040] The backbone network Backbone includes: a first convolutional layer Conv, a second convolutional layer Conv, a first feature extraction module C2f_Pconv, a third convolutional layer Conv, a second feature extraction module C2f_Pconv, a fourth convolutional layer Conv, a third feature extraction module C2f_Pconv, a fifth convolutional layer Conv, a fourth feature extraction module C2f_Pconv, a multi-scale attention module EMA and an SPPF module connected in sequence.

[0041] The backbone network Backbone is used to extract image features, wherein the first, second and third convolutional layers Conv are used to increase the receptive field, the first, second, third and fourth feature extraction modules C2f_Pconv are used for feature extraction, the multi-scale attention module EMA is used to adjust feature weights, and the SPPF module is used to splice features of different receptive fields.

[0042] Further, the neck network Neck includes: a first upsampling module, a first splicing unit, a fifth feature extraction module C2f_Pconv, a second upsampling module, a second splicing unit, a sixth feature extraction module C2f_Pconv, a sixth convolutional layer Conv, a third splicing unit, a seventh feature extraction module C2f_Pconv, a seventh convolutional layer Conv, a fourth splicing unit and an eighth feature extraction module C2f_Pconv connected in sequence;

[0043] Wherein, the input end of the first up-sampling module is connected to the output end of the SPPF module;

[0044] The input end of the first splicing unit is connected to the output end of the SPPF module

[0045] The output end of the third feature extraction module C2f_Pconv is connected to the input end of the first splicing unit;

[0046] An output end of the second up-sampling module is connected to an input end of the third splicing unit.

[0047] Furthermore, the neck network Neck is used to fuse the shallow geometric features and deep semantic features of the backbone network, so that the network can take into account the detailed geometric information while maintaining the semantic understanding of the image. Among them, the upsampling module and the convolution layer Conv are used to maintain the uniformity of the feature shape, the splicing unit is used to merge the feature information, and the feature extraction module C2f_Pconv is used to further learn the features.

[0048] Furthermore, the head network Head includes: a first Head module, a second Head module and a third Head module; wherein the input end of the first Head module is connected to the output end of the sixth feature extraction module C2f_Pconv; the input end of the second Head module is connected to the output end of the seventh feature extraction module C2f_Pconv; and the input end of the third Head module is connected to the output end of the eighth feature extraction module C2f_Pconv.

[0049] Furthermore, if Figure 5 As shown, the first Head module, the second Head module and the third Head module have the same internal structure, and the first Head module includes: a parallel category branch and a position branch;

[0050] The category branch is composed of the first convolutional layer Conv in the category branch, the second convolutional layer Conv in the category branch and a convolution operation;

[0051] The position branch is composed of the first convolutional layer Conv in the position branch, the second convolutional layer Conv in the position branch and a convolution operation;

[0052] The input ends of the category branch and the position branch are both connected to the input end of the first Head module, and the output results of the category branch and the position branch are spliced ​​to form the output of the first Head module.

[0053] Furthermore, the head network Head is used to generate a network pre-selected box. Different Head modules are used to generate pre-selected boxes for different scales by connecting different positions of the neck network. In each Head module, the category branch is used to calculate the category probability prediction of the pre-selected box, and the position branch is used to calculate the midpoint position of the pre-selected box and the side length of the pre-selected box.

[0054] Further, the internal structures of the first feature extraction module C2f_Pconv, the second feature extraction module C2f_Pconv, the third feature extraction module C2f_Pconv, the fourth feature extraction module C2f_Pconv, the fifth feature extraction module C2f_Pconv, the sixth feature extraction module C2f_Pconv, the seventh feature extraction module C2f_Pconv and the eighth feature extraction module C2f_Pconv are the same, and the first feature extraction module C2f_Pconv includes:

[0055] The eighth convolutional layer Conv, the Split layer, the fifth splicing unit and the ninth convolutional layer Conv are connected in sequence;

[0056] The Split layer is a splitting unit that evenly splits the input features into two parts according to the channel dimension, one part is directly input into the input layer of the fifth splicing unit, and the other part is input into the input layer of the fifth splicing unit through n serially connected PConv layers;

[0057] The n serially connected PConv layers are numbered in order from 1 to n. The output data of the even-numbered PConv layers are evenly split into two parts, one of which is directly input into the input layer of the fifth concatenation unit, and the other is input into the subsequent PConv layers.

[0058] Furthermore, the working process of the first feature extraction module C2f_Pconv includes:

[0059] (1) After the feature is input into the C2f_Pconv module, it is first divided into two parts, one is directly input into the fifth concatenation unit, and the other is processed by two layers of Pconv in series;

[0060] (2) The feature-processed data is further split into two parts, one of which is directly input into the fifth concatenation unit, and the other is further processed by two layers of Pconv in series;

[0061] (3) Step (2) is repeated n / 2 times, and the output of the last step is directly input to the input end of the fifth splicing unit;

[0062] (4) The output end of the fifth splicing unit is the output end of the first feature extraction module C2f_Pconv, and the output end of the fifth splicing unit outputs the final feature extraction result.

[0063] It should be understood that how to obtain calculation results faster while using the same computing equipment has always been a hot topic for the deployment and application of neural networks on construction sites. The original YOLOv8 network, which is mainly composed of convolution modules Conv and C2f modules, can guarantee powerful feature extraction capabilities, but there are a large number of redundant features, which introduce unnecessary model parameters to a certain extent, increase the amount of calculation, and make the original network still have a lot of room for improvement in speed. In response to the above problems, researchers have designed lightweight network structures such as ShuffleNet V2, MobileNetV3, and GhostNet. These networks use deep convolution (DWConv) or group convolution (GConv) to extract image features, which greatly reduces the number of floating-point operations (FLOPs) consumed by model operations. However, when these lightweight convolution operations are used to replace the original ordinary convolution operations, it is often impossible to obtain an increase in network speed that matches the FLOPs. The reason is that although these convolution structures can effectively reduce the FLOPs of the network, they also lead to more frequent memory accesses, significantly reducing the number of floating-point operations per second (FLOPS) of the computing device, and ultimately resulting in the inability to effectively improve the network's operating speed. This is particularly true for small-volume network models.

[0064] Based on this, PConv was introduced to eliminate this difference. While reducing network FLOPs, it can also reduce the number of network memory accesses to maintain extremely high FLOPS and obtain faster network operation speed. PConv proposes different solutions to the problem of feature redundancy in the network. Figure 2 As shown in , it applies regular Conv to some input channels for spatial feature extraction, and the rest of the channels remain unchanged. For continuous or regular memory access, the first or last continuous c p A channel is computed as a representative of the entire feature map. Without loss of generality, the input and output feature maps have the same number of channels.

[0065] Therefore, the FLOPs of PConv are only:

[0066] h*w*k 2 *c p 2 (1)

[0067] Among them, w and h are the width and height of the feature map, k is the size of the convolution kernel, and c p is the number of channels for partial convolution. p When the number of channels is 1 / 4 of the total number of channels c, the FLOPs of PConv is only 1 / 16 of that of ordinary convolution. In addition, PConv also has a smaller memory access amount:

[0068] h*w*2c p +k 2 *c p 2 ≈h*w*2c p (2)

[0069] At this time, the memory access amount of PConv is only 1 / 4 of that of ordinary convolution.

[0070] In the original network, as shown in Figure 3(a), considering that the stacked Bottleneck in the C2f module uses a large number of Conv modules, which occupies most of the model parameters, PConv is used to replace the Conv operation used in the Bottleneck module in the original network to form a C2f_PConv module with lighter weight and faster response, as shown in Figure 3(b).

[0071] Furthermore, if Figure 2 As shown, the PConv layer includes: splitting the features of the input end of the PConv layer in the channel dimension to split the first part of the features and the second part of the features, performing convolution processing on the first part of the features to obtain the convolution result, keeping the second part of the features unchanged, and finally splicing the convolution result with the second part of the features, and outputting the splicing result through the output end of the PConv layer.

[0072] Furthermore, if Figure 4 As shown, the multi-scale attention module EMA includes:

[0073] The Groups grouping unit has an input end that is an input end of the multi-scale attention module EMA. The Groups grouping unit reorganizes the input features and reorganizes the input feature shape (B, C, H, W) into (B*G, C / / G, H, W), where B represents the batch size, C represents the number of channels, W represents the channel width, H represents the channel height, and G represents the number of groups.

[0074] The output end of the Groups grouping unit is connected to the input ends of the first branch, the second branch, the third branch, the first Re_weight layer and the second Re_weight layer respectively;

[0075] The first branch includes: a first average pooling layer, a sixth splicing unit, a tenth convolution layer Conv, a first activation function layer, a first Re_weight layer, a GroupNorm layer, a second average pooling layer, a second activation function layer, a first Matmul layer, an adder, a third activation function layer and a second Re_weight layer connected in sequence; the input end of the second Re_weight layer is also connected to the output end of the Groups grouping unit, and the output end of the second Re_weight layer is the output end of the multi-scale attention module EMA;

[0076] The second branch includes: a third average pooling layer, an input end of the third average pooling layer is connected to an output end of the Groups grouping unit, an output end of the third average pooling layer is connected to an input end of the sixth splicing unit, and an output end of the tenth convolutional layer Conv is further connected to an input end of the first Re_weight layer through a fourth activation function layer;

[0077] The third branch includes: an eleventh convolutional layer Conv, a fourth average pooling layer, a fifth activation function layer and a second Matmul layer connected in sequence, the input end of the eleventh convolutional layer Conv is connected to the output end of the Groups grouping unit, the output end of the second Matmul layer is connected to the input end of the adder; the input end of the second Matmul layer is also connected to the input end of the GroupNorm layer; the output end of the eleventh convolutional layer Conv is also connected to the input end of the first Matmul layer; the input end of the first Re_weight layer is also connected to the output end of the Groups grouping unit.

[0078] Furthermore, the internal working process of the first Re_weight layer is consistent with that of the second Re_weight layer, and the first Re_weight layer is used to multiply the input data according to the corresponding dimension.

[0079] Furthermore, the GroupNorm layer divides the input data into multiple groups, calculates the mean and variance of each group respectively, and normalizes the elements of each group.

[0080] Furthermore, the internal working process of the first Matmul layer is consistent with that of the second Matmul layer, and the first Matmul layer is used to perform matrix multiplication operation on the input data groups.

[0081] Furthermore, if Figure 6As shown, the SPPF module includes a twelfth convolutional layer Conv, a first maximum pooling layer, a second maximum pooling layer, and a third maximum pooling layer connected in sequence, wherein the output end of the twelfth convolutional layer Conv, the output end of the first maximum pooling layer, the output end of the second maximum pooling layer, and the output end of the third maximum pooling layer are also connected to the input end of the seventh splicing unit, the output end of the seventh splicing unit is connected to the thirteenth convolutional layer Conv, and the output end of the thirteenth convolutional layer Conv is the output end of the SPPF module.

[0082] Furthermore, the internal working processes of the first splicing unit, the second splicing unit, the third splicing unit, the fourth splicing unit, the fifth splicing unit and the seventh splicing unit are consistent, and the first splicing unit is used to connect the input data according to the channel dimension. The sixth splicing unit is used to splice the input data.

[0083] It should be understood that in actual on-site bridge inspection tasks, bridge crack images are often subject to a lot of noise interference outdoors, which sometimes leads to false detections. To solve this problem, it is necessary to suppress the weight of these similar but unimportant features and increase the weight ratio of the laser stripe location.

[0084] The efficient multi-scale attention module (EMA) based on cross-space learning used in this invention is a fine-grained long-range spatial and channel dual-effect attention module. It refers to the parallel structure design of CA, and innovatively reshapes the channel dimension into a batch dimension, uses the 1×1 branch in the CA module to design a channel learning method, and also designs a cross-space learning method, such as Figure 4 Although the structure of this attention module is relatively complex, it does not bring high computational complexity because it mainly uses segmented local operations. At the same time, it also provides extremely effective global attention, which can well adapt to the needs of lightweight and high-precision network improvement.

[0085] The main features of the Multi-Scale Attention Module (EMA) include:

[0086] 1. Feature grouping: The input feature map is divided into multiple sub-feature groups, each of which learns different semantics, which helps to allocate and process the model on more GPU resources, strengthens feature learning in semantic areas, and compresses noise.

[0087] 2. Parallel subnetworks: EMA uses three parallel paths to extract the attention weight descriptors of the grouped feature maps. Two of the paths are 1x1 convolution branches (the first branch and the second branch), and the third path is a 3x3 convolution branch (the third branch). One-dimensional global average pooling operations are used in the first and second branches to encode channel information in two spatial directions respectively. The third branch captures multi-scale feature representations through 3x3 convolutions.

[0088] 3. Cross - spatial learning: EMA enriches feature aggregation by providing a cross - spatial information aggregation method in different spatial dimension directions. The outputs of the first branch and the second branch encode global spatial information through two - dimensional global average pooling, while the output of the third branch is directly transformed into the corresponding dimensional shape. Then, these outputs are aggregated through a matrix dot - product operation to generate the first spatial attention map. Finally, the output feature maps within each group are aggregated through the Sigmoid function of the two generated spatial attention weight values, capturing pixel - level pairing relationships and highlighting the global context of all pixels. As Figure 4 shown, C / / G represents the number of channels in each group after grouping, and X / Y Avg Pool represents the pooling operation in the corresponding direction.

[0089] Furthermore, the multi - scale attention module EMA has a working process that includes:

[0090] For any input feature map, first, the feature map is split into G (G << C) groups according to the channel dimension, and the weight prediction of attention is performed for each group separately.

[0091] First, three parallel first branches, second branches, and third branches are used to obtain the attention weights of the features;

[0092] Among them, the first branch and the second branch decode the feature map from different directions; the third branch provides a local receptive field through a 3×3 convolution;

[0093] In the first branch and the second branch, first, the global position information in one - dimensional directions is extracted through average pooling operations in the x and y directions respectively. After that, the two - direction information is concatenated, and the information in different channels at each position in each direction is blended through a 1×1 convolution to obtain a long vector. Subsequently, the completed long vector is split back into two - direction vectors, and the two - dimensional weight distributions in each direction are obtained by using the Sigmoid function respectively. Then, the values of the feature map are adjusted through multiplication to adaptively calibrate the channel relationship, realizing different cross - channel interaction features between the first branch and the second branch, and at the same time ensuring fine - grained feature information.

[0094] In the third branch, the original input features will also be processed through a 3×3 convolution to expand the feature space and obtain a local receptive field.

[0095] In the above process, the feature map realizes the shallow fusion in two one - dimensional spatial directions and the deep fusion in the channel direction. After that, further weight extraction of the spatial features is required to achieve richer feature aggregation.

[0096] The multi-scale attention module EMA uses two branches, and adopts a cross-space feature fusion scheme based on the different focuses of the two branches. For each branch, the overall feature information and long-range dependencies are encoded through global pooling, and the linear transformation is fitted through the Softmax function, which is then multiplied with the complete feature map of the other branch to obtain the weight map in the spatial dimension. The two feature maps are then added together, followed by the Sigmoid function to derive the attention map that retains the precise spatial position information. Finally, the attention map is used to guide the input feature map to obtain the final output feature information.

[0097] In this network, the EMA module is added before the SPPF layer of the backbone network to guide the focus of feature information extraction of the backbone network.

[0098] Furthermore, the loss function used in the network training process is used to suppress the gradient when the difference between the prediction box and the target box is less than a set threshold, so that the network focuses on the recognition of difficult-to-predict samples, wherein the specific expression of the loss function is:

[0099]

[0100] Among them, λ is the suppression factor, which is used to adjust the value of the function suppression gradient range, α is the proportional factor, which is used to adjust the impact of edge length loss, and L SEIoU represents the loss function, IoU represents the intersection-over-union ratio between the prediction box and the target box, ρ 2 (b,b gt ) represents the Euclidean distance between the prediction box and the midpoint of the target box, b represents the midpoint of the prediction box, and b gt represents the midpoint of the target box, ρ 2 (w,w gt ) represents the width difference between the prediction box and the target box, w represents the prediction box width, w gt represents the target box height, ρ 2 (h,h gt ) represents the height difference between the prediction box and the target box, h represents the height of the prediction box, and h gt Indicates the target box height, w c Indicates the minimum bounding box width between the prediction box and the target box, h c It represents the minimum bounding box height between the prediction box and the target box, α represents the distance loss scaling factor, and λ represents the distance loss suppression factor.

[0101] The Smooth expression is expressed as follows:

[0102]

[0103] Smooth is used to monitor edge length loss, Smooth(x) represents the smoothing function, and x represents the function input.

[0104] Furthermore, the fusion features are identified to obtain a number of prediction frames, and the prediction frames are screened by non-maximum suppression to obtain bridge crack recognition results, which specifically include:

[0105] (1): Arrange the pre-selected boxes generated by the network in descending order of their confidence;

[0106] (2): Remove the confidence values ​​lower than the set confidence threshold N th Pre-selected box;

[0107] (3): Keep the box with the highest confidence and save it as the successfully predicted target box;

[0108] (4): Use the pre-selected box with the highest confidence to calculate the intersection over union (EDIoU) with other pre-selected boxes, and remove the boxes with an EDIoU greater than the IoU threshold N. IoU Pre-selected box;

[0109]

[0110] Among them, w i With h i They represent the center point positions of different bounding boxes in the width and height directions respectively, β represents the scaling factor of the distance term; EDIoU represents the IoU evaluation standard, IoU represents the intersection-over-union ratio of the two shapes involved in the calculation, β represents the scaling factor of the distance term, and ρ 2 (w 1 ,w 2 ) represents the Euclidean distance between the center points of the two shapes projected on the width dimension, ρ 2 (h 1 ,h 2 ) represents the Euclidean distance between the center points of the two shapes projected in the height dimension, w c Indicates the width of the minimum bounding box of the two shapes, h c Indicates the height of the minimum bounding box of the two shapes,

[0111] (5): Repeat steps (3) to (4) until all pre-selected boxes are processed.

[0112] In the original YOLOv8 network, NMS (non-maximum suppression) is used to filter the pre-selected boxes generated by the network to obtain the bounding box containing the target. The screening quality of NMS often directly affects the prediction effect of the final target box.

[0113] Due to the shape of the cracks and the operation of expanding the width of the bounding box during the annotation process, the network model is more likely to have overlapping occlusions when generating prediction boxes, which also increases the difficulty of network NMS processing. During the experiment, it is easy to cause adjacent cracks to retain only a single box after processing, and a single crack is mistakenly identified as two cracks, etc.

[0114] Therefore, in the present invention, the screening index IoU used by NMS is reconstructed and the distance loss term in EIou is introduced. However, this distance loss term is calculated by the center point distance of the two shapes and the diagonal length of the minimum bounding box. However, due to the large length-to-width ratio of the long crack bounding box, the formula term based on the center point distance is easily affected by the long side, resulting in insufficient sensitivity to changes in the short side. Therefore, the distance loss term is split in different directions to enhance the sensitivity of the bounding box to changes in the short side.

[0115] The present invention names the modified IoU as EDIoU. It is described as follows: The YOLOv8 model provides multiple models for different recognition speed and accuracy requirements. Different models of networks have different network sizes to freely obtain a balance between detection speed and accuracy. The present invention aims to deploy the model on edge computing devices and maintain good real-time performance and positioning accuracy. Therefore, the smallest model YOLOv8n in the YOLOv8 series is used.

[0116] The YOLOv8 backbone network is used to extract the feature information of the image. Among them, the C2f module is the most important feature extraction module of the network. It is essentially a stacked residual module that continuously divides the input information into two parts. For each segmentation, one part continues to pass through the residual bottleneck module to extract deep network features, and the other part is directly connected to the splicing part. Finally, the information of the residual modules at each level is integrated to complete the feature extraction. The SPPF module obtains local features of different scales through multi-segment maximum pooling, and splices them without changing the size of the feature map to increase the global receptive field.

[0117] The neck module adopts a simple PANet structure. Through top-to-bottom and bottom-to-up paths, it fuses high-level semantic features and low-level positioning features extracted from the Backbone network to enhance the detection effect of targets of different sizes, communicate feature information of different levels of abstraction, and integrate them into detection heads of different scales.

[0118] The detection head is responsible for the output of information. It adopts the design of decoupling head to decouple the information in the neck module into category detection and location detection parts, which fully solves the spatial misalignment problem in the traditional coupled detection head. At the same time, detection heads of different scales are connected to feature maps of different sizes, corresponding to targets of different sizes. Each grid of the feature map will output a pre-selected information, including the predicted category, target confidence, and the center coordinates of the bounding box and its width and height. Finally, according to the confidence and position information of the pre-selected boxes, the non-maximum suppression (NMS) method is used to screen these pre-selected boxes to obtain the predicted target.

[0119] In order to better adapt to the real-time bridge quality inspection task, enhance the real-time performance of feature point recognition on edge computing devices, especially CPU devices, and ensure positioning accuracy, the present invention has made some targeted improvements, such as Figure 1 As shown. First, the C2f module in the network is improved by combining the partial convolution Pconv with a lightweight structure, and C2f_Ponv is proposed, which enables the model to simultaneously obtain a reduction in floating-point operations and memory access, fully improving the network's computing speed. Afterwards, in order to reduce the accuracy reduction caused by network lightweighting, an efficient multi-scale attention module (EMA) is introduced into the network, which can help the network better highlight important features, improve the recognition ability of the network model in a strong noise environment, and only bring a small amount of additional parameters. In the final output part, the center point of the prediction box output by the model is used as the feature point of the bridge crack, and the center point is screened using the bounding box information to obtain the final prediction result.

[0120] In the task of pavement crack detection, it is usually not necessary to accurately locate the cracks, but only the size and number of the cracks on the pavement need to be relatively precise.

[0121] In pavement crack images, since most cracks are long and thin with a high aspect ratio, in order to reduce the difficulty of annotation, precise annotation along the crack width is usually not performed when annotating large quantities of images. A common operation is to expand the bounding box outward to a certain size, which causes large fluctuations in the data when annotating the bounding box, making it difficult to unify the expansion outside the bounding box, affecting the training effect of the network.

[0122] Therefore, in order to solve the problem that the data bounding box annotation standard is difficult to unify and the annotation results have large fluctuations during fast manual annotation, the present invention reconstructs the bounding box loss function. In the original YOLOv8 network, CIoU is used as the loss function. In this function, the aspect ratio is used as the loss term, but due to the lack of direct numerical constraints, the effective adjustment of the predicted bounding box is sometimes hindered during the process of adjusting the aspect ratio of the network.

[0123] To use a more intuitive EIoU loss replacement, the expression is as follows:

[0124]

[0125] In the formula, IoU represents the intersection-over-union ratio between the predicted box and the target box, and the expression is A and B represent two arbitrary shapes. ρ(·) represents the Euclidean distance between b and b gt Represent the center points of the prediction box and the target box respectively, w c With h c Respectively represent the width and height of the minimum bounding box, w and h represent the width and height of the prediction box, w gt With h gt Indicates the width and height of the target box.

[0126] In order to further reduce the impact of labeling errors on data training, the core idea of ​​Smooth loss is integrated. By suppressing the gradient when the difference between the prediction box and the target box is small, the network focuses on the recognition of difficult-to-predict samples, reducing the impact of small fluctuations in sample recognition caused by labeling errors, thereby improving the network convergence speed and stability.

[0127] Aiming at the limitations of the existing bridge crack feature recognition algorithm, the improvement strategy of the recognition process is analyzed from multiple perspectives, and a set of lightweight bridge crack feature extraction task solutions is proposed. First, in view of the shortcomings of the existing algorithm in detecting few crack types and failing to identify crack intersections, the key point detection and crack intersection identification tasks are unified through target detection, and a rich bridge defect data set is collected.

[0128] The present invention is based on the YOLOv8n algorithm and improves the network model according to the requirements of the bridge quality identification task. The C2f_Pconv operator is combined to improve the network operation speed, and the EMA attention module is introduced to adjust the backbone network feature weights. Finally, a multi-objective bridge crack feature extraction solution that can run on edge computing devices and maintain good real-time performance and accuracy is formed.

[0129] The data set used in the present invention uses a mobile device equipped with a camera device to collect data on cracks in bridge pavements under different scenes and different lighting conditions. During the collection process, the camera is kept level with the ground to ensure clear image capture. After completing the data set image collection, 2,000 images of different crack types and quantities are selected based on the number of cracks contained in the image. The cracks and intersections in the selected images are labeled using labelImg software. It is required that the cracks be completely wrapped when annotating the cracks, and different crack objects are segmented according to the crack intersections. When annotating the intersections, the intersections are framed with several fixed-size square bounding boxes to ensure that the final annotation is accurate and without objection.

[0130] Since YOLOv8 uses txt format data and labelImg outputs data sets in xml format, the present invention further uses python to write data set format conversion code to obtain a data set in txt format, and divides the data set into a training set and a validation set according to a ratio of 8:2.

[0131] Due to the small number of data sets, the present invention performs data enhancement on the data sets. Data enhancement is used during data set construction and training, including cropping, translation, rotation, brightness change, Gaussian noise addition, and Mosica algorithm to splice data images, etc., to effectively improve the generalization ability of the model.

[0132] The model used in the present invention is designed based on the deep learning framework Pytorch. In order to achieve better training speed, a higher batch size is set. The operating system used during training is Ubuntu 22.04, equipped with 120G memory, the CPU is 16vCPU Intel (R) Xeon (R) Platinum 8352V CPU, the GPU is RTX 4090, and the video memory is 24GB. At the same time, in order to better verify the computing speed of the network under low-end computing power devices, the present invention uses different devices for inference testing, and the CPU device used in the test is 12th Gen Intel (R) Core (TM) i5-12400.

[0133] In model training, the input image is compressed and padded to a size of 640*640, and after data enhancement, it is used as network input. The present invention uses the SGD optimizer under the Pytorch framework, the initial learning rate is set to 0.1, and the weight decay strategy is used. During the training process, the learning rate gradually decays until it drops to 0.01 times the initial value. Loss uses the default value, the batch size is set to 16, and the number of training rounds is set to 700 rounds. When the model effect does not improve after 50 rounds of training, it automatically stops.

[0134] In the target detection task, the real-time performance of the model is usually evaluated using the frames per second (FPS), and the evaluation of detection accuracy usually uses precision, recall, and mean average precision (mAP) as indicators. Their calculation formulas are as follows:

[0135]

[0136] Among them, samples with annotated boxes near the predicted boxes and whose calculated intersection over union (IoU) values ​​are greater than the set IoU threshold are considered to be correctly predicted. TP represents the number of samples with correct predictions, FP represents the number of predicted boxes that do not meet the correct prediction criteria, and FN represents the number of annotated boxes that do not have corresponding correct predicted boxes.

[0137]

[0138] Where N represents the number of categories, AP (average precision) represents the area under the joint curve of precision and recall (PR), and AP c Represents the AP value of class c. mAP0.5 represents the mAP value when the IoU threshold is 0.5, and mAP0.5:0.95 represents the IoU threshold from 0.5 to 0.95, and the mAP is calculated every 0.05, and the average value is finally added.

[0139] mAP comprehensively reflects the precision and recall rate of target detection. In the crack feature point recognition task, the higher the mAP, the lower the missed detection and false detection rate of the graphics, which directly indicates that the recognition of the weld is better, and to a certain extent, it indicates that the positioning accuracy of the crack feature points is higher.

[0140] The validation set was tested using this weight, and the evaluation results of the above indicators are shown in Table 1. The model's Precision and Recal reached 0.975 and 0.966 respectively, indicating that the model can extract image features well, accurately identify and classify bridge crack feature points, and only rarely miss or misdetect. mAP0.5:0.95 reached 0.576, and the model can still maintain a high prediction result under different detection standards, further reflecting the positioning performance of the model. At the same time, the CPU (12th Gen Intel(R) Core(TM) i5-12400F) device was used to test the running speed of the model. The model's inference speed was 47.8ms, and the FPS reached 20.9HZ, which can meet the real-time requirements in the bridge crack identification process. Figure 7 This is a schematic diagram of the improved YOLOV8 network recognition results of Example 1.

[0141] Table 1 Performance parameters related to target detection of the algorithm of the present invention

[0142]

[0143]

[0144] Embodiment 2

[0145] This embodiment provides a bridge crack identification system, including:

[0146] The recognition module is configured to: obtain a bridge image; input the bridge image into a trained bridge crack recognition network, wherein the trained bridge crack recognition network is used to extract features from the input bridge image using a C2f_Ponv module, fuse the extracted features using a multi-scale attention module EMA, recognize the fused features, obtain several prediction frames, and screen the prediction frames using non-maximum suppression to obtain bridge crack recognition results; the loss function used in the network training process is used to suppress the gradient when the difference between the prediction frame and the target frame is less than a set threshold, so that the network focuses on the recognition of difficult-to-predict samples.

[0147] It should be noted that the above identification module corresponds to the steps in Embodiment 1, and the examples and application scenarios implemented by the above module and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above module, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0148] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0150] Embodiment 3

[0151] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0152] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0153] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0154] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0155] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0156] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] Embodiment 4

[0158] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A bridge crack identification method, characterized in that: include: Get bridge images; The bridge image is input into the trained bridge crack recognition network, wherein the trained bridge crack recognition network is used to extract features from the input bridge image using the C2f_Ponv module, and to fuse the extracted features using the multi-scale attention module EMA, to identify the fused features, to obtain several prediction boxes, and to screen the prediction boxes using non-maximum suppression to obtain the bridge crack recognition results; the loss function used in the network training process is used to suppress the gradient when the difference between the prediction box and the target box is less than the set threshold, so that the network focuses on the recognition of difficult-to-predict samples.

2. A bridge crack identification method as claimed in claim 1, characterized in that: The trained bridge crack recognition network includes: The backbone network Backbone, neck network Neck and head network Head are connected in sequence; The backbone network Backbone includes: a first convolution layer Conv, a second convolution layer Conv, a first feature extraction module C2f_Pconv, a third convolution layer Conv, a second feature extraction module C2f_Pconv, a fourth convolution layer Conv, a third feature extraction module C2f_Pconv, a fifth convolution layer Conv, a fourth feature extraction module C2f_Pconv, a multi-scale attention module EMA and an SPPF module connected in sequence; The neck network Neck includes: a first upsampling module, a first splicing unit, a fifth feature extraction module C2f_Pconv, a second upsampling module, a second splicing unit, a sixth feature extraction module C2f_Pconv, a sixth convolutional layer Conv, a third splicing unit, a seventh feature extraction module C2f_Pconv, a seventh convolutional layer Conv, a fourth splicing unit and an eighth feature extraction module C2f_Pconv connected in sequence; Among them, the input end of the first up-sampling module is connected to the output end of the SPPF module; the input end of the first splicing unit is connected to the output end of the SPPF module; the output end of the third feature extraction module C2f_Pconv is connected to the input end of the first splicing unit; the output end of the second up-sampling module is connected to the input end of the third splicing unit; The head network Head includes: a first Head module, a second Head module and a third Head module; wherein the input end of the first Head module is connected to the output end of the sixth feature extraction module C2f_Pconv; the input end of the second Head module is connected to the output end of the seventh feature extraction module C2f_Pconv; and the input end of the third Head module is connected to the output end of the eighth feature extraction module C2f_Pconv.

3. A bridge crack identification method as claimed in claim 2, characterized in that: The first feature extraction module C2f_Pconv includes: an eighth convolution layer Conv, a Split layer, a fifth concatenation unit, and a ninth convolution layer Conv connected in sequence; wherein the Split layer is a splitting unit, which evenly splits the input feature into two parts according to the channel dimension, one part is directly input into the input layer of the fifth concatenation unit, and the other part is input into the input layer of the fifth concatenation unit through n serially connected PConv layers; the n serially connected PConv layers are numbered in order from 1 to n, and the output data of the even-numbered PConv layers are evenly split into two parts, one part is directly input into the input layer of the fifth concatenation unit, and the other part is input into the subsequent PConv layers; The first feature extraction module C2f_Pconv, the working process includes: (1) After the feature is input into the C2f_Pconv module, it is first divided into two parts, one is directly input into the fifth concatenation unit, and the other is processed by two layers of Pconv in series; (2) The feature-processed data is further split into two parts, one of which is directly input into the fifth concatenation unit, and the other is further processed by two layers of Pconv in series; (3) Step (2) is repeated n / 2 times, and the output of the last step is directly input to the input end of the fifth splicing unit; (4) The output end of the fifth splicing unit is the output end of the first feature extraction module C2f_Pconv, and the output end of the fifth splicing unit outputs the final feature extraction result; The PConv layer includes: splitting the features of the input end of the PConv layer in the channel dimension to split the first part of the features and the second part of the features, performing convolution processing on the first part of the features to obtain the convolution result, keeping the second part of the features unchanged, and finally splicing the convolution result with the second part of the features, and outputting the splicing result through the output end of the PConv layer.

4. A bridge crack identification method as claimed in claim 2, characterized in that: The multi-scale attention module EMA includes: A Groups grouping unit, wherein the input end of the Groups grouping unit is the input end of the multi-scale attention module EMA; the Groups grouping unit reorganizes the input features; the output end of the Groups grouping unit is respectively connected to the input ends of the first branch, the second branch, the third branch, the first Re_weight layer, and the second Re_weight layer; The first branch includes: a first average pooling layer, a sixth splicing unit, a tenth convolution layer Conv, a first activation function layer, a first Re_weight layer, a GroupNorm layer, a second average pooling layer, a second activation function layer, a first Matmul layer, an adder, a third activation function layer and a second Re_weight layer connected in sequence; the input end of the second Re_weight layer is also connected to the output end of the Groups grouping unit, and the output end of the second Re_weight layer is the output end of the multi-scale attention module EMA; The second branch includes: a third average pooling layer, an input end of the third average pooling layer is connected to an output end of the Groups grouping unit, an output end of the third average pooling layer is connected to an input end of the sixth splicing unit, and an output end of the tenth convolutional layer Conv is further connected to an input end of the first Re_weight layer through a fourth activation function layer; The third branch includes: an eleventh convolutional layer Conv, a fourth average pooling layer, a fifth activation function layer and a second Matmul layer connected in sequence, the input end of the eleventh convolutional layer Conv is connected to the output end of the Groups grouping unit, the output end of the second Matmul layer is connected to the input end of the adder; the input end of the second Matmul layer is also connected to the input end of the GroupNorm layer; the output end of the eleventh convolutional layer Conv is also connected to the input end of the first Matmul layer; the input end of the first Re_weight layer is also connected to the output end of the Groups grouping unit.

5. A bridge crack identification method as claimed in claim 4, characterized in that: The multi-scale attention module EMA has the following working process: for any input feature map, the feature map is first split into G groups according to the channel dimension, and the attention weight is predicted for each group; first, the attention weight of the feature is obtained by using three parallel first branches, second branches and third branches; wherein the first branch and the second branch decode the feature map from different directions; the third branch provides a local field of view through a 3×3 convolution; In the first branch and the second branch, the global position information in the one-dimensional direction is first extracted through the average pooling operation in the x and y directions respectively, and then the two-dimensional information is spliced, and the information in different channels at each directional position is fused through a 1×1 convolution to obtain a long vector; then the fused long vector is split into two directional vectors again, and the Sigmoid function is used to obtain the two-dimensional weight distribution in each direction; then, the value of the feature map is readjusted through multiplication to achieve adaptive calibration of the channel relationship, so as to achieve different cross-channel interaction features between the first branch and the second branch, while also ensuring fine-grained feature information; In the third branch, the original input features will also be processed by 3×3 convolution to expand the feature space and obtain a local field of view.

6. A bridge crack identification method as claimed in claim 1, characterized in that: The loss function used in the network training process is used to suppress the gradient when the difference between the prediction box and the target box is less than the set threshold, so that the network focuses on the recognition of difficult-to-predict samples. The specific expression of the loss function is: Among them, λ is the suppression factor, which is used to adjust the value of the function suppression gradient range, α is the proportional factor, which is used to adjust the impact of edge length loss, and L SEIoU represents the loss function, IoU represents the intersection-over-union ratio between the prediction box and the target box, ρ 2 (b,b gt ) represents the Euclidean distance between the prediction box and the midpoint of the target box, b represents the midpoint of the prediction box, and b gt represents the midpoint of the target box, ρ 2 (w,w gt ) represents the width difference between the prediction box and the target box, w represents the prediction box width, w gt represents the target box height, ρ 2 (h,h gt ) represents the height difference between the prediction box and the target box, h represents the height of the prediction box, and h gt Indicates the target box height, w c Indicates the minimum bounding box width between the prediction box and the target box, h c represents the minimum bounding box height between the prediction box and the target box, α represents the distance loss scaling factor, and λ represents the distance loss suppression factor; The Smooth expression is as follows: Smooth is used to monitor edge length loss, Smooth(x) represents the smoothing function, and x represents the function input.

7. A bridge crack identification method as claimed in claim 1, characterized in that: The fusion features are identified to obtain a number of prediction frames, and the prediction frames are screened by non-maximum suppression to obtain the bridge crack recognition results, which specifically include: (1): Arrange the pre-selected boxes generated by the network in descending order of their confidence; (2): Remove the confidence values ​​lower than the set confidence threshold N th Pre-selected box; (3): Keep the box with the highest confidence and save it as the successfully predicted target box; (4): Use the pre-selected box with the highest confidence to calculate the intersection-over-union (EDIoU) with other pre-selected boxes, and remove the boxes with an intersection-over-union (EDIoU) greater than the IoU threshold N. IoU Pre-selected box; Among them, w i With h i Respectively represent the center point positions of different bounding boxes in the width and height directions, β represents the scaling factor of the distance term; IoU represents the intersection-over-union ratio of the two shapes involved in the calculation, β represents the scaling factor of the distance term, ρ 2 (w 1 ,w 2 ) represents the Euclidean distance between the center points of the two shapes projected on the width dimension, ρ 2 (h 1 ,h 2 ) represents the Euclidean distance between the center points of the two shapes projected in the height dimension, w c Indicates the width of the minimum bounding box of the two shapes, h c Indicates the height of the minimum bounding box of the two shapes, (5): Repeat steps (3) to (4) until all pre-selected boxes are processed.

8. A bridge crack identification system, characterized in that: include: A recognition module is configured to: acquire a bridge image; The bridge image is input into the trained bridge crack recognition network, wherein the trained bridge crack recognition network is used to extract features from the input bridge image using the C2f_Ponv module, and to fuse the extracted features using the multi-scale attention module EMA, to identify the fused features, to obtain several prediction boxes, and to screen the prediction boxes using non-maximum suppression to obtain the bridge crack recognition results; the loss function used in the network training process is used to suppress the gradient when the difference between the prediction box and the target box is less than the set threshold, so that the network focuses on the recognition of difficult-to-predict samples.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 7 is performed.

Images