Complex environment pavement crack detection method and device, mobile equipment and medium

Abstract

The invention provides a complex environment pavement crack detection method and device, mobile equipment and a medium, and belongs to the field of image classification, and the method comprises the steps: inputting an obtained to-be-detected pavement image into a completely trained pavement crack detection network, carrying out the convolution normalization processing of the to-be-detected pavement image, and obtaining an input feature, performing feature separation reconstruction convolution on the input features to obtain reconstructed convolution features, performing flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, and performing improved visual feature extraction on the flexible attention features to obtain output features; and carrying out classification prediction on the output features and outputting a pavement crack detection result. According to the method, redundant features can be effectively inhibited through feature separation reconstruction convolution, the model efficiency is improved, environmental noise in the features can be effectively eliminated through a flexible attention mechanism, the false detection rate is reduced, and the application requirement of highway pavement crack detection is met.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a method, device, mobile equipment and medium for detecting road cracks in a complex environment. Background Art

[0002] The detection of pavement cracks is a key point in highway maintenance. This technology can provide effective information for the maintenance and repair of highways. Timely repair of damaged pavement according to the detection results can effectively prevent the occurrence of traffic accidents, which has important practical significance for the development of the transportation industry. The initial pavement crack detection method used a method based on digital image processing to detect pavement cracks. Although the method based on digital image processing has a good detection speed, it is easily disturbed by external environments such as light, shadow, and obstacles, resulting in a high false detection rate. It also relies on manually designed operators for detection and has poor versatility. With the development of deep learning, deep learning has strong analysis and learning capabilities, can accurately capture deep-level features in data, and can be applied to the automated detection of pavement cracks.

[0003] However, although the existing deep learning methods have made good progress in the field of pavement crack detection, in complex road environments, for example, when there is external environmental interference such as light and shadow or obstacles in the detection image, the existing deep learning methods have a high false detection rate for pavement cracks due to environmental interference. In addition, the existing deep learning algorithms usually have complex structures and long detection times, making them difficult to deploy on mobile devices, and cannot meet the application requirements of pavement crack detection in actual highway application scenarios.

[0004] Therefore, the existing technology has technical problems such as high false detection rate of pavement cracks due to environmental interference in complex road environments, long detection time due to complex structures, and difficulty in meeting application requirements, which needs to be improved. Summary of the invention

[0005] In view of this, it is necessary to provide a method, device, mobile equipment and medium for detecting pavement cracks in a complex environment, so as to solve the technical problems existing in the prior art that in a complex road environment, environmental interference will lead to a high false detection rate of pavement cracks, and the existing model has a complex structure and a long detection time, which is difficult to meet application requirements.

[0006] In order to solve the above problems, the present invention provides a method for detecting pavement cracks in a complex environment, comprising: The acquired road surface image to be detected is input into a well-trained road crack detection network, the road surface image to be detected is convoluted and normalized to obtain input features, the input features are subjected to feature separation and reconstruction convolution to obtain reconstructed convolution features, the reconstructed convolution features are subjected to flexible attention extraction to obtain flexible attention features, and the flexible attention features are subjected to improved visual feature extraction to obtain output features; The output features are classified and predicted to output the pavement crack detection results.

[0007] In a possible implementation, performing feature separation and reconstruction convolution on the input features to obtain reconstructed convolution features includes: Perform spatial reconstruction on the input features to obtain spatial reconstruction features; The spatial reconstruction features are reconstructed through channels to obtain the reconstructed convolution features.

[0008] In a possible implementation, spatially reconstructing the input features to obtain spatially reconstructed features includes: The input features are evaluated by scale factors and spatial pixel variance measurements to obtain preliminary feature weights; Performing normalization mapping and threshold gating on the preliminary feature weights to obtain a first gating weight and a second gating weight; Multiplying the input feature by the first gating weight and the second gating weight respectively to obtain a first weighted feature and a second weighted feature; The first weighted feature and the second weighted feature are cross-reconstructed and spliced ​​to obtain a spatial reconstruction feature.

[0009] In a possible implementation, channel reconstruction is performed on the spatial reconstruction feature to obtain a reconstructed convolution feature, including: The spatial reconstruction feature is divided into a first channel feature and a second channel feature, and the first channel feature and the second channel feature are convoluted and compressed to obtain a first compressed feature and a second compressed feature; Perform group convolution and point convolution on the first compressed feature to obtain high-level information features; Perform point convolution on the second compressed feature to obtain shallow detail features; Perform global average pooling on high-level information features and shallow-level detail features to obtain a global channel descriptor, and generate a feature vector based on the global channel descriptor; According to the feature vector, the high-level information features and shallow-level detail features are fused to obtain the reconstructed convolutional features.

[0010] In a possible implementation, performing flexible attention extraction on the reconstructed convolutional features to obtain flexible attention features includes: Perform multiple residual feature extraction on the reconstructed convolution features to obtain residual extraction features; Perform horizontal maximum pooling on the residual extraction features to obtain the first pooling features, and perform vertical maximum pooling on the residual extraction features to obtain the second pooling features; The first pooling feature is subjected to two-dimensional convolution and feature-aware normalization, and then multiplied and merged with the first pooling feature to obtain a first fused feature. The second pooling feature is subjected to two-dimensional convolution and feature-aware normalization, and then multiplied and merged with the second pooling feature to obtain a second fused feature. The first fusion feature and the second fusion feature are added and merged, and then the combined features are concatenated with the residual extraction feature to obtain a concatenated feature, and soft threshold feature mapping is performed on the concatenated feature to obtain a soft threshold mapping feature; The soft threshold mapping feature and the reconstructed convolution feature are added together to obtain the flexible attention feature.

[0011] In a possible implementation, the flexible attention feature is improved to extract visual features to obtain output features, including: Embedding the flexible attention features to obtain embedded features; The embedded features are sandwich-layouted and channel-enhanced, and then batch normalized, ReLu activated, and average pooled to obtain the output features.

[0012] In one possible implementation, the sandwich layout feature channel enhancement includes cascaded attention extraction, several feed-forward network layers, and token interaction operations set before each feed-forward network layer.

[0013] On the other hand, the present invention also provides a method for detecting pavement cracks in a complex environment, comprising: An image feature extraction unit is used to input the acquired road surface image to be detected into a well-trained road crack detection network, perform feature separation and reconstruction convolution on the road surface image to be detected to obtain reconstructed convolution features, perform flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, and perform improved visual feature extraction on the flexible attention features to obtain output features; The classification prediction output unit is used to classify and predict the output features to output the pavement crack detection results.

[0014] On the other hand, the present invention also provides a mobile device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned complex environment pavement crack detection method is implemented.

[0015] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned method for detecting pavement cracks in a complex environment is implemented.

[0016] The beneficial effects of the present invention are as follows: in the complex environment pavement crack detection method provided by the present invention, the acquired pavement image to be detected is first input into a well-trained pavement crack detection network, the pavement image to be detected is subjected to convolution normalization processing to obtain input features, the input features are subjected to feature separation and reconstruction convolution to obtain reconstructed convolution features, the reconstructed convolution features are subjected to flexible attention extraction to obtain flexible attention features, the flexible attention features are subjected to improved visual feature extraction to obtain output features; and then the output features are classified and predicted to output the pavement crack detection results. The present invention can effectively suppress redundant features and improve model efficiency through feature separation and reconstruction convolution, and can effectively remove environmental noise in features and reduce false detection rate through the flexible attention mechanism, thereby meeting the application requirements of highway pavement crack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for detecting pavement cracks in a complex environment provided by the present invention; Figure 2 It is a schematic diagram of the process of feature separation and reconstruction convolution in an embodiment of the present invention; Figure 3 A schematic diagram of a process of space reconstruction according to an embodiment of the present invention; Figure 4 A schematic diagram of a process of channel reconstruction according to an embodiment of the present invention; Figure 5 A schematic diagram of a process of flexible attention extraction according to an embodiment of the present invention; Figure 6 A schematic diagram of a process for improving visual feature extraction according to an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an embodiment of a device for detecting pavement cracks in a complex environment provided by the present invention; Figure 8 A schematic structural diagram of an embodiment of a mobile device provided by the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise specified, "multiple" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone.

[0021] The descriptions of "first" and "second" in the embodiments of the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0022] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0023] The present invention provides a method, device, mobile equipment and medium for detecting pavement cracks in a complex environment, which are described below respectively.

[0024] Figure 1 A flow chart of an embodiment of a method for detecting cracks in a pavement in a complex environment provided by the present invention is shown in FIG. Figure 1 As shown, the method for detecting pavement cracks in complex environments includes: S101, inputting the acquired road surface image to be detected into a well-trained road crack detection network, performing convolution normalization processing on the road surface image to be detected to obtain input features, performing feature separation and reconstruction convolution on the input features to obtain reconstructed convolution features, performing flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, and performing improved visual feature extraction on the flexible attention features to obtain output features; S102: classify and predict the output features to output the pavement crack detection results.

[0025] Compared with the prior art, the complex environment pavement crack detection method provided by the embodiment of the present invention first inputs the acquired pavement image to be detected into a well-trained pavement crack detection network, performs convolution normalization processing on the pavement image to be detected to obtain input features, performs feature separation and reconstruction convolution on the input features to obtain reconstructed convolution features, performs flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, performs improved visual feature extraction on the flexible attention features to obtain output features; then classifies and predicts the output features to output the pavement crack detection results. The present invention can effectively suppress redundant features and improve model efficiency through feature separation and reconstruction convolution, and can effectively remove environmental noise in features through the flexible attention mechanism, reduce the false detection rate, and meet the application requirements of highway pavement crack detection.

[0026] In some embodiments of the present invention, Figure 2 FIG. 1 is a flow chart of feature separation and reconstruction convolution in an embodiment of the present invention. Figure 2 As shown, the input features are subjected to feature separation and reconstruction convolution to obtain reconstructed convolution features, including: S201, spatially reconstructing the input features to obtain spatially reconstructed features; S202, performing channel reconstruction on the spatial reconstruction features to obtain reconstructed convolution features.

[0027] Specifically, the embodiment of the present invention uses the YOLOv5s algorithm as the benchmark algorithm and improves it so that it can adapt to the application scenario requirements of highway pavement crack detection. In the neck network, the traditional standard convolution is prone to generate redundant features when processing features. In this regard, the embodiment uses feature separation and reconstruction convolution to replace the standard convolution, so as to suppress redundant features, thereby enhancing the model feature expression ability while reducing the model complexity. Feature separation and reconstruction convolution consists of two parts: spatial reconstruction and channel reconstruction. The purpose of spatial reconstruction is to separate information-rich features and features with less information content at the spatial level for processing, and to generate richer features through reconstruction and fusion. Channel reconstruction processes high-level information features and shallow detail features separately on different channels, while extracting high-level information and avoiding the loss of detail information. The embodiment can significantly reduce the model calculation cost, reduce redundant features, and enhance the expression ability of features by feature separation and reconstruction.

[0028] In some embodiments of the present invention, Figure 3 FIG. 1 is a schematic diagram of a flow chart of space reconstruction according to an embodiment of the present invention. Figure 3 As shown, the input features are spatially reconstructed to obtain spatially reconstructed features, including: S301, evaluating the scale factor and measuring the spatial pixel variance of the input features to obtain preliminary feature weights; S302, performing normalization mapping and threshold gating on the preliminary feature weights to obtain a first gating weight and a second gating weight; S303, performing product operations on the input feature, the first gating weight and the second gating weight respectively to obtain a first weighted feature and a second weighted feature; S304: Cross-reconstruct and concatenate the first weighted feature and the second weighted feature to obtain a spatial reconstruction feature.

[0029] Specifically, in the spatial reconstruction module, spatial redundancy is suppressed by separating and reconstructing. The separation operation can separate the features with rich features from the features with less information content, and the reconstruction operation can merge the two features to generate richer features. The embodiment first uses the scale factor in the group normalization layer to evaluate the information content in the input feature map, and then uses the trainable affine parameter The spatial pixel variance of each batch and channel is measured to obtain the preliminary feature weight. The preliminary feature weight is then mapped to the preset range through Sigmoid normalization, and the threshold gating mechanism is used to divide the preliminary feature weight into the first gating weight and the second gating weight according to the preset threshold, and the input feature is multiplied by the first gating weight and the second gating weight to obtain the first weighted feature. and the second weighted feature Finally, the two weighted features are spliced ​​and fused through cross reconstruction to obtain the spatial reconstruction feature .

[0030] In some embodiments of the present invention, Figure 4 FIG. 1 is a flow chart of channel reconstruction according to an embodiment of the present invention. Figure 4 As shown, the spatial reconstruction features are channel reconstructed to obtain the reconstructed convolution features, including: S401, dividing the spatial reconstruction feature into a first channel feature and a second channel feature, and convolutionally compressing the first channel feature and the second channel feature to obtain a first compressed feature and a second compressed feature; S402, performing group convolution and point convolution on the first compressed feature to obtain a high-level information feature; S403, performing point convolution on the second compressed feature to obtain a shallow detail feature; S404, performing global average pooling on high-level information features and shallow-level detail features to obtain a global channel descriptor, and generating a feature vector according to the global channel descriptor; S405, according to the feature vector, the high-level information feature and the shallow-level detail feature are fused to obtain the reconstructed convolution feature.

[0031] Specifically, in the channel reconstruction module, the embodiment adopts a segmentation, transformation and fusion method to reduce channel redundancy. First, the spatial reconstruction feature obtained by the embodiment The channel is divided into two parts according to the preset ratio, and then compressed using a 1×1 convolution kernel to obtain the first compressed features of the upper and lower parts. and the second compression feature For the first compression feature , the embodiment uses efficient convolution GWC (Groupwise Convolution) and PWC (Pointwise Convolution) to extract high-level information features of the upper layer, where WC makes up for the damage of GWC to the channel group and reduces the information loss between channels. For the second compression feature , the embodiment uses PWC point convolution to generate shallow detail features of the lower layer as a supplement to the upper features. Then the upper and lower features are globally averaged pooled to obtain their corresponding global channel descriptors and , and the global channel descriptor is superimposed and processed by SoftMax to generate the corresponding feature vector and , under the guidance of the feature vector, the high-level information features and shallow-level detail features are fused in a channel-by-channel manner to obtain the reconstructed convolutional features.

[0032] In some embodiments of the present invention, Figure 5 FIG. 1 is a flow chart of flexible attention extraction according to an embodiment of the present invention. Figure 5 As shown, flexible attention features are extracted from the reconstructed convolution features to obtain flexible attention features, including: S501, performing multiple residual feature extraction on the reconstructed convolution feature to obtain a residual extraction feature; S502, performing horizontal maximum pooling on the residual extraction feature to obtain a first pooling feature, and performing vertical maximum pooling on the residual extraction feature to obtain a second pooling feature; S503, performing two-dimensional convolution and feature-aware normalization on the first pooled feature, and then multiplying and merging the first pooled feature with the first pooled feature to obtain a first fused feature, and performing two-dimensional convolution and feature-aware normalization on the second pooled feature, and then multiplying and merging the second pooled feature with the second pooled feature to obtain a second fused feature; S504, adding and merging the first fusion feature and the second fusion feature and concatenating them with the residual extraction feature to obtain a concatenated feature, and performing soft threshold feature mapping on the concatenated feature to obtain a soft threshold mapping feature; S505, adding and merging the soft threshold mapping feature and the reconstructed convolution feature to obtain a flexible attention feature.

[0033] Specifically, when it comes to road crack detection in complex environments, it is easy to have a high false detection rate due to interference from environmental noise. Most existing detection methods only mechanically deepen the number of network layers in the hope of enhancing detection accuracy to combat the interference of environmental noise. However, the deepening of the number of network layers makes the model more complex and interference factors are still difficult to be effectively suppressed. Its real-time performance and accuracy cannot meet the requirements of road detection. In order to solve this problem, an embodiment of the present invention designs a flexible attention mechanism (FAM) based on a residual network and a soft threshold to achieve noise removal in the model channel and enhance feature extraction capabilities.

[0034] In the FAM module, the embodiment first uses multiple residual blocks to extract deep residual extraction features of the feature map. The residual block is the core of the deep residual network ResNet. It solves the gradient explosion and gradient vanishing problems through jump connections and additive structures, while reducing the number of parameters and calculations, and improving the network's feature expression ability. The residual block reduces the image resolution and obtains crack features through continuous downsampling, and restores the image and integrates global information through continuous upsampling.

[0035] Then, the embodiment uses maximum pooling to perform independent directional feature perception on the residual extraction features in the horizontal and vertical directions, respectively, to obtain the first pooling feature and the second pooling feature. Among them, one spatial direction, horizontal and vertical, is used to capture long-range dependencies, and the other spatial direction is used to retain accurate location information. Subsequently, the first pooling feature and the second pooling feature are processed by two-dimensional convolution and feature perception normalization respectively, and then added and merged with various original first pooling features and second pooling features to obtain the first fusion feature and the second fusion feature, so as to achieve the extraction and fusion of multiple semantics in the channel from both local and global dimensions. Finally, the first fusion feature and the second fusion feature are added and merged and spliced ​​with the residual extraction feature to obtain the splicing feature, and the soft threshold is used for output adjustment and weight allocation to obtain the soft threshold mapping feature, and the soft threshold mapping feature and the reconstructed convolution feature are added and merged to output the flexible attention feature.

[0036] The flexible attention mechanism based on residual network and soft threshold in the embodiment is essentially a nonlinear feature mapper. In the soft threshold feature mapping, the embodiment learns a set of variable thresholds for each feature map of different scales, and then combines it with the original feature map to change the features unrelated to road cracks to values ​​infinitely close to 0, while retaining the features related to road cracks. The process formula is expressed as follows:

[0037] in, is the input feature map, is the input feature map, is a learnable soft threshold. In order to ensure that the soft threshold is not set too large so as not to cause the output feature value of the attention mechanism to be 0, and to ensure that the soft threshold is a positive number, its formula is expressed as follows:

[0038]

[0039] in, It is the absolute average of the input feature map, which is obtained by first taking the absolute value and then performing global average pooling in the attention mechanism. is the output value of the FC+BN+SiLU layer, is a scaling factor, which is used to limit the size of the soft threshold. At the same time, the sigmoid layer in the attention mechanism finally scales the scaling factor to a The value of .

[0040] Embodiments can make full use of rich context information through FAM, accurately capture the location information of cracks, effectively identify and remove interference information, and thus avoid interference from irrelevant information. Adding FAM to the model can enable the model to better locate crack information and effectively improve the anti-interference ability of the model.

[0041] In some embodiments of the present invention, Figure 6 FIG. 1 is a flow chart of improving visual feature extraction according to an embodiment of the present invention. Figure 6 As shown in the figure, the flexible attention feature is improved to extract visual features to obtain output features, including: S601, embedding the flexible attention feature to obtain an embedded feature; S602, after sandwich layout feature channel enhancement on the embedded features, batch normalization, ReLu activation function and average pooling are performed to obtain output features.

[0042] In some embodiments of the present invention, the sandwich layout feature channel enhancement includes cascaded attention extraction, several feed-forward network layers, and token interaction operations set before each feed-forward network layer.

[0043] Specifically, in order to save computing resources, reduce energy consumption, improve model deployment efficiency and compression transmission effect, an efficient solution is provided for resource-constrained environments such as embedded devices, mobile devices and edge computing. In the backbone part of the model, an improved visual feature extraction is introduced to improve the backbone network.

[0044] The embodiment first embeds 16×16 overlapping image blocks into The 3D tokens are used to improve the model's expressiveness, and then the processed embedded features are processed through a three-layer sandwich layout module for feature channel enhancement to enhance the communication efficiency between channels. Through the sandwich layout module, the number of tokens between modules is reduced by 4 times under downsampling. The overall module uses BatchNorm and ReLU activation functions to improve the model's reasoning speed and adaptability, and finally outputs features through average pooling.

[0045] In each sandwich layout module, the embodiment uses CGA (Cascaded Group Attention) and a memory-efficient feedforward network FFN (Feed Forward Network) to interact between channels. Its unit structure is to add an independent attention layer between two layers of feedforward network layers, and its formula is expressed as:

[0046] in, It is The layout module input features, is the corresponding output feature, To transform into A single self-attention needs to go through N layers of feedforward networks before and after. This design can effectively reduce the memory consumption caused by self-attention, and the superposition of feedforward network layers makes the communication between different features more convenient and efficient. Then an additional token interaction is applied before each FFN to introduce the inductive bias of local structural information to enhance the model performance.

[0047] In the conventional multi-head self-attention mechanism, the excessive redundancy of the attention heads leads to low computational efficiency. For this, the embodiment adopts the CGA cascade attention mechanism, which provides different segmentations of the complete features as input to each head, thereby explicitly decomposing the attention calculation into different heads. Formally, this attention can be expressed as:

[0048]

[0049] in, is the input feature No. The segmentation representation is and , is the total number of heads, , and represents the projection layer that maps input features to different subspaces, It is a linear layer responsible for projecting the output features to the same dimension as the input. The CGA structure uses Q, K, and V layers to learn feature projections with rich information to gradually expand its capacity, and calculates the attention map of each head in a cascade manner. It adds the output of each head to the subsequent head to gradually optimize the feature representation. The formula is expressed as:

[0050] When calculating attention, replace As the The new input features of the heads are used again after Q-projection, so that CGA can better capture global and local information. The cascade structure provides different features for each head, thereby improving the diversity of the attention map, and group attention can save parameters and computation by reducing input and output channels. In addition, cascading attention heads can increase the depth of the network and further improve the model capacity without introducing any additional parameters.

[0051] In order to verify the effect of the present invention, the embodiment experimentally verifies the scheme of the present invention. The operating environment of the experiment is the Window11 operating system, the hardware environment is i7-12700KF (CPU), NVIDIA GeForce RTX 4050 (GPU), and the algorithm framework is Python3.8, Pytorch1.8 and CUDA11.4. The Stochastic Gradient Descent (SGD) algorithm is the experimental optimizer, the image size is uniformly adjusted to 640×640, the initial learning rate is 0.01, the weight decay coefficient is 0.0005, the momentum is 0.937, and the maximum number of iterations is 200.

[0052] Among them, in the process of constructing training data, there are some problems with the public road crack data set, such as fewer crack images in complex scenes, unstable image resolution, poor annotation effect, etc. The embodiment uses a road crack data set collected and produced by itself, and the image acquisition uses a road detection drone for low-altitude shooting. The acquisition work focuses on crack images at different times, weather and backgrounds. In order to achieve effective crack detection in complex environments and avoid overfitting, this patent will randomly select images from the collected data set for rotation, cutting, random brightness enhancement and contrast enhancement, and obtain a total of 6780 crack image data. Then, the data set is divided into a training set, a validation set and a test set, with the ratios of 0.64, 0.16 and 0.20 respectively. Before model training, the obtained images are normalized and reduced to a size of 640×640 to ensure that the YOLO model performs best in training. Labelimg software is used to annotate the image data, and the annotation information output is adjusted to the txt format corresponding to YOLOv5.

[0053] Then the embodiment uses accuracy P, recall R, mean average precision mAP, detection speed FPS and parameter quantity Parameters as evaluation indicators of algorithm performance. Accuracy represents the proportion of the number of correct detections to the total number of detections; recall represents the proportion of predicted targets to the proportion of actual correct samples; mAP is the mean of average precision; FPS represents the number of frames processed per second; Parameters represents the number of parameters of the model. Accuracy P, recall R and mean average precision mAP are calculated as follows:

[0054]

[0055]

[0056]

[0057] In the above formula, Indicates the number of cracks successfully identified, represents the number of other targets identified as cracks, represents the number of crack samples that were not successfully identified, is the number of categories, represents the average precision, which is geometrically expressed as the area enclosed by the precision minus the recall curve.

[0058] It has been verified that compared with the single-stage benchmark algorithm YOLOv5s, this scheme reduces the model size by 6.5%, increases the recall rate by 6 percentage points, and increases the mAP by 4.5 percentage points. The present invention has significant improvements in both model size and detection accuracy. Compared with YOLOv3 and YOLOv4, the model size of the algorithm of the present invention is less than 6.6% and 6.3% of YOLOv3 and YOLOv4, but the detection accuracy is improved by 9.4% and 3.3%. Compared with the two-stage algorithms Faster R-CNN and Mask R-CNN, although the two-stage algorithms Faster R-CNN and Mask R-CNN have certain advantages in detection accuracy, they are difficult to be applied to mobile device deployment due to their complex algorithm structure, large amount of calculation, and slow detection speed.

[0059] In addition, in actual highway application scenarios, due to external interference such as light and shadow mottle, lane lines and fallen leaves, the two-stage algorithms Faster R-CNN and Mask R-CNN have false detections, and light and leaves are detected as road cracks. The single-stage algorithms YOLOv3, YOLOv4 and YOLOv5s have missed detections, and some crack information cannot be fully identified. In comparison, the algorithm model of the embodiment of the present invention can accurately identify cracks interfered by lane lines, and can effectively distinguish between interference objects and cracks.

[0060] In summary, in order to improve model efficiency, reduce false positive rate, and meet application requirements, in the complex environment pavement crack detection method provided by the present invention, the acquired pavement image to be detected is first input into a well-trained pavement crack detection network, the pavement image to be detected is subjected to convolution normalization processing to obtain input features, the input features are subjected to feature separation and reconstruction convolution to obtain reconstructed convolution features, the reconstructed convolution features are subjected to flexible attention extraction to obtain flexible attention features, the flexible attention features are subjected to improved visual feature extraction to obtain output features; then the output features are classified and predicted to output the pavement crack detection results. The present invention can effectively suppress redundant features and improve model efficiency through feature separation and reconstruction convolution, and can effectively remove environmental noise in features through the flexible attention mechanism, reduce false positive rate, and meet the application requirements of highway pavement crack detection.

[0061] In order to better implement the complex environment pavement crack detection method in the embodiment of the present invention, based on the complex environment pavement crack detection method, correspondingly, Figure 7 As shown, the present invention also provides a device for detecting cracks in a road surface in a complex environment. The device 700 for detecting cracks in a road surface in a complex environment includes: The image feature extraction unit 701 is used to input the acquired road surface image to be detected into a well-trained road crack detection network, perform feature separation and reconstruction convolution on the road surface image to be detected to obtain reconstructed convolution features, perform flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, and perform improved visual feature extraction on the flexible attention features to obtain output features; The classification prediction output unit 702 is used to classify and predict the output features and output the pavement crack detection results.

[0062] The complex environment pavement crack detection device 700 provided in the above embodiment can implement the technical solution described in the above complex environment pavement crack detection method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above complex environment pavement crack detection method embodiment, which will not be repeated here.

[0063] like Figure 8 As shown, the present invention also provides a mobile device 800. The mobile device 800 includes a processor 801, a memory 802 and a display 803. Figure 8 Only some of the components of the mobile device 800 are shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may be implemented instead.

[0064] In some embodiments, the processor 801 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 802, such as the complex environment pavement crack detection method of the present invention.

[0065] In some embodiments, the memory 802 may be an internal storage unit of the mobile device 800, such as a hard disk or memory of the mobile device 800. In other embodiments, the memory 802 may also be an external storage device of the mobile device 800, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the mobile device 800.

[0066] Furthermore, the memory 802 may include both an internal storage unit of the mobile device 800 and an external storage device. The memory 802 is used to store application software installed in the mobile device 800 and various data.

[0067] In some embodiments, the display 803 may be an LED display, a liquid crystal display, or a touch-sensitive liquid crystal display, etc. The display 803 is used to display information of the mobile device 800 and to display a visual user interface. The components 801-803 of the mobile device 800 communicate with each other via a system bus.

[0068] In one embodiment, when the processor 801 executes the complex environment pavement crack detection program in the memory 802, the following steps may be implemented: The acquired road surface image to be detected is input into a well-trained road crack detection network, the road surface image to be detected is convoluted and normalized to obtain input features, the input features are subjected to feature separation and reconstruction convolution to obtain reconstructed convolution features, the reconstructed convolution features are subjected to flexible attention extraction to obtain flexible attention features, and the flexible attention features are subjected to improved visual feature extraction to obtain output features; The output features are classified and predicted to output the pavement crack detection results.

[0069] It should be understood that: when the processor 801 executes the complex environment pavement crack detection program in the memory 802, in addition to the above functions, it can also implement other functions, and the details can be found in the description of the corresponding method embodiments above.

[0070] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions of the complex environment pavement crack detection method provided by the above-mentioned method embodiments.

[0071] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0072] The above is a detailed introduction to the complex environment pavement crack detection method, device, mobile device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting road cracks in a complex environment, characterized in that: include: Inputting the acquired road surface image to be detected into a well-trained road crack detection network, performing convolution normalization processing on the road surface image to be detected to obtain input features, performing feature separation and reconstruction convolution on the input features to obtain reconstructed convolution features, performing flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, and performing improved visual feature extraction on the flexible attention features to obtain output features; The output features are classified and predicted to output pavement crack detection results.

2. The complex environment pavement crack detection method according to claim 1 is characterized in that: The step of performing feature separation and reconstruction convolution on the input feature to obtain a reconstructed convolution feature includes: Performing spatial reconstruction on the input features to obtain spatial reconstruction features; Channel reconstruction is performed on the spatial reconstruction features to obtain reconstructed convolution features.

3. The complex environment pavement crack detection method according to claim 1 is characterized in that: The spatially reconstructing the input features to obtain spatially reconstructed features includes: Performing scale factor evaluation and spatial pixel variance measurement on the input features to obtain preliminary feature weights; Performing normalization mapping and threshold gating on the preliminary feature weights to obtain a first gating weight and a second gating weight; Multiplying the input feature by the first gating weight and the second gating weight respectively to obtain a first weighted feature and a second weighted feature; The first weighted feature and the second weighted feature are cross-reconstructed and spliced ​​to obtain a spatial reconstruction feature.

4. The complex environment pavement crack detection method according to claim 1, characterized in that: The performing channel reconstruction on the spatial reconstruction feature to obtain a reconstructed convolution feature includes: The spatial reconstruction feature is divided into a first channel feature and a second channel feature, and the first channel feature and the second channel feature are convoluted and compressed to obtain a first compressed feature and a second compressed feature; Performing group convolution and point convolution on the first compressed features to obtain high-level information features; Performing point convolution on the second compressed feature to obtain shallow detail features; Performing global average pooling on the high-level information features and the shallow-level detail features to obtain a global channel descriptor, and generating a feature vector according to the global channel descriptor; The high-level information feature and the shallow-level detail feature are fused according to the feature vector to obtain a reconstructed convolution feature.

5. The complex environment pavement crack detection method according to claim 1, characterized in that: The step of performing flexible attention extraction on the reconstructed convolutional features to obtain flexible attention features includes: Performing multiple residual feature extraction on the reconstructed convolution feature to obtain a residual extraction feature; Performing horizontal maximum pooling on the residual extraction feature to obtain a first pooling feature, and performing vertical maximum pooling on the residual extraction feature to obtain a second pooling feature; Performing two-dimensional convolution and feature-aware normalization on the first pooled feature, and then multiplying and merging the first pooled feature with the first pooled feature to obtain a first fused feature; performing two-dimensional convolution and feature-aware normalization on the second pooled feature, and then multiplying and merging the second pooled feature with the second pooled feature to obtain a second fused feature; The first fusion feature and the second fusion feature are added and combined, and then the combined features are spliced ​​with the residual extraction feature to obtain a spliced ​​feature, and soft threshold feature mapping is performed on the spliced ​​feature to obtain a soft threshold mapping feature; The soft threshold mapping feature and the reconstructed convolution feature are added and combined to obtain a flexible attention feature.

6. The complex environment pavement crack detection method according to claim 1, characterized in that: The step of extracting the improved visual features from the flexible attention features to obtain output features includes: Embedding the flexible attention feature to obtain an embedded feature; The embedded features are subjected to sandwich layout feature channel enhancement and then batch normalization, ReLu activation function and average pooling to obtain output features.

7. The complex environment pavement crack detection method according to claim 6, characterized in that: The sandwich layout feature channel enhancement includes cascaded attention extraction, a plurality of feedforward network layers, and a token interaction operation arranged before each of the feedforward network layers.

8. A method for detecting road cracks in a complex environment, characterized in that: include: An image feature extraction unit is used to input the acquired road surface image to be detected into a well-trained road crack detection network, perform feature separation and reconstruction convolution on the road surface image to be detected to obtain reconstructed convolution features, perform flexible attention extraction on the reconstructed convolution features to obtain flexible attention features, and perform improved visual feature extraction on the flexible attention features to obtain output features; The classification prediction output unit is used to classify and predict the output features to output pavement crack detection results.

9. A mobile device comprising a processor, a memory and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the complex environment pavement crack detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting pavement cracks in a complex environment according to any one of claims 1 to 7 is implemented.

Images