Road defect real-time detection method, system, device and storage medium

Abstract

The application discloses a road defect real-time detection method, system, device and storage medium, wherein the method comprises the following steps: acquiring a to-be-detected road image; the to-be-detected road image is collected by a camera for real-time detection of a road video stream; the to-be-detected road image is input into a trained road defect detection model to output a road defect detection result; wherein the trained road defect detection model is realized by using an improved YOLOv8s network, a coordination attention mechanism layer CA is added in a Backbone network and a Neck network of the YOLOv8s network, and a P2CA module is further added in the Neck network of the original YOLOv8s network.

Description

Technical Field

[0001] This invention relates to the field of computer vision target detection technology, and in particular to a method, system, device and storage medium for real-time detection of road defects. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Highway maintenance refers to the upkeep, repair, and maintenance of highways to ensure their safety, smooth traffic flow, and comfort, extend their service life, and improve their service level. Highway maintenance is an indispensable part of the transportation system and a crucial means of ensuring highway safety and maximizing their benefits. The ability to quickly and accurately obtain road surface information is key to highway maintenance management.

[0004] In 2021, Aaquib Javed et al. trained a regional convolutional neural network on a road defect dataset and, by combining it with hardware, finally realized a real-time pothole detection system. However, the system only detected potholes, and the hardware requirements were relatively high. In 2022, Vung Pham et al., based on the YOLOv7 model, combined with methods such as class label smoothing and image enhancement to detect road defects. However, the model only detected four types of defects, lacking detection diversity. For small targets and inconspicuous defects such as cracks, there were missed detections. In 2022, Li Deke et al. invented an artificial intelligence-based road crack detection method and system. First, images were acquired, then the images were deblurred, and finally, the images were input into a neural network for training. However, the model was slow and could not meet the requirements for real-time detection; the model only detected cracks, resulting in a single detection method.

[0005] Traditional manual road defect detection has several shortcomings: harsh working environments jeopardize the safety of inspectors and disrupt normal traffic flow; it is time-consuming, labor-intensive, and inefficient, making timely and periodic inspections difficult; and it is susceptible to human error, hindering objective and accurate assessment of road damage. With the rapid development of detection and information technologies, road defect detection is increasingly characterized by high speed, automation, and intelligence, utilizing high-performance sensors for automatic road damage detection. However, due to the vast areas of roads to be inspected, these sensors are expensive and difficult to maintain. With the development of artificial intelligence, target-based road defect detection methods can automate and efficiently detect road defects, reducing the burden of manual inspection, improving the efficiency and quality of road maintenance, and reducing traffic accidents, which is of great significance for promoting road traffic safety and economic development. Currently, road defect detection technology still needs further improvement in the identification and classification of different types of road defects. Real-time road defect detection requires faster algorithms and processing speeds. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method, system, equipment, and storage medium for real-time detection of road defects, thereby improving the efficiency and quality of road maintenance.

[0007] On the one hand, it provides a method for real-time detection of road defects;

[0008] Real-time road defect detection methods include:

[0009] Acquire images of the road to be detected; the images of the road to be detected are acquired by a camera that detects road video streams in real time;

[0010] The road image to be detected is input into the trained road defect detection model, and the road defect detection result is output.

[0011] The trained road defect detection model is implemented using an improved YOLOv8s network. The improved YOLOv8s network adds a Coordinating Attention (CA) layer to the Backbone and Neck networks of the original YOLOv8s network. The improved YOLOv8s network also adds a P2CA module to the Neck network of the original YOLOv8s network.

[0012] On the other hand, a real-time road defect detection system was provided;

[0013] A real-time road defect detection system includes:

[0014] The acquisition module is configured to: acquire an image of the road to be detected; the image of the road to be detected is acquired by a camera that detects road video streams in real time;

[0015] The detection module is configured to: input the road image to be detected into the trained road defect detection model, and output the road defect detection results;

[0016] The trained road defect detection model is implemented using an improved YOLOv8s network. The improved YOLOv8s network adds a Coordinating Attention (CA) layer to the Backbone and Neck networks of the original YOLOv8s network. The improved YOLOv8s network also adds a P2CA module to the Neck network of the original YOLOv8s network.

[0017] Furthermore, an electronic device is also provided, including:

[0018] Memory, used for non-transitory storage of computer-readable instructions; and

[0019] Processor, for executing the computer-readable instructions,

[0020] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0021] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0022] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0023] One of the above technical solutions has the following advantages or beneficial effects:

[0024] 1. This invention uses YOLOv8s as the basic framework and integrates the CA (Coordinate Attention) module in the Backbone and Neck networks to capture long-range dependent features along one spatial direction and retain precise location information along another spatial direction, thereby improving the network's feature representation ability and enhancing the representation of the target of interest.

[0025] 2. For road defects that are not obvious and have small targets, this invention proposes a P2CA module, which is integrated into the Neck network to fuse the features of the second layer (P2), increasing the detection scale of the model to capture more feature information and reduce the false negative rate.

[0026] 3. This invention reorganizes and divides the training data from the CRDDC2022 competition, proposing a dataset with six types of road defects: longitudinal cracks (D00), transverse cracks (D10), alligator cracks (D20), potholes (D40), lane line blurring (D44), and pedestrian crossing blurring (D43). This improves the diversity of road defect model detection.

[0027] 4. Compared with existing road defect detection models, the model trained by this invention achieves faster speed, higher accuracy, and more detection categories while maintaining lightweight design; it fully meets the requirements of real-time detection; it is portable and scalable, easier to integrate with hardware devices, and simple to deploy. Attached Figure Description

[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0029] Figure 1 The YOLOv8s network structure is shown in Example 1;

[0030] Figure 2 This is the improved YOLOv8s network structure of Example 1;

[0031] Figure 3 This refers to the Focus module in Example 1;

[0032] Figure 4 This refers to the Conv module in Example 1;

[0033] Figure 5 The C2f structure is shown in Example 1;

[0034] Figure 6 This refers to the CA (Coordinate Attention) module in Example 1;

[0035] Figure 7 This is the SPPF structure of Example 1;

[0036] Figures 8(a)-8(f) Analysis of the road defect detection effect of the model in Example 1;

[0037] Figure 9 The PR curve of the model in Example 1 was obtained on the test set. Detailed Implementation

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] Example 1

[0040] This embodiment provides a method for real-time detection of road defects;

[0041] Real-time road defect detection methods include:

[0042] S101: Acquire the image of the road to be detected; the image of the road to be detected is acquired by a camera that detects road video streams in real time;

[0043] S102: Input the road image to be detected into the trained road defect detection model and output the road defect detection result;

[0044] The trained road defect detection model is implemented using an improved YOLOv8s network. The improved YOLOv8s network adds a Coordinate Attention Mechanism (CA) layer to the Backbone and Neck networks of the original YOLOv8s network. The improved YOLOv8s network also adds a P2CA module to the Neck network of the original YOLOv8s network.

[0045] Furthermore, such as Figure 2 As shown, the network structure of the trained road defect detection model includes:

[0046] The input layer, backbone network, neck network, and head module are connected in sequence.

[0047] Furthermore, the Backbone network includes: a Focus layer, a Conv1 layer, a C2f1 layer, a CA1 layer, a Conv2 layer, a C2f2 layer, a CA2 layer, a Conv3 layer, a C2f3 layer, a CA3 layer, a Conv4 layer, a C2f4 layer, a CA4 layer, and an SPPF layer connected in sequence.

[0048] The P2 feature layer consists of Conv1, C2f1, and CA1 layers; the P3 feature layer consists of Conv2, C2f2, and CA2 layers; the P4 feature layer consists of Conv3, C2f3, and CA3 layers; and the P5 feature layer consists of Conv4, C2f4, and CA4 layers.

[0049] Further, the Neck network includes: an upsampling layer Upsampling1, a fusion layer concat1, a C2f5 layer, an upsampling layer Upsampling2, a fusion layer concat2, a C2f6 layer, an upsampling layer Upsampling3, a fusion layer concat3, a C2f7 layer, a CA5 layer, a Conv5 layer, a fusion layer concat4, a C2f8 layer, a CA6 layer, a Conv6 layer, a fusion layer concat5, a C2f9 layer, a CA7 layer, a Conv7 layer, a fusion layer concat6, a C2f10 layer, and a CA8 layer connected in sequence.

[0050] Specifically, the output of layer C2f1 is connected to the input of layer concat3, the output of layer C2f2 is connected to the input of layer concat2, the output of layer C2f3 is connected to the input of layer concat1, the output of layer SPPF is connected to the input of layer Upsampling1, the output of layer SPPF is connected to the input of layer concat6, the output of layer C2f5 is connected to the input of layer concat5, and the output of layer C2f6 is connected to the input of layer concat4.

[0051] Furthermore, the Head module includes: Conv8 layer, Conv9 layer, Conv10 layer, and Conv11 layer; wherein, the input terminal of Conv8 layer is connected to the output terminal of CA5 layer, the input terminal of Conv9 layer is connected to the output terminal of CA6 layer, the input terminal of Conv10 layer is connected to the output terminal of CA7 layer, and the input terminal of Conv11 layer is connected to the output terminal of CA8 layer.

[0052] The P2CA module consists of the concat2 fusion layer, the C2f6 layer, the upsampling3 upsampling layer, the concat3 fusion layer, the C2f7 layer, the CA5 layer, the Conv5 layer, and the Conv8 layer.

[0053] The P2CA module is used to fuse the P2 feature layers from the Backbone network to improve the model's feature capture capability. A prediction layer from the P2 feature layers is added to the Head module. Before prediction, the CA attention module is used to assign higher weights to the parts that need attention.

[0054] Unlike the original model, the detection scale of the model has been increased, which improves the model's performance for detecting small and inconspicuous targets.

[0055] Furthermore, the internal structures of the fusion layers concat1, concat2, concat3, concat4, concat5, and concat6 are the same, and the fusion layer concat1 is used for serial splicing.

[0056] Furthermore, such as Figure 3 As shown, the Focus layer includes: four parallel branches, each branch's input is connected to the input of the Focus layer, each branch has a Slice layer, each branch's output is connected to the input of the Concat7 fusion layer, the output of the Concat7 fusion layer is connected to the input of the Conv12 layer, and the output of the Conv12 layer is connected to the output of the Focus layer.

[0057] Furthermore, such as Figure 4 As shown, the internal structures of Conv1, Conv2, Conv3, Conv4, Conv5, Conv6, Conv7, Conv8, Conv9, Conv10, Conv11, and Conv12 are consistent; among them, Conv1 includes a two-dimensional convolutional layer, a batch normalization layer, and an activation function layer connected in sequence.

[0058] Furthermore, such as Figure 5 As shown, the internal structures of layers C2f1, C2f2, C2f3, C2f4, C2f5, C2f6, C2f7, C2f8, C2f9, and C2f10 are the same. Layer C2f1 includes: a convolutional layer Conv13, a split layer, n Bottleneck layers, a fusion layer concat7, and a convolutional layer Conv14 connected in sequence.

[0059] Each Bottleneck layer includes: a convolutional layer Conv15, a convolutional layer Conv16, and an adder connected in sequence, wherein the input of the convolutional layer Conv15 is also connected to the input of the adder;

[0060] The outputs of the Split layer and the outputs of the n Bottleneck layers are all connected to the input of the fusion layer concat7.

[0061] Furthermore, such as Figure 6 As shown, the internal structures of layers CA1, CA2, CA3, CA4, CA5, CA6, CA7, and CA8 are the same. Layer CA1 includes:

[0062] The input layer, residual layer, re-weight layer, and output layer are connected in sequence.

[0063] The output of the Residual layer is also connected to the input of the one-dimensional horizontal average pooling layer and the input of the one-dimensional vertical average pooling layer, respectively.

[0064] The outputs of the one-dimensional horizontal average pooling layer and the one-dimensional vertical average pooling layer are both connected to the input of the concat8 fusion layer, and the output of the concat8 fusion layer is connected to the input of the two-dimensional convolutional layer.

[0065] The output of the two-dimensional convolutional layer is connected to the input of the batch normalization layer, and the output of the batch normalization layer is connected to the input of the nonlinear layer.

[0066] The output of the nonlinear layer is connected to the input of two two-dimensional convolutional layers, the output of each of the two two-dimensional convolutional layers is connected to the corresponding activation function layer, and the output of the two activation function layers is connected to the input of the re-weight layer.

[0067] To understand this, "residual" refers to adding skip connections across layers in a model to preserve information from the original input as it flows through multiple layers. Specifically, in neural networks using attention mechanisms, each layer transforms and processes the input before passing the output to the next layer. If skip connections are added during this process, information from the original input is added back to the output after the transformations and processing. This skip connection aims to alleviate the vanishing gradient problem in deep neural networks and also helps improve the training speed and performance of the model. By preserving information from the original input at each layer, the model can more easily learn higher-level features, thus improving its performance. In CA attention mechanisms, skip connections are typically added after the input features are summed with the attention-weighted features to ensure that information from the original input is still preserved after attention weighting.

[0068] To understand, re-weighting refers to weighting different parts of the input so that more attention is paid to certain parts when computing the output. Specifically, the attention mechanism divides the input into multiple parts and assigns a weight to each part. These weights are typically calculated by weighted averaging of the input, where each weight represents the importance of the corresponding input part. These weighted input parts are then used to compute the output, giving more attention to those parts with higher weights. Re-weighting is usually performed at each time step so that the model can dynamically adjust to different input parts. This allows the model to better adapt to different inputs and perform better on different tasks.

[0069] Furthermore, the CA1 layer's operation includes:

[0070] First, two one-dimensional global average pooling operations are used to aggregate the input features in the vertical and horizontal directions into two independent orientation-aware feature maps.

[0071] Then, the two independent orientation-aware feature maps are encoded into two attention maps, each of which captures the long-range dependencies of the input feature map along a spatial direction, so as to preserve the location information in the generated attention map.

[0072] Finally, the two attention maps are applied to the input feature map via multiplication to emphasize the representation of interest. This captures not only cross-channel information but also orientation-aware and position-sensitive information, which helps the model more accurately locate and identify objects of interest.

[0073] Furthermore, the CA1 layer's operation includes:

[0074] First, the input feature map is divided into two directions, width w and height h, and global average pooling is performed separately for each direction. The two spatial ranges of the pooling kernel are (H, 1) or (1, W). Each channel is then encoded using the following formula:

[0075]

[0076]

[0077] Where w and h represent the width and height of the feature map; W and H represent the pooling kernel size; This represents the output of the c-th channel at height h; The output of the c-th channel with width w; x c This represents the squeeze operation function.

[0078] Next, the feature maps in the width and height directions of the global receptive field are concatenated together, and then fed into the F1 function of a shared 1×1 convolutional module to reduce its dimension to the original size. Then, the batch-normalized feature maps are fed into the Sigmoid activation function to obtain a form like... Feature maps f, f∈R C / r×(H+W) The formula is shown below:

[0079] f=δ(F1([z h , z w (3)

[0080] Where F1 is a 1×1 convolution module function; z h , z wThe result is obtained from formulas (1) and (2); [z h , z w ] represents the cascading operation along the spatial dimension; δ is the nonlinear activation function; f is the intermediate feature map that encodes spatial information in the horizontal and vertical directions; r is the reduction rate used to control the block size (squeeze function) in the SE block.

[0081] Next, the feature map f is convolved with a 1×1 kernel according to its original height and width to obtain feature maps f with the same number of channels as the original. h and f w After passing through the Sigmoid activation function, the attention weights g of the feature map in height and width are obtained respectively. h And the attention weights g in the width direction w The formula is shown below:

[0082] g h =σ(F h (f h (4)

[0083] g w =σ(F w (f w (5)

[0084] Among them, f h and f w These are two independent tensors obtained by decomposing f along spatial dimensions; F h and F w σ is a 1×1 convolution transformation function; σ is the sigmoid function.

[0085] After calculation, the attention weights g of the input feature map in the height direction will be obtained. h And the attention weights g in the width direction w ;

[0086] Finally, by multiplicatively weighting the original feature map, we obtain the final feature map with attention weights in the width and height directions, as shown in the following formula:

[0087]

[0088] Where i and j represent coordinate position information; x c This represents the squeeze operation function; The attention weight feature map in the height direction obtained by formula (4); The attention weight feature map in the width direction obtained by formula (5) is shown.

[0089] The CA attention mechanism can capture not only cross-channel information, but also orientation and position awareness information, helping the model to more accurately locate and identify targets of interest.

[0090] Furthermore, such as Figure 7 As shown, the SPPF layer includes:

[0091] The Conv17, first max pooling layer, second max pooling layer, third max pooling layer, fusion layer concat9 and Conv18 are connected in sequence. The outputs of Conv17, the first max pooling layer and the second max pooling layer are all connected to the input of fusion layer concat9.

[0092] Furthermore, the SPPF layer includes:

[0093] First, a 1×1 Conv layer is passed to reduce the number of channels. Then, three 5×5 max pooling kernels are processed sequentially. The first max pooling layer represents a larger scale. The second max pooling layer further pools based on the first max pooling, so the resulting scale will be further reduced. The third max pooling layer is processed in the same way. Finally, concat is used to fuse the results.

[0094] Furthermore, the working principle of the trained road defect detection model includes:

[0095] The input layer is used to input the road image to be detected;

[0096] The Backbone network is used for the original 640×640×3 image input Focus structure. It uses a slice operation to first transform the image into a 320×320×12 feature map, then concatenates it, and finally performs a convolution operation to transform it into a 320×320×64 feature map.

[0097] The feature map obtained from the Conv structure is fed into the C2f structure. The feature map after Conv processing is split into two parts. The first part is processed by n Bottleneck modules and then concatenated with the second part. The feature map after each Bottleneck structure is involved in the concatenation with the second part.

[0098] After processing by the Focus, Conv, and C2f structures, the feature information of the P2 feature layer is obtained; the CA attention mechanism layer assigns higher weights to the feature information that the P2 feature layer needs to pay attention to.

[0099] Similarly, the P3, P4, and P5 feature layers are also obtained after processing by the Conv, C2f, and CA modules; after being processed by the SPPF structure at the end of the Backbone network, they are fed into the Neck network.

[0100] The Neck network first upsamples the feature maps from the Backbone network and then fuses them with the P4 feature layer. After C2f and upsampling, the maps enter the P2CA module for feature processing. First, the P3 feature layer is fused, then after C2f and upsampling, it is fused with the P2 feature layer to capture the feature information from the P2 feature layer. After that, it passes through C2f and the CA module. After the CA module, a prediction of the P2 feature layer is added to the Head section. Finally, a Conv layer is connected to fuse the features from the P2CA module. After C2f and the CA module, a prediction of the P3 layer features is performed in the Head section. Similarly, after fusing the P4 layer features, a prediction layer for P4 is added to the Head section; after fusing the P5 layer features, a prediction layer for P5 is added to the Head section.

[0101] The Head module has been changed from the original prediction layer for feature layers P3, P4, and P5 from the Backbone to the prediction layer for feature layers P2, P3, P4, and P5, and an additional prediction layer for feature layer P2 from the Backbone has been added.

[0102] Furthermore, the training process of the trained road defect detection model includes:

[0103] Construct training and testing sets; both the training and testing sets consist of road images with known road defect labels.

[0104] The training set is input into the road defect detection model to train the model. When the total loss function value of the model no longer decreases, the preliminary trained road defect detection model is obtained.

[0105] The test set is input into the initially trained road defect detection model to test the model. When the test results meet the test criteria, the test is stopped, and the final trained road defect detection model is obtained.

[0106] Furthermore, the construction of the training and test sets includes: dividing and integrating existing public datasets to enhance the diversity of road defect data in the datasets.

[0107] Furthermore, the road defect labels include: longitudinal cracks, transverse cracks, crazing, potholes, blurred lane lines, and blurred pedestrian crossings.

[0108] Furthermore, the testing criteria include: precision, recall, average precision, F1-Score, frames per second, and model size.

[0109] It should be understood that YOLOv8 is an object detection algorithm, the latest version of the YOLO (You Only Look Once) series of algorithms. Developed by Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao, its primary goal is to improve the accuracy and speed of object detection. YOLOv8 employs an anchor-free detection approach, using a Feature Pyramid Network (FPN) to process feature maps at different scales and replacing the ReLU function with an activation function similar to Swish. Furthermore, YOLOv8 utilizes some optimization techniques from previous versions, including BatchNormalization, Dropout, data augmentation, and model distillation. It also provides models of different sizes (N / S / M / L / X) based on scaling factors to meet the needs of various scenarios. This invention primarily focuses on improvements to YOLOv8, mainly because it has fewer parameters and faster speed, facilitating industrial deployment and real-time camera detection. The YOLOv8 network structure mainly consists of four parts: Input, Backbone, Neck, and Head, as follows... Figure 1 As shown.

[0110] YOLOv8 makes several improvements over YOLOv5. The Backbone and Neck sections replace the original C3 structure with a more gradient-rich C2f structure, and adjust the number of channels for different model scales, significantly improving model performance. The Head section undergoes major changes compared to YOLOv5, adopting the currently mainstream decoupled head structure to separate classification and detection tasks. It also changes from Anchor-Based to Anchor-Free. For Loss calculation, it employs a Task AlignedAssigner positive sample allocation strategy and introduces Distribution Focal Loss, which effectively improves accuracy.

[0111] When detecting road defects, the angle of the detection camera and the complex road environment generate a large number of small and inconspicuous targets, leading to missed or false detections and affecting the efficiency and quality of road maintenance. However, directly applying the original YOLOv8 model to road defect detection still has certain shortcomings in practical applications. This invention improves the YOLOv8 model. The improved network structure is as follows: Figure 2 As shown.

[0112] First, to enhance the focus on regions of interest, this invention integrates the Coordinate Attention (CA) mechanism into the Backbone and Neck networks. In the Backbone network, a CA attention mechanism layer is connected after each C2f module. These four CA attention mechanism layers correspond to the P2, P3, P4, and P5 feature layers. Similarly, in the Neck network, before Head prediction, three CA attention mechanism layers are connected after each of the three C2f modules, corresponding to the prediction outputs of the P3, P4, and P5 feature layers.

[0113] Secondly, to improve the model's ability to capture inconspicuous features, this invention proposes a P2CA module, such as... Figure 2 As shown, its purpose is to extract features from the P2 feature layer and add the prediction output of the P2 feature layer to the Head part, thus realizing the model's multi-scale prediction. The P2CA module also embeds a CA attention mechanism layer, which assigns higher weights to the features that the P2 feature layer needs to pay attention to.

[0114] Finally, this invention compiled a road defect dataset containing six categories and divided it into training and testing sets. The training set contained 6471 images, and the testing set contained 1610 images. The pixel size was resized to 640×640 before training and testing. The operating system used in the experiment was CentOS Linux 8, and the GPU was an NVIDIA A100-SXM4-40GB. The PyTorch 1.11 deep learning framework and CUDA 11.3 were used, with Python 3.8 as the programming language. No pre-trained model was used; the batch size was set to 32, and 32 training samples were taken from the training set for each training iteration. The SGD optimizer was used, with a momentum of 0.937, a weight decay coefficient of 0.0005, and initial and final learning rates of 0.01. After 150 rounds of training, Mosaic data augmentation was disabled in the last 10 rounds.

[0115] The following section explains the function and principle of each module in the model, as well as the data flow.

[0116] First, the 640×640×3 training set images are fed into the Backbone network for training. They are first processed through a Focus structure, such as... Figure 3As shown, a slicing-like operation is used to split the high-resolution feature map into multiple low-resolution feature maps, i.e., sampling every other column and then concatenating them. The original 640×640×3 image input to the Focus structure is first sliced ​​into a 320×320×12 feature map using the slice operation. After concatenation, it undergoes a convolution operation, finally becoming a 320×320×64 feature map. This method can reduce the information loss caused by downsampling.

[0117] The feature map processed by the Focus structure is then passed through the Conv structure, such as... Figure 4 As shown, it consists of 2D convolutional layers (Conv2d), batch normalization (BatchNorm2d), and SiLu activation function.

[0118] The feature map obtained from the Conv structure is fed into the C2f structure, such as... Figure 5 As shown, the feature map after Conv processing is split into two parts. The first part is processed by n Bottleneck modules and then concatenated with the second part. The feature map processed by each Bottleneck structure participates in the concatenation with the second part. The Bottleneck can optionally include shortcut connections. The C2f structure obtains richer gradient flow information while maintaining lightweight design.

[0119] After processing using the Focus, Conv, and C2f structures, the feature information of layer P2 is obtained. A CA attention mechanism layer is then used to assign higher weights to the feature information that layer P2 needs to focus on. The CA structure is as follows: Figure 6 As shown. The CA attention mechanism layer aims to enhance the network's ability to learn and express features; it can handle any intermediate feature tensor in the network:

[0120] X = x 1, x 2, …, c ]∈R H×W×C

[0121] After transformation, output a tensor of the same size:

[0122] Y = y l, y 2, …,y c ]∈R H×W×C

[0123] Similarly, feature layers P3, P4, and P5 are also obtained after processing by the Conv, C2f, and CA modules. Finally, after processing with the SPPF structure at the end of the Backbone network, they are fed into the Neck network. The SPPF structure is as follows: Figure 7 As shown, the process first uses a 1×1 Conv layer to reduce the number of channels. Then, three 5×5 max-pooling convolutional kernels are performed sequentially. The first max-pooling represents a larger scale, the second max-pooling further reduces the scale based on the first, and the third is processed similarly. Finally, concatenation is performed to fuse the results. The SPPF structure enhances the feature map's feature representation capability while maintaining speed.

[0124] In the Neck network, the feature maps from the Backbone network are first upsampled and then fused with the P4 feature layer. After C2f and upsampling, they enter the P2CA module proposed in this invention for feature processing. First, the P3 feature layer is fused, and after C2f and upsampling, it is fused with the P2 feature layer to capture the feature information from the P2 feature layer. Then, after C2f and the CA module, a prediction of the P2 layer is added to the Head section after the CA module, changing the original model's prediction only for the P3, P4, and P5 layers. Finally, a Conv layer is connected, fusing the features from the P2CA module. After C2f and the CA module, a prediction of the P3 layer features is performed in the Head section. Similarly, after fusing the P4 layer features, a prediction layer for P4 is added to the Head section; after fusing the P5 layer features, a prediction layer for P5 is added to the Head section.

[0125] In the Head section, the original P3, P4, P5 prediction layers were changed to P2, P3, P4, P5 prediction layers, adding a prediction for the P2 feature layer. This increases the model's detection scale to capture more feature information and reduce the false negative and false positive rates.

[0126] This invention evaluates model performance using precision (P), recall (R), average precision (AP), F1-Score, frames per second (FPS), and model size. Precision P is the probability that a predicted positive sample is actually a positive sample, and recall R is the probability that a sample actually being positive is predicted to be positive.

[0127] The method for calculating P:

[0128]

[0129] How to calculate R:

[0130]

[0131] TP stands for True Positive, meaning the sample is correctly identified as positive by the classifier; FP stands for False Positive, meaning the sample is correctly identified as positive by the classifier but is actually negative; and FN stands for False Negative, meaning the sample is correctly identified as negative by the classifier but is actually positive. Mean average precision (mAP) is the average of the average precision (AP) across all classes. Based on different IoU thresholds, mAP is categorized into mAP@0.5, mAP@0.5, and 0.95 for model evaluation.

[0132] AP calculation method:

[0133]

[0134] How mAP is calculated:

[0135]

[0136] Where C represents the number of categories.

[0137] How to calculate F1 score:

[0138]

[0139] FPS calculation method:

[0140]

[0141] This invention compares four lightweight object detection models: YOLOv3-tiny, YOLO5s, TPH-YOLOv5, and YOLO8s. As shown in Table 1, the improved model outperforms all other models. AP is improved by 2.6% compared to the original YOLOv8s model, and mAP@0.5:0:95 is improved by 1.0%, achieving a maximum FPS of 526.3, fully meeting the requirements for real-time (>30 FPS) detection.

[0142] Table 1 shows the experimental results of the improved YOLOv8s.

[0143]

[0144] like Figures 8(a)-8(f) The image shows the model's detection performance on six types of road defects: longitudinal cracks (D00), transverse cracks (D10), alligator cracks (D20), potholes (D40), lane line blurring (D44), and pedestrian crossing blurring (D43).

[0145] like Figure 9As shown, the PR curves obtained by the model on the test set are plotted, with the horizontal axis representing recall (R) and the vertical axis representing precision (P). Each curve represents the relationship between the model's detection precision (P) and recall (R) for each class. The mAP@0.5 for all classes reaches 0.636, which is higher than the original model.

[0146] Example 2

[0147] This embodiment provides a real-time road defect detection system;

[0148] A real-time road defect detection system includes:

[0149] The acquisition module is configured to: acquire an image of the road to be detected; the image of the road to be detected is acquired by a camera that detects road video streams in real time;

[0150] The detection module is configured to: input the road image to be detected into the trained road defect detection model, and output the road defect detection results;

[0151] The trained road defect detection model is implemented using an improved YOLOv8s network. The improved YOLOv8s network adds a Coordinating Attention (CA) layer to the Backbone and Neck networks of the original YOLOv8s network. The improved YOLOv8s network also adds a P2CA module to the Neck network of the original YOLOv8s network.

[0152] It should be noted that the acquisition module and detection module described above correspond to steps S101 to S102 in Embodiment 1. The examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the modules described above, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0153] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0154] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0155] Example 3

[0156] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0157] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0158] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0159] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0160] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0161] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0162] Example 4 This example also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Example 1.

[0163] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for real-time detection of road defects, characterized in that, The method comprises the following steps: acquiring a to-be-detected road image; the to-be-detected road image is collected by a camera for real-time detection of a road video stream; inputting the to-be-detected road image into a trained road defect detection model to output a road defect detection result; wherein the trained road defect detection model is realized by using an improved YOLOv8s network, the improved YOLOv8s network adds a coordination attention mechanism layer CA in a Backbone network and a Neck network of the YOLOv8s network, and the improved YOLOv8s network further adds a P2CA module in the Neck network of the original YOLOv8s network; the network structure of the trained road defect detection model comprises: an input layer, a Backbone network, a Neck network and a Head module connected in sequence; the Backbone network comprises: a Focus layer, a Conv1 layer, a C2f1 layer, a CA1 layer, a Conv2 layer, a C2f2 layer, a CA2 layer, a Conv3 layer, a C2f3 layer, a CA3 layer, a Conv4 layer, a C2f4 layer, a CA4 layer and a SPPF layer connected in sequence; wherein the Conv1 layer, the C2f1 layer and the CA1 layer constitute a P2 feature layer; the Neck network comprises: an up-sampling layer Upsampling1, a fusion layer concat1, a C2f5 layer, an up-sampling layer Upsampling2, a fusion layer concat2, a C2f6 layer, an up-sampling layer Upsampling3, a fusion layer concat3, a C2f7 layer, a CA5 layer, a Conv5 layer, a fusion layer concat4, a C2f8 layer, a CA6 layer, a Conv6 layer, a fusion layer concat5, a C2f9 layer, a CA7 layer, a Conv7 layer, a fusion layer concat6, a C2f10 layer and a CA8 layer connected in sequence; the Head module comprises: a Conv8 layer, a Conv9 layer, a Conv10 layer and a Conv11 layer; wherein an input end of the Conv8 layer is connected with an output end of the CA5 layer, an input end of the Conv9 layer is connected with an output end of the CA6 layer, an input end of the Conv10 layer is connected with an output end of the CA7 layer, and an input end of the Conv11 layer is connected with an output end of the CA8 layer; wherein the fusion layer concat2, the C2f6 layer, the up-sampling layer Upsampling3, the fusion layer concat3, the C2f7 layer, the CA5 layer, the Conv5 layer and the Conv8 layer jointly constitute a P2CA module; the P2CA module is used for fusing the P2 feature layer from the Backbone network, and a prediction layer from the P2 feature layer is added in the Head module, and before prediction, a CA attention module is combined to assign weights.

2. The method of claim 1, wherein the road defect is detected in real time. The Backbone network, wherein the Conv2 layer, the C2f2 layer and the CA2 layer constitute a P3 feature layer; wherein the Conv3 layer, the C2f3 layer and the CA3 layer constitute a P4 feature layer; and wherein the Conv4 layer, the C2f4 layer and the CA4 layer constitute a P5 feature layer. The Neck network, wherein an output end of the C2f1 layer is connected with an input end of a fusion layer concat3, an output end of the C2f2 layer is connected with an input end of a fusion layer concat2, an output end of the C2f3 layer is connected with an input end of a fusion layer concat1, an output end of the SPPF layer is connected with an input end of an up-sampling layer Upsampling1, an output end of the SPPF layer is connected with an input end of a fusion layer concat6, an output end of the C2f5 layer is connected with an input end of a fusion layer concat5, and an output end of the C2f6 layer is connected with an input end of a fusion layer concat4.

3. The method of claim 1, wherein the road defect is detected in real time. The CA1 layer, comprising: an input layer, a Residual layer, a Re-weight layer and an output layer connected in sequence; an output end of the Residual layer is further connected with an input end of a one-dimensional horizontal average pooling layer and an input end of a one-dimensional vertical average pooling layer; an output end of the one-dimensional horizontal average pooling layer and an output end of the one-dimensional vertical average pooling layer are both connected with an input end of a fusion layer concat8, and an output end of the fusion layer concat8 is connected with an input end of a two-dimensional convolution layer; an output end of the two-dimensional convolution layer is connected with an input end of a batch normalization layer, and an output end of the batch normalization layer is connected with an input end of a nonlinear layer; an output end of the nonlinear layer is connected with input ends of two two-dimensional convolution layers, respectively, and output ends of the two two-dimensional convolution layers are respectively connected with corresponding activation function layers, and output ends of the two activation function layers are connected with an input end of the Re-weight layer.

4. The method of claim 1, wherein the road defect is detected in real time. The working process of the CA1 layer, comprising: firstly, two one-dimensional global average pooling operations are used to respectively aggregate input features in vertical and horizontal directions into two independent direction perception feature maps; then, the two independent direction perception feature maps are respectively encoded into two attention maps, each of which captures long-range dependencies of the input feature map along a spatial direction to reserve position information in the generated attention map; finally, the two attention maps are applied to the input feature map through multiplication to emphasize the representation of interest.

5. The method of claim 1, wherein the road defect is detected in real time. The working process of the CA1 layer, comprising: firstly, the input feature map is divided into two directions of width w and height h for global average pooling, and two spatial range sizes of a pooling kernel are (H, 1) or (1, W); each channel is encoded according to the following formula: (1)； (2)； wherein, , denotes the width and height of the feature map; , denotes the size of the pooling kernel; the output of the i-th channel at height h; denotes the squeeze operation function;​ Then, the feature maps of the width and height of the global receptive field are spliced together, and then they are sent to the convolution module with shared convolution kernel function, which reduces the dimension to , and then the feature maps after batch normalization are sent to the Sigmoid activation function to obtain feature maps , , as shown in the following formula:​ (3)； wherein, a convolution module function; (1), (2) are obtained; denotes a concatenation operation along the spatial dimension; is a nonlinear activation function; is an intermediate feature map encoding spatial information in the horizontal and vertical directions; is a reduction rate used to control the block size of the squeeze function; Then, the feature map is obtained by performing convolution on the feature map in the height direction and the width direction respectively The convolution kernel is 1×1, and the channel number of the feature map obtained by convolution is the same as the original one And After the Sigmoid activation function, the attention weight of the feature map in the height direction and the attention weight in the width direction are obtained respectively And The formula is as follows: ； ； wherein, and are two independent tensors obtained by decomposing f along the spatial dimensions; is a 1 x 1 convolutional transform function; is a sigmoid function; After calculation, the attention weight of the input feature map in the height direction will be obtained and the attention weight in the width direction ; finally, the final feature map with attention weight in the width and height directions is obtained through multiplication weighting calculation on the original feature map, and the formula is as follows: (6)； wherein, represents an extrusion operation function; is the attention weight feature map in the height direction obtained by formula (4); is the attention weight feature map in the width direction obtained by formula (5).

6. The method of claim 2, wherein the road defect is detected in real time. The working principle of the trained road defect detection model, comprising: an input layer for inputting a road image to be detected; Backbone network, used for original image input Focus structure, using slice operation, first into feature map, after splicing, through a convolution operation, finally into feature map; The feature map obtained by the Conv structure is input into the C2f structure, and the feature map processed by the Conv is divided into two parts, wherein the first part is processed by the n Bottleneck modules and then spliced and fused with the second part, and the feature map processed by each Bottleneck structure participates in the splicing and fusion with the second part; After processing by the Focus, Conv and C2f structures, the feature information of the P2 feature layer is obtained; the CA attention mechanism layer is used to give higher weights to the feature information that needs to be focused on the P2 feature layer; Similarly, the P3 feature layer, the P4 feature layer and the P5 feature layer are obtained by processing the Conv, C2f and CA modules; and after processing by the SPPF structure at the end of the Backbone network, the processed result is input into the Neck network; The Neck network is used to first up-sample the feature map from the Backbone network, and then fuse the P4 feature layer; after C2f and up-sampling, the feature is processed by the P2CA module; first, the P3 feature layer is fused, and after C2f and Upsampling, the P2 feature layer is fused to capture the feature information from the P2 feature layer; then, after C2f and the CA module, a prediction of the P2 feature layer is added in the Head part, and finally a Conv layer is connected to fuse the features from the P2CA module; after C2f and the CA module, a prediction of the P3 layer is made in the Head part; by the same principle, after fusing the P4 layer feature, a prediction layer of the P4 is added in the Head part; after fusing the P5 layer feature, a prediction layer of the P5 is added in the Head part; The Head module changes the original prediction layer of the feature layer P3, P4 and P5 from the Backbone part to the prediction layer of the feature layer P2, P3, P4 and P5, and adds a prediction layer of the P2 feature layer from the Backbone part.

7. A real-time road defect detection system characterized in that, It comprises: An acquisition module configured to acquire a to-be-detected road image; the to-be-detected road image is collected by a camera for real-time detection of a road video stream; A detection module configured to input the to-be-detected road image into a trained road defect detection model to output a road defect detection result; The trained road defect detection model is implemented by using an improved YOLOv8s network, wherein the improved YOLOv8s network adds a coordination attention mechanism layer CA in the Backbone network and the Neck network of the YOLOv8s network, and the improved YOLOv8s network further adds a P2CA module in the Neck network of the original YOLOv8s network; The network structure of the trained road defect detection model comprises: an input layer, a Backbone network, a Neck network and a Head module connected in sequence; The Backbone network comprises: a Focus layer, a Conv1 layer, a C2f1 layer, a CA1 layer, a Conv2 layer, a C2f2 layer, a CA2 layer, a Conv3 layer, a C2f3 layer, a CA3 layer, a Conv4 layer, a C2f4 layer, a CA4 layer, and a SPPF layer connected in sequence. The Conv1 layer, the C2f1 layer, and the CA1 layer constitute a P2 feature layer. The Neck network comprises: an up-sampling layer Upsampling1, a fusion layer concat1, a C2f5 layer, an up-sampling layer Upsampling2, a fusion layer concat2, a C2f6 layer, an up-sampling layer Upsampling3, a fusion layer concat3, a C2f7 layer, a CA5 layer, a Conv5 layer, a fusion layer concat4, a C2f8 layer, a CA6 layer, a Conv6 layer, a fusion layer concat5, a C2f9 layer, a CA7 layer, a Conv7 layer, a fusion layer concat6, a C2f10 layer, and a CA8 layer connected in sequence. The Head module comprises: a Conv8 layer, a Conv9 layer, a Conv10 layer, and a Conv11 layer. The input end of the Conv8 layer is connected to the output end of the CA5 layer, the input end of the Conv9 layer is connected to the output end of the CA6 layer, the input end of the Conv10 layer is connected to the output end of the CA7 layer, and the input end of the Conv11 layer is connected to the output end of the CA8 layer. The fusion layer concat2, the C2f6 layer, the up-sampling layer Upsampling3, the fusion layer concat3, the C2f7 layer, the CA5 layer, the Conv5 layer, and the Conv8 layer jointly constitute a P2CA module. The P2CA module is configured to fuse the P2 feature layer from the Backbone network, and add a prediction layer from the P2 feature layer to the Head module. Before prediction, a CA attention module is combined to assign weights. 8.An electronic device, comprising: a memory configured to non-transitorily store computer-readable instructions; and a processor configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, perform the method of any one of claims 1-6.

9. A storage medium characterized by, non-transitorily storing computer-readable instructions, wherein the non-transitory computer-readable instructions, when executed by a computer, perform the method of any one of claims 1-6.

Images