A lightweight object segmentation method for end-to-end road damage detection

Abstract

The present invention discloses a lightweight end-to-end road damage detection target segmentation method. First, images in a data set are passed through a backbone feature extraction network Shuffle-ECANet to obtain features, and then passed through a multi-scale feature fusion network to obtain feature maps containing different scales; the feature maps are then passed through a lightweight multi-branch channel attention network LMCA-Net to obtain an image that finally identifies the damage. The weights of the network are obtained through training. During actual detection, the real-time image captured by the camera can be input into the target segmentation model, and the weights obtained above can be imported to identify and segment the road damage. The present invention can quickly, automatically and accurately identify and classify specific types of road damage, and still has good performance effects in target segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and in particular relates to a road damage detection target segmentation method. Background Art

[0002] At present, China already has a huge highway and road network. Every year, relevant departments have to invest a lot of manpower and financial resources to solve the problem of road damage. If road defects cannot be discovered in a timely and accurate manner, road conditions will further deteriorate and even easily lead to traffic accidents. Therefore, how to quickly, conveniently and accurately identify road damage has become a major issue in the current field of computer vision.

[0003] Early road damage detection mainly relied on manual inspection, which had problems such as poor safety, low efficiency and high cost. Later, automated road inspection technology was derived, relying on sensor equipment mounted on road patrol vehicles for detection, but there were still problems such as high hardware cost and low recognition accuracy. Currently, road damage detection mainly relies on methods in the field of computer vision, based on neural network models, and uses image recognition to accurately determine the type of road damage.

[0004] Current research in target segmentation for road damage identification: In order to solve the problem of only classifying images with or without road damage and to be able to detect road damage at the pixel level more intuitively through the network, Pereira et al. used U-Net for semantic segmentation of road and pothole images. Their network structure is divided into two parts: encoder and decoder for feature extraction, feature fusion and result prediction. In order to design a more advanced semantic segmentation model to improve the detection rate, Fan et al. proposed a new semantic segmentation pothole detection method, which uses a spatial pyramid pooling module composed of serial hollow convolutions, and uses the channel attention mechanism to enhance the feature extraction process and integrate spatial context information, which helps to detect multi-scale road potholes. To solve the problem of difficult road crack detection, Zhang et al. improved the crack road detection performance of AD-Net by adding hollow convolutions between the encoder and decoder and introducing deep supervision in the decoder stage. In order to enhance the feature representation ability and reduce the impact of interference factors such as shadows and noise on road crack detection, Fang et al. improved the performance of AD-Net by configuring transformer blocks in the encoder layer.

[0005] At present, road damage detection still has some shortcomings in target segmentation. The main problem is that although the detection accuracy is high, the detection speed is slow, and the two cannot be well balanced. It requires multi-stage operations, often pre-processing and post-processing of images, and the real-time detection performance is poor, and it is impossible to achieve an end-to-end lightweight model network. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a lightweight end-to-end road damage detection target segmentation method. First, the images in the data set are passed through the backbone feature extraction network Shuffle-ECANet to obtain features, and then passed through the multi-scale feature fusion network to obtain feature maps containing different scales; the feature maps are then passed through the lightweight multi-branch channel attention network LMCA-Net to obtain the final image that identifies the damage. The weights of the network are obtained through training. During actual detection, the real-time image taken by the camera can be input into the target segmentation model, and the weights obtained above can be imported to identify and segment the road damage. The present invention can quickly, automatically and accurately identify and classify specific types of road damage, and still has good performance effects in target segmentation.

[0007] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0008] Step 1: The images in the dataset are passed through the backbone feature extraction network Shuffle-ECANet to obtain features, and the input image size is a×b×3;

[0009] Replace the backbone feature extraction network of YOLOv5 with Shuffle-ECANet. Shuffle-ECANet consists of ShuffleNetV2 and ECA-Net attention mechanism. ECA-Net converts the W*H*C feature map extracted by the Shuffle layer into a 1*1*C feature map through global average pooling and 1*1 convolution dimensionality reduction.

[0010] Step 2: Then, through the multi-scale feature fusion network, feature maps containing different scales are obtained;

[0011] A feature pyramid structure is used to implement a multi-scale feature fusion network; based on the feature pyramid network FPN, an adaptive attention module AAM and a feature enhancement module FEM are added; the input image is subjected to multiple convolutions to generate multi-scale feature maps C1, C2, C3, C4, and C5, and the corresponding top-down feature maps are M5, M4, and M3. Through the above Shuffle-ECANet module, 1*1*C can be extracted. C5 generates a feature map M6 through AAM, and M6 is summed with M5 and propagated through a top-down path to fuse with other features at a lower level, and FEM is added in each feature map Mx, x=3,4,5 module;

[0012] Step 3: The feature map is then passed through the lightweight multi-branch channel attention network LMCA-Net to obtain the final recognized image, which has the same size as the input image, a×b×3;

[0013] The specific steps of the lightweight multi-branch channel attention network LMCA-Net are:

[0014] Convolve the input feature map F with convolution kernels of sizes 3×3, 5×5, and 7×7 to obtain three feature maps F1, F2, and F3, and then sum them to obtain F' of shape C×H×W;

[0015] Pass F' into a pooling layer and perform average pooling along the H and W dimensions;

[0016] Use 1D convolution to map the original C dimension to Z dimension information, and then use three 1D convolutions to change from Z dimension to the original C dimension to complete the information extraction of channel dimension;

[0017] Step 4: Steps 1 to 3 form a target segmentation model, train the target segmentation model, and obtain a trained target segmentation model for use in damage identification;

[0018] Step 5: Input the real-time image captured by the camera into the trained target segmentation model to detect and segment road damage.

[0019] Preferably, the input image size is 416×416×3.

[0020] The beneficial effects of the present invention are as follows:

[0021] 1. The present invention aims to quickly, automatically and accurately identify and classify specific types of road damage, while still having good performance results in target segmentation.

[0022] 2. The feature extraction and multi-scale fusion network designed by the present invention are more conducive to target recognition and classification at different distances and angles.

[0023] 3. The embedded lightweight attention module ECA-Net proposed in this paper can improve detection accuracy with fewer parameters than previous studies.

[0024] 4. Various ablation experiment results evaluating the backbone network ShuffleNetV2, attention mechanism and other widely used target detection models show that the present invention can achieve significant performance improvements with very little computation and is easier to implant in mobile devices.

[0025] 5. The present invention can also achieve high-efficiency detection, allowing continuous video frames to be detected and classified in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The figure is a flow chart of the method of the present invention.

[0027] Figure 2This is a schematic diagram of the structure of the backbone feature extraction network Shuffle-ECANet of the present invention.

[0028] Figure 3 Schematic diagram of the feature pyramid structure of the multi-scale feature fusion network of the present invention.

[0029] Figure 4 Schematic diagram of the lightweight multi-branch channel attention network LMCA-Net of the present invention.

[0030] Figure 5 It is a structural expansion diagram of the multi-branch channel of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0032] In order to solve the problem of high computational cost and low lightness in target recognition in road damage detection, a compromise between accuracy and computational cost is achieved. Due to the limited memory and computing resources of embedded devices, the efficiency and lightness of the network itself are crucial, so the network calculation amount should be reduced as much as possible while ensuring accuracy. Since the overall calculation of the neural network mainly depends on the number of parameters of the backbone network, it is crucial to choose a lightweight backbone network. In addition, features of different scales have different functions. Shallow features have higher resolution and contain more detailed information, while deep features have more semantic information but poor perception of details. In order to improve the performance of detection, fusing features of different scales is an important method. Finally, target segmentation requires the addition of an attention mechanism in the network to improve detection efficiency.

[0033] Based on the characteristics of lightweight, high efficiency, high performance, and end-to-end, the present invention uses a lightweight backbone feature extraction network to ensure efficient automatic feature extraction while making the model parameters smaller. In addition, the present invention proposes to use a multi-scale feature fusion network to enrich the diversity of road damage features. Finally, a lightweight embedded attention network is introduced to assign weights to multi-scale convolution kernels according to the object size, thereby enhancing feature information and improving detection accuracy with smaller parameters.

[0034] like Figure 1 As shown, a lightweight end-to-end road damage detection target segmentation method includes the following steps:

[0035] Step 1: First, the weight model is obtained by training the neural network. In this process, the dataset is firstly passed through the backbone feature extraction network Shuffle-ECANet to obtain features, and the input image size is 416×416×3.

[0036] Step 2, then pass through the proposed multi-scale feature fusion network to obtain feature maps containing different scales.

[0037] Step 3: After passing through the lightweight multi-branch channel attention network LMCA-Net, the final image (416×416×3) is obtained.

[0038] Step 4: Put it into the training model for training to obtain the weight model.

[0039] Step 5: Installed in an embedded device, the real-time image is captured by a camera, transmitted to the prediction model, the trained weights are called, and the final result is predicted.

[0040] like Figure 2 As shown, on the backbone feature extraction network, the original YOLOv5 backbone feature extraction network adopts the C3 structure, which will bring a large number of parameters, resulting in slow detection speed and low efficiency. The present invention refers to a lightweight backbone feature extraction network, replaces the original YOLOv5 backbone feature extraction network with Shuffle-ECANet, and the detection speed is much faster than the original YOLOv5. It includes two parts: ShuffleNetV2 and ECA-Net attention mechanism. ShuffleNetV2 is a relatively advanced lightweight CNN network at present, which introduces a channel splitting method to reduce the degree of fragmentation and uses channel random operations to mix functions. The channel attention mechanism has been proven to have great potential in improving the performance of CNN. The present invention introduces a lightweight attention network ECA-Net to provide faster speed and better performance for mobile devices. ECA-Net is an extremely lightweight and efficient attention mechanism module that can capture local cross-channel interactions. ECA-Net uses a one-dimensional sparse convolution filter to generate channel weights to replace the complete connection layer in SENet. This solves the problem caused by the dimensionality reduction operation, greatly reduces the complexity of the network, and can maintain the same performance as the original. This Shuffle-ECANet backbone network can greatly improve the degree of lightweight and provide support for end-to-end lightweight detection.

[0041] like Figure 3As shown, there are targets of different sizes in the image, and different targets have different features. Simple targets can be distinguished by shallow features; complex targets can be distinguished by deep features. Therefore, the present invention requires a feature pyramid structure to realize multi-scale target fusion. The present invention adds an adaptive attention module (AAM) and a feature enhancement module (FEM) on the basis of the traditional feature pyramid network FPN. The former can reduce feature channels and reduce the loss of contextual information in high-level feature maps. The latter part improves the inference speed and optimizes the computing performance. FEM uses dilated convolution to adaptively learn different receptive fields in each feature map according to the different scales of the detected damage, thereby improving the accuracy of multi-scale target detection and recognition. The input image generates multi-scale feature maps C1, C2, C3, C4, C5 through multiple convolutions, and C5 generates feature map M6 through AAM. M6 is summed with M5 and propagated through a top-down path to fuse with other features at a lower level, and finite element analysis is performed after each fusion by expanding the receptive field. And FEM is added to each feature map Mx module to enhance the representation of the feature pyramid.

[0042] like Figure 4 and Figure 5 As shown, different sizes of convolution kernels will have other effects on objects of different scales. Usually, an attention mechanism is usually added to add weights to the convolution kernel to improve its ability to distinguish information during the CNN design process. Multi-scale convolution kernels are essential for obtaining more feature information because objects are of different sizes when performing road damage detection. In the present invention, a LMCA-Net is cited to embed fewer attention modules in the network to improve detection efficiency. First, the input feature map F is convolved with convolution kernels of sizes 3×3, 5×5, and 7×7 to obtain three feature maps F1, F2, and F3, and then summed to obtain F' with a shape of C×H×W. Next, enter a pooling layer and perform average pooling (Global Average Pooling) along the H and W dimensions, so that the vector can represent the importance of the information of each channel. Next, a 1D convolution is used to map the original C dimension to Z-dimensional information. After that, three 1D convolutions are used to change from the Z dimension to the original C dimension. This completes the information extraction of the channel dimension. Compared with the fully connected layer in the linear transformation, 1D convolution can effectively capture the information of cross-channel interaction while significantly reducing the number of parameters. Softmax is used for normalization. At this time, each channel corresponds to a score representing the importance of its channel.

Claims

1. A lightweight end-to-end road damage detection target segmentation method, characterized in that: The steps include: Step 1: The images in the dataset are passed through the backbone feature extraction network Shuffle-ECANet to obtain features, and the input image size is a×b×3; Replace the backbone feature extraction network of YOLOv5 with Shuffle-ECANet. Shuffle-ECANet consists of ShuffleNetV2 and ECA-Net attention mechanism. ECA-Net converts the W*H*C feature map extracted by the Shuffle layer into a 1*1*C feature map through global average pooling and 1*1 convolution dimensionality reduction. Step 2: Then, through the multi-scale feature fusion network, feature maps containing different scales are obtained; A feature pyramid structure is used to realize a multi-scale feature fusion network; On the basis of the feature pyramid network FPN, an adaptive attention module AAM and a feature enhancement module FEM are added; the input image is to generate multi-scale feature maps C1, C2, C3, C4, C5 through multiple convolutions, and the corresponding top-down feature maps are M5, M4, M3. Through the above Shuffle-ECANet module, 1*1*C can be extracted. C5 generates feature map M6 through AAM. M6 is summed with M5 and propagated through a top-down path to fuse with other features at a lower level, and FEM is added in each feature map Mx, x=3,4,5 module; Step 3: The feature map is then passed through the lightweight multi-branch channel attention network LMCA-Net to obtain the final recognized image, which has the same size as the input image, a×b×3; The specific steps of the lightweight multi-branch channel attention network LMCA-Net are: Convolve the input feature map F with convolution kernels of sizes 3×3, 5×5, and 7×7 to obtain three feature maps F1, F2, and F3, and then sum them to obtain F' of shape C×H×W; Pass F' into a pooling layer and perform average pooling along the H and W dimensions; Use 1D convolution to map the original C dimension to Z dimension information, and then use three 1D convolutions to change from Z dimension to the original C dimension to complete the information extraction of channel dimension; Step 4: Steps 1 to 3 form a target segmentation model, train the target segmentation model, and obtain a trained target segmentation model for use in damage identification; Step 5: Input the real-time image captured by the camera into the trained target segmentation model to detect and segment road damage.

2. A lightweight end-to-end road damage detection target segmentation method according to claim 1, characterized in that: The input image size is 416×416×3.

Images