A Road Manhole Cover Status Detection Method Based on Improved YOLOX

Abstract

The application provides a road manhole cover state detection method based on improved YOLOX, comprising the following steps: S1, establishing a data set required by a model, collecting road manhole cover images under various conditions, and performing image processing; S2, establishing a YOLOX model, obtaining an anchor-free detector; S3, inserting an efficient channel attention module (ECANet) before the input of the Decoupled Head module, and further extracting channel features; S4, using a PyTorch framework to perform a training strategy of a frozen backbone network, and realizing a model; S5, obtaining a final road manhole cover state detection result through classification and regression. The application subdivides the manhole cover state into three categories of normal, damaged and subsided, and adds the attention module ECANet before the decoupled head of the YOLOX model to further extract channel features, thereby providing a new method for efficiently determining the position of the road manhole cover and identifying the state thereof.

Description

Technical Field

[0001] This invention relates to the field of combining deep learning neural networks with road anomaly detection, and more specifically, to a road manhole cover status detection method based on an improved YOLOX. Background Technology

[0002] With the rapid pace of urbanization, the construction of municipal public facilities has also developed rapidly. Manhole covers, as important urban public facilities, are widely used in municipal, telecommunications, gas, traffic management, and power industries. However, because manhole covers are laid on road surfaces, they often face damage and dents, not only detracting from the city's appearance but also becoming a common cause of road safety issues. Furthermore, road condition detection is an important application of many Intelligent Transportation Systems (ITS), and real-time and effective feedback on road conditions can provide a certain degree of safety protection for drivers. In summary, efficient, feasible, and timely knowledge of manhole cover conditions can not only assist the government in maintaining road traffic safety and become part of smart city development, but also support applications including autonomous driving within intelligent transportation systems.

[0003] In the field of road anomaly detection, the traditional method is to manually inspect for potential hazards, which is not only time-consuming and labor-intensive but also poses safety threats to workers. However, with the development of sensors, LiDAR and other high-precision devices are increasingly being used to detect road anomalies. For example, Mobile LiDAR is used to automatically detect manhole covers, Arduino is used to collect data, and IoT-based detection and monitoring systems for manhole covers are built. However, compared to image-based machine learning algorithms, sensor-based research is often more expensive in terms of equipment and computing costs. Moreover, with technological advancements, especially with the help of convolutional neural networks (CNNs), the performance of image-based target detection has greatly improved. This makes using deep learning algorithms to obtain the location and status of manhole covers in real time and accurately more cost-effective. Many scholars have begun to try using aerial photography or remote sensing images to train models for detecting manhole covers. Although these studies have achieved some results and have advantages such as low cost, wide detection range, and high detection accuracy, aerial images cannot detect damage and settlement of manhole covers and are easily affected by buildings and vegetation. Some scholars directly extract the training data needed by the model from Google Street View, but this method is relatively passive in acquisition and not real-time enough.

[0004] Dashcams, as common in-vehicle devices, capture road conditions while a car is in motion. This method of image acquisition is not only convenient and inexpensive but also subjectively guarantees image quality. In numerous studies on real-time road surface condition (RSC) monitoring, using dashcams to acquire images is a common approach. Manhole cover detection, as part of road condition detection, was studied using dashcam images and a self-made dataset. Based on the advanced anchorless frame detector YOLOX, an attention mechanism was incorporated into the model training. Experiments demonstrated that the improved model can effectively identify the location and state of manhole covers, achieving a more balanced detection accuracy and speed compared to Faster-RCNN, SDD, and other YOLO models. Summary of the Invention

[0005] This invention provides a road manhole cover status detection method based on improved YOLOX, which can better identify the location of manhole covers and determine their status.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] The improved YOLOX-based road manhole cover condition detection method includes the following steps:

[0008] S1: Establish the dataset needed for the model, collect images of road manhole covers under various different conditions, and perform image processing;

[0009] S2: Establish the YOLOX model to obtain an anchorless detector;

[0010] S3: Insert an efficient channel attention module (ECANet) before the input of the Decoupled Head module to further extract channel features;

[0011] S4: The training strategy of freezing the backbone network was implemented using the PyTorch framework to realize the model;

[0012] S5: The final detection results of the road manhole cover status are obtained through classification and regression.

[0013] Preferably, the establishment of the dataset in step S1 specifically involves:

[0014] To build the dataset needed for the model, the experiment used a dashcam to autonomously capture and collect 637 road images. Each image included one or more manhole cover instances, with a resolution of 3200×1800. Among them, there were 246 damaged manhole cover instances, 149 subsided ones, and 345 normal ones. Due to differences in shooting time, road conditions, and location, the collected road manhole cover images included a variety of situations, such as manhole covers obscured by other vehicles or shadows, manhole covers in inconspicuous locations, manhole covers with cracks in the surrounding road surface, manhole covers with road markings on their surface, and manhole covers that were not fully visible in the frame. This diversity makes the dataset itself relatively robust.

[0015] Preferably, the image data processing in step S1 specifically includes:

[0016] Although we intentionally collected images with variations, in order to further improve the robustness of the model, we used image processing techniques, including filtering transformation and noise transformation, to augment the dataset. Each category was augmented by a factor of two, and the Training-Val and Test sets were randomly divided in a ratio of 8:1:1, i.e., 1548 images for the training set, 172 images for the validation set, and 192 images for the test set.

[0017] Preferably, in step S2, the anchorless detector YOLOX is constructed as follows:

[0018] The backbone uses a Focus network structure to concentrate the width and height information of the image into the channels. Specifically, it takes one value for every other pixel, thus obtaining four independent feature layers, which are then stacked in the channels. A Decoupled Head is used to decouple the classification and regression tasks into two parallel branches. Mosaic and MixUp are added to the enhancement strategy. The number of predicted values ​​at each location is reduced, and four values ​​are predicted directly. SimOTA is defined to dynamically match positive samples for targets of different sizes. The final result is the constructed anchor-free detector.

[0019] Preferably, in step S3, the Decoupled Head inserts a high-efficiency channel attention module, specifically as follows:

[0020] The efficient channel attention module (ECANet) is a local cross-channel interaction strategy proposed on top of SENet that does not require dimensionality reduction and can be efficiently implemented through one-dimensional convolutions. After global average pooling (GAP), the fully-connected (FC) layers in SENet are replaced with a fast 1D convolution of size k. To avoid manually adjusting k through cross-validation, ECANet proposes a method for generating adaptive convolution kernels, the kernel size of which can be adaptively determined by a nonlinear mapping of the channel dimension.

[0021] The calculation process for GAP is as follows:

[0022]

[0023] Where W and H represent width and height respectively, and χ ij The feature value represents row i and column j. The nonlinear mapping adaptive calculation process for determining the value of k is as follows:

[0024]

[0025] Where C represents the channel dimension; |t| odd γ represents the nearest odd number t; γ and b are linear functions. The parameters are set to constants 2 and 1 respectively.

[0026] Preferably, the training strategy for freezing the backbone network in step S4 is as follows:

[0027] During training, the total number of training iterations (epochs) was set to 200, the learning rate to 0.001, and the weight decay to 0.0005. To avoid memory overflow, different batch sizes (2, 4, and 8) were set according to the model structure. Finally, stochastic gradient descent (SGD) and cosine annealing were used to optimize the training process. Due to the use of pre-trained weights, a training strategy of freezing the backbone network was implemented in the first 50 iterations.

[0028] Preferably, the final detection result is obtained through classification and regression in step S5, specifically as follows:

[0029] Two subnetworks are used for bounding box regression and classification, respectively. These subnetworks include a localization subnetwork and a classification subnetwork. The localization subnetwork predicts a preliminary representative set of points. and a refined set of representative prediction points Then according to The system generates directional target detection bounding box predictions, while the classification subnet combined with the localization subnet outputs a preliminary representative set of predicted points. The classification prediction results are generated, and the final prediction result is obtained by fusing the two prediction results.

[0030] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0031] 1. Self-made manhole cover detection dataset. The study used dashcam footage to capture road conditions and compiled 637 images. The condition of the manhole covers was further refined, mainly divided into three categories: normal, broken, and down.

[0032] 2. Innovative and improved manhole cover detection model. Based on the advanced anchorless detector YOLOX, the study incorporates an attention mechanism to further extract features and improve the model's accuracy. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention.

[0034] Figure 2 This is a structural diagram of an efficient channel attention module.

[0035] Figure 3 This is a diagram of the algorithm structure of the present invention.

[0036] Figure 4 This is a comparison chart of the improved model's predictions and the baseline.

[0037] Figure 5 A comparative visualization of prediction heatmaps for different attention modules. Detailed Implementation

[0038] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0039] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0040] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0041] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] Example 1

[0043] This embodiment provides a road manhole cover status detection method based on an improved YOLOX, such as... Figure 1 As shown, it includes the following steps:

[0044] S1: Establish the dataset needed for the model, collect images of road manhole covers under various different conditions, and perform image processing;

[0045] S2: Establish the YOLOX model to obtain an anchorless detector;

[0046] S3: Insert an efficient channel attention module (ECANet) before the input of the Decoupled Head module to further extract channel features;

[0047] S4: The training strategy of freezing the backbone network was implemented using the PyTorch framework to realize the model;

[0048] S5: The final detection results of the road manhole cover status are obtained through classification and regression.

[0049] The creation of the dataset in step S1 is as follows:

[0050] To build the dataset needed for the model, the experiment used a dashcam to autonomously capture and collect 637 road images. Each image included one or more manhole cover instances, with a resolution of 3200×1800. Among them, there were 246 damaged manhole cover instances, 149 subsided ones, and 345 normal ones. Due to differences in shooting time, road conditions, and location, the collected road manhole cover images included a variety of situations, such as manhole covers obscured by other vehicles or shadows, manhole covers in inconspicuous locations, manhole covers with cracks in the surrounding road surface, manhole covers with road markings on their surface, and manhole covers that were not fully visible in the frame. This diversity makes the dataset itself relatively robust.

[0051] The image data processing in step S1 is specifically as follows:

[0052] Although we intentionally collected images with variations, in order to further improve the robustness of the model, we used image processing techniques, including filtering transformation and noise transformation, to augment the dataset. Each category was augmented by a factor of two, and the Training-Val and Test sets were randomly divided in a ratio of 8:1:1, i.e., 1548 images for the training set, 172 images for the validation set, and 192 images for the test set.

[0053] In step S2, the anchorless detector YOLOX is constructed as follows:

[0054] Deep learning-based object detection algorithms can generally be divided into two categories: two-stage algorithms and single-stage algorithms. Two-stage detectors, represented by R-CNN, offer high accuracy but are slow. These detectors generate a series of candidate regions of interest (ROIs) that may contain the target, and then use these regions to classify and locate the foreground objects to obtain the results. In contrast, single-stage detectors, represented by the YOLO series and SSD, simplify the process into a regression task. The end-to-end design simplifies the algorithm structure and significantly improves detection speed. However, these detectors typically require setting a large number of prior boxes to ensure a sufficiently high IoU (intersection over union) rate with the ground-truth object. Furthermore, cluster analysis is needed before detection to find suitable anchor sizes and aspect ratios. Therefore, the academic community began to develop research on anchor-free detectors. YOLOX is an anchor-free detector that is an improvement on the YOLO series models. The backbone uses a Focus network structure to concentrate the width and height information of the image into the channels. Specifically, it takes one value for every other pixel, thus obtaining four independent feature layers, which are then stacked in the channels. A Decoupled Head is used to decouple the classification and regression tasks into two parallel branches. Mosaic and MixUp are added to the enhancement strategy. The number of predicted values ​​at each location is reduced, and four values ​​are predicted directly. SimOTA is defined to dynamically match positive samples for targets of different sizes. The final result is the constructed anchor-free detector.

[0055] In step S3, an efficient channel attention module is inserted into the Decoupled Head, specifically as follows:

[0056] In object detection tasks, the head is often used to determine the classification of the object and the location of the predicted bounding box. Classification tasks focus more on the texture information of the object, while regression tasks focus more on the edge information of the object. This information is usually distributed in the feature channels. Therefore, we insert an efficient channel attention module (ECANet) before the input of the Decoupled Head module to obtain cross-channel information and further extract channel features to help the model locate and identify the object more accurately.

[0057] The efficient channel attention module is a local cross-channel interaction strategy proposed on the basis of SE that does not require dimensionality reduction and can be efficiently implemented through one-dimensional convolution. Figure 2The diagram illustrates the ECANet model. After global average pooling (GAP), the fully-connected (FC) layers in SENet are replaced with a fast 1D convolution of size k to avoid the dimensionality decay caused by FC layers affecting the learning of channel attention weights. In the 1D convolution, the kernel size k represents the coverage of local cross-channel interactions, i.e., how much of the domain participates in the attention prediction of a channel. To avoid manually adjusting k through cross-validation, ECANet proposes a method for generating adaptive convolution kernels, where the kernel size can be adaptively determined by a non-linear mapping of the channel dimension.

[0058] The calculation process for GAP is as follows:

[0059]

[0060] Where W and H represent width and height respectively, and χ ij The feature value represents row i and column j. The nonlinear mapping adaptive calculation process for determining the value of k is as follows:

[0061]

[0062] Where C represents the channel dimension; |t| odd γ represents the nearest odd number t; γ and b are linear functions. The parameters are set to constants 2 and 1 respectively. The final algorithm structure diagram is as follows. Figure 3 .

[0063] The training strategy for freezing the backbone network in step S4 is as follows:

[0064] During training, the total number of training iterations (epochs) was set to 200, the learning rate to 0.001, and the weight decay to 0.0005. To avoid memory overflow, different batch sizes (2, 4, and 8) were set according to the model structure. The experiments selected accuracy (AP), average accuracy (mAP), number of parameters (params), computational cost (GFlops), and frames per second (FPS) as evaluation metrics for comparison and evaluation. The calculation methods for each evaluation metric are as follows:

[0065]

[0066]

[0067] Params = (C in K 2 +1)*C out

[0068] GFLOPs = 10 9FLOPs

[0069] FLOPs = 2 * H * W * (C in K 2 +1)*C out

[0070]

[0071] In the formula, p represents precision. Recall rate is represented by H and W, which represent width and height, respectively. C in C out The input and output channels are represented by K, the kernel size is K, the frame is the number of images detected by the model, and time is the total detection time.

[0072] In step S5, the final detection result is obtained through classification and regression, specifically as follows:

[0073] Two subnetworks are used for bounding box regression and classification, respectively. These subnetworks include a localization subnetwork and a classification subnetwork. The localization subnetwork predicts a preliminary representative set of points. and a refined set of representative prediction points Then according to The system generates directional target detection bounding box predictions, while the classification subnet combined with the localization subnet outputs a preliminary representative set of predicted points. The classification prediction results are generated, and the final prediction result is obtained by fusing the two prediction results.

[0074] Following the strategy of configuring different network structures based on image width and height in the YOLOv5 model, this paper provides several optional structures, including four standard network structures: YOLOX-s, YOLOX-m, YOLOX-l, and YOLOX-x, and two lightweight network structures: YOLOX-Nano and YOLOX-Tiny. In this paper, the lighter YOLOX-s standard network is selected as the baseline for experiments.

[0075] Figure 4 This demonstrates a comparison of the improved model's predictions with the baseline. Through intuitive comparison, we find that the baseline is more prone to confusing down and broken predictions, such as... Figure 4 The first line, which leads to more misclassifications in the baseline model; furthermore, like... Figure 4 As shown in row 2, the baseline also has a relatively higher false negative rate, especially for the augmented image. Overall, by adding the ECA attention module, the improved model achieves better prediction results.

[0076] Although we identified the strategy of using attention mechanisms to further extract features and improve model detection accuracy, there are many types of attention models, each focusing on different features. To select a more suitable type, we experimented with three of the most common attention models: SE, CBAM, and ECA. Table 1 shows the differences in the improvement of the entire network using different attention modules. The SE module performed the worst, even having a negative effect, while the CBAM module performed better in broken classes. Overall, ECA provided the greater improvement.

[0077] The reason ECA performs better overall is likely related to the characteristics of the YOLOX model and the ECA module. In the YOLOX model, the Focus module concentrates the width and height information of the input image into the channels, while the ECA module is known for its convolutional feature, which extracts information across channels. This may play a positive role in the model's better target identification. To more intuitively explain the prediction performance of the three types of attention modules, I also created a heatmap of the predicted values. Figure 5 This is the result of the visualization comparison.

[0078] To evaluate the performance of the proposed model, this paper compares it with most mainstream object detectors under the same training conditions, including the classic two-stage detector Faster R-CNN, the lightweight SSD model, the CenterNet detector which also has an anchor box-less structure, the more industrially used YOLOv3 model, and other YOLO series models.

[0079] Table 2 presents the experimental comparison results. It is clear that our model effectively detects the location and state of manhole covers, with the best performance in normal, settled, and average accuracy states. In particular, the detection performance for the most difficult settled state is significantly higher than other models, achieving an AP value of 92.2%. Furthermore, although the detection performance for damaged manhole covers is not the best, it is only less than 0.1 percentage points lower than Faster R-CNN, while the detection speed is much faster than Faster R-CNN, achieving a better balance between speed and accuracy. Regarding the model structure, since the ECA module is a lightweight attention module, it only adds 0.003G of computation, thus not significantly burdening the entire network and affecting the detection speed.

[0080] Table 1. Differences in the improvement of the entire network by different attention modules.

[0081]

[0082] Table 2. Comparison of experimental results for different models

[0083]

[0084] The same or similar labels correspond to the same or similar parts;

[0085] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0086] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A road manhole cover state detection method based on improved YOLOX, characterized in that, Comprise the following steps: S1: the data set needed to establish the model, collect a variety of different situations of road manhole cover image, and image processing; S2: the establishment of YOLOX model, get a kind of anchor frame detector, specifically: In the main part using Focus network structure, in order to the width and height information of the picture is concentrated in the channel, the specific implementation mode is every other pixel to take a value, in this way, four independent feature layer is obtained, finally in the channel stack these independent feature layer; Using Decoupled Head for two parallel branch decoupling classification and regression task; Mosaic and MixUp are added to the enhancement strategy; Reduce the number of prediction value of each position, and directly predict four values; Define SimOTA, for different size of target dynamic matching positive sample; Finally get the constructed anchor frame detector; S3: before the input of Decoupled Head module, insert efficient channel attention module, further extract channel features, specifically: Efficient channel attention module is a kind of local cross channel interaction strategy without dimensionality reduction based on SE, which is realized efficiently by one-dimensional convolution; After global average pooling, a fast 1D convolution with size k is used to replace the full connection layer in SE Net; The size of convolution kernel is determined by the nonlinear mapping of channel dimension; The calculation process of GAP is: wherein, , respectively represent width and height, represent the eigenvalues of i rows and j columns; the nonlinear mapping adaptive calculation process for determining k value is: where C denotes the channel dimension; denotes the nearest odd t; and are parameters of the linear function, which are set to constants 2 and 1, respectively; S4: the training strategy of freezing the backbone network is used to implement the model using the PyTorch framework; S5: through classification and regression to get the final road manhole cover state detection result.

2. The road manhole cover condition detection method based on the improved YOLOX according to claim 1, characterized in that, The establishment of data set in step S1, specifically: Use the dashboard to shoot and collect 637 road images, each image includes one or more manhole cover instances, the resolution size is 3200x1800, including 246 damaged manhole cover instances, 149 subsidence and 345 normal.

3. The method for detecting the state of road manhole covers based on improved YOLOX according to claim 1, characterized in that, The image data processing in step S1, specifically: Image processing methods including filtering transformation and noise transformation are used to expand the data set, each category is expanded twice, and Train-Val and Test set are randomly divided according to the ratio of 8:1:1, that is, 1548 training sets, 172 validation sets and 192 test sets.

4. The method for detecting the state of road manhole covers based on improved YOLOX according to claim 1, characterized in that, The training strategy of frozen backbone network in step S4, specifically: In the training process, the total training iteration number epochs is set to 200, the learning rate is 0.001, the weight decay is 0.0005, and the stochastic gradient descent method and cosine annealing algorithm are used to optimize the training process.

5. The method for detecting the state of road manhole covers based on improved YOLOX according to claim 1, characterized in that, The final detection result is obtained by classification and regression in step S5, specifically: The two subnets respectively perform bounding box regression and classification, and the two subnets include a positioning subnetwork and a classification subnetwork, wherein the positioning subnetwork predicts a preliminary representative point set and a refined representative point set According to a directional target bounding box prediction result is generated, and the classification subnetwork combines the preliminary representative point set output by the positioning subnetwork to generate a classification prediction result, and finally the final prediction result is obtained by fusing the two prediction results.

Images