A method and system for detecting abnormal road conditions

By combining a road anomaly detection model with deformable convolutional residual modules, coordinate attention mechanisms, and direct-connection feature pyramid networks, the problem of irregular target shapes, varying sizes, and complex backgrounds in road inspections is solved, achieving efficient detection of various types and sizes of anomalies and accurate classification of small foreign objects.

CN118570639BActive Publication Date: 2026-11-10SHANDONG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410699891.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-31
Publication Date
2026-11-10
Estimated Expiration
2044-05-31

AI Technical Summary

Technical Problem

Existing technologies for detecting road anomalies suffer from problems such as irregular target shapes, varying sizes, complex and variable image backgrounds, and diverse data sources, resulting in poor detection performance. In particular, during routine inspections, the variety of anomalies and their varying sizes make effective detection difficult, and during key inspections, it is impossible to effectively distinguish the materials of tiny foreign objects.

Method used

A road anomaly detection model is adopted, which combines deformable convolutional residual modules, coordinate attention mechanisms, and direct-connected feature pyramid networks to enhance feature extraction and multi-scale feature fusion capabilities. Furthermore, a fine-grained classification module is introduced on this basis, and classification performance is enhanced through attention pruning and discarding operations.

Benefits of technology

It achieves efficient detection of abnormal road conditions, accurately identifies various types and sizes of abnormalities, and accurately distinguishes the materials of tiny foreign objects during key inspections, thus improving detection accuracy and recall rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570639B_ABST
    Figure CN118570639B_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for detecting abnormal road conditions, comprising: acquiring a road image to be processed; using an anomaly detection model to detect the road image to be processed to obtain an initial detection target; cropping the initial detection target image from the road image to be processed; classifying the initial detection target image using a fine-grained classification model to obtain the final anomaly detection target. This solves the problem of diverse types and sizes of abnormal conditions in daily inspections, making them difficult to detect effectively, as well as the problem of not being able to effectively distinguish the materials of small foreign objects in key inspections.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for detecting abnormal road conditions. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Road inspection work consists of two parts: routine inspection and key inspection. Routine inspection mainly detects abnormalities such as street vendors occupying the road, foreign objects on the road, and road defects. Key inspection mainly detects small foreign objects on the road and classifies them by material.

[0004] Currently, due to the following difficulties in the process of detecting abnormal road conditions, existing models cannot achieve good detection results in inspection tasks. In daily inspections, there are various types and sizes of abnormal conditions, making it difficult to detect them effectively. In key inspections, there is a problem that the materials of small foreign objects cannot be effectively distinguished.

[0005] The specific difficulties include:

[0006] (1) The shapes of the targets in abnormal road conditions are irregular, and the aspect ratios of different types of abnormal conditions vary greatly;

[0007] (2) There is no fixed scene for road inspection. All images are captured dynamically, and the background of the images is complex and varied with a lot of noise interference.

[0008] (3) The data sources for road inspection include various methods such as inspection vehicle photography, drone photography, surveillance photography, and manual photography. The angle and size of the images taken by different methods are different.

[0009] Furthermore, due to the diverse data sources and varying shooting angles of road inspection data, road anomalies are distributed across various sizes. Shallow image features contain more detailed information such as color, texture, edges, and corners, while deep image features contain more abstract semantic information. Shallow features have a smaller receptive field, focusing more on detailed information and are more suitable for detecting small targets, while deep abstract features have a larger receptive field and are more suitable for detecting large targets. Traditional feature fusion methods cannot effectively integrate feature information at each scale, potentially leading to the loss of detailed information on large targets or the loss of information on small targets. Summary of the Invention

[0010] To address the aforementioned issues, this invention proposes a method and system for detecting abnormal road conditions. It proposes an anomaly detection model and, based on this model, improves the classification performance of the anomaly detection model by adaptively enabling a fine-grained classification module. This solves the problems of diverse types and sizes of anomalies in daily inspections, making them difficult to detect effectively, as well as the inability to effectively distinguish the materials of tiny foreign objects during key inspections.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a method for detecting abnormal road conditions, comprising:

[0013] Acquire the road image to be processed;

[0014] An anomaly detection model is used to detect the road image to be processed, and the initial detection target is obtained.

[0015] The anomaly detection model includes: extracting multi-scale feature maps from the road image to be processed; using a deformable convolutional residual module to extract features from the last layer feature map; concatenating the extracted features with the original feature map and processing them with coordinate attention to obtain the maximum scale feature map; performing adaptive spatial feature fusion on the intermediate scale feature map and the maximum scale feature map; and then using this to perform target detection.

[0016] The initial target image is cropped from the road image to be processed, and the initial target image is classified using a fine-grained classification model to obtain the final anomaly detection target;

[0017] The fine-grained classification model includes: enhancing the initial target image under attention guidance and extracting the enhanced feature map and attention map; classifying the enhanced feature map and attention map after fusion; attention guidance involves obtaining the key region image by attention cropping the attention map and obtaining the discarded region image by attention discarding the attention map, and guiding the enhancement operation based on the key region image and the discarded region image.

[0018] As an alternative implementation, the process of obtaining the maximum scale feature map includes: the deformable convolutional residual module is composed of two deformable convolutional residuals connected together; after the last layer feature map is processed by n deformable convolutional residual modules for feature extraction, it is concatenated with the original feature map to obtain a complete feature map; and the complete feature map is processed by coordinate attention to obtain the maximum scale feature map.

[0019] As an alternative implementation, the process of performing adaptive spatial feature fusion on intermediate-scale feature maps and maximum-scale feature maps includes:

[0020] The multi-scale feature maps extracted from the road image to be processed include feature map C1, feature map C2, feature map C3, feature map C4, and feature map C5;

[0021] For the intermediate-scale feature maps C3 and C4, feature scaling is performed through linear interpolation upsampling and convolution downsampling. After scaling to a uniform size, an adaptive spatial feature fusion operation is then performed.

[0022] The final fusion result used for object detection is obtained by performing feature scaling and adaptive spatial feature fusion operations on the fusion result of the maximum scale feature map with feature map C3 and feature map C4.

[0023] As an alternative implementation, the attention cropping operation includes: normalizing the attention map to between 0 and 1, selecting the attention map with a high attention level exceeding a set threshold, cropping the corresponding part of the initial detection target image under the guidance of the selected attention map, and scaling it to the same size as the initial detection target image, thereby obtaining the key region image.

[0024] As an alternative implementation, the attention discarding operation includes: normalizing the attention map to between 0 and 1, selecting the attention map with a high attention level exceeding a set threshold, covering the corresponding part of the initial detection target image with a mask under the guidance of the selected attention map, discarding the feature information of the corresponding region, thereby obtaining the discarded region image.

[0025] As an alternative implementation, multi-scale feature maps are extracted from the enhanced initial detection target image, including feature map C1, feature map C2, feature map C3, feature map C4 and feature map C5. After feature map C2, feature map C3, feature map C4 and feature map C5 are processed by a feature pyramid structure, feature map P2, feature map P3, feature map P4 and feature map P5 are obtained. Each layer of feature map is then concatenated by a global average pooling operation to obtain the enhanced feature map.

[0026] As an alternative implementation, when extracting multi-scale feature maps, a coordinate attention mechanism is added to the last layer to aggregate global spatial features and channel features.

[0027] Secondly, the present invention provides a road anomaly detection system, comprising:

[0028] The data acquisition module is configured to acquire images of the road to be processed;

[0029] The target detection module is configured to use an anomaly detection model to detect the road image to be processed, and obtain the initial detection targets;

[0030] The anomaly detection model includes: extracting multi-scale feature maps from the road image to be processed; using a deformable convolutional residual module to extract features from the last layer feature map; concatenating the extracted features with the original feature map and processing them with coordinate attention to obtain the maximum scale feature map; performing adaptive spatial feature fusion on the intermediate scale feature map and the maximum scale feature map; and then using this to perform target detection.

[0031] The fine-grained classification module is configured to crop out the initial detection target image from the road image to be processed, and classify the initial detection target image using a fine-grained classification model to obtain the final anomaly detection target;

[0032] The fine-grained classification model includes: enhancing the initial target image under attention guidance and extracting the enhanced feature map and attention map; classifying the enhanced feature map and attention map after fusion; attention guidance involves obtaining the key region image by attention cropping the attention map and obtaining the discarded region image by attention discarding the attention map, and guiding the enhancement operation based on the key region image and the discarded region image.

[0033] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0034] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0035] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] To address the challenge of effectively detecting diverse and varying sizes of anomalies during routine inspections, this invention proposes a Road Anomaly Detection Network (RADNet). First, a Deformable Feature Extraction Module (DFE) is introduced to enhance the model's feature extraction capabilities for irregular targets. Second, Coordinate Attention (CA) is introduced into the backbone network to improve the model's feature learning ability. Finally, a Directly Connected Feature Pyramid Network (DCFPN) is proposed to optimize the model's multi-scale feature fusion performance through cross-layer feature adaptive fusion, enabling the model to capture all effective information.

[0038] The Directly Connected Feature Pyramid Network (DCFPN) reduces the semantic gap between non-adjacent layers by directly fusing feature layers. This allows deep features to be fused with shallow detailed information, and shallow features to be fused with deep semantic information, thus greatly enriching the network's information flow. Furthermore, adaptive fusion operations suppress useless information from different layers and enhance the expression of key information, achieving better multi-scale feature fusion results.

[0039] To address the issue of effectively distinguishing the material composition of minute foreign objects during key inspections, this invention proposes the RADNet-FG algorithm based on the RADNet algorithm. This algorithm improves RADNet's classification performance by adaptively enabling the Fine Grained Module (FG). First, a data augmentation module is constructed to enhance the original image under attention-guided processing through attention cropping and attention discarding. Second, a feature learning module is constructed to extract feature maps using the ResNet-FPN-CA backbone network. Finally, a bilinear attention pooling mechanism is used to fuse the feature maps and attention maps to generate fused features. Fine-grained classification is then performed based on these fused features, enabling the detection of road anomalies during both routine and key inspections.

[0040] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0042] Figure 1This is a diagram of the RADNet structure provided in Embodiment 1 of the present invention;

[0043] Figure 2 This is a structural diagram of the DFE module provided in Embodiment 1 of the present invention;

[0044] Figure 3 This is a structural diagram of the CA module provided in Embodiment 1 of the present invention;

[0045] Figure 4 This is a DCFPN structure diagram provided in Embodiment 1 of the present invention;

[0046] Figure 5 This is a diagram of the RADNet-FG structure provided in Embodiment 1 of the present invention;

[0047] Figure 6 This is a diagram of the fine-grained classification model architecture provided in Embodiment 1 of the present invention;

[0048] Figure 7 This is a diagram illustrating the attention-guided data augmentation process provided in Embodiment 1 of the present invention.

[0049] Figure 8 This is a diagram of the ResNet-50 network structure provided in Embodiment 1 of the present invention;

[0050] Figure 9 This is a diagram of the FPN-ResNet-50 network structure provided in Embodiment 1 of the present invention;

[0051] Figure 10 This is a diagram of the FPN-ResNet-50 network structure with an enhanced attention mechanism provided in Embodiment 1 of the present invention.

[0052] Figures 11(a)-(c) show a comparison of the detection results of RADNet and YOLOv8 provided in Embodiment 1 of the present invention;

[0053] Figures 12(a)-(c) show a comparison of the detection results of RADNet and YOLOv8 provided in Embodiment 1 of the present invention.

[0054] Figures 13(a)-(c) show a comparison of the detection results of RADNet and YOLOv8 provided in Embodiment 1 of the present invention.

[0055] Figures 14(a)-(c) show a comparison of the detection results of RADNet-FG and RADNet provided in Embodiment 1 of the present invention;

[0056] Figures 15(a)-(c) show a comparison of the detection results of RADNet-FG and RADNet provided in Embodiment 1 of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0060] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0061] Example 1

[0062] This embodiment provides a method for detecting abnormal road conditions, including:

[0063] Acquire the road image to be processed;

[0064] An anomaly detection model is used to detect the road image to be processed, and the initial detection target is obtained.

[0065] The anomaly detection model includes: extracting multi-scale feature maps from the road image to be processed; using a deformable convolutional residual module to extract features from the last layer feature map; concatenating the extracted features with the original feature map and processing them with coordinate attention to obtain the maximum scale feature map; performing adaptive spatial feature fusion on the intermediate scale feature map and the maximum scale feature map; and then using this to perform target detection.

[0066] The initial target image is cropped from the road image to be processed, and the initial target image is classified using a fine-grained classification model to obtain the final anomaly detection target;

[0067] The fine-grained classification model includes: enhancing the initial target image under attention guidance and extracting the enhanced feature map and attention map; classifying the enhanced feature map and attention map after fusion; attention guidance involves obtaining the key region image by attention cropping the attention map and obtaining the discarded region image by attention discarding the attention map, and guiding the enhancement operation based on the key region image and the discarded region image.

[0068] The original YOLOv8s model cannot achieve good detection results in road inspection tasks because there are three main challenges in detecting road anomalies:

[0069] (1) The shapes of road anomaly targets are irregular, and the aspect ratios of different types of anomalies vary greatly, so it is necessary to accurately extract the features of road anomaly targets.

[0070] (2) There is no fixed scene for road inspection. All images are captured dynamically. The background of the images is complex and varied, with a lot of noise interference. It is necessary to accurately detect abnormal road conditions in complex backgrounds.

[0071] (3) The data sources for road inspection include various methods such as inspection vehicle photography, drone photography, surveillance photography, and manual photography. The images taken by different methods have different perspectives and sizes, and it is necessary to detect road anomalies at different scales.

[0072] To address the above issues and considering both the accuracy and real-time performance of road inspection, this embodiment uses YOLOv8s as the baseline network and improves upon it to propose the Road Anomaly Detection Network (RADNet). Figure 1 As shown, firstly, a Deformable Feature Extraction Module (DFE) is proposed to enhance the model's feature extraction capability; secondly, a Coordinate Attention (CA) module is introduced into the backbone network to improve the model's feature learning capability; finally, a Directly Connected Feature Pyramid Network (DCFPN) is proposed to enhance the model's multi-scale feature fusion capability.

[0073] The three improvements will be described in detail below.

[0074] (1) Deformable Feature Extraction Module (DFE).

[0075] like Figure 2 As shown, the DFE module can adaptively adjust the receptive field of the model to better fit the target region, effectively enhancing the feature extraction capability for irregular targets.

[0076] Specifically, the input feature map is first divided into two gradient flows by a regular convolution. The upper part is processed by n deformable convolutional residual modules (DCN Blocks) for deep feature extraction, while the lower part retains the original feature information. Finally, the deep features extracted by the n deformable convolutional residual modules (DCN Blocks) are concatenated with the original feature map to obtain a complete feature map. At this point, the feature map contains rich contextual information.

[0077] The deformable convolutional residual module (DCNBlock) is the core component of the DFE module, consisting of two deformable convolutional DCNv2 residual connections. Deformable convolution networks (DCNs) add an offset to each sampling point of the convolution kernel, allowing the kernel shape to automatically adjust to fit different target shapes instead of a fixed rectangle. In practical applications, the region extracted by deformable convolutions may exceed the target area, introducing irrelevant noise. To address this, DCNv2 adds weight coefficients to each sampling point, focusing only on useful information and removing noise interference.

[0078] (2) Coordinate Attention Mechanism (CA)

[0079] Road inspections often involve complex and varied backgrounds, requiring the overcoming of background noise to accurately detect road anomalies. To improve the feature learning ability of the detection model, this embodiment introduces Coordinate Attention (CA) into the backbone network. By weighting features in both spatial and channel dimensions, the model focuses more on regions of interest, extracting useful information and suppressing useless information.

[0080] like Figure 3 As shown, Coordinate Attention (CA) is a hybrid attention mechanism. First, through two one-dimensional feature encoding processes, spatial features are aggregated along the horizontal and vertical directions of the complete feature map output by the DFE module to capture long-range dependencies and precise location information. Then, the spatial information is embedded into channel attention to extract key information in both channel and location dimensions, thereby obtaining the maximum scale feature map.

[0081] (3) Directly Connected Feature Pyramid Network (DCFPN)

[0082] Road inspection data comes from diverse sources and is captured from various angles, with road anomalies distributed across a wide range of sizes. Shallow image features contain more detailed information such as color, texture, edges, and corners, while deep image features contain more abstract semantic information. Shallow features have a smaller receptive field, focusing more on detailed information and are better suited for detecting small targets. Deep, abstract features, on the other hand, have a larger receptive field and are better suited for detecting large targets. Traditional feature fusion methods cannot effectively integrate feature information at each scale, potentially leading to the loss of detailed information on large targets or the loss of information on small targets.

[0083] To enhance the multi-scale feature fusion capability of the detection model, this embodiment proposes a Directly Connected Feature Pyramid Network (DCFPN), which enables the model to capture all effective information by directly fusing multi-scale features from the backbone network.

[0084] Directly Connected Feature Pyramid Network (DCFPN) structure as follows Figure 4 As shown, the feature information of each layer of the neck network is directly fused through the adaptive spatial feature fusion operation ASFF. The feature fusion operation is divided into feature scaling and adaptive fusion.

[0085] Specifically:

[0086] In the backbone network, multi-scale feature maps are extracted from the input road image to be processed, including feature map C1, feature map C2, feature map C3, feature map C4 and feature map C5. The last feature map C5 is processed by the DFE module and coordinate attention CA to obtain the maximum scale feature map.

[0087] Then, feature scaling is performed on the intermediate-scale feature maps C3 and C4 extracted from the backbone network. The feature maps from different layers are scaled to a uniform size through linear interpolation upsampling and convolution downsampling, and then fed into the ASFF module for adaptive fusion.

[0088] The ASFF adaptive fusion process is shown in Equation (1). By spatially weighting the features on different feature layers to filter out interference information, the spatial dimension adaptive feature fusion is achieved, which suppresses useless information from different feature layers and enhances the expression of useful information.

[0089]

[0090] in, This represents the feature vector of the nth layer feature map and the lth layer feature map at position (i, j). and β ij l For the trained spatial weight coefficients, This is the result of the fusion.

[0091] Finally, the ASFF adaptive fusion process is repeated on the maximum scale feature map and the result after fusing feature map C3 and feature map C4 to obtain the final fusion result, which is then used for target detection to obtain the initial detection target.

[0092] Compared to FPN, DCFPN reduces the semantic gap between non-adjacent layers by directly fusing feature layers. This allows deep features to be fused with shallow detailed information, and shallow features to be fused with deep semantic information, thus greatly enriching the network's information flow. Furthermore, DCFPN suppresses useless information from different layers through ASFF adaptive fusion, enhancing the expression of key information and achieving better multi-scale feature fusion results.

[0093] Because some road sections contain tiny foreign objects such as screws and rubber rods, which seriously threaten road safety, regular focused inspections are necessary to detect and classify these small foreign objects. General object detection algorithms primarily focus on coarse-grained identification of medium to large-sized targets. However, relying solely on general object detection algorithms is insufficient to solve the above problems. Therefore, this embodiment proposes a fine-grained classification model FG (Fine Grained Module). If the initial detected target obtained by the anomaly detection model is lower than the confidence level, the fine-grained classification model FG is continued for classification. This enhances the fine-grained classification capability based on the excellent localization capability of the RADNet algorithm, forming the enhanced algorithm RADNet-FG. Figure 5 As shown, where, Figure 5 The RADNet network in the example Figure 1 As shown, the FG module is as follows Figure 6 As shown.

[0094] In RADNet-FG, firstly, a data augmentation module for the FG model is constructed, which enhances the original image under attention guidance through attention cropping and attention dropout; secondly, a feature learning module for the FG model is constructed, which extracts augmented feature maps through the backbone network ResNet-FPN-CA; finally, the augmented feature maps and attention maps are fused through a bilinear attention pooling mechanism to generate fused features, and fine-grained classification is performed based on the fused features.

[0095] Fine-grained classification tasks often involve high visual similarity between different categories of objects, making the creation of fine-grained classification datasets more time-consuming compared to regular classification tasks. Therefore, extracting more feature information from limited data is crucial for improving fine-grained classification performance. This embodiment proposes a fine-grained classification model, FG, which can expand the original dataset through attention-guided mapping, such as... Figure 6As shown, it consists of two parts: a data augmentation module and a feature learning module. These two parts will be described in detail below.

[0096] (1) Data Enhancement Module.

[0097] Conventional data augmentation methods may introduce uncontrollable noise interference and are not suitable for fine-grained classification tasks. This embodiment addresses this issue by introducing an attention-guided data augmentation module, which can selectively enhance the detailed features of the target.

[0098] Figure 7 This paper demonstrates the attention-guided data augmentation process. The initial target image is first fed into the backbone network for feature extraction, resulting in a feature map of the target. Then, a 1×1 convolution and batch normalization operation are performed on the feature map to obtain M dimensionality-reduced attention maps (M is set to 32 by default), where each attention map corresponds to a detailed feature of the target. However, the resulting attention maps still contain a lot of invalid and interfering information; therefore, further attention augmentation operations are needed.

[0099] This embodiment primarily employs two attention enhancement methods: attention cropping and attention discarding. To amplify important details, the generated attention map is first cropped. The cropping process involves: first, normalizing the attention map to a range of 0-1; then, selecting attention maps with high attention levels according to a set threshold; and finally, cropping the corresponding portion of the original initial target image under the guidance of the selected attention map, scaling it to the same size as the original initial target image. The resulting image of the key region is then used as augmented data for training.

[0100] However, performing attention cropping alone will cause the network to overemphasize information in key regions and neglect features in other regions. Therefore, attention discarding is performed after attention cropping to enhance the network's attention to non-key regions. The attention discarding process is as follows: First, the attention map is normalized to the range of 0-1. Then, attention maps for key regions are selected according to a set threshold. Finally, guided by the selected attention map, the corresponding parts of the original initial detection target image are masked, and the feature information of the corresponding regions is discarded. The resulting discarded region image is used as augmented data for training.

[0101] (2) Feature learning module.

[0102] The feature learning module mainly consists of a backbone network, responsible for extracting image features. The quality of the backbone network directly affects the performance of the final classification model. Therefore, this embodiment selects the high-performance ResNet-50 as the backbone network and improves upon it by adding a CA attention mechanism and an FPN structure, making it more suitable for fine-grained classification tasks of small targets.

[0103] With the development of deep learning, the number of network layers has gradually increased. However, excessively deep network structures can lead to the vanishing stochastic gradient problem. To address this issue, residual units were developed. Residual units add cross-layer connections to the deep network structure, allowing inputs to be directly passed across layers and added to the results of convolutions.

[0104] ResNet-50 is a deep residual network built on residual units, consisting of 49 convolutional layers and one fully connected layer. The network structure is as follows: Figure 8 As shown in the figure, the convolutional layers can be divided into 5 groups, namely C1, C2, C3, C4, and C5. The first group C1 consists of 7×7 convolutions, batch normalization (BN) modules, ReLU activation functions, and max pooling modules. The second, third, fourth, and fifth parts have similar structures, consisting of residual units, 1×1 convolutions, 3×3 convolutions, and max pooling modules.

[0105] Therefore, using ResNet-50 as the backbone network, multi-scale feature maps of the enhanced initial target image are extracted, including feature map C1, feature map C2, feature map C3, feature map C4 and feature map C5;

[0106] Convolutional layers of different depths extract different feature information. Deeper features contain more semantic information, while shallower features contain more localization information. If only deep features are used, shallow detailed features will be ignored. At the same time, multiple convolution downsampling will continuously reduce the image resolution and cause the loss of feature information.

[0107] To effectively utilize deep semantic information and shallow localization information and address the feature information attenuation problem caused by multiple downsampling in convolutional neural networks, this embodiment adds a Feature Pyramid Network (FPN) structure to the ResNet-50 backbone network, forming an FPN-ResNet-50 backbone network structure. The improved backbone network is as follows: Figure 9As shown, the feature maps C2, C3, C4, and C5 extracted by the backbone network ResNet-50 are fused into four feature maps P2, P3, P4, and P5 through an FPN structure. Each feature map is then concatenated using global average pooling. The resulting enhanced feature map is used by the bilinear attention pooling module and the data augmentation module. Fine-grained classification is then performed on the fused enhanced feature map and attention map.

[0108] Meanwhile, to further enhance the feature extraction capability of the backbone network, this embodiment adds a coordinate attention module (CA) to the last layer of the backbone network. For example... Figure 10 As shown, the CA module can aggregate global spatial features and channel features, enabling the generated feature map C5 to comprehensively consider global detailed information.

[0109] The method described in this embodiment can detect road anomalies during routine and key inspections. Experiments on a large dataset show that the method described in this embodiment achieves excellent results.

[0110] 1. RADNet algorithm.

[0111] (1) Ablation test results.

[0112] Table 1 shows the statistical results of the ablation experiments, where √ indicates that relevant methods were included in the model.

[0113] Table 1 Ablation Experiment Results

[0114]

[0115] As can be seen, the Deformable Feature Extraction (DFE) module has strong feature extraction capabilities, improving detection performance. After introducing the DFE module, the model's precision, recall, and mAP improved by 1.8%, 0.6%, and 0.6% respectively compared to the baseline network YOLOv8. The Coordinate Attention (CA) module enhances the model's focus on foreground targets and suppresses background noise interference. After introducing the CA module, the model's precision, recall, and mAP improved by 1.7%, 1.5%, and 1.8% respectively compared to the baseline network YOLOv8. The Directly Connected Feature Pyramid Network (DCFPN) enhances the model's feature fusion capabilities. After introducing the DCFPN network, the model's precision, recall, and mAP improved by 0.2%, 2.6%, and 0.2% respectively compared to the baseline network YOLOv8. The Focaler-IOU loss function can improve the problem of uneven sample distribution. After introducing the Focaler-IOU loss function, the model precision, recall and mAP are improved by 1.4%, 1.4% and 1.6% respectively compared with the baseline network YOLOv8.

[0116] (2) Visualize the experimental results.

[0117] The detection performance of the RADNet algorithm and the baseline YOLOv8 algorithm is visually compared to demonstrate the superiority of the improved algorithm. The comparison results are shown in Figures 11-13, where (a) is the original image, (b) is the detection result of YOLOv8, and (c) is the detection result of RADNet. As shown in Figures 11(a)-(c), YOLOv8 misdetected the road markings in the background as road debris, while RADNet did not misdetect any. This is because the coordinate attention module in RADNet suppresses noise information, enhances the model's feature learning ability, and improves the false detection phenomenon. As shown in Figures 12(a)-(c), YOLOv8 failed to detect the banner target and the road debris target, while RADNet correctly detected them. This is because the direct-connected feature pyramid module in RADNet enhances the model's multi-scale information fusion ability, strengthens the feature representation of target information, and improves the false detection phenomenon. As shown in Figures 13(a)-(c), compared with YOLOv8, the target bounding boxes located by RADNet fit the target shape better. The reason for this improvement is that the deformable feature extraction module in RADNet adjusts the distribution of convolutional kernels, enhancing the model's ability to extract features from targets of specific shapes and improving localization accuracy. In summary, the combined application of these three improvements gives RADNet strong feature extraction and fusion capabilities, enabling it to accurately detect abnormal road conditions.

[0118] 2. RADNet-FG algorithm.

[0119] (1) Comparison of experimental results.

[0120] Table 2 shows the experimental results of the RADNet-FG algorithm, RADNet algorithm, and YOLOv8 algorithm on the TPFOD road surface image dataset. It can be seen that after introducing fine-grained classification processing, the detection capability of RADNet-FG for small road debris is significantly improved. The model accuracy reaches 95.6%, recall reaches 48.3%, and mAP reaches 90.1%, which are improvements of 5.0%, 3.5%, and 15.4% respectively compared to RADNet without fine-grained classification processing, and improvements of 10.6%, 2.2%, and 21.3% compared to YOLOv8.

[0121] Table 2. Experimental results comparing RADNet-FG and RADNet

[0122]

[0123] (2) Visualize the experimental results.

[0124] Figures 14-15 illustrate the material detection performance of the RADNet-FG algorithm and the RADNet algorithm for small foreign objects on the road. (a) is the original image, (b) is the RADNet detection result, and (c) is the RADNet-FG detection result. As shown in Figures 14(a)-(c), the RADNet algorithm misdetects metal as rubber, while the RADNet-FG algorithm can accurately classify the material of small foreign objects. As shown in Figures 15(a)-(c), the RADNet algorithm misdetects potholes as rubber, which will trigger false alarms, while the RADNet-FG algorithm can correctly distinguish between foreign objects and potholes, reducing false alarms.

[0125] Example 2

[0126] This embodiment provides a road anomaly detection system, including:

[0127] The data acquisition module is configured to acquire images of the road to be processed;

[0128] The target detection module is configured to use an anomaly detection model to detect the road image to be processed, and obtain the initial detection targets;

[0129] The anomaly detection model includes: extracting multi-scale feature maps from the road image to be processed; using a deformable convolutional residual module to extract features from the last layer feature map; concatenating the extracted features with the original feature map and processing them with coordinate attention to obtain the maximum scale feature map; performing adaptive spatial feature fusion on the intermediate scale feature map and the maximum scale feature map; and then using this to perform target detection.

[0130] The fine-grained classification module is configured to crop out the initial detection target image from the road image to be processed, and classify the initial detection target image using a fine-grained classification model to obtain the final anomaly detection target;

[0131] The fine-grained classification model includes: enhancing the initial target image under attention guidance and extracting the enhanced feature map and attention map; classifying the enhanced feature map and attention map after fusion; attention guidance involves obtaining the key region image by attention cropping the attention map and obtaining the discarded region image by attention discarding the attention map, and guiding the enhancement operation based on the key region image and the discarded region image.

[0132] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0133] In further embodiments, the following is also provided:

[0134] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0135] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0136] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0137] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0138] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0139] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0140] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0141] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal road conditions, characterized in that, include: Acquire the road image to be processed; An anomaly detection model is used to detect the road image to be processed, and the initial detection target is obtained. The anomaly detection model includes: extracting multi-scale feature maps from the road image to be processed; using a deformable convolutional residual module to extract features from the last layer feature map; concatenating the extracted features with the original feature map and performing coordinate attention processing to obtain the maximum-scale feature map; performing an adaptive spatial feature fusion operation on the intermediate-scale feature map and the maximum-scale feature map; and then performing target detection based on the fused feature map. The process of performing the adaptive spatial feature fusion operation on the intermediate-scale feature map and the maximum-scale feature map includes: The multi-scale feature maps extracted from the road image to be processed include feature map C1, feature map C2, feature map C3, feature map C4, and feature map C5; For the intermediate-scale feature maps C3 and C4, feature scaling is performed through linear interpolation upsampling and convolution downsampling. After scaling to a uniform size, an adaptive spatial feature fusion operation is then performed. The fusion result of the maximum scale feature map, feature map C3, and feature map C4 is subjected to feature scaling and adaptive spatial feature fusion operations to obtain the final fusion result used for target detection. The initial target image is cropped from the road image to be processed, and the initial target image is classified using a fine-grained classification model to obtain the final anomaly detection target; The fine-grained classification model includes: enhancing the initial target image under attention guidance and extracting the enhanced feature map and attention map; classifying the enhanced feature map and attention map after fusion; attention guidance involves obtaining the key region image by attention cropping the attention map and obtaining the discarded region image by attention discarding the attention map, and guiding the enhancement operation based on the key region image and the discarded region image.

2. The method for detecting abnormal road conditions as described in claim 1, characterized in that, The process of obtaining the maximum scale feature map includes: the deformable convolutional residual module is composed of two deformable convolutional residuals connected together; after the last layer feature map is processed by n deformable convolutional residual modules for feature extraction, it is concatenated with the original feature map to obtain the complete feature map; and the complete feature map is processed by coordinate attention to obtain the maximum scale feature map.

3. The method for detecting abnormal road conditions as described in claim 1, characterized in that, The attention cropping operation includes: normalizing each attention map to between 0 and 1, selecting the attention map with an attention score higher than a set threshold from multiple attention maps, cropping the corresponding part of the initial detection target image under the guidance of the selected attention map, and scaling it to the same size as the initial detection target image, thereby obtaining the key region image; The attention discarding operation includes: normalizing each attention map to the range of 0-1, selecting the attention map with an attention score higher than a set threshold from multiple attention maps, and covering the corresponding part of the initial detection target image with a mask under the guidance of the selected attention map, discarding the feature information of the corresponding region, thereby obtaining the discarded region image.

4. The method for detecting abnormal road conditions as described in claim 1, characterized in that, Multi-scale feature maps are extracted from the enhanced initial target image, including feature map C1, feature map C2, feature map C3, feature map C4 and feature map C5. After feature map C2, feature map C3, feature map C4 and feature map C5 are processed by feature pyramid structure, feature map P2, feature map P3, feature map P4 and feature map P5 are obtained. Each layer of feature map is then concatenated by global average pooling to obtain the enhanced feature map.

5. The method for detecting abnormal road conditions as described in claim 4, characterized in that, When extracting multi-scale feature maps, a coordinate attention mechanism is added to the last layer to aggregate global spatial features and channel features.

6. A road anomaly detection system employing the road anomaly detection method as described in any one of claims 1-5, characterized in that, include: The data acquisition module is configured to acquire images of the road to be processed; The target detection module is configured to use an anomaly detection model to detect the road image to be processed, and obtain the initial detection targets; The anomaly detection model includes: extracting multi-scale feature maps from the road image to be processed; using a deformable convolutional residual module to extract features from the last layer feature map; concatenating the extracted features with the original feature map and processing them with coordinate attention to obtain the maximum scale feature map; performing an adaptive spatial feature fusion operation on the intermediate scale feature map and the maximum scale feature map; and then performing target detection based on the feature map after the fusion operation. The fine-grained classification module is configured to crop out the initial detection target image from the road image to be processed, and classify the initial detection target image using a fine-grained classification model to obtain the final anomaly detection target; The fine-grained classification model includes: enhancing the initial target image under attention guidance and extracting the enhanced feature map and attention map; classifying the enhanced feature map and attention map after fusion; attention guidance involves obtaining the key region image by attention cropping the attention map and obtaining the discarded region image by attention discarding the attention map, and guiding the enhancement operation based on the key region image and the discarded region image.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-5.

Citation Information

Patent Citations

  • Target detection method for large-size aerial remote sensing image

    CN113989645A

  • Fruit detection method and system in complex orchard environment

    CN114494151A