Pothole detection method and device based on deep learning, electronic equipment and program product

Abstract

The application discloses a kind of road surface pit detection method, device, electronic equipment and program product based on deep learning.The detection method is realized based on the road surface pit detection model trained.The model introduces MSEA module in neck network, can model multiscale space structure and channel dependence relationship jointly, improve the robustness of model to real pit discrimination.At the same time, LG- FEAF module is set in backbone network, can construct efficient and robust cross-scale feature modeling mechanism, enhance the detection ability of model to multiscale and boundary fuzzy pit.In addition, detection network also uses MFAFDet, in view of the problem that traditional detection network is insufficient in multiscale feature fusion and feature alignment capability, multiscale feature extraction, channel attention, spatial attention and residual connection are ingeniously fused to improve the identification ability to road surface pit.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a road pit detection method based on deep learning, a road pit detection device, an electronic device and a computer program product. BACKGROUND

[0002] With the rapid construction of urban transportation infrastructure in China and the improvement of digital level of road maintenance and management, intelligent road surface state monitoring technology plays an increasingly important role in smart transportation and urban fine management. As a typical and high-risk road disease, road pit not only affects the smoothness and safety of vehicle traffic, but also accelerates the damage of road structure, and therefore becomes a key monitoring object of highway maintenance and safety supervision.

[0003] In recent years, artificial intelligence technologies such as deep learning have been gradually applied to the automatic identification and detection of road pits, providing a new technical approach to improve road inspection efficiency. However, the multi-source interference factors in complex urban road environment still have a significant impact on the detection results. In particular, non-disease features such as oil stains, water reflection, tire marks, shadows and road debris are often visually similar to pits, which can easily cause model misjudgment; at the same time, complex lighting conditions such as rainy days and night further weaken the image difference, making the false alarm and missed detection problems more prominent. SUMMARY

[0004] The application provides a road pit detection method based on deep learning, a road pit detection device, an electronic device and a computer program product, which can effectively deal with multi-element interference in complex background, and further improve the robustness of the detection model for road pit detection.

[0005] In a first aspect, the application provides a road pit detection method based on deep learning, comprising:

[0006] extracting features of a to-be-detected image based on a backbone network of a pre-trained road pit detection model, to obtain image features; the to-be-detected image comprises a road;

[0007] fusing the image features based on a neck network of the road pit detection model, to obtain target fusion features;

[0008] detecting the target fusion features based on a detection network of the road pit detection model, to obtain a detection result of road pits in the to-be-detected image;

[0009] The neck network is provided with an MSEA module, and the MSEA module comprises a multi-scale convolution structure, a CAB branch, an SAB branch, a hybrid weighting layer and a first splicing layer; for a first input feature input into the MSEA module:

[0010] The multi-scale feature extraction and adaptive fusion are performed on the first input feature through a multi-scale convolution structure to obtain first multi-scale fusion features;

[0011] The first multi-scale fusion features are processed through the CAB branch to obtain corresponding target channel weights;

[0012] The first multi-scale fusion features are processed through the SAB branch to obtain corresponding target spatial weights;

[0013] The multi-scale fusion is mixed and weighted through the target channel weights and the target spatial weights in the mixed weighting layer to obtain mixed enhancement features;

[0014] The mixed enhancement features are spliced with the first input features in the first splicing layer to obtain first output features corresponding to the first input features.

[0015] In a second aspect, the present application provides a road pit detection device, comprising:

[0016] An extraction module is configured to extract features of a to-be-detected image based on a backbone network of a pre-trained road pit detection model to obtain image features, wherein the to-be-detected image comprises a road;

[0017] A fusion module is configured to fuse the image features based on a neck network of the road pit detection model to obtain target fusion features;

[0018] A detection module is configured to detect the target fusion features based on a detection network of the road pit detection model to obtain a detection result of a road pit in the to-be-detected image;

[0019] The neck network is provided with an MSEA module, and the MSEA module comprises a multi-scale convolution structure, a CAB branch, an SAB branch, a mixed weighting layer, and a first splicing layer. For first input features input into the MSEA module, the fusion module comprises a fusion unit, and the fusion unit is configured to:

[0020] The multi-scale feature extraction and adaptive fusion are performed on the first input feature through a multi-scale convolution structure to obtain first multi-scale fusion features;

[0021] The first multi-scale fusion features are processed through the CAB branch to obtain corresponding target channel weights;

[0022] The first multi-scale fusion features are processed through the SAB branch to obtain corresponding target spatial weights;

[0023] The multi-scale fusion is mixed and weighted through the target channel weights and the target spatial weights in the mixed weighting layer to obtain mixed enhancement features;

[0024] The mixed enhanced feature is spliced with the first input feature in the first splicing layer to obtain a first output feature corresponding to the first input feature.

[0025] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0026] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program implements the steps of the method of the first aspect when executed by a processor.

[0027] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements the steps of the method of the first aspect when executed by one or more processors.

[0028] The first aspect of the present application has the beneficial effects compared with the prior art: the present application realizes detection based on a pre-trained road pit detection model. First, the backbone network extracts deep features of the input image, captures multi-level representations of the road area and potential pit targets; then, the neck network effectively fuses features of different levels to form fused features containing rich context information; finally, the detection network analyzes the fused features and outputs the detection results, completing pit positioning and identification.

[0029] To improve the detection ability of the model in complex backgrounds and reduce the possibility of missed detection or false detection, a new multi-scale enhanced attention (MSEA) module is introduced into the model. The module includes a multi-scale convolution structure, a CAB branch, a SAB branch, a hybrid weighting layer and a splicing layer. The multi-scale convolution structure performs multi-scale feature extraction and adaptive fusion to generate first multi-scale fusion features, realizing sufficient capture of pit details of different scales. The first multi-scale fusion features are respectively input into the CAB and SAB branches to learn feature dependency in the channel dimension and mine key region responses in the spatial dimension, thereby obtaining target channel weights and spatial weights, so that the model can accurately focus on pit key features under multi-scale, boundary blur and environmental interference.

[0030] Subsequently, the mixed weighting layer acts the above-mentioned channel and spatial weight on the first multi-scale fusion feature, further strengthens the discriminative ability of the feature, significantly improves the sensitivity to the real pit and groove in the presence of noise interference (such as oil stains, water reflection, etc.), and reduces the risk of false detection and missed detection. Finally, the weighting result is integrated through the splicing layer to form a more comprehensive and detailed feature representation, which provides a more robust input for subsequent detection, so that the model performs better in complex road environment.

[0031] It can be understood that the beneficial effects of the above-mentioned second aspect to the fifth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0033] Figure 1 is a network structure schematic diagram of the road pit detection model provided by the embodiments of the present application;

[0034] Figure 2 is a flowchart of the road pit detection method based on deep learning provided by the embodiments of the present application;

[0035] Figure 3 is a network structure schematic diagram of the MSEA module provided by the embodiments of the present application;

[0036] Figure 4 is a network structure schematic diagram of the LG-FEAF module provided by the embodiments of the present application;

[0037] Figure 5 is a network structure schematic diagram of the MFAFDet provided by the embodiments of the present application;

[0038] Figure 6 is a structure schematic diagram of the road pit detection device provided by the embodiments of the present application;

[0039] Figure 7 is a structure schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0040] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, technologies, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0041] As a typical and high-risk road disease, road potholes not only affect the smoothness and safety of vehicle traffic, but also accelerate the damage of road structure, so it becomes the key monitoring object of highway maintenance and safety supervision. However, in the complex urban road environment, potholes often coexist with non-disease factors such as oil stains, water reflection, tire marks, shadow blocking, fallen leaves and other non-disease factors. These interferences are highly similar to the actual pothole in color, texture and geometric shape, which can easily cause false positives or missed detection of the detection model. Especially in rainy days, at night or in low light conditions, water reflection and pothole water are almost indistinguishable in visual characteristics, and the dark information of the shadow area is also easy to be misjudged as a concave disease.

[0042] Under the action of these multiple source interferences, the traditional detection method is difficult to effectively distinguish the real disease and the high similarity pseudo feature in the feature level, which leads to the wrong guidance of attention response, and the focusing ability of the model on the key pothole feature is significantly reduced, thereby reducing the robustness and reliability of the overall detection.

[0043] To cope with the feature confusion problem caused by light changes, water reflection, shadow blocking and background clutter in complex road environment, and to improve the robustness and reliability of the model for road pothole detection, the present application proposes a road pothole detection model, which includes a backbone network, a neck network and a detection head. Specifically, the backbone network is used for feature extraction of the input image to obtain image features; the neck network is used for feature fusion of the image features to obtain target fusion features; and the detection head is used for detection of the target fusion features to obtain the detection result of the road pothole in the image to be processed.

[0044] Among them, the MSEA module in the neck network includes a multi-scale convolution structure, a CAB branch, a SAB branch, a hybrid weighting layer and a first splicing layer. The design of this module aims to significantly improve the discrimination robustness of the model for real potholes by jointly modeling the multi-scale spatial structure and channel dependence, so as to effectively cope with multiple source interferences in complex environment.

[0045] Firstly, the multi-scale convolution structure performs multi-scale feature extraction and adaptive fusion on the input features to generate the first multi-scale fusion features. This enables the model to effectively capture the detailed information of the road potholes from different scales, improving the recognition ability of potholes of different sizes, especially in a multi-scale background, it can accurately extract and locate the target features. Based on the accurate perception of potholes of various sizes, the model can effectively improve its performance in diverse environments.

[0046] Then, the generated multi-scale features are passed to the CAB branch and the SAB branch to perform channel learning and spatial learning, respectively. In the CAB branch, the model learns the channel dependency of the input features to generate target channel weights, focusing on the feature channels that are crucial for pothole detection and suppressing irrelevant channel interference, which helps the model reduce background noise and irrelevant information interference. In the SAB branch, the model learns the spatial dependency of the input features to generate target spatial weights, further improving the fine attention to the pothole area, especially when the boundary is fuzzy or the background is complex, it can effectively distinguish real potholes from environmental interference. Through the joint action of the two branches, the model can accurately capture the key spatial and channel features of the potholes, thereby achieving more accurate pothole detection in a variable environment.

[0047] In the hybrid weighting layer, the model applies the target channel weights and spatial weights to the first multi-scale fusion features for weighted processing, further improving the discriminative ability of the features. Weighted operation helps the model maintain high sensitivity to real potholes in complex backgrounds, avoiding false positives or false negatives caused by noise factors such as oil stains, water reflections, etc., enhancing the recognition accuracy and robustness of the model to actual potholes.

[0048] Finally, the first concatenation layer concatenates the weighted features to provide more comprehensive and detailed feature representation for subsequent detection. This process significantly improves the model's recognition ability for multi-scale potholes, especially in environments with fuzzy boundaries or complex backgrounds, greatly enhancing the robustness and accuracy of the detection model in different scenarios, ensuring effective identification of road potholes in various practical application scenarios.

[0049] In some embodiments, the multi-scale convolution structure includes a multi-scale convolution branch, an adaptive learning branch, an adaptive fusion layer, and a first convolution layer. This structure design can effectively deal with pits and grooves of different sizes and improve detection accuracy. Specifically, the multi-scale convolution structure can use multiple depth separable convolutions with different kernel sizes, such as 3x3, 5x5, 7x7, and 9x9 depth separable convolutions, to extract rich feature information at different scales. Different scale kernels can capture the features of small and large pits and grooves, enabling the model to accurately identify diverse pit and groove patterns in complex backgrounds. The multi-scale convolution structure not only includes a multi-scale convolution branch to extract features of different scales, but also effectively fuses features through an adaptive learning branch and an adaptive fusion layer, dynamically adjusts feature weights according to the different characteristics of the input data, and finally integrates these multi-scale information through the first convolution layer, enhancing the robustness and sensitivity to details of the model, especially when dealing with complex backgrounds and multi-scale pits and grooves.

[0050] In some embodiments, the CAB branch includes three channel sub-branches, a weight branch, a second splicing layer, and a first activation function layer, and each channel sub-branch includes a pooling layer, an MLP layer, and a branch weighting layer. This branch combines multiple pooling methods and weight weighting mechanisms, significantly improving the ability to accurately capture road pit and groove features.

[0051] Specifically, the CAB branch includes three channel sub-branches, each of which uses different types of pooling layers, such as global average pooling, global maximum pooling, and channel standardization pooling, to extract global information and local features in different ways, thereby enhancing the model's sensitivity to various features. Each channel sub-branch also shares an MLP layer to further enhance the expression ability of the features through nonlinear transformation. The output after pooling is passed to the branch weighting layer for weighting, where the output of the branch weighting layer is provided by the weight branch, which learns the features output by the MLP layer of each channel sub-branch to obtain corresponding weights to dynamically adjust the importance of different channel features. Finally, the features after weighting are integrated through the second splicing layer and activated through the first activation function layer, ensuring that the network can more accurately capture pit and groove features. This structure can effectively fuse information from different scale and type pools and significantly improve the model's detection ability in complex backgrounds, especially in noisy and complex background environments, where the model can exhibit stronger robustness and accuracy.

[0052] In some embodiments, the SAB branch structure is diverse and flexible, which can effectively deal with the detection task of road potholes in complex environments. Specifically, the SAB branch includes a mixed pooling structure, a second convolutional layer, a spatial separable convolution structure, a third concatenation layer, and a second activation function layer. The mixed pooling structure combines multiple pooling methods, such as global average pooling and max pooling, to extract global and local features from multiple levels, thereby improving the model's perception ability of pothole areas. The second convolutional layer further refines feature extraction and enhances the ability to capture details.

[0053] Particularly noteworthy is the spatial separable convolution structure in the SAB branch, which can model the surface defects of road potholes more finely through separate horizontal and vertical convolution operations. This separable convolution approach not only reduces computational complexity but also allows for accurate modeling of the edges and linear features of potholes, thereby improving the model's ability to recognize different forms of potholes. In complex backgrounds, spatial separable convolution can effectively distinguish between potholes and environmental noise, thereby enhancing the model's robustness and accuracy.

[0054] Finally, based on the third concatenation layer, different features are integrated, and the second activation function layer is used to activate the features, which can further strengthen the model's expression ability, enabling the entire SAB branch to provide accurate pothole detection results in various disturbances and complex environments. Overall, the SAB branch can effectively enhance the model's detection performance in complex road environments through multi-level feature extraction and spatial modeling, especially when dealing with the linear structure and edge features of potholes.

[0055] In addition, road potholes in actual urban road scenes generally exhibit significant multi-scale differences and boundary blur characteristics, posing a significant challenge to intelligent detection algorithms. On the one hand, the scale of pothole targets in images varies greatly, influenced by factors such as vehicle speed, camera installation height, shooting angle, and resolution. The visual size can range from a few pixels of minor peeling to dozens of pixels of large-area collapse. On the other hand, the pothole morphology is complex and lacks regular geometric structure, with edges often exhibiting gradual transitions, irregular breaks, or partial occlusions due to rain erosion, vehicle crushing, or material aging, resulting in very blurred boundaries. In complex environmental conditions such as uneven lighting, shadow coverage, or rainwater reflection, the texture contrast between the pothole area and the normal road surface is further reduced, and the appearance becomes more unstable. These characteristics make traditional detection methods have obvious shortcomings in feature discriminability, spatial positioning accuracy, and deformation robustness, making it difficult to effectively cope with the diversification and complex environment in reality.

[0056] In some embodiments, in order to solve this problem, a local-global feature extraction and adaptive fusion (LG-FEAF) module is introduced in the backbone network of the model. Through the synergistic effect of the local branch and the global branch, the LG-FEAF module can capture the fine texture details and macro context information of the pits and grooves respectively, and improve the adaptability of the model to pits and grooves of different sizes and their complex forms.

[0057] Specifically, the global branch enhances feature extraction from the macro level and the important feature level through multi-scale sub-branches and attention sub-branches. The multi-scale sub-branch includes a multi-scale pooling fusion structure, a flattening layer, an aggregation layer, and a CBS layer. By extracting features at different scales and fusing them, the model can capture the change information of the pits and grooves at different scales, ensuring that the detection can maintain high accuracy when facing large-area pits and grooves or small defects. At the same time, the attention sub-branch helps the model automatically focus on the key areas of the pits and grooves through the query layer, matrix calculation layer, and normalization layer, suppressing irrelevant background interference and further improving the ability to capture details, especially in the case of fuzzy boundaries and complex backgrounds.

[0058] The local branch is responsible for extracting the detailed texture information of the pits and grooves. Through fine local feature learning, it enhances the model's ability to recognize pits and grooves in complex backgrounds, especially when dealing with irregularly shaped pits and grooves with fuzzy boundaries. Through the fourth splicing layer, local features and global context can be effectively fused, allowing the LG-FEAF module to generate more comprehensive and rich feature representations, providing more accurate inputs for subsequent detection, and improving the model's robustness in multi-scale and different environmental conditions. This cross-scale and adaptive feature modeling mechanism enables the LG-FEAF module to perform well in complex environments and pits and grooves with fuzzy boundaries, thereby greatly improving the overall performance and robustness of the model.

[0059] In the task of road pit and groove recognition, although multi-scale fusion structures such as FPN and PAN are widely used, their fusion mechanisms still have problems. These architectures lack differentiated modeling of different levels of features, leading to insufficient cross-layer information exchange, loss of shallow features, and difficulty in effective feedback of deep features. In addition, dynamic alignment and selective modulation mechanisms are not introduced, leading to spatial misalignment and semantic deviation of features before fusion, especially in complex environments such as rainwater reflection, shadow obstruction, and oil stain interference, which can easily cause false positives and missed detections, affecting the robustness of the model.

[0060] In some embodiments, to solve this problem, the detection network is provided with a Multi-Field Adaptive Fusion Detection Head (MFAFDet) to improve the robustness and accuracy of the model through an innovative fusion strategy. The MFAFDet includes a third convolutional layer, an MFAF module, a fourth convolutional layer, a bounding box regression identification branch, and a class prediction branch, aiming to solve the problems of insufficient information interaction, feature alignment, and semantic deviation in traditional architectures.

[0061] Based on the adaptive fusion mode of the MFAF module, the MFAFDet can not only effectively solve the problems of "top-down semantic discontinuity" and "bottom-up detail attenuation", but also accurately identify pits and grooves in complex environments, especially in small size and edge blurred pit and groove detection. Finally, combined with the bounding box regression and class prediction branches, the MFAFDet can provide more accurate and stable output for subsequent detection tasks, significantly improving the robustness and practicality of the detection system.

[0062] In some embodiments, the MFAF module skillfully fuses multi-scale feature extraction, channel attention, spatial attention, and residual connection, thereby significantly improving the identification ability of road pit and groove. This module performs three key operations in parallel: first, it uses convolution kernels of different sizes (k=3, k=5, and k=7) to extract multi-scale features, capturing rich information from local details to global context; second, it uses global average pooling combined with MLP to build a channel attention mechanism, automatically enhancing the feature response of important channels; finally, it uses a 1x1 convolution to generate a spatial attention map, highlighting the location information of the disease area. These branches and residual connections each use independent and learned different weights, ensuring that each process can focus on optimizing a specific type of feature representation.

[0063] The outputs of the parallel processes are fused with the original reduced dimension feature maps through adaptively learned weights, preserving the original information, enhancing the discriminative ability of the features, and improving the recognition accuracy of the model in complex backgrounds. The improved fusion structure design not only strengthens the expression ability of the model, but also improves its performance in various challenging conditions. By adaptively fusing features of different scales and perspectives, the MFAF module can effectively alleviate the spatial misplacement problem in multi-scale feature fusion, solving the problems of "top-down semantic discontinuity" and "bottom-up detail attenuation". Based on this fusion mode, the MFAFDet can accurately identify pits and grooves in complex environments, especially in small size and edge blurred pit and groove detection. Finally, combined with the bounding box regression and class prediction branches, the MFAFDet provides more accurate and stable output for subsequent detection tasks, significantly improving the robustness and practicality of the detection model.

[0064] In some embodiments, in view of the detection advantages of the YOLO series model, the road pothole detection model can be improved based on the model of the YOLO series. Preferably, YOLOv8s. Illustratively, if YOLOv8s is used as the basis network for improvement, through the MSEA module, the LG-FEAF detection head, and the MFAFDet improvement, the network structure of the road pothole detection model can refer to Figure 1 .

[0065] In some embodiments, in order to ensure that the trained road pothole detection model can meet the expectations and accurately understand and detect road potholes of different scales in different scenarios, a special data set can be created for training the road pothole detection model. Specifically, a data set named road pothole (Pothole Detection Dataset, PDD) is created, which is collected by a vehicle-mounted camera under various environmental conditions, and includes a total of 3000 images with a resolution of 1280x720. Each image has been manually labeled.

[0066] Illustratively, in order to construct a high-quality road pothole data set, in addition to ensuring sufficient number of images, samples from different regions and roads should be collected as much as possible to cover a variety of real scenarios. For example, the data set can include images collected from typical road sections such as urban trunk roads, expressways, urban-rural junctions, and tunnels, focusing on common pothole diseases in urban roads and highways, covering various typical pothole forms such as partial spalling, loose surface, deep collapse, and edge cracking, thereby improving the diversity and representativeness of the data set and providing a more comprehensive and reliable sample basis for model training.

[0067] In addition, the data set should also widely cover different road types (such as asphalt concrete, cement concrete, composite pavement), traffic load levels, climate conditions (sunny, rainy, night), and geographic regions (urban center, suburb, industrial area, etc.). Especially, it should include a large number of pothole samples under complex visual interference such as strong light, shadow obstruction, water reflection, and dust covering, significantly improving the challenge and generalization evaluation value of the data set. This data set provides rich and representative visual resources for the training and validation of deep learning models in variable real environments.

[0068] Such a data set can support research and application in the fields of intelligent transportation systems, autonomous vehicles, and road safety monitoring, to improve the safety and efficiency of roads.

[0069] To ensure the effectiveness of model training and the reliability of evaluation results, the data set in the present application is divided into a training set (2400 images) for training and optimizing the algorithm, a validation set (300 images) for hyperparameter tuning and model selection, and a test set (300 images) for independent evaluation of the detection performance of the algorithm on unseen samples, comprehensive verification of its accuracy, robustness, and potential for actual deployment. A scientific data division strategy not only improves the efficiency and stability of model training, but also lays a solid foundation for the practical application and evaluation of the algorithm in the management of sidewalks in smart cities.

[0070] Preferably, during the process of dividing the data set, special attention is paid to maintaining the balance of sample distribution between the data sets, ensuring that they have similar statistical characteristics in terms of road pit type, size, background environment, etc. This avoids evaluation bias due to uneven sample distribution and makes the validation results more representative and reliable.

[0071] In some embodiments, to comprehensively and accurately measure the performance of each version of the road pit detection model, after training at least one version of the road pit detection model based on the training set to obtain convergence, the converged model can be evaluated by the validation set to avoid model overfitting, verify the generalization of the model, and ensure that the model can run stably after deployment.

[0072] Specifically, the performance of the road pit detection model on the validation set can be evaluated according to pre-set conditions.

[0073] In some embodiments, to verify the effectiveness of the proposed algorithm, a corresponding experimental platform is established. Specifically, the model running environment of the experimental platform includes an Intel Xeon Platinum 8255C processor, 314 GB of memory, an NVIDIA Tesla V100 32 GB graphics card, and a CentOS 8.5.2 (64-bit) operating system. The deep neural network is built based on the PyTorch framework, the input image size is [640, 640], and a multi-scale training strategy is used. The batch size of the experiment is set to 64, the training is performed for 200 epochs, the SGD optimizer is used, the initial learning rate is 0.01, and the cosine decay strategy is used for optimization.

[0074] In some embodiments, to comprehensively evaluate the detection performance of the model, the experiment selects four indicators: F1-Score, mean average precision (mAP), parameter quantity (Params), and computational quantity (GFLOPs). F1-Score integrates precision and recall to measure the comprehensive performance and stability of the model. Average precision (AP) is obtained by calculating the area under the precision-recall curve, and mAP is the average of all class APs. Parameter quantity reflects the complexity of the model, and computational quantity measures the computational complexity of the model. The calculation process of these indicators will be introduced in detail below.

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] wherein TP, FP and FN represent the number of positive samples predicted correctly, misjudged and missed, respectively.

[0081] That is, after verifying each version of the road pit recognition model through the validation set, the performance of each version of the road pit recognition model can be comprehensively evaluated based on the above several indicators, so as to determine the road pit recognition model with the best performance from each version as the trained road pit recognition model.

[0082] In some embodiments, the road pit recognition model training is optimized in combination with classification loss and regression loss. The classification loss adopts binary cross-entropy loss (BCE Loss) to determine the anchor box category; the regression loss is composed of SIoU Loss and DFLLoss to measure the error between the predicted box and the real box. The positive and negative sample matching strategy adopts the TAL dynamic matching method to optimize the target allocation and improve the recognition accuracy.

[0083] Based on the network structure of the road pit detection model of the foregoing embodiments, the present application proposes a road pit detection method based on deep learning.

[0084] The deep learning-based road surface pit detection method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and the like. The embodiments of the present application do not make any limitation on the specific type of electronic devices.

[0085] To illustrate the technical solutions proposed in the present application, the electronic device will be taken as the execution subject to describe each embodiment.

[0086] Figure 2 A schematic flowchart of the deep learning-based road surface pit detection method provided in the present application is shown, which includes the following steps:

[0087] In step 210, the electronic device performs feature extraction on the to-be-detected image based on the backbone network of the pre-trained road surface pit detection model, to obtain image features.

[0088] In step 220, the electronic device performs fusion on the image features based on the neck network of the road surface pit detection model, to obtain target fusion features.

[0089] In step 230, the electronic device performs detection on the target fusion features based on the detection network of the road surface pit detection model, to obtain the detection result of the road surface pit in the to-be-detected image.

[0090] The neck network is provided with an MSEA module, which includes a multi-scale convolution structure, a CAB branch, an SAB branch, a hybrid weighting layer, and a first splicing layer. For the first input features input into the MSEA module:

[0091] In step A1, the electronic device performs multi-scale feature extraction and adaptive fusion on the first input features through the multi-scale convolution structure, to obtain first multi-scale fusion features.

[0092] In step A2, the electronic device processes the first multi-scale fusion features through the CAB branch, to obtain corresponding target channel weights.

[0093] In step A3, the electronic device processes the first multi-scale fusion features through the SAB branch, to obtain corresponding target spatial weights.

[0094] In step A4, the electronic device performs hybrid weighting on the multi-scale fusion through the target channel weights and the target spatial weights in the hybrid weighting layer, to obtain hybrid enhanced features.

[0095] Step A5, the electronic device splices the mixed enhanced feature and the first input feature in the first splicing layer to obtain the first output feature corresponding to the first input feature.

[0096] In the MSEA module, the electronic device performs multi-scale feature extraction and adaptive fusion on the first input feature through a multi-scale convolution structure to generate first multi-scale fusion features. This step extracts details and global information in the image through convolution kernels of different scales, enhancing the model's perception ability for pits and grooves of different sizes. Then, the electronic device processes the first multi-scale fusion features through the CAB branch to obtain target channel weights. This branch can focus on feature channels that are crucial to pit and groove detection by learning the channel dependency of the features, and suppress the interference of irrelevant features. The electronic device also processes the first multi-scale fusion features through the SAB branch to obtain target spatial weights. The SAB branch enhances the model's fine attention to pit and groove regions by learning spatial dependency, especially in cases of ambiguous boundaries or complex backgrounds. Subsequently, the electronic device weights the multi-scale fusion features through the target channel weights and the target spatial weights in the mixed weighting layer to obtain the mixed enhanced feature. This weighting operation can effectively fuse channel and spatial features, strengthen the response of key regions, and improve the discriminability of the features. Finally, the electronic device splices the mixed enhanced feature and the first input feature in the first splicing layer to obtain the first output feature corresponding to the first input feature. The spliced features contain more abundant information, providing more accurate input for subsequent detection and classification tasks. The entire process effectively improves the model's detection ability for road pit and groove in complex scenes by gradually extracting, weighting, and fusing features at different levels, especially in handling multi-scale, ambiguous boundary, and complex background pits and grooves, showing significant advantages.

[0097] In this embodiment, the model first extracts deep-level features from the input image through the backbone network, capturing multi-level representations of the road region and potential pit and groove targets. Then, the neck network effectively fuses features at different levels to form fusion features containing rich contextual information. Finally, the detection network analyzes the fusion features and outputs the detection results, completing pit and groove positioning and recognition.

[0098] In this process, the model introduces the MSEA module to perform multi-scale feature extraction and adaptive fusion on the first input feature through a multi-scale convolution structure to generate first multi-scale fusion features, achieving sufficient capture of pit and groove details at different scales. The first multi-scale fusion features are input into the CAB branch and the SAB branch, respectively, to learn feature dependency in the channel dimension and to mine key region response in the spatial dimension, thereby obtaining target channel weights and spatial weights, enabling the model to accurately focus on pit and groove key features under multi-scale, ambiguous boundary, and environmental interference.

[0099] Subsequently, the mixed weighting layer applies the above-mentioned channel and spatial weights to the first multi-scale fusion feature, further enhancing the discriminative ability of the feature, significantly improving the sensitivity to real pits in the presence of noise interference (such as oil stains, water reflection, etc.), and reducing the risk of false detection and missed detection. Finally, the weighted results are integrated by the concatenation layer to form a more comprehensive and detailed feature representation, providing a more robust input for subsequent detection, so that the model performs better in complex road environments.

[0100] In some embodiments, based on the specific structure of the aforementioned multi-scale convolution structure, step A1 specifically comprises:

[0101] Step A11, the electronic device performs convolution operations of different scales on the first input feature through the multi-scale convolution branch, and stacks the convolution results of different scales to obtain multi-scale stacked features.

[0102] The electronic device performs convolution operations of different scales on the first input feature through the multi-scale convolution branch. First, the electronic device can perform a group of CBS operations on the first input feature, that is, after extracting features through convolution, performing batch normalization processing to ensure the consistency of feature distribution, and then introducing non-linear transformation through an activation function to enhance the expression ability and training stability of the network. Then, four different convolution kernel sizes (3x3, 5x5, 7x7, and 9x9) are used for multi-scale feature extraction through depth separable convolution, and the convolution results of each scale are stacked together to form multi-scale stacked features. This multi-scale feature extraction method enables the model to effectively capture local details and global context information in the image, thereby enhancing its perception of pits of different sizes and improving the recognition effect of diversified pit shapes.

[0103] Step A12, the electronic device generates multi-scale dynamic weights by adaptively learning the first input feature through the adaptive learning branch.

[0104] The adaptive learning branch realizes dynamic weight generation through a global pooling layer, a convolution layer, an activation function, and an unsqueeze layer. First, the global pooling layer extracts global information of the first input feature, compresses the spatial dimension into a single value, and thus obtains a summary representation of the global feature. Then, the convolution layer processes the global pooled feature to further extract higher-level semantic information and adjust the expression ability of the feature as needed. After convolution, the feature is nonlinearly transformed by the activation function, increasing the nonlinear expression ability of the model and enabling it to capture more complex feature patterns. Finally, the unsqueeze operation is used to adjust the dimension of the feature, adding an additional dimension to enable the data to be weighted with other features. These steps enable the adaptive learning branch to generate dynamic weights, automatically adjust the importance of features of different scales, and thus provide a basis for subsequent feature fusion and weighting, thereby enhancing the model's perception and discrimination ability for multi-scale pit and groove features.

[0105] In step A13, the electronic device adaptively fuses the multi-scale stacked features based on the dynamic weights of the multi-scale features through an adaptive fusion layer, and adjusts the number of channels of the adaptive weighted fusion result through a first convolution layer to obtain first multi-scale fusion features.

[0106] In step A13, the electronic device adaptively fuses the multi-scale stacked features based on the dynamic weights of the multi-scale features through an adaptive fusion layer, and adjusts the number of channels of the adaptive weighted fusion result through a first convolution layer to obtain first multi-scale fusion features. This operation can fuse features of multiple scales into a unified feature representation, and adjust the number of channels to make the features more refined, thereby improving the model's perception ability and recognition accuracy of pits and grooves.

[0107] In this embodiment, the model can fully fuse features from different scales and weight each scale of feature according to the dynamic weights of adaptive learning. The multi-scale convolution operation extracts feature information of different scales, and the adaptive learning mechanism adjusts the weights of each scale according to the importance of the feature, ensuring that the model can pay more attention to key features. Adaptive weighted fusion further optimizes the fusion effect of multi-scale features, and adjusts the number of channels to make the features more refined and efficient. The whole process not only improves the model's detection ability for different scale pits and grooves, but also enhances its robustness and accuracy in detecting complex backgrounds and diverse features.

[0108] In some embodiments, based on the specific structure of the aforementioned CAB branch; step A2 specifically includes:

[0109] Step A21, in each channel sub-branch, the electronic device performs a pooling operation on the first multi-scale fusion feature through a pooling layer to obtain a corresponding pooling result; performs nonlinear channel weight learning on the pooling result through an MLP layer to obtain a corresponding deep feature; and performs a weighting operation on the deep feature based on a primary channel weight through a branch weighting layer to obtain a corresponding channel sub-branch output; the primary channel weight is obtained by performing a splicing operation, a convolution operation and a normalization operation on the deep features of each channel sub-branch through a weight branch.

[0110] In each channel sub-branch, the electronic device first performs a pooling operation on the first multi-scale fusion feature through a pooling layer. The pooling operation can select a global average pooling, a global maximum pooling or a channel standardization pooling operation. Different channel sub-branches adopt different pooling methods. The pooling operation can compress the feature from a global perspective, effectively extract potential important features, and at the same time avoid excessive computational burden. Then, the pooling result is transmitted to the MLP layer. The MLP layer performs nonlinear channel weight learning on the pooling result of each channel sub-branch. The model learns the weights of different channels through nonlinear transformation of the multilayer perceptron and generates a deep feature representation. The deep feature contains higher-level semantic information, which can further enhance the expression ability of the model and provide more accurate feature representation for subsequent feature fusion and detection.

[0111] In the weight branch, the deep features input by each channel sub-branch are first spliced, and then the primary channel weight is generated through convolution and normalization operations. The splicing operation integrates different features, the convolution operation is used to extract higher-level feature representation, and the normalization operation ensures the consistency of the scale of the features to prevent gradient problems during training. The generated primary channel weight is then input to the branch weighting layer of each channel sub-branch. In the branch weighting layer of each channel sub-branch, the deep feature is weighted based on the primary channel weight, and finally the output of each channel sub-branch is obtained. Through weighting optimization of the expression ability of the feature, the model can more accurately focus on important features and improve the detection effect.

[0112] Step A22, the electronic device splices the outputs of each channel sub-branch through a second splicing layer, and activates the corresponding splicing result through a first activation function layer to obtain a target channel weight.

[0113] Finally, the electronic device splices the outputs of each channel sub-branch through a second splicing layer to integrate the feature information of different channels. After the spliced result is processed through the first activation function layer, the target channel weight is obtained. Through the introduction of nonlinear transformation by the activation function, the feature expression ability of the model can be further strengthened, and the target channel weight for strengthening the first multi-scale fusion feature is obtained.

[0114] In this embodiment, the model can effectively extract and weight the feature information of each channel. First, the pooling and MLP layers learn and compress the features of different channels, then the importance of different channels is weighted through the branch weighting layer, and the representation of the features is further optimized through the concatenation layer and the activation layer, so that the final target channel weight can accurately reflect the importance of different features in the model. The CAB branch effectively improves the model's ability to accurately identify pit and groove features, especially in multi-scale and complex backgrounds, which can enhance the robustness and adaptability of the model.

[0115] In some embodiments, based on the specific structure of the aforementioned SAB branch, step A3 specifically includes:

[0116] Step A31, the electronic device performs average pooling operation and maximum pooling operation on the first multi-scale fusion features respectively through the mixed pooling structure, and splices the corresponding pooling results to obtain a mixed pooling result.

[0117] When processing the first multi-scale fusion features, the electronic device first performs average pooling operation and maximum pooling operation on them respectively through the mixed pooling structure. By capturing different information in the input features through two different types of pooling methods, average pooling helps to extract global information, while maximum pooling emphasizes local significant features. Then, the pooling results are spliced together to obtain a mixed pooling result, which provides more rich feature representation for subsequent feature learning.

[0118] Step A32, the electronic device performs large-scale convolution operation on the mixed pooling result through the second convolution layer to obtain a basic spatial feature.

[0119] Then, the electronic device performs large-scale convolution operation on the mixed pooling result through the second convolution layer to obtain a basic spatial feature. By further extracting the global spatial information of the features through large-scale convolution (such as 7x7, 9x9 or 11x11, etc.), the understanding of the entire image structure by the model can be enhanced.

[0120] Step A33, the electronic device performs separable horizontal convolution operation and vertical convolution operation on the basic spatial feature through the spatial separable convolution structure to obtain corresponding deep spatial features.

[0121] Subsequently, the electronic device performs separable horizontal convolution operation and vertical convolution operation on the basic spatial feature through the spatial separable convolution structure to generate corresponding deep spatial features. By modeling the spatial structure of the pit and groove features in detail, the electronic device can effectively reduce the computational complexity while enhancing the ability to capture spatial information based on horizontal and vertical separation convolution operations.

[0122] Step A34: The electronic device stitches the deep spatial features with the basic spatial features through the third stitching layer, and activates the stitching result through the second activation function layer to obtain the target spatial weights.

[0123] Finally, the electronic device stitches the deep spatial features with the basic spatial features through a third stitching layer, and activates the stitched result through a second activation function layer to generate target spatial weights. The stitching and activation operations further integrate deep and shallow spatial features, ensuring that the model can learn more accurately and focus on important spatial information.

[0124] In this embodiment, the model can extract rich spatial features from multiple angles and scales. Hybrid pooling provides different perspectives for feature fusion, while convolutional layers and spatially separable convolutional layers further enhance the capture and refined modeling of spatial information. Concatenation and activation operations ensure the effective fusion of various features, and the resulting target spatial weights improve the model's ability to recognize pit features in complex backgrounds, enhancing its robustness and accuracy.

[0125] In some embodiments, see Figure 3 , Figure 3 A schematic diagram of the network structure of the MSEA module is shown. This is for the first input feature. , among which, among which For batch size, For the number of channels, The spatial dimensions are characterized by their spatial dimensions.

[0126] First, the number of channels in the input feature map is compressed using a set of CBS (Combined Channel Blocks). ,in To achieve the desired dimensionality reduction, this operation not only introduces a nonlinear transformation but also preserves spatial information.

[0127]

[0128] Next, the compressed first input features Parallel applications N Separable convolutions at different depths can be used to capture multi-scale local patterns. Let the th... n The kernel size of each branch is Then its output is:

[0129]

[0130] Subsequently, the output feature maps of all branches are stacked along the channel dimension to form a multi-scale stacked feature, which is essentially a multi-scale feature set:

[0131]

[0132] To realize adaptive fusion, the first input feature is learned by an adaptive learning branch The dynamic weight vector of each scale is generated:

[0133]

[0134]

[0135] The generated weights are used to weight the multi-scale stacked features, and the fused features are restored to the original channel number through another convolution operation , to obtain the first multi-scale fusion feature:

[0136]

[0137]

[0138] To further enhance the correlation features of the channel dimension, an improved Channel Attention Block (CAB) branch is designed, which can more effectively aggregate statistical information.

[0139] In the CAB branch, the first multi-scale fusion feature is respectively executed global average pooling (GAP), global maximum pooling (GMP) and channel standardization pooling (Std Pooling), to obtain three global descriptions:

[0140]

[0141]

[0142]

[0143] Then, the three pooling results are sent to a shared MLP, which includes two layers of 1x1 convolution network (including ReLU activation), to learn nonlinear channel weights and obtain corresponding deep features.

[0144]

[0145]

[0146]

[0147] Finally, the three deep features are spliced, and the spliced results are normalized by the Softmax function to generate the primary channel weight, and the primary channel weight is multiplied by each deep feature, and the products are summed to obtain the final target channel weight .

[0148]

[0149]

[0150]

[0151]

[0152] To enhance spatial localization capabilities, a Spatial Attention Block (SAB) branch is proposed.

[0153] In the SAB branch, the input is processed along the channel dimension. Max pooling and average pooling are performed to generate two two-dimensional spatial description graphs.

[0154]

[0155]

[0156] Then, the two pooling operations are concatenated along the channel dimension to obtain a hybrid pooling result, and the basic spatial features are learned through a 7×7 convolutional layer.

[0157]

[0158] In addition, spatially separable convolutional structures are introduced, namely separable horizontal and vertical convolutions ( k ×1 and 1× k ), explicitly model the linear structure of surface defects to obtain deep spatial features.

[0159]

[0160]

[0161] Finally, the outputs of the basic convolution and the directional convolution are combined and activated by the Sigmoid function to generate the target space weights.

[0162]

[0163] Based on the two weights output from the CAB and SAB branches, a hybrid weighting is performed on the first multi-scale fusion feature to obtain the modulated hybrid enhancement feature. .

[0164] To further improve the expressive power and stability of the model, A 1×1 convolutional layer is applied to the first output feature and residually connected to the original input feature, thereby achieving a more comprehensive feature representation.

[0165]

[0166] In some embodiments, based on the specific structure of the aforementioned LG-FEAF module, the electronic device performs the following operations on the second input feature of the input LG-FEAF module:

[0167] Step B1, in the local branch, the electronic device performs a multi-scale feature extraction operation on the second input feature to obtain local features.

[0168] In the local branch, the multi-scale feature extraction operation can be realized by applying a dilated convolution with different dilation rates (increasing). Through these operations, the model can extract features at multiple scales, capturing multi-level information from details to the global. Then, the features at each scale are spliced, and then a channel shuffling operation is performed to promote information interaction between feature groups and further enhance the relevance between different features. Finally, the mixed features are fused through a convolution operation to obtain refined local features, ensuring that the model has higher precision and robustness when processing complex details.

[0169] Step B2, in the global branch, the electronic device sequentially performs a multi-scale pooling operation, a splicing operation, and a flattening operation on the second input feature through a multi-scale pooling fusion structure and a flattening layer in series to obtain corresponding keys and values; in the matrix calculation layer, a similarity matrix is calculated between the second input feature processed by the query layer and the keys; in the aggregation layer, the similarity matrix processed by the normalization layer is aggregated with the values, and then passed through the CBS layer to obtain corresponding global features.

[0170] In the global branch, the electronic device first sequentially performs a multi-scale pooling operation, a splicing operation, and a flattening operation on the second input feature through a multi-scale pooling fusion structure and a flattening layer in series to obtain corresponding “keys” (Key) and “values” (Value). Specifically, the multi-scale pooling operation is first performed on the second input feature to extract features at different scales, which can capture feature information at different sizes and levels of detail. Then, the features after each scale pooling are spliced to combine feature information at multiple scales to increase the diversity and richness of the features. Finally, the flattened operation is performed on the spliced features to form a one-dimensional array to adapt to subsequent matrix operations, thereby obtaining two groups of feature representations of “keys” and “values”.

[0171] Next, at the matrix computation layer, the electronic device processes the second input feature through the query layer, generating a query vector (Query). The query vector is compared with the previously generated "keys" to generate a similarity matrix. The similarity matrix reflects the relationship or matching degree between the input feature and the "keys", and calculates which features are most relevant to the current feature. This operation can be considered as part of the self-attention mechanism, aiming to dynamically adjust the attention allocation of the model by calculating the relationship between features.

[0172] At the aggregation layer, the similarity matrix processed by the normalization layer is weighted and aggregated with the "values". Each element of the similarity matrix represents the importance or weight of the corresponding feature, and through weighted aggregation, the model can strengthen or suppress each feature according to its importance. Finally, after the convolution, batch normalization and activation function operations of the CBS layer, the model will obtain the corresponding global feature. The CBS layer further extracts features through convolution operations, batch normalization ensures the standardization of features, and the activation function introduces non-linear transformation, enabling the network to learn more complex feature patterns.

[0173] The global branch effectively integrates information from different scales by combining the similarity calculation of queries and keys through multi-scale pooling and flattening operations, and optimizes the expression of features through adaptive weighted aggregation. Through such operations, the model can more accurately capture global context information while avoiding the limitations of single-scale features. The calculation of the similarity matrix enables the model to dynamically adjust the importance of different features, thereby improving the attention to key features and enhancing the discriminability of global features. This design is particularly suitable for handling pit and groove detection tasks in complex backgrounds, and can improve the robustness and accuracy of the model in diverse and changing environments.

[0174] Step B3, the locally weighted and globally weighted features are spliced at the fourth splicing layer to obtain the second output feature corresponding to the second input feature.

[0175] By weighting the local features and global features according to their importance and semantics through adaptive weights (the sum of the weights of the two is 1), the relevance and importance of the features can be dynamically considered. The weighted local features and global features can integrate their respective advantages, with local features focusing on detailed information and global features providing overall context. Then, these weighted features are spliced into comprehensive feature representations at the fourth splicing layer to form the second output feature, providing more rich and efficient information for subsequent processing and decision-making.

[0176] In this embodiment, the operations of local branches, global branches, and the stitching layer work together to extract and process local and global features respectively. This ensures that the electronic device can comprehensively understand both the details and the overall information in the input data. Local branches focus on learning detailed features, capturing local information in the image, while global branches provide global context, helping the model understand a wider range of background and semantics. The stitching layer operations weight and fuse these two aspects, ensuring full integration of information. This allows the model to handle both detailed and global features when processing complex tasks, improving detection accuracy and robustness.

[0177] In some embodiments, see Figure 4 , Figure 4 A schematic diagram of the network structure of the LG-FEAF module is shown. (This is for the second input feature.) , among which, among which For batch size, For the number of channels, The spatial dimensions are defined by B, C, H, and W, where B, C, H, and W represent the batch size, number of channels, height, and width, respectively. The LG-FEAF module aims to enhance both local detail and global contextual information through a dual-path structure.

[0178] To capture local patterns under different receptive fields, four dilated convolution operations with increasing dilation rates are used:

[0179]

[0180]

[0181]

[0182]

[0183] The obtained features are then concatenated along the channel dimension:

[0184]

[0185] Then, a channel shuffle operation is applied, with a group size of [number missing]. To facilitate information exchange between groups:

[0186]

[0187] Finally, a refined local feature representation is obtained by fusing the mixed features through a 1×1 convolution:

[0188]

[0189] To model global spatial dependencies, a key-value attention mechanism based on max-pooling pyramid is designed.

[0190] (1) Query, key and value generation.

[0191] Query is directly generated from input extraction, flattened to spatial-channel matrix:

[0192]

[0193] Key and value are extracted by shared QKV Block. This module contains four max-pooling operations with kernel size and stride of 2x2, 4x4, 6x6, 8x8, respectively, to capture multi-granularity down-sampling context.

[0194] Define max-pooling operation set Then:

[0195]

[0196] Concatenate and flatten the outputs of each pooling:

[0197]

[0198] where is the total number of down-sampling positions.

[0199] Similarly, Exchange dimensions to match the attention calculation format:

[0200]

[0201]

[0202] (2) Attention weight and global feature aggregation.

[0203] Calculate the similarity matrix between query and key:

[0204]

[0205] Apply Softmax normalization to get attention weight:

[0206]

[0207] Weighted aggregation of value vector to generate global features containing context:

[0208]

[0209] Restore spatial structure:

[0210]

[0211] Feature modulation by CBS with 1x1 convolution:

[0212]

[0213] At the aggregation layer, the local features and global features are weighted by learnable parameters respectively, and then the weighted results are added to realize complementary feature fusion:

[0214]

[0215] This fusion strategy takes into account the local detail perception and global structure modeling, significantly improving the model's expression ability for complex textures and long-range dependencies.

[0216] In some embodiments, based on the specific structure of the foregoing MFAFDet, the electronic device performs the following operations on the target fusion features input into the MFAFDet:

[0217] Step C1, the electronic device performs channel compression on the target fusion features through a third convolutional layer to obtain compressed features.

[0218] Step C2, the electronic device performs multi-scale feature extraction and fusion on the compressed features through the MFAF module to obtain comprehensive features.

[0219] Step C3, the electronic device performs channel expansion on the comprehensive features through a fourth convolutional layer to obtain expanded features.

[0220] Step C4, the electronic device performs bounding box identification on the expanded features through the bounding box regression identification branch to output the predicted bounding box.

[0221] Step C5, the electronic device performs class prediction on the expanded features through the class prediction branch to output the predicted road pit slot class.

[0222] The electronic device first performs channel compression on the target fusion feature through a third convolutional layer, maps the high-dimensional feature to a lower-dimensional space, thereby reducing the amount of calculation and highlighting more discriminative core features. Subsequently, the compressed feature enters the MFAF module, through mechanisms such as multi-scale feature extraction, channel attention, spatial attention, and residual fusion, the sufficient fusion of multi-scale information is completed, and a comprehensive feature with more semantic expression ability is generated. Then, the electronic device performs channel expansion on the comprehensive feature through a fourth convolutional layer, maps the feature to restore to the dimension required by the detection head, to ensure that the subsequent prediction task has enough feature capacity and expression ability. In the bounding box regression recognition branch, the expanded feature is further analyzed to predict the specific position of the target area, and the accurate bounding box is output through regression. In the category prediction branch, the expanded feature is used to judge which category of disease the detected target belongs to, and finally the corresponding road pit category prediction is given.

[0223] The branch network structures of the bounding box regression detection branch and the category prediction branch can be the same; for example, the branch network structure includes three convolutional layers, and the difference lies in that the output channel number of the last convolutional layer of the bounding box regression detection branch is set to The output channel number of the last convolutional layer corresponding to the bounding box regression detection branch is set to n c .

[0224] In this embodiment, the electronic device improves the feature processing efficiency through channel compression, enhances the discriminative ability of the feature through multi-scale fusion, and ensures the richness of the feature required for prediction through channel expansion, while the classification and regression branches are responsible for the final target recognition and positioning. Through this series of closely connected operations, the model can realize high-precision detection of pit targets in complex road environments, has strong semantic understanding ability and mature spatial positioning ability, thereby significantly improving the overall detection performance and robustness.

[0225] In some embodiments, based on the specific structure of the MFAF module, step C2 specifically includes:

[0226] Step C21, in the multi-scale extraction branch, the electronic device performs multi-scale feature extraction operation, splicing operation and convolution operation on the compressed feature to obtain second multi-scale fusion feature.

[0227] Step C22, in the channel attention branch, the electronic device weights the compressed feature based on the weight obtained by performing the channel attention mechanism on the compressed feature to obtain the channel enhanced feature.

[0228] Step C23, in the spatial attention branch, the compressed feature is weighted based on the weight obtained by performing the spatial attention mechanism on the compressed feature to obtain the spatial enhanced feature.

[0229] Step C23, in the fifth splicing layer, the electronic device splices the second multi-scale fusion feature, the channel enhancement feature, the spatial enhancement feature and the compression feature transmitted through the residual path based on the adaptive learning weight respectively, to obtain a comprehensive feature.

[0230] In the multi-scale extraction branch, the electronic device first performs a multi-scale feature extraction operation on the compression feature, jointly models the input feature in fine granularity and coarse granularity through convolution of different scales, then splices the multi-scale convolution results to converge information from different receptive fields, and finally performs further fusion on the spliced features through convolution operation, thereby obtaining the second multi-scale fusion feature, so that the model has both local detail capture ability and global structure understanding ability. In the channel attention branch, the electronic device learns the channel weight by performing channel attention mechanism (global average pooling + MLP + Sigmoid activation function) on the compression feature, and then weights the compression feature based on the weight, to obtain the channel enhancement feature, so that the model can highlight key channels and suppress invalid or interfering channels, thereby improving the discriminability in the channel direction. In the spatial attention branch, the electronic device learns the importance weight of each position through the spatial attention mechanism (convolution + Sigmoid activation function), and weights the compression feature using the weight, to obtain the spatial enhancement feature, so that the model can focus more accurately on the key area in the image, especially when the background is complex or the target boundary is blurred. Finally, in the fifth splicing layer, the electronic device splices the second multi-scale fusion feature, the channel enhancement feature, the spatial enhancement feature and the compression feature transmitted through the residual path based on the adaptive learning weight respectively, thereby obtaining a comprehensive feature, providing richer, more stable and more discriminative feature expression for subsequent detection.

[0231] In some embodiments, referring to Figure 5 , Figure 5A network structure diagram of MFAFDet is shown. MFAFDet skillfully integrates multi-scale feature extraction, channel attention, spatial attention, and residual connection to improve the recognition ability of road potholes. During processing, the input feature map is first reduced in channel dimension by 1x1 convolution to reduce computational complexity. Subsequently, three key operations are performed in parallel: multi-scale features are extracted using convolution kernels of different sizes (k=3, k=5, k=7) to capture information from local details to global context; a channel attention mechanism is constructed by combining global average pooling with MLP to adaptively enhance the feature response of important channels; and a spatial attention map is generated using 1x1 convolution to highlight the location information of the disease area. Each of these three branches and the residual connection uses different weights learned independently to ensure that each process focuses on optimizing a specific type of feature representation. Finally, the outputs of these parallel processes are fused with the original reduced feature map through adaptively learned weights, preserving the original information while enhancing the discriminative ability of the features, enabling the model to accurately identify road potholes in complex backgrounds. This design not only enhances the model's expressive power but also improves its performance in various challenging conditions. Finally, the enhanced features are fed into the bounding box (Bbox) and class (Cls) prediction branches to improve positioning accuracy and reduce computational complexity, enabling effective identification of road potholes in small size, low contrast, and complex backgrounds.

[0232] In some embodiments, the detection method of each of the above embodiments is implemented based on a trained road pothole detection model. The road pothole detection model can be realized through the cooperative application of the MSEA module, the LG-FEAF module, and the MFAFDet, which has the advantage of modular design. The introduction of the MSEA module in the neck network can jointly model the multi-scale spatial structure and channel dependency, thereby improving the model's discriminant robustness for real potholes. At the same time, the LG-FEAF module set in the backbone network can construct an efficient and robust cross-scale feature modeling mechanism, enhancing the model's detection ability for multi-scale and boundary fuzzy potholes. In addition, the detection network also uses MFAFDet, which skillfully integrates multi-scale feature extraction, channel attention, spatial attention, and residual connection to improve the recognition ability of road potholes, addressing the problem of insufficient multi-scale feature fusion and feature alignment capability in traditional detection networks.

[0233] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0234] The deep learning-based road pothole detection method corresponding to the above embodiments, Figure 6A structural block diagram of the road surface pit detection device 6 provided by the embodiment of the present application is shown, and only parts related to the embodiment of the present application are shown for ease of description.

[0235] With reference to Figure 6 The road surface pit detection device 6 comprises:

[0236] An extraction module 61 is configured to perform feature extraction on a to-be-detected image based on a backbone network of a pre-trained road surface pit detection model, to obtain image features, wherein the to-be-detected image comprises a road.

[0237] A fusion module 62 is configured to perform fusion on the image features based on a neck network of the road surface pit detection model, to obtain target fusion features.

[0238] A detection module 63 is configured to perform detection on the target fusion features based on a detection network of the road surface pit detection model, to obtain a detection result of a road surface pit in the to-be-detected image.

[0239] The neck network is provided with an MSEA module, and the MSEA module comprises a multi-scale convolution structure, a CAB branch, an SAB branch, a hybrid weighting layer, and a first splicing layer; the fusion module 62 comprises a fusion unit, and the fusion unit is configured to:

[0240] For a first input feature input into the MSEA module:

[0241] The multi-scale convolution structure is configured to perform multi-scale feature extraction and adaptive fusion on the first input feature, to obtain first multi-scale fusion features.

[0242] The CAB branch is configured to process the first multi-scale fusion features, to obtain corresponding target channel weights.

[0243] The SAB branch is configured to process the first multi-scale fusion features, to obtain corresponding target spatial weights.

[0244] The hybrid weighting layer is configured to perform hybrid weighting on the multi-scale fusion features by using the target channel weights and the target spatial weights, to obtain hybrid enhanced features.

[0245] The first splicing layer is configured to splice the hybrid enhanced features and the first input feature, to obtain first output features corresponding to the first input feature.

[0246] The remaining modules are configured to set the method steps corresponding to each embodiment, and details are not described herein.

[0247] It should be noted that the information interaction and execution process between the above devices / units, and the specific functions and technical effects brought by the same, can be referred to the method embodiment part, and details are not described herein.

[0248] Figure 7 A schematic diagram of a physical layer of an electronic device is provided for an embodiment of the present application. As shown, the electronic device 7 of this embodiment includes at least one processor 70 (only one processor is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, the processor 70 implementing the steps of any of the above-described embodiments of the method for detecting a pothole on a road surface based on deep learning when executing the computer program 72, such as the steps 210-230 shown. Figure 7 Figure 7 Figure 2

[0249] The processor 70 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0250] The memory 71 can be an internal storage unit of the electronic device 7 in some embodiments, such as a hard disk or a memory of the electronic device 7. The memory 71 can also be an external storage device of the electronic device 7 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7.

[0251] Further, the memory 71 can include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store operating systems, application programs, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0252] ​​​Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software functional unit. In addition, the specific name of each functional unit and module is only for the convenience of mutual distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above-mentioned system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0253] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps in each method embodiment described above.

[0254] The embodiment of the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to execute the steps in each method embodiment described above.

[0255] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the embodiment of the present application realizes all or part of the processes in the above-mentioned embodiment method, which can be completed by a computer program instructing related hardware. The above-mentioned computer program can be stored in a computer readable storage medium, and the computer program can realize the steps in each method embodiment described above when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc.

[0256] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0257] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0258] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely schematic. The division of the above modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0259] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0260] The above embodiments are only used to illustrate but not limit the technical solutions of the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and such modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for detecting a pothole based on deep learning, characterized by, The method comprises: extracting features of a to-be-detected image based on a backbone network of a pre-trained road pit detection model to obtain image features; the to-be-detected image comprises a road; fusing the image features based on a neck network of the road pit detection model to obtain target fusion features; detecting the target fusion features based on a detection network of the road pit detection model to obtain a detection result of a road pit in the to-be-detected image; wherein the neck network is provided with an MSEA module, the MSEA module comprises a multi-scale convolution structure, a CAB branch, an SAB branch, a hybrid weighting layer, and a first splicing layer; for a first input feature input into the MSEA module: performing multi-scale feature extraction and adaptive fusion on the first input feature through the multi-scale convolution structure to obtain a first multi-scale fusion feature; processing the first multi-scale fusion feature through the CAB branch to obtain a corresponding target channel weight; processing the first multi-scale fusion feature through the SAB branch to obtain a corresponding target spatial weight; performing hybrid weighting on the multi-scale fusion through the target channel weight and the target spatial weight in the hybrid weighting layer to obtain a hybrid enhanced feature; splicing the hybrid enhanced feature and the first input feature in the first splicing layer to obtain a first output feature corresponding to the first input feature.

2. The road surface depression detection method according to claim 1, characterized by, The multi-scale convolution structure comprises a multi-scale convolution branch, an adaptive learning branch, an adaptive fusion layer, and a first convolution layer, and the multi-scale feature extraction and adaptive fusion on the first input feature through the multi-scale convolution structure to obtain a first multi-scale fusion feature comprises: performing convolution operations of different scales on the first input feature through the multi-scale convolution branch, and stacking the convolution results of different scales to obtain multi-scale stacked features; generating multi-scale dynamic weights through adaptive learning on the first input feature through the adaptive learning branch; performing adaptive weighted fusion on the multi-scale stacked features based on the multi-scale dynamic weights through the adaptive fusion layer, and adjusting the number of channels of the adaptive weighted fusion result through the first convolution layer to obtain the first multi-scale fusion feature.

3. The method of claim 1, wherein the step of detecting the road surface depression is characterized by, The CAB branch comprises three channel sub-branches, a weight branch, a second splicing layer, and a first activation function layer, each channel sub-branch comprises a pooling layer, an MLP layer, and a branch weighting layer, the pooling layers of different channel sub-branches are different, and the MLP layers of each channel sub-branch are shared; the processing of the first multi-scale fusion feature through the CAB branch to obtain the corresponding target channel weight comprises: In each of the channel sub-branches, a pooling operation is performed on the first multi-scale fusion feature by the pooling layer to obtain a corresponding pooling result; a nonlinear channel weight learning is performed on the pooling result by the MLP layer to obtain a corresponding deep feature; a weighting operation is performed on the deep feature based on a primary channel weight by the branch weighting layer to obtain a corresponding channel sub-branch output; the primary channel weight is obtained by performing a concatenation operation, a convolution operation and a normalization operation on the deep features of each channel sub-branch by the weight branch; The channel sub-branch outputs are concatenated by the second concatenation layer, and the corresponding concatenation result is activated by the first activation function layer to obtain the target channel weight.

4. The method of claim 1, wherein, The SAB branch includes a hybrid pooling structure, a second convolution layer, a spatial separable convolution structure, a third concatenation layer and a second activation function layer; the processing of the first multi-scale fusion feature by the SAB branch to obtain the corresponding target spatial weight includes: An average pooling operation and a maximum pooling operation are respectively performed on the first multi-scale fusion feature by the hybrid pooling structure, and the corresponding pooling results are concatenated to obtain a hybrid pooling result; A large-scale convolution operation is performed on the hybrid pooling result by the second convolution layer to obtain a basic spatial feature; A separable horizontal convolution operation and a vertical convolution operation are respectively performed on the basic spatial feature by the spatial separable convolution structure to obtain a corresponding deep spatial feature; Each of the deep spatial features and the basic spatial feature are concatenated by the third concatenation layer, and the concatenation result is activated by the second activation function layer to obtain the target spatial weight.

5. The method of pothole detection as defined in claim 1, wherein, The backbone network is provided with an LG-FEAF module, and the LG-FEAF module includes a local branch, a global branch and a fourth concatenation layer; the global branch includes a multi-scale sub-branch and an attention sub-branch, the multi-scale sub-branch includes a multi-scale pooling fusion structure, a flattening layer, an aggregation layer and a CBS layer; the attention sub-branch includes a query layer, a matrix calculation layer and a normalization layer; for the second input feature input into the LG-FEAF module: In the local branch, a multi-scale feature extraction operation is performed on the second input feature to obtain a local feature; In the global branch, a multi-scale pooling operation, a concatenation operation and a flattening operation are sequentially performed on the second input feature by the multi-scale pooling fusion structure and the flattening layer in series to obtain a corresponding key and value; In the matrix calculation layer, a similarity matrix is calculated by the second input feature processed by the query layer and the key; In the aggregation layer, the similarity matrix processed by the normalization layer and the value are aggregated, and the CBS layer is passed through to obtain a corresponding global feature; In the fourth concatenation layer, the local feature and the global feature weighted based on an adaptive weight are concatenated to obtain a second output feature corresponding to the second input feature.

6. The method of detecting a pothole of any one of claims 1-5, wherein, The detection network is provided with an MFAFDet, which includes a third convolutional layer, an MFAF module, a fourth convolutional layer, a bounding box regression identification branch, and a category prediction branch; for a target fusion feature input into the MFAFDet: The target fusion feature is channel-compressed through the third convolutional layer to obtain a compressed feature; The compressed feature is subjected to multi-scale feature extraction and fusion through the MFAF module to obtain a comprehensive feature; The comprehensive feature is channel-extended through the fourth convolutional layer to obtain an extended feature; The extended feature is subjected to bounding box identification through the bounding box regression identification branch to output a predicted bounding box; The extended feature is subjected to category prediction through the category prediction branch to output a predicted road pit category.

7. The method of claim 6, wherein, The MFAF module includes a multi-scale extraction branch, a channel attention branch, a spatial attention branch, and a fifth splicing layer, and the comprehensive feature is obtained by subjecting the compressed feature to multi-scale feature extraction and fusion through the MFAF module, including: In the multi-scale extraction branch, multi-scale feature extraction operation, splicing operation, and convolution operation are performed on the compressed feature to obtain a second multi-scale fusion feature; In the channel attention branch, the compressed feature is weighted based on the weight obtained by performing channel attention mechanism on the compressed feature to obtain a channel-enhanced feature; In the spatial attention branch, the compressed feature is weighted based on the weight obtained by performing spatial attention mechanism on the compressed feature to obtain a spatial-enhanced feature; In the fifth splicing layer, the second multi-scale fusion feature, the channel-enhanced feature, the spatial-enhanced feature, and the compressed feature based on residual transmission, which are respectively weighted based on adaptive learning weight, are spliced to obtain the comprehensive feature.

8. A pothole detection apparatus, characterized by, including: The extraction module is configured to perform feature extraction on a to-be-detected image based on a backbone network of a pre-trained road pit detection model to obtain an image feature; The to-be-detected image includes a road; The fusion module is configured to perform fusion on the image feature based on a neck network of the road pit detection model to obtain a target fusion feature; The detection module is configured to perform detection on the target fusion feature based on a detection network of the road pit detection model to obtain a detection result of a road pit in the to-be-detected image; The neck network is provided with an MSEA module, and the MSEA module includes a multi-scale convolution structure, a CAB branch, an SAB branch, a hybrid weighting layer, and a first splicing layer; for a first input feature input into the MSEA module, the fusion module includes a fusion unit, and the fusion unit is configured to: perform multi-scale feature extraction and adaptive fusion on the first input feature through the multi-scale convolution structure to obtain a first multi-scale fusion feature; process the first multi-scale fusion feature through the CAB branch to obtain corresponding target channel weight; process the first multi-scale fusion feature through the SAB branch to obtain corresponding target spatial weight; The mixed weighting layer performs mixed weighting on the multi-scale fusion through the target channel weight and the target space weight to obtain a mixed enhanced feature. The first splicing layer splices the mixed enhanced feature and the first input feature to obtain a first output feature corresponding to the first input feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the deep learning-based road surface pit detection method in any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the deep learning-based road surface pit detection method in any one of claims 1 to 7.

Images