Manhole cover damage detection method, device, electronic equipment and program product based on deep learning

Abstract

The present application discloses a method, device, electronic device and program product for detecting manhole cover damage based on deep learning. The detection method sets feature extraction modules of different scales in the backbone network through the manhole cover detection model. Each scale feature extraction module is provided with an MSRGA submodule. The MSRGA submodule captures multi-scale damage features of different shapes and sizes based on two attention mechanisms, channel and space, and simultaneously considers contextual information, which can reduce background interference and improve the detection effect of manhole cover damage in complex backgrounds. The backbone network is also provided with a CSFF module. The CSFF module extracts local details and multi-scale contextual information simultaneously through convolution branches and SPD convolution branches to effectively capture subtle and complex damage features on the manhole cover surface, enhance feature representation, and improve the recognition accuracy and reliability of manhole cover damage.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a manhole cover damage detection method, a manhole cover damage detection device, an electronic device and a computer program product based on deep learning. Background Art

[0002] With the continuous advancement of urbanization and the gradual improvement of municipal infrastructure, manhole covers, as an indispensable component of urban roads, have a direct impact on the safety of citizens' travel, life, and property. Manhole covers not only serve important functions such as drainage and passage, but also play a vital role in protecting underground pipelines, maintaining smooth roads, and ensuring the smooth operation of cities. However, with the increasing service life of manhole covers, changes in environmental conditions, and the impact of external forces, incidents such as cover displacement, damage, and even loss are becoming more common, posing serious risks to pedestrians, vehicles, and public safety.

[0003] Although deep learning can currently provide new technical support for automated detection of manhole cover damage, since manhole covers are distributed in diverse scenarios such as urban roads, squares, parks, and around buildings, the images collected in different scenarios are easily affected by the corresponding environment, reducing the accuracy of manhole cover damage detection. Summary of the Invention

[0004] The present application provides a manhole cover damage detection method, a manhole cover damage detection device, an electronic device and a computer program product based on deep learning, which can capture multi-scale damage features of different shapes and sizes, reduce interference in complex scenes, and thus significantly improve the accuracy and reliability of manhole cover damage detection.

[0005] In a first aspect, the present application provides a method for detecting manhole cover damage based on deep learning, comprising:

[0006] The backbone network of the pre-trained manhole cover detection model extracts features from the image to be detected to obtain image features; the image to be detected includes a manhole cover;

[0007] The neck network based on the manhole cover detection model fuses image features to obtain target fusion features;

[0008] The detection network based on the manhole cover detection model detects the target fusion features and obtains the detection results of the manhole cover in the image to be detected;

[0009] Among them, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; the feature extraction module is provided with an MSRGA submodule, which extracts and fuses multi-scale features through the channel attention mechanism and the spatial attention mechanism, and the image features are obtained based on the fused multi-scale features.

[0010] In a second aspect, the present application provides a manhole cover damage detection device, comprising:

[0011] An extraction module is used to extract features from an image to be detected based on a backbone network of a pre-trained manhole cover detection model to obtain image features; the image to be detected includes a manhole cover;

[0012] The fusion module is used to fuse image features based on the neck network of the manhole cover detection model to obtain fused features;

[0013] The detection module is used to detect the fused features based on the detection network of the manhole cover detection model to obtain the detection results of the manhole cover in the image to be detected;

[0014] Among them, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; the feature extraction module is provided with an MSRGA submodule, which extracts and fuses multi-scale features through the channel attention mechanism and the spatial attention mechanism, and the image features are obtained based on the fused multi-scale features.

[0015] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0018] Compared with the prior art, the present application has the following beneficial effects: the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2. By introducing the Multi-scale Residual Global Attention (MSRGA) submodule into the feature extraction module, multi-scale features are extracted and fused through the channel attention mechanism and the spatial attention mechanism, damage features of different sizes and shapes are highlighted and enhanced, and the representation ability of the backbone network is enhanced to reduce the interference of complex backgrounds on the identification of manhole cover damage. The image features extracted by this backbone network will contain rich and discriminative manhole cover damage features, so that the feature fusion based on it can more comprehensively integrate the effective information for identifying manhole cover damage, significantly improving the detection accuracy of manhole cover damage in complex scenarios.

[0019] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 Schematic diagram of the network structure of the manhole cover detection model provided in the embodiment of the present application;

[0022] Figure 2 is an example graph in the MCD dataset provided in the embodiments of the present application;

[0023] Figure 3 1 is a flowchart of a method for detecting manhole cover damage based on deep learning provided in an embodiment of the present application;

[0024] Figure 4 Schematic diagram of the network structure of the MSRGA submodule provided in an embodiment of the present application;

[0025] Figure 5 Schematic diagram of the network structure of the MSWFF module provided in the embodiment of the present application;

[0026] Figure 6 Schematic diagram of the network structure of the CSFF module provided in an embodiment of the present application;

[0027] Figure 7 Schematic diagram of the structure of the manhole cover damage detection device provided in an embodiment of the present application;

[0028] Figure 8 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0030] Damaged manhole cover identification faces numerous challenges in complex background environments, such as vehicle obstruction, pedestrian interference, vegetation cover, shadow variations, and uneven lighting. These factors degrade image quality and complicate feature extraction. Furthermore, the similarity of background texture and color to the manhole cover surface can easily lead to confusion between damaged areas and further complicate detection. These complex factors require a more robust and accurate model to ensure precise identification and classification of damaged manhole covers.

[0031] To address this issue, this application proposes a manhole cover detection model, which includes a backbone network, a neck network, and a detection head. Specifically, the backbone network extracts features from the input image to obtain image features; the neck network fuses these image features to obtain target fused features; and the detection head detects the target fused features to obtain detection results for damaged manhole covers in the processed image.

[0032] To ensure accurate manhole cover damage detection in complex environments, the backbone network includes m feature extraction modules at different scales, where m is a positive integer and m ≥ 2. By introducing a Multi-scale Residual Global Attention (MSRGA) submodule into the feature extraction module, multi-scale features are extracted and fused through channel and spatial attention mechanisms. This highlights and enhances damage features of varying sizes and shapes, strengthening the backbone network's representational capabilities and reducing the interference of complex backgrounds on manhole cover damage identification. The image features extracted by this backbone network contain rich and discriminative manhole cover damage features, enabling feature fusion based on them to more comprehensively integrate effective information for identifying manhole cover damage, significantly improving the accuracy of manhole cover damage detection in complex scenarios.

[0033] In some embodiments, the MSRGA submodule consists of a multi-scale channel attention (Efficient Channel Attention, ECA) fusion block and a multi-scale spatial attention (Efficient Spatial Attention, ESA) fusion block, connected using a residual structure. By incorporating multi-scale feature extraction into ECA and ESA, the model effectively captures key information across different receptive fields. The residual connection not only enhances model stability but also preserves original input features when either attention mechanism fails, ensuring information integrity.

[0034] In some embodiments, the multi-scale ECA fusion block consists of a first convolutional layer, a multi-scale feature extraction structure, and an ECA structure. The multi-scale feature extraction structure includes at least two feature extraction layers at different scales, each consisting of a series of adaptive pooling and depthwise separable convolutions. This structure effectively captures both global and local information through the multi-scale feature extraction layers, and, through the ECA structure, enhances inter-channel feature interaction, improving feature expression capabilities.

[0035] In some embodiments, the multi-scale ESA fusion block consists of a pooling structure, a first concatenation layer, a parallel convolution structure, and an ESA structure. The pooling structure includes a multi-scale adaptive pooling layer. Each pooling layer at each scale consists of a series of average pooling and max pooling, efficiently acquiring dual-channel spatial features. The parallel convolution structure, consisting of at least two convolutional layers with different convolution kernels, extracts spatial information of different receptive fields from the concatenated dual-channel spatial features. The ESA structure further optimizes feature representation, improving the accuracy of spatial attention.

[0036] In some embodiments, a multi-scale ECA fusion block, connected in series, works in conjunction with a multi-scale ESA fusion block. The former utilizes multi-scale adaptive pooling to efficiently capture global and local information, while the latter extracts deep feature information through convolution at different scales. This achieves complementary enhancement of global and local deep information, improving feature representation. Further incorporating residual connections preserves the original information and learns complex mapping relationships, thereby enhancing the model's robustness in complex backgrounds and effectively improving the accuracy of manhole cover damage detection. The MSRGA submodule enables the model to focus on key damage features while integrating global contextual information, reducing background interference, improving detection efficiency and reliability, and enabling stronger generalization capabilities in complex environments.

[0037] In addition to complex environmental factors, the wide variety of manhole covers and their diverse forms of damage also pose challenges to damage detection. Rainwater, sewage, power, and telecommunications manhole covers come in varying materials, shapes, and sizes, along with a wide range of damage types, including cracks, fractures, and corrosion. This requires models to possess greater adaptability and generalization capabilities. The complex environment, combined with the diverse manhole covers and damage patterns, further increases the difficulty of detection.

[0038] In some embodiments, to address this challenge, the neck network may include n feature fusion modules at different scales, where n is a positive integer not less than 2. Each feature fusion module is equipped with a multi-scale weighted feature fusion (MSWFF) module to align and weight the initial fusion features output by the feature fusion module to generate target fusion features corresponding to each scale.

[0039] The MSWFF module combines features at different scales to enhance the capture of both local details and global context, thereby improving the model's recognition of various damage types. This mechanism ensures the detection model can accurately adapt to different scenarios, even in complex environments with diverse manhole covers and damage patterns, achieving more efficient and reliable manhole cover damage detection.

[0040] In some embodiments, the MSWFF module consists of a scale alignment layer, a convolutional stitching layer, an ECA layer, and an ESA layer. The scale alignment layer lays the foundation for the effective fusion of features at different scales, while the convolutional stitching layer further integrates multi-scale information to achieve full feature interaction. Through adaptive adjustment of the ECA and ESA layers, the module accurately allocates channel weights and spatial weights, enhancing the ability to capture local details and global contextual information, ensuring effective identification of all types of damage.

[0041] The network structure design of the MSWFF module not only improves the robustness and accuracy of detection, but also can efficiently handle multi-scale problems ranging from subtle defects to large-scale damage, making the model more adaptable and generalizable in complex environments.

[0042] The diverse nature of manhole cover damage makes detailed features unclear, a major challenge in manhole cover damage identification, especially in complex real-world scenarios. Due to factors such as material differences, aging, obstruction by stains, and varying lighting, surface damage features such as cracks and deformations on manhole covers are often difficult to clearly capture and accurately distinguish, leading to increased uncertainty in feature extraction and classification for detection algorithms.

[0043] In some embodiments, to address uncertainty in feature extraction and classification, the backbone network introduces a Convolution-Space-to-Depth Feature Fusion (CSFF) module. The CSFF module consists of a convolution branch, a SPD convolution branch, a second concatenation layer, an Effective Squeeze and Extraction (ESE) layer, and a second convolution layer.

[0044] By extracting features from two branches, the CSFF module simultaneously captures local details and multi-scale contextual information, effectively extracting subtle and complex damage features from manhole cover surfaces. This ensures efficient and accurate feature extraction even in the presence of interference from stains, lighting changes, and other factors in the detection image. The ESE layer adaptively adjusts between channels, highlighting important features and suppressing redundant information, further enhancing focus on subtle features. Ultimately, the fused features are mapped to a higher-dimensional space, increasing feature richness and differentiation, thereby highlighting the boundaries between different damage types and helping to accurately distinguish between different types of manhole cover damage.

[0045] In some embodiments, the manhole cover detection model can be improved based on the YOLO series models, given the following advantages of the YOLO series models:

[0046] Both are end-to-end, single-stage detection frameworks, enabling efficient object detection. Compared to two-stage detectors (such as Faster R-CNN), YOLO directly regresses bounding boxes and categories through a single forward pass, improving detection speed and making it suitable for real-time applications. Its architecture has been continuously optimized, with features such as the introduction of anchor-free mechanisms and feature pyramids (FPN and PAN), enhancing its detection capabilities for small and multi-scale objects. Furthermore, the YOLO series models have been continuously optimized for lightweightness, computational efficiency, and robustness, making them highly adaptable to both embedded devices and cloud-based inference scenarios. YOLOv8 is a major upgrade to the YOLO series, supporting tasks such as object detection, image classification, and instance segmentation. Its architecture consists of a backbone network, a neck network, and a detection head. The backbone network uses a C2f module to improve feature extraction efficiency, while the neck network utilizes a PANet architecture to enhance multi-scale feature fusion. The detection head incorporates an anchor-free design and a Dependent Fluent (DFL) loss to improve detection accuracy and flexibility. In addition, YOLOv8 offers five model variants: n / s / m / l / x, adapting to different scenarios. For the manhole cover damage detection task, YOLOv8s was selected for optimization to balance accuracy and real-time performance, improving detection results.

[0047] Based on this, YOLOv8s can be preferred when building a manhole cover detection model. For example, if the YOLOv8s basic network is improved through the MSRGA submodule, MSWFF module and CSFF module, the network structure of the manhole cover detection model can be referred to. Figure 1 .

[0048] The backbone network of this manhole cover detection model is equipped with feature extraction modules of different scales. Each feature extraction module of each scale is equipped with an MSRGA submodule. The MSRGA submodule captures multi-scale damage features of different shapes and sizes based on two attention mechanisms, channel and space, and simultaneously considers contextual information, which can reduce background interference and improve the detection effect of manhole cover damage in complex backgrounds. The backbone network is also equipped with a CSFF module. The CSFF module simultaneously extracts local details and multi-scale contextual information through convolution branches and SPD convolution branches to effectively capture subtle and complex damage features on the surface of manhole covers, enhance feature representation, and improve the recognition accuracy and reliability of manhole cover damage. In addition, the neck network is equipped with an MSWFF module after the feature fusion module of each scale, which helps to combine the initial fusion features of different scales to enhance the local detail information and global contextual information of local features, thereby improving the recognition ability of different types of manhole covers corresponding to various forms of damage.

[0049] In some embodiments, to ensure that the trained manhole cover detection model can meet expectations and accurately understand and detect manhole covers of different scales in different scenarios, a dedicated dataset can be created for training the manhole cover detection model. Specifically, a dataset called Manhole Cover Damage (MCD) is created. This dataset is collected by a vehicle-mounted camera under various environmental conditions and includes a total of 2,000 images with a resolution of 1280×720, each of which is manually annotated. Figure 2 , Figure 2 An example image of a training sample is shown.

[0050] For example, in order to provide a high-quality dataset, in addition to ensuring the number of images, images covering different scenes such as urban roads, squares, parks, and areas around buildings can also be collected.

[0051] For example, the image may contain common obstructions, such as vehicles, people, vegetation, warning objects (warning triangles), etc.

[0052] For example, the types of manhole covers in the image should cover as many types of manhole covers as possible, including those for rainwater, sewage, electricity, and communications, of different materials, shapes, and sizes.

[0053] For example, the types of manhole cover damage in the image should cover as many types of damage as possible, such as cracks, fractures, corrosion, etc. of different shapes and sizes.

[0054] For example, each scene can include multiple time periods and weather conditions such as day, night, sunny, rainy and foggy days to increase the complexity and comprehensiveness of the scene. Each image can be configured with a corresponding txt label file that details the location and category of the manhole cover.

[0055] Such datasets can support research and applications in areas such as intelligent transportation systems, autonomous vehicles, and road safety monitoring to improve road safety and traffic efficiency.

[0056] In order to ensure the effectiveness of model training and the reliability of evaluation results, each image in the dataset of this application is divided into a training set and a validation set in a ratio of 8:2.

[0057] Preferably, when dividing the dataset, pay special attention to maintaining a balanced sample distribution between the training set and the validation set, ensuring that both have similar statistical characteristics in terms of manhole cover type, size, background environment, etc. This can avoid evaluation bias caused by uneven sample distribution and make the validation results more representative and reliable.

[0058] In some embodiments, in order to comprehensively and accurately measure the performance of each version of the manhole cover detection model, after at least one version of the manhole cover detection model is converged based on the training set, the converged model can be evaluated through the validation set to avoid model overfitting, verify the generalization of the model, and ensure that the model can run stably after deployment.

[0059] Specifically, the performance of the manhole cover detection model on the validation set can be evaluated according to preset conditions.

[0060] For example, these preset conditions may include performance indicators such as the intersection-over-union ratio, detection accuracy, and recall rate of the manhole cover detection device for manhole cover damage.

[0061] IoU is an indicator that evaluates the degree of overlap between the predicted bounding box and the true bounding box. It calculates the ratio of the intersection and union of the predicted bounding box and the true bounding box. It plays a key role in determining whether it is a correct detection. The calculation formula of IoU can be written as:

[0062]

[0063] Precision (p), also known as the precision rate, refers to the ratio of correct positive predictions to all positive predictions, as shown in the formula:

[0064]

[0065] Recall (R), also known as the recall rate, refers to the ratio of correctly predicted positive results to all actual positive results, as shown in the formula:

[0066]

[0067] Average Precision (AP) is calculated from precision and recall. A line graph of precision is drawn based on the recall value, and the area under the line is calculated, as shown in the formula:

[0068]

[0069] The mean average precision (mAP) refers to the average of the average precision AP of C different manhole cover damage categories, as shown in the formula:

[0070]

[0071] That is to say, after verifying each version of the manhole cover detection model through the validation set, the manhole cover detection model of each version can be comprehensively evaluated based on the above indicators, so as to determine the manhole cover detection model with the best performance from each version as the trained manhole cover detection model.

[0072] In some embodiments, the model runs on an Intel Xeon Platinum 8255C processor, 314GB of RAM, an NVIDIA Tesla V100 32GB graphics card, and a CentOS 8.5.2 (64-bit) operating system. The deep neural network is built using the PyTorch framework, with input image dimensions of [640, 640], and a multi-scale training strategy. The experiment uses a batch size of 64, 300 epochs, and an AdamW optimizer with an initial learning rate of 0.01 and a cosine decay strategy for optimization.

[0073] In some embodiments, manhole cover detection model training combines classification and regression loss optimization. The classification loss uses binary cross entropy loss (BCE Loss) to determine the anchor box category; the regression loss consists of SIoU Loss and DFLLoss to measure the error between the predicted box and the ground-truth box. The positive and negative sample matching strategy uses TAL dynamic matching to optimize target allocation and improve detection accuracy.

[0074] Based on the network structure of the manhole cover detection model in the previous embodiment, this application proposes a manhole cover damage detection method based on deep learning.

[0075] The deep learning-based manhole cover damage detection method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0076] In order to illustrate the technical solution proposed in this application, each embodiment will be described below using an electronic device as the execution entity.

[0077] Figure 3 A schematic flow chart of a deep learning-based manhole cover damage detection method provided by the present application is shown. The deep learning-based manhole cover damage detection method includes:

[0078] Step 310: The electronic device extracts features from the image to be detected based on the backbone network of the pre-trained manhole cover detection model.

[0079] Step 320: The electronic device fuses the image features based on the neck network of the manhole cover detection model to obtain fused features.

[0080] Step 330: The electronic device detects the fused features based on the detection network of the manhole cover detection model to obtain a detection result of the manhole cover in the image to be detected.

[0081] In this embodiment, to ensure the accuracy of manhole cover damage detection in complex environments, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2. By introducing the MSRGA submodule into the feature extraction module, multi-scale features are extracted and fused through the channel attention mechanism and the spatial attention mechanism, damage features of different sizes and shapes are highlighted and enhanced, and the representation ability of the backbone network is enhanced to reduce the interference of complex backgrounds on manhole cover damage identification. The image features extracted by this backbone network will contain rich and discriminative manhole cover damage features, so that feature fusion based on them can more comprehensively integrate and identify effective information for manhole cover damage, significantly improving the detection accuracy of manhole cover damage in complex scenarios.

[0082] In some embodiments, the MSRGA submodule includes a multi-scale ECA fusion block and a multi-scale ESA fusion block, and the multi-scale ECA fusion block and the multi-scale ESA fusion block are connected based on a residual structure. For the first input feature input to the MSRGA submodule, the electronic device may perform the following operations:

[0083] Step A1: The electronic device performs an ECA-based multi-scale weighted fusion operation on the first input feature through a multi-scale ECA fusion block to obtain a first fused feature.

[0084] Step A2: The electronic device performs an ESA-based multi-scale weighted fusion operation on the first fusion feature through a multi-scale ESA fusion block to obtain a second fusion feature.

[0085] Step A3: The electronic device concatenates the second fusion feature with the first input feature through a residual structure to obtain a first output feature corresponding to the first input feature.

[0086] The multi-scale ECA fusion block captures both global information and local details, while the multi-scale ESA fusion block with multi-scale convolution captures deeper feature information. This allows the serialized multi-scale ECA and ESA fusion blocks to dynamically fuse global and local deep features, focusing on capturing multi-scale damage signatures of varying sizes and shapes. Furthermore, residual connections are combined to preserve the original information and learn complex mapping relationships, ensuring accurate detection of manhole cover damage even in complex backgrounds.

[0087] In some embodiments, the multi-scale ECA fusion block includes a first convolutional layer, a multi-scale feature extraction structure, and an ECA structure; the multi-scale feature extraction structure includes feature extraction layers at at least two scales; for the first input feature, the electronic device may perform the following operations:

[0088] Step A11: The electronic device performs a channel compression operation on the first input feature through the first convolution layer to obtain a first convolution feature.

[0089] The first convolutional layer can use a 1×1 convolution combined with the Sigmoid Linear Unit (SiLU) activation function. The 1×1 convolution is used to adjust the number of channels and fuse local information, while the SiLU activation function enhances gradient flow and improves nonlinear expression capabilities, contributing to efficient and effective feature extraction.

[0090] Step A12: The electronic device sequentially performs adaptive pooling operations and depthwise separable convolution operations on the first convolutional features through the feature extraction layers at each scale to obtain convolutional features at each scale.

[0091] To extract global features at different scales, a feature extraction layer with different kernels can be used. This feature extraction layer consists of adaptive pooling and depthwise separable convolution. Taking the extraction of features at three scales as an example, the kernel sizes of the feature extraction layer can be set to 1, 3, and 5, respectively. The adaptive pooling and depthwise separable convolution corresponding to each convolution kernel extract the first convolution feature, thereby obtaining convolution features at the corresponding scale. Adaptive pooling is used to extract global features at different scales, while depthwise separable convolution further enhances the expressiveness of local features. These two operations yield fused global and local features at the corresponding scales (1, 3, and 5), effectively capturing multi-scale information. This multi-scale feature extraction approach fully captures both global and local information, enhancing the model's multi-scale feature expression capabilities.

[0092] Step A13: The electronic device performs an ECA operation on the convolution features at each scale through the ECA structure, and performs weighted fusion on the first input features using the obtained channel weights to obtain a first fused feature.

[0093] The electronic device performs channel attention on convolution features of different scales through the ECA structure to calculate channel weights; then, the calculated channel weights are used to perform weighted fusion on the first input features to enhance the information expression of important channels and suppress the influence of secondary channels, thereby obtaining the optimized first fusion feature.

[0094] In this embodiment, the electronic device effectively captures both global and local information through adaptive pooling at different scales and depthwise separable convolution within a multi-scale feature extraction architecture. Furthermore, channel weights are calculated within the ECA architecture to enhance the representation of important features. Ultimately, a weighted fusion optimization results in a first fused feature, improving the model's accuracy and robustness in identifying manhole cover damage.

[0095] In some embodiments, the multi-scale ESA fusion block includes a pooling structure, a first splicing layer, a parallel convolution structure, and an ESA structure; for the first fusion feature, the electronic device performs the following steps:

[0096] Step A21: The electronic device performs an average pooling operation and a maximum pooling operation on the first fusion feature through a pooling structure to obtain a first pooling result and a second pooling result.

[0097] Average pooling smoothes features and preserves overall information, while maximum pooling emphasizes significant features and focuses on key areas. Using these two pooling methods, the device obtains both the first and second pooling results, providing the foundation for subsequent feature fusion.

[0098] Step A22: The electronic device splices the first pooling result and the second pooling result through the first splicing layer to obtain a first splicing feature.

[0099] This splicing operation can effectively combine the information extracted by different pooling methods, making the feature expression richer and providing diverse information input for subsequent convolution operations.

[0100] Step A23: The electronic device performs at least two convolution operations on the first splicing feature through a parallel convolution structure to obtain at least two corresponding convolution results.

[0101] The parallel convolution architecture can include convolutional layers with different kernel sizes. Through different convolution operations, multi-level and multi-scale feature representations can be obtained. These convolution operations can include convolutions of different sizes. The convolutions can be standard convolutions, depthwise separable convolutions, or dilated convolutions to enhance the ability to capture local details and global information, ultimately resulting in at least two different convolution results.

[0102] Step A24: The electronic device performs an ESA operation on each convolution result through the ESA structure, and performs weighted fusion on the first fusion feature with the obtained spatial weight to obtain a second fusion feature.

[0103] The electronic device uses the ESA structure to perform an ESA operation on each convolution result to calculate spatial weights. These weights are then used to weightedly fuse the first fused features to produce the second fused features. The ESA structure adaptively adjusts spatial attention to increase focus on key areas. The ESA mechanism enhances the selectivity of channel information, thereby improving the overall feature representation and enhancing the accuracy and robustness of detection tasks.

[0104] In this embodiment, the electronic device achieves multi-level feature extraction and optimization of the first fused feature through operations such as pooling, concatenation, parallel convolution, and spatial attention fusion. The combination of average and maximum pooling helps preserve overall information while highlighting key features. The concatenation operation fuses different pooling results, enhancing the richness of feature expression. The parallel convolution structure uses multiple convolution operations to obtain multi-scale features, improving the ability to capture deep local details and global information. Finally, the ESA structure, combined with the ESA mechanism, calculates spatial weights and performs weighted fusion on the first fused feature, further enhancing the focus on key information to obtain the second fused feature, thereby improving the model's feature expression capability and detection accuracy.

[0105] In some embodiments, Figure 4 The network structure diagram of the MSRGA submodule is shown. Figure 4 The network structure of , first perform channel dimensionality reduction to reduce the amount of calculation and highlight important features to obtain the first convolution feature Y :

[0106]

[0107] in, It is a 3×3 convolution layer used to convert the number of channels of the input C Reduce to , σ is the SiLU activation function, which is used to enhance feature nonlinearity.

[0108] Extracting structure pairs through multi-scale features Y Perform multi-scale processing, that is, use adaptive pooling of different scales and corresponding depth convolution to extract multi-scale context information:

[0109]

[0110] The kernel size is Adaptive pooling operation is used to extract global and local context. It is a depth-separable convolution of the corresponding kernel size, which further captures the local features after pooling. These features have different receptive fields. , focusing on the local and global information of the features respectively, and obtaining the convolution features at each scale. After splicing the convolution features at different scales, we get the spliced ​​features , through 1×1 convolution Restore the original number of channels and perform the channel attention mechanism:

[0111]

[0112] The channel attention mechanism determines the channel weights that include the weights of each channel , and X The weighted fusion operation is performed on each channel feature to obtain the first fusion feature X MSCA :

[0113]

[0114] where ⊙ represents element-wise multiplication.

[0115] based on X MSCA Calculate the global average pooling and global maximum pooling to obtain the spatial features of the two channels. The first pooling result and the second pooling result :

[0116]

[0117] Then, the two pooling results are spliced ​​to obtain the first splicing feature :

[0118]

[0119] exist Multi-scale convolution is applied to extract features of different receptive fields, that is, the convolution results , i= {1,3,5,7}:

[0120]

[0121] After the convolution results are concatenated, a 1×1 convolution is performed to generate a spatial attention mechanism:

[0122]

[0123] The spatial attention mechanism can weight each spatial position of the concatenated convolution results to obtain the second fusion feature :

[0124]

[0125] Combining channel attention and spatial attention, while adding residual connections, the residual connections can retain the original features X , while alleviating the vanishing gradient problem and ensuring the effectiveness of the attention module.

[0126]

[0127] The MSRGA submodule extracts and captures multi-scale damage features of varying sizes and shapes. It also incorporates residual connections to preserve original information and learn complex mapping relationships, effectively detecting manhole cover damage in complex environments. A global attention mechanism enables the model to focus on key damage features while simultaneously considering the overall context, reducing background interference and improving the robustness of feature representation, enabling more efficient and reliable detection results in complex environments.

[0128] In some embodiments, the neck network includes n feature fusion modules of different scales, where n is a positive integer and n≥2; each feature fusion module corresponds to a MSWFF module, and the MSWFF modules corresponding to each scale are used to align and weightedly fuse the initial fusion features output by each feature fusion module to obtain the target fusion features corresponding to each scale.

[0129] Exemplarily, n can be consistent with the aforementioned m, that is, the number of feature extraction modules set in the backbone network is the same as the number of feature fusion modules set in the neck network.

[0130] In some embodiments, the MSWFF module includes a scale alignment layer, a convolutional splicing layer, an ECA layer, and an ESA layer. The MSWFF module performs the following steps on the initial fused features output by each feature fusion module:

[0131] Step B1: At the current scale, the electronic device aligns the scales of the initial fusion features with the current scale through the scale alignment layer to obtain aligned features.

[0132] The electronic device uses the scale alignment layer to adjust the scale of each initial fused feature to align it with the current scale of the MSWFF module, resulting in aligned features. This alignment ensures that when subsequent features are fused, they effectively match in spatial dimensions, avoiding information loss or uneven fusion due to scale differences.

[0133] Step B2: The electronic device performs a convolution operation and a splicing operation on each alignment feature through a convolutional splicing layer to obtain a second splicing feature.

[0134] The convolution operation is used to extract local features and enhance the representation ability, while the splicing operation integrates features from different sources to improve the expressive richness of the features, thereby obtaining the second splicing feature.

[0135] Step B3: The electronic device performs a weighted fusion operation based on a channel attention operation on the second splicing feature through the ECA layer to obtain a third fusion feature, and splices the second splicing feature with the third fusion feature to obtain a third splicing feature.

[0136] The electronic device performs a weighted fusion operation on the second concatenated features through the ECA layer, using a channel-attention mechanism. This dynamically adjusts the importance of different channels, highlights key features, and suppresses redundant information. The second concatenated features are then concatenated with the fused third fused features to enrich the feature expression and provide a more comprehensive feature representation, resulting in the third concatenated features.

[0137] Step B4: The electronic device performs a spatial attention-based weighted fusion operation on the third fusion feature operation through the ESA layer to obtain the target fusion feature at the current scale.

[0138] The electronic device performs a weighted fusion operation based on spatial attention on the third fused feature through the ESA layer. This mechanism enhances the model's focus on key areas and reduces interference from irrelevant background. Ultimately, a more discriminative fused feature of the target is extracted at the current scale, providing a more accurate feature representation for subsequent detection tasks.

[0139] It can be considered that the MSWFF module includes MSWFF branches of different scales, which correspond one-to-one to each initial fusion module.

[0140] In this embodiment, the electronic device first adjusts the scale of the initial fused features through the scale alignment layer to match the current scale of the MSWFF module, ensuring spatial alignment of the features. Subsequently, the convolutional stitching layer performs convolution and stitching operations on the aligned features to enhance feature representation. Next, the ECA layer uses a channel-wise attention mechanism to perform a weighted fusion of the stitched features, highlighting key channel information. These features are then further fused with the original stitched features to obtain a richer feature representation. Finally, the ESA layer uses a spatial attention mechanism to weight the fused features, focusing the model more closely on key areas and reducing background interference, thereby obtaining more discriminative target fused features at the current scale.

[0141] In some embodiments, the neck network can output fused object features corresponding to each scale, and a separate decoupled detection head is set up for each scale to optimize the independence of classification and localization tasks. Accordingly, referring to Figure 1, the detection head processes the fused object features of different scales separately. The classification branch improves category discrimination, while the regression branch optimizes bounding box prediction accuracy. This enhances the model's detection capabilities for objects of different sizes, improving detection accuracy and stability.

[0142] In some embodiments, Figure 5 Figure 2 shows the network structure diagram of the MSWFF module. Figure 5 The network structure of MSWFF branches of different scales has the same subsequent operations except for the difference in scale alignment.

[0143] The initial fusion features for three different scales include: High-resolution initial fusion features: , with a resolution of ; Medium resolution initial fusion features: , with a resolution of ; Low-resolution initial fusion features: , with a resolution of For ease of description, the MSWFF branches corresponding to high, medium, and low resolutions can be denoted as P3 branch, P4 branch, and P5 branch, respectively.

[0144] First, the features of different branches are resized and channel compressed through spatial downsampling, upsampling interpolation or convolutional layers to complete feature alignment:

[0145]

[0146] For each initial fusion feature, the same feature alignment operation is different in different branches. Specifically:

[0147] P3 branch:

[0148]

[0149] P4 branch:

[0150]

[0151] P5 branch:

[0152]

[0153] The alignment features of each branch input can be expressed as 、 as well as , for different branches, i The value of is different. i ={3, 4, 5}.

[0154] Since the operations performed by subsequent branches are consistent, each branch is abstracted as a general branch for description, that is, each branch performs the operations performed by the general branch.

[0155] Splice the alignment features corresponding to the general branch to obtain the second splicing feature F i ; Through the efficient channel attention mechanism ECA F i Perform weighted fusion operation to obtain the third splicing feature F ECA-i :

[0156]

[0157] in, GAP is global average pooling, is a 1×1 convolution, σ is Sigmoid Activation function.

[0158] Then, the spatial weights are generated by efficient spatial attention ESA W i :

[0159]

[0160] in: It is a 1×1 convolution.

[0161] Finally, using spatial weights Fusion F ECA , for the third fusion feature after fusion Perform dimension expansion and output the corresponding target fusion features.

[0162]

[0163] It can be understood that in the mathematical expression of the above general branch, i The parameters depend on the specific branch. i Different values ​​indicate corresponding parameters of different branches.

[0164] In some embodiments, the backbone network further includes a CSFF module, the CSFF module including a convolution branch, an SPD convolution branch, a second splicing layer, an ESE layer, and a second convolution layer; for the second input feature of the CSFF module:

[0165] Step C1: The electronic device performs a convolution operation on the second input feature through a convolution branch to obtain a second convolution feature.

[0166] The electronic device performs a standard convolution operation on the second input feature through the convolution branch to extract local features and enhance feature expression capabilities, thereby obtaining a second convolution feature, which provides a basis for subsequent feature fusion.

[0167] Step C2: The electronic device performs a space-to-depth conversion operation and a convolution operation on the second input feature through the SPD convolution branch to obtain a rearranged feature.

[0168] The electronic device performs a space-to-depth conversion operation on the second input feature through the SPD convolution branch to rearrange the feature space distribution, improve the compactness of the feature expression, and extract deep information in combination with the convolution operation to obtain the rearranged feature.

[0169] Step C3: The electronic device performs a splicing operation on the second convolution feature and the rearrangement feature through the second splicing layer to obtain a fourth splicing feature.

[0170] The electronic device splices the second convolutional feature and the rearrangement feature through the second splicing layer to integrate the advantages of the two features, form a richer representation capability, and obtain the fourth splicing feature.

[0171] In step C4, the electronic device performs a global average pooling operation and a convolution operation on the fourth splicing feature through the ESE layer, and weights the fourth splicing feature with the calculated channel attention weight to obtain an initial loss feature.

[0172] The electronic device performs a global average pooling operation and a convolution operation on the fourth spliced ​​feature through the ESE layer, calculates the channel attention weight, and weights the fourth spliced ​​feature to highlight the key features and suppress redundant information to obtain the initial loss feature.

[0173] Step C5: The electronic device convolves the initial loss feature through a second convolutional layer to obtain a second output feature corresponding to the second input feature.

[0174] The electronic device convolves the initial loss features through the second convolutional layer to further optimize the feature expression, obtains the second output features corresponding to the second input features, and finally constructs the image features based on the fused multi-scale features and the second output features, providing more accurate representation information for the detection task.

[0175] In this embodiment, the electronic device sequentially performs standard convolution and SPD transformation on the second input features to extract local features and rearrange spatial information, thereby enhancing feature representation. Subsequently, features from different branches are fused through a concatenation operation to enrich feature representation. Next, the ESE layer utilizes global average pooling and channel-wise attention weighting to highlight key information and optimize feature distribution. Finally, the second output features are further processed by the second convolutional layer to obtain the final second output features. These features are then combined with multi-scale features to construct image features, providing more accurate feature representation for detection tasks.

[0176] In some embodiments, Figure 6 The network structure diagram of the CSFF module is shown. Figure 6 The network structure of the second input feature , the convolution branch reduces the input second input feature by a 3×3 convolution operation with a stride of 2 X The size of the channel is compressed C comp , get the second convolution feature .

[0177]

[0178] At the same time, the SPD convolution branch is used to input X Perform space-to-depth conversion operations to avoid the loss of fine-grained spatial information, and use a 3×3 convolution to compress the number of channels and integrate features.

[0179]

[0180] Next, the output second convolution feature of the above two branches is and rearrangement features Merge to get the fourth splicing feature Y ECA , the ESE layer calculates the channel attention weights through global average pooling and a 1×1 convolution, and applies the channel attention weights to Y ECA On top, we get the initial loss characteristics.

[0181]

[0182] Finally, a 3×3 convolutional layer is used to map the initial loss feature into a higher-dimensional space to obtain the final second output feature. .

[0183]

[0184] In some embodiments, the detection methods of the above-mentioned embodiments are implemented based on a trained manhole cover detection model. The manhole cover detection model is provided with feature extraction modules of different scales in the backbone network, and each scale feature extraction module is provided with an MSRGA submodule. The MSRGA submodule captures multi-scale damage features of different shapes and sizes based on two attention mechanisms, channel and space, and simultaneously considers contextual information, which can reduce background interference and improve the detection effect of manhole cover damage under complex backgrounds. The backbone network is also provided with a CSFF module. The CSFF module extracts local details and multi-scale contextual information simultaneously through convolution branches and SPD convolution branches to effectively capture subtle and complex damage features on the surface of the manhole cover, enhance feature representation, and improve the recognition accuracy and reliability of manhole cover damage. In addition, the neck network is provided with an MSWFF module after the feature fusion module of each scale, which helps to combine the initial fusion features of different scales to enhance the local detail information and global contextual information of the local features, thereby improving the recognition ability of various damage forms of different types of manhole covers.

[0185] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0186] Corresponding to the manhole cover damage detection method based on deep learning in the above embodiment, Figure 7 A structural block diagram of the manhole cover damage detection device 7 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0187] Reference Figure 7 , the manhole cover damage detection device 7 includes:

[0188] An extraction module 71 is configured to extract features from an image to be detected based on a pre-trained backbone network of a manhole cover detection model to obtain image features; the image to be detected includes a manhole cover;

[0189] A fusion module 72 is used to fuse image features based on the neck network of the manhole cover detection model to obtain fused features;

[0190] A detection module 73 is configured to detect the fused features based on the detection network of the manhole cover detection model to obtain a detection result of the manhole cover in the image to be detected;

[0191] Among them, the backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; the feature extraction module is provided with an MSRGA submodule, which extracts and fuses multi-scale features through the channel attention mechanism and the spatial attention mechanism, and the image features are obtained based on the fused multi-scale features.

[0192] Optionally, the MSRGA submodule includes a multi-scale ECA fusion block and a multi-scale ESA fusion block, and the multi-scale ECA fusion block and the multi-scale ESA fusion block are connected based on a residual structure; the extraction module 71 includes a first extraction unit, which is used to:

[0193] For the first input feature of the MSRGA submodule:

[0194] Performing an ECA-based multi-scale weighted fusion operation on the first input feature through a multi-scale ECA fusion block to obtain a first fused feature;

[0195] Performing an ESA-based multi-scale weighted fusion operation on the first fusion feature through the multi-scale ESA fusion block to obtain a second fusion feature;

[0196] The second fusion feature is concatenated with the first input feature through the residual structure to obtain the first output feature corresponding to the first input feature.

[0197] Optionally, the multi-scale ECA fusion block includes a first convolutional layer, a multi-scale feature extraction structure, and an ECA structure; the multi-scale feature extraction structure includes feature extraction layers at at least two scales; and the extraction unit is specifically configured to:

[0198] For the first input feature:

[0199] Performing a channel compression operation on the first input feature through the first convolutional layer to obtain a first convolutional feature;

[0200] The first convolution feature is sequentially subjected to adaptive pooling and depth-wise separable convolution operations through the feature extraction layer at each scale to obtain the convolution features at each scale.

[0201] The ECA operation is performed on the convolution features at each scale through the ECA structure, and the first input features are weightedly fused with the obtained channel weights to obtain the first fused features.

[0202] Optionally, the multi-scale ESA fusion block includes a pooling structure, a first splicing layer, a parallel convolution structure, and an ESA structure; the extraction unit is specifically used to:

[0203] For the first fusion feature:

[0204] Performing average pooling and maximum pooling operations on the first fusion feature through the pooling structure to obtain a first pooling result and a second pooling result;

[0205] The first pooling result and the second pooling result are spliced ​​together through the first splicing layer to obtain a first splicing feature;

[0206] Performing at least two convolution operations on the first concatenated features respectively through a parallel convolution structure to obtain at least two corresponding convolution results;

[0207] The ESA structure performs an ESA operation on each convolution result, and uses the obtained spatial weight to perform weighted fusion on the first fusion feature to obtain the second fusion feature.

[0208] Optionally, the neck network includes n feature fusion modules of different scales, where n is a positive integer and n≥2; each feature fusion module corresponds to a MSWFF module, and the MSWFF modules corresponding to each scale are used to align and weightedly fuse the initial fusion features output by each feature fusion module to obtain the target fusion features corresponding to each scale.

[0209] Optionally, the MSWFF module includes a scale alignment layer, a convolutional splicing layer, an ECA layer, and an ESA layer, and the fusion module 72 includes a fusion unit for:

[0210] The scale of each initial fusion feature is aligned with the current scale of the MSWFF module through the scale alignment layer to obtain each aligned feature;

[0211] The convolution operation and the splicing operation are performed on each aligned feature through the convolution splicing layer to obtain the second splicing feature;

[0212] Performing a weighted fusion operation based on a channel attention operation on the second spliced ​​feature through the ECA layer to obtain a third fused feature, and splicing the second spliced ​​feature with the third fused feature to obtain a third spliced ​​feature;

[0213] The third fusion feature operation is performed with a weighted fusion operation based on spatial attention through the ESA layer to obtain the target fusion feature at the current scale.

[0214] Optionally, the extraction module 71 includes a second extraction unit, configured to:

[0215] The backbone network also includes a CSFF module, which includes a convolution branch, an SPD convolution branch, a second splicing layer, an ESE layer, and a second convolution layer; for the second input feature of the CSFF module:

[0216] Performing a convolution operation on the second input feature through the convolution branch to obtain a second convolution feature;

[0217] The SPD convolution branch performs a space-to-depth conversion operation and a convolution operation on the second input feature to obtain a rearranged feature;

[0218] Performing a splicing operation on the second convolutional feature and the rearranged feature through the second splicing layer to obtain a fourth splicing feature;

[0219] Perform global average pooling and convolution operations on the fourth splicing feature through the ESE layer, and weight the calculated channel attention weight on the fourth splicing feature to obtain the initial loss feature;

[0220] The initial loss feature is convolved through the second convolutional layer to obtain a second output feature corresponding to the second input feature, and the image feature is obtained based on the fused multi-scale feature and the second output feature.

[0221] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0222] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided in one embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one processor is shown in the figure), a memory 81, and a computer program 82 stored in the memory 81 and executable on at least one processor 80. When the processor 80 executes the computer program 82, the steps in any of the above-mentioned embodiments of the method for detecting manhole cover damage based on deep learning are implemented, for example Figure 3 Steps 310-330 are shown.

[0223] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0224] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the electronic device 8.

[0225] Furthermore, the memory 81 may include both an internal storage unit of the electronic device 8 and an external storage device. The memory 81 is used to store operating devices, application programs, boot loaders, data, and other programs, such as program code of computer programs. The memory 81 may also be used to temporarily store data that has been output or is about to be output.

[0226] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0227] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps in the above-mentioned various method embodiments.

[0228] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0229] If this integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The above-mentioned computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The above-mentioned computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The above-mentioned computer-readable medium can include at least: any entity or device capable of carrying computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. Examples include a USB flash drive, a removable hard drive, a magnetic disk, or an optical disk.

[0230] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0231] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0232] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the above modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0233] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting manhole cover damage based on deep learning, characterized in that: include: The backbone network of the pre-trained manhole cover detection model extracts features from the image to be detected to obtain image features; The image to be detected includes a manhole cover; fusing the image features based on the neck network of the manhole cover detection model to obtain target fusion features; The detection network based on the manhole cover detection model detects the target fusion features to obtain a detection result of the manhole cover in the image to be detected; The backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; the feature extraction module is provided with an MSRGA submodule, which extracts and fuses multi-scale features through a channel attention mechanism and a spatial attention mechanism, and the image features are obtained based on the fused multi-scale features; The MSRGA submodule includes a multi-scale ECA fusion block and a multi-scale ESA fusion block, and the multi-scale ECA fusion block is connected with the multi-scale ESA fusion block based on a residual structure; for the first input feature of the MSRGA submodule: Performing an ECA-based multi-scale weighted fusion operation on the first input feature through the multi-scale ECA fusion block to obtain a first fused feature; Performing an ESA-based multi-scale weighted fusion operation on the first fused features through the multi-scale ESA fusion block to obtain a second fused feature; splicing the second fused feature with the first input feature through a residual structure to obtain a first output feature corresponding to the first input feature; The multi-scale ECA fusion block includes a first convolutional layer, a multi-scale feature extraction structure, and an ECA structure; the multi-scale feature extraction structure includes feature extraction layers at at least two scales; for the first input feature: Performing a channel compression operation on the first input feature through the first convolutional layer to obtain a first convolutional feature; Performing adaptive pooling operations and depth-wise separable convolution operations on the first convolutional features in sequence through the feature extraction layers at each scale to obtain convolutional features at each scale; An ECA operation is performed on the convolution features at each scale through the ECA structure, and the first input features are weightedly fused using the obtained channel weights to obtain the first fused features.

2. The method for detecting damage to a manhole cover according to claim 1, wherein: The multi-scale ESA fusion block includes a pooling structure, a first splicing layer, a parallel convolution structure, and an ESA structure; for the first fusion feature: Performing an average pooling operation and a maximum pooling operation on the first fusion feature through the pooling structure to obtain a first pooling result and a second pooling result; Splicing the first pooling result and the second pooling result through the first splicing layer to obtain a first splicing feature; Performing at least two convolution operations on the first splicing feature respectively through the parallel convolution structure to obtain at least two corresponding convolution results; The ESA operation is performed on each of the convolution results through the ESA structure, and the first fusion features are weightedly fused with the obtained spatial weights to obtain the second fusion features.

3. The method for detecting damage to a manhole cover according to claim 1 or 2, wherein: The neck network includes n feature fusion modules of different scales, where n is a positive integer and n≥2; each feature fusion module corresponds to an MSWFF module, which is used to align and weightedly fuse the initial fusion features output by each feature fusion module to obtain the target fusion features corresponding to each scale.

4. The method for detecting damage to a manhole cover according to claim 3, wherein: The MSWFF module includes a scale alignment layer, a convolutional splicing layer, an ECA layer, and an ESA layer. At each scale, the MSWFF module performs the following steps on the initial fusion features output by each feature fusion module: Aligning the scale of each of the initial fusion features with the current scale through the scale alignment layer to obtain each aligned feature; Performing a convolution operation and a splicing operation on each of the alignment features through the convolution splicing layer to obtain a second splicing feature; Performing a weighted fusion operation based on a channel attention operation on the second spliced ​​feature through the ECA layer to obtain a third fused feature, and splicing the second spliced ​​feature with the third fused feature to obtain a third spliced ​​feature; A spatial attention-based weighted fusion operation is performed on the third fusion feature operation through the ESA layer to obtain the target fusion feature at the current scale.

5. The method for detecting damage to a manhole cover according to any one of claims 1 or 2, wherein: The backbone network further includes a CSFF module, which includes a convolution branch, an SPD convolution branch, a second splicing layer, an ESE layer, and a second convolution layer; for the second input feature of the CSFF module: Performing a convolution operation on the second input feature through the convolution branch to obtain a second convolution feature; Performing a space-to-depth conversion operation and a convolution operation on the second input feature through the SPD convolution branch to obtain a rearranged feature; Performing a splicing operation on the second convolutional features and the rearranged features through the second splicing layer to obtain a fourth splicing feature; Performing a global average pooling operation and a convolution operation on the fourth splicing feature through the ESE layer, and weighting the fourth splicing feature with the calculated channel attention weight to obtain an initial loss feature; The initial loss feature is convolved by the second convolutional layer to obtain a second output feature corresponding to the second input feature, and the image feature is obtained based on the fused multi-scale feature and the second output feature.

6. A manhole cover damage detection device, characterized in that: include: An extraction module is used to extract features from the image to be detected based on the backbone network of the pre-trained manhole cover detection model to obtain image features; The image to be detected includes a manhole cover; A fusion module, configured to fuse the image features based on the neck network of the manhole cover detection model to obtain fused features; A detection module, configured to detect the fused features based on the detection network of the manhole cover detection model to obtain a detection result of the manhole cover in the image to be detected; The backbone network includes m feature extraction modules of different scales, where m is a positive integer and m≥2; the feature extraction module is provided with an MSRGA submodule, which extracts and fuses multi-scale features through a channel attention mechanism and a spatial attention mechanism, and the image features are obtained based on the fused multi-scale features; The MSRGA submodule includes a multi-scale ECA fusion block and a multi-scale ESA fusion block, wherein the multi-scale ECA fusion block is connected with the multi-scale ESA fusion block based on a residual structure; the extraction module includes a first extraction unit, wherein the first extraction unit is used to: For the first input feature of the MSRGA submodule: Performing an ECA-based multi-scale weighted fusion operation on the first input feature through the multi-scale ECA fusion block to obtain a first fused feature; Performing an ESA-based multi-scale weighted fusion operation on the first fused features through the multi-scale ESA fusion block to obtain a second fused feature; splicing the second fused feature with the first input feature through a residual structure to obtain a first output feature corresponding to the first input feature; The multi-scale ECA fusion block includes a first convolutional layer, a multi-scale feature extraction structure, and an ECA structure; the multi-scale feature extraction structure includes feature extraction layers at at least two scales; the first extraction unit is specifically used to: For the first input feature: Performing a channel compression operation on the first input feature through the first convolutional layer to obtain a first convolutional feature; Performing adaptive pooling operations and depth-wise separable convolution operations on the first convolutional features in sequence through the feature extraction layers at each scale to obtain convolutional features at each scale; An ECA operation is performed on the convolution features at each scale through the ECA structure, and the first input features are weightedly fused using the obtained channel weights to obtain the first fused features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the deep learning-based manhole cover damage detection method according to any one of claims 1 to 5 is implemented.

8. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the computer program is executed by a processor, the deep learning-based manhole cover damage detection method according to any one of claims 1 to 5 is implemented.

Images