Road surface scattering detection method and device based on deep learning, electronic equipment and program product

Abstract

The invention discloses a pavement throwing detection method and device based on deep learning, electronic equipment and a program product. The method is realized through a trained detection model, the model adopts an LMSADet detection head, a multi-scale feature extraction and space attention mechanism is introduced into a task branch, and multi-scale modeling is decoupled from a backbone network and a neck network and integrated to the detection head so as to fit a detection task to directly optimize local details and scale differences of a throwing target. In order to suppress background interference and improve the recognition effect of fuzzy boundaries, the neck network is added into an MSHA module so as to efficiently capture the semantic relation between the thrown object and the background and enhance the regional understanding ability. A C3ESP module is introduced into the backbone network, deep features are extracted through stacking depth separable convolution, and information loss is avoided in combination with residual optimization fusion; meanwhile, a PEMA attention mechanism is introduced, the importance of different receptive field features is dynamically adjusted, the model focuses on key features, data information is captured more comprehensively, and therefore the detection performance is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a road surface throwing detection method based on deep learning, a road surface throwing detection device, an electronic device and a computer program product. BACKGROUND

[0002] With the continuous advancement of urbanization process, municipal infrastructure construction is gradually improved. Road surface throwing, as an indispensable important part of urban roads, its safety is directly related to the travel and life and property safety of citizens. Road surface throwing not only bears important functions such as drainage and traffic, but also plays an important role in protecting underground pipelines, maintaining road flatness and ensuring city operation. However, with the increase of the service life of road surface throwing, the change of environmental conditions and the influence of external damage, road surface throwing displacement, damage and even loss frequently occur, which brings serious hidden dangers to pedestrians, vehicles and public safety.

[0003] At present, although deep learning can provide new technical support for automatic detection of road surface throwing damage, because road surface throwing is distributed in diversified scenes such as urban roads, squares, parks and building periphery, the images collected in different scenes are easily affected by the corresponding environment, which reduces the accuracy of road surface throwing detection. SUMMARY The present application provides a road surface throwing detection method based on deep learning, a road surface throwing detection device, an electronic device and a computer program product, which can enhance the perception ability of the model to small targets in complex background, and thus improve the accuracy of road surface throwing detection.

[0004] In a first aspect, the present application provides a road surface throwing detection method based on deep learning, comprising: extracting features of an image based on a backbone network of a pre-trained road surface throwing detection model, to obtain image features; the image to be detected includes a road surface; fusing the image features based on a neck network of the road surface throwing detection model, to obtain target fusion features; detecting the target fusion features based on a detection network of the road surface throwing detection model, to obtain a detection result of road surface throwing in the image to be detected; Wherein, the detection network includes an LMSADet detection head, which enhances the target fusion features based on multi-scale fusion and spatial attention, and detects the enhanced target fusion features to obtain the detection result. In the case that the image to be detected includes road surface throwing, the detection result includes a predicted bounding box of road surface throwing and a predicted throwing category.

[0005] Optionally, the LMSADet detection head comprises a bounding box regression detection branch for outputting a predicted bounding box, and a category prediction branch for outputting a predicted scattering category, the branch network structures of the bounding box regression detection branch and the category prediction branch are the same; the branch network structure comprises a multi-scale convolution structure, a first splicing layer, a spatial attention structure, a channel fusion structure, a second splicing layer, a first DWCBS layer, a second DWCBS layer, and a first convolution layer; the target fusion feature input to the bounding box regression detection branch or the category prediction branch: The target fusion feature is subjected to at least two scale series convolution operations through the multi-scale convolution structure, to obtain series convolution features corresponding to each scale; the series convolution operation comprises a depth separable convolution operation and a convolution operation executed in sequence; The features corresponding to each scale are spliced in the channel dimension through the first splicing layer, to obtain first splicing features; The first splicing features are subjected to a channel compression operation, a channel expansion operation, and a normalization operation in sequence through the spatial attention structure, to obtain spatial attention weights corresponding to each scale; For each scale, the corresponding spatial attention weight and the corresponding mixed convolution feature are weighted and fused through the channel fusion structure, to obtain a corresponding spatial enhancement feature; The spatial enhancement features of each scale are spliced through the second splicing layer, to obtain attention perception features; The attention perception features are subjected to two DWCBS operations and a convolution operation in sequence through the first DWCBS layer, the second DWCBS layer, and the first convolution layer, to obtain a predicted bounding box or a predicted scattering category; The output channel number of the first convolution layer corresponding to the bounding box regression detection branch is set to The output channel number of the first convolution layer corresponding to the category prediction branch is set to n c .

[0006] Optionally, the neck network is provided with an MSHA module, the MSHA module comprises an MSCA submodule and an MSSA submodule, the MSCA submodule and the MSSA submodule are connected in series, the MSCA submodule is used for performing channel attention-based multi-scale feature extraction and fusion in the channel attention dimension; the MSSA submodule is used for performing spatial attention-based multi-scale feature extraction and fusion in the spatial dimension.

[0007] Optionally, the MSCA submodule comprises a first mixed pooling layer, a third splicing layer, a dimension adjustment layer, a multi-scale convolution layer, a fourth splicing layer, a second convolution layer, a first activation layer, and a first fusion layer; the first input feature input to the MSHA module: The first input feature is respectively subjected to an average pooling operation and a maximum pooling operation by the first mixed pooling layer to obtain a first pooling result and a second pooling result; The first pooling result and the second pooling result are spliced by the splicing layer to obtain a first mixed pooling result; The first mixed pooling result after the dimension adjustment is respectively subjected to different scale convolution operations by the multi-scale convolution layer to obtain first convolution features corresponding to each scale; The first convolution features of each scale are spliced by the third splicing layer to obtain third splicing features; The third splicing features subjected to the convolution of the second convolution layer are subjected to an activation operation by the first activation layer to obtain channel weights; The first input feature and the channel weights are fused by the fusion layer to obtain multi-scale channel enhancement features.

[0008] Optionally, the MSSA sub-module includes a second mixed pooling layer, a fifth splicing layer, a multi-scale convolution weighting structure, a sixth splicing layer, a second activation layer, and a second fusion layer; for the multi-scale channel enhancement features input into the MSSA sub-module: The multi-scale channel enhancement features are respectively subjected to an average pooling operation and a maximum pooling operation by the second mixed pooling layer to obtain a third pooling result and a fourth pooling result; The third pooling result and the fourth pooling result are spliced by the fifth splicing layer to obtain a second mixed pooling result; At each scale, the second mixed pooling result is sequentially subjected to a convolution operation and a weighting operation of the corresponding scale by the multi-scale convolution weighting structure to obtain a corresponding weighted feature; the weight corresponding to the weighting operation is obtained by learning; The weighted features of each scale are spliced by the sixth splicing layer to obtain a weighted fusion feature; The weighted fusion feature is subjected to an activation operation by the second activation layer to obtain a spatial weight; The multi-scale channel enhancement features and the spatial weight are fused by the fusion layer to obtain multi-scale channel-spatial enhancement features.

[0009] Optionally, the backbone network is provided with a C3ESP module, and the C3ESP module includes a third convolution layer, a residual branch, a multi-scale perception branch, a seventh splicing layer, and a fourth convolution layer; for the second input feature input into the C3ESP module: The second input feature is subjected to a channel compression operation by the third convolution layer to obtain a compressed feature; The compressed feature is subjected to a multi-scale deep feature extraction operation by the multi-scale perception branch to obtain a deep feature; The deep feature and the compressed feature are spliced by the seventh splicing layer to obtain seventh splicing features; The channel expansion operation is performed on the seventh spliced feature through a fourth convolutional layer to obtain a second output feature corresponding to a second input feature.

[0010] Optionally, the multi-scale perception branch includes n ESPBlocks connected in series, the ESPBlock includes a progressive depth separable convolution structure, an eighth splicing layer, a fifth convolutional layer, a PEMA layer, and a ninth splicing layer; for a third input feature input into the ESPBlock: The progressive multi-level depth separable convolution operation is performed on the third input feature through the progressive depth separable convolution structure to obtain depth convolution features of each level; the input corresponding to the first level is the third input feature, and the input of the non-first level is the depth convolution feature corresponding to the previous level; The depth convolution features of each level are spliced through the eighth splicing layer to obtain fused depth convolution features; The channel fusion operation is performed on the fused depth convolution features through the fifth convolutional layer; The mixed attention mechanism is performed on the fused depth convolution features after the channel fusion through the PEMA layer to obtain a mixed enhanced feature; The third input and the mixed enhanced feature are spliced through the ninth splicing layer to obtain a third output feature corresponding to the third input feature.

[0011] In a second aspect, the present application provides a road surface scattering detection device, comprising: An extraction module is configured to extract features of a to-be-detected image based on a backbone network of a pre-trained road surface scattering detection model to obtain image features, wherein the to-be-detected image includes a road surface. A fusion module is configured to fuse the image features based on a neck network of the road surface scattering detection model to obtain fused features. A detection module is configured to detect the fused features based on a detection network of the road surface scattering detection model to obtain a detection result of road surface scattering in the to-be-detected image. The detection network includes an LMSADet detection head, which enhances target fused features based on multi-scale fusion and spatial attention, and detects the enhanced target fused features to obtain the detection result. In the case where the to-be-detected image includes road surface scattering, the detection result includes a predicted bounding box of the road surface scattering and a predicted scattering category.

[0012] In a third aspect, the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the first aspect.

[0013] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the method of the first aspect.

[0014] In a fifth aspect, the present application provides a computer program product, the computer program product comprising a computer program, the computer program being executed by one or more processors to implement the steps of the method of the first aspect.

[0015] The present application has the beneficial effects that: the present scheme realizes the detection process based on a pre-trained road surface spill detection model. First, the backbone network extracts deep features of the input image, capturing multi-level representations of the road surface region and potential spill targets. Then, the neck network effectively fuses features of different levels to form target fusion features containing rich context information. Finally, the detection network analyzes the fusion features and outputs the detection results, which can give the corresponding prediction bounding box and prediction category in the presence of road surface spills.

[0016] Among them, the model introduces a lightweight multi-scale perception (Lightweight Multi-Scale Aware Detection, LMSADet) detection head. Unlike traditional methods that place multi-scale modeling in the backbone network or neck network, integrating multi-scale mechanisms into the detection head allows the model to be more directly oriented towards the detection task itself. This design can more accurately optimize the local details and scale differences of road surface spill targets, avoiding unnecessary global calculations at early and middle feature layers, thereby effectively reducing computational complexity while maintaining detection accuracy. Further combined with a spatial attention mechanism, the LMSADet detection head can highlight key regional features, suppress background interference, and improve the ability to identify fuzzy boundaries.

[0017] In summary, the model integrated with the LMSADet detection head not only improves the detection accuracy and robustness of road surface spills, but also significantly optimizes the computational efficiency, achieving efficient and lightweight precise detection, thereby achieving the purpose of balancing the real-time and accuracy requirements in practical applications.

[0018] It can be understood that the beneficial effects of the above-mentioned second aspect to the fifth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 is a network structure schematic diagram of a road surface scattering detection model provided by the embodiments of the present application; Figure 2 is a flowchart of a road surface scattering detection method based on deep learning provided by the embodiments of the present application; Figure 3 is a network structure schematic diagram of an LMSADet detection head provided by the embodiments of the present application; Figure 4 is a network structure schematic diagram of an MSHA module provided by the embodiments of the present application; Figure 5 is a network structure schematic diagram of a C3ESP module provided by the embodiments of the present application; Figure 6 is a structure schematic diagram of a road surface scattering detection device provided by the embodiments of the present application; Figure 7 is a structure schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary details.

[0022] Since the road surface scattering is distributed in various scenes such as urban roads, squares, parks, and building perimeters, the images collected in different scenes are easily affected by the corresponding environment, reducing the accuracy of road surface scattering detection.

[0023] In order to solve this problem, the present application researches and finds that in the related art, the multi-scale modeling method is mainly concentrated in the backbone network or the neck network (such as FPN, PANet structure, etc.), although it can improve the detection performance through multi-level feature fusion, but this kind of method often introduces a large amount of additional calculation overhead, including complex up-sampling, down-sampling or cross-layer connection operation, which is difficult to meet the real-time and lightweight requirements in the vehicle-mounted deployment scene.

[0024] To further meet the real-time and lightweight requirements in the vehicle deployment scenario, the application proposes a road litter detection model, which includes a backbone network, a neck network and a detection head. Specifically, the backbone network is used for feature extraction of the input image to obtain image features; the neck network is used for feature fusion of the image features to obtain target fusion features; and the detection head is used for detection of the target fusion features to obtain the detection result of the road litter in the image to be processed. In the presence of road litter, the corresponding prediction bounding box and prediction category can be given.

[0025] Among them, the road litter detection model adopts the LMSADet detection head, which integrates a multi-scale mechanism into the detection head. Compared with the traditional method of placing multi-scale modeling in the backbone network or the neck network, the model can be more directly oriented towards the detection task itself. The LMSADet detection head can more accurately optimize the local details and scale differences of the road litter target, avoid unnecessary global calculation in the early and middle feature layers, and effectively reduce the computational complexity while maintaining the detection accuracy. In addition, based on the spatial attention mechanism, the LMSADet detection head can highlight key regional features, suppress background interference, and improve the recognition ability of fuzzy boundaries.

[0026] As can be seen, the road litter detection model not only improves the detection accuracy and robustness of road litter, but also significantly optimizes the model calculation efficiency, realizes efficient and lightweight accurate detection, and meets the real-time and accuracy requirements in practical applications.

[0027] In some embodiments, the LMSADet detection head includes a bounding box regression detection branch for outputting a prediction bounding box, and a category prediction branch for outputting a prediction litter category. The branch network structures of the bounding box regression detection branch and the category prediction branch are the same, which can realize the sharing of different task-specific feature extraction methods, guarantee the consistency and sufficiency of feature utilization, and avoid redundant calculation caused by repeated construction of complex structures, thereby improving the detection efficiency.

[0028] To enable different branches to achieve different detection tasks, in the final output stage, the two branches are distinguished in the output channel number of the first convolutional layer: the output channel number of the first convolutional layer of the bounding box regression detection branch is set to , which is used to finely regress the position of the prediction bounding box; and the output channel number of the first convolutional layer of the category prediction branch is set to nc (total number of categories), which is used to output the prediction probability of each category. In this way, different task targets obtain consistent representation ability through the same feature processing path, and then complete their own prediction tasks with the help of the difference in the output channel number of the first convolutional layer.

[0029] The design of the branch network structure focuses on the cooperation of multi-scale and attention mechanism. By combining multi-scale convolution, splicing and spatial attention mechanism, the model can capture target details at different scales and highlight key areas while suppressing irrelevant backgrounds. In addition, in order to balance the lightweight design of the model and the detection accuracy, the DWCBS (Deep Separable Convolution + BatchNorm + Activation) structure and residual fusion are used to effectively reduce the amount of calculation while avoiding information loss and ensuring high-precision detection. Specifically, the branch network structure includes a multi-scale convolution structure, a first splicing layer, a spatial attention structure, a channel fusion structure, a second splicing layer, a first DWCBS layer, a second DWCBS layer, and a first convolution layer. In this embodiment, the LMSADet detection head realizes efficient decoupling of boundary box regression and class classification through the method of "shared enhancement module + differentiated output convolution", which can not only improve the precision and robustness of the model in road debris detection, but also ensure the lightweight of the model and the real-time of the detection.

[0030] In the scene of road debris identification, complex road environment and unclear ground conditions can cause significant interference, affecting the detection accuracy and reliability of the model. Various visual elements (such as traffic signs, lane lines, vehicle shadows, etc.) commonly seen on urban roads or highways may be similar to the appearance of debris in images, increasing the risk of false positives. In addition, changes in lighting conditions (such as day, night, and lighting differences under different weather conditions) can cause image quality to decline, and low-light conditions may produce noise or blurring, making it difficult to accurately identify small-sized debris. Changes in ground texture and color can also affect the visibility of debris, and dark or light-colored debris can be easily overlooked in the corresponding background. Partial occlusion and incomplete view (such as occlusion by other objects or limited camera angle) further increase the difficulty of identification, while ground stains and water reflection can cause false features, interfering with model judgment. Dynamic environmental changes (such as dust and fog generated during vehicle travel) can also make images unclear, reducing identification ability.

[0031] In some embodiments, to address the above problems, the neck network is provided with an MSHA module, which includes an MSCA submodule and an MSSA submodule. The MSCA submodule and the MSSA submodule are connected in series, the MSCA submodule is used to perform channel attention-based multi-scale feature extraction and fusion in the channel attention dimension, and the MSSA submodule is used to perform spatial attention-based multi-scale feature extraction and fusion in the spatial dimension. Through multi-scale feature extraction, global and local feature fusion, context information capture, and channel and spatial attention combination, the performance and reliability of the model are significantly improved.

[0032] In some embodiments, the MSCA submodule sequentially comprises a first mixed pooling layer, a third concatenation layer, a dimension adjustment layer, a multi-scale convolution layer, a fourth concatenation layer, a second convolution layer, a first activation layer, and a first fusion layer. This structure integrates multi-scale feature extraction into the channel attention and spatial attention mechanism, enabling the model to flexibly obtain key information under different receptive fields, improving the ability to balance details and global features, and enhancing robustness and discriminability in complex background environments.

[0033] In some embodiments, the MSSA submodule comprises a second mixed pooling layer, a fifth concatenation layer, a multi-scale convolution weighting structure, a sixth concatenation layer, a second activation layer, and a second fusion layer. This structure efficiently extracts global context information through adaptive pooling, while combining multi-scale convolution to obtain local detail features, achieving dynamic fusion of global and local features, thereby significantly enhancing feature expression ability and improving adaptability to complex scenes.

[0034] In some embodiments, in the MSHA module, the MSCA and MSSA submodules each undertake different feature enhancement tasks and form complementary advantages.

[0035] Among them, the MSCA submodule introduces multi-scale feature extraction into the channel attention and spatial attention mechanism, combining mixed pooling, concatenation, convolution, and fusion operations, enabling the model to simultaneously obtain local details and global context information under different receptive fields. This design not only strengthens key region features but also effectively suppresses background interference, improving the model's robustness in complex road surface environments.

[0036] The MSSA submodule, on the other hand, focuses on the dynamic fusion of global and local features. By extracting global semantics through adaptive pooling, and combining a multi-scale convolution weighting structure to extract local features, and through concatenation, activation, and fusion operations to achieve efficient complementarity of the two types of features. This module enables the model to capture large-scale context relationships while maintaining sensitivity to subtle litter features, thereby significantly enhancing overall feature expression ability.

[0037] The MSCA module focuses on attention enhancement under multi-scale receptive fields, highlighting key features and suppressing interference; the MSSA module focuses on dynamic balance of global and local features, ensuring information integrity and expression ability. The synergistic effect of the two modules enables the detection model to maintain lightweight while balancing precision, robustness, and efficiency, significantly improving the practical effect of road litter detection.

[0038] The size of the road surface scattering object ranges widely, from small stones to large piles of discarded construction materials. The images have large differences. The small scattering objects occupy a small number of pixels, and the model is difficult to detect and identify. The large piles of discarded construction materials cover most of the image area, and it is difficult to capture details. The model needs to capture the fine features of small-scale scattering objects and understand the overall distribution and morphological structure of large-scale scattering objects, which requires higher feature extraction network. Single-scale feature representation cannot balance detail and overall information, and effective feature extraction needs to switch between different scales to accurately identify and classify various scattering objects.

[0039] In some embodiments, to solve the above problems, a C3ESP module is introduced in the backbone network. The C3ESP module includes a third convolutional layer, a residual branch, a multi-scale perception branch, a seventh splicing layer, and a fourth convolutional layer. Through the multi-scale perception branch, the feature information is gradually extracted and deepened under the premise of ensuring computational efficiency. At the same time, the residual branch is used to realize efficient feature fusion, avoid information loss in the multi-layer transmission process, and thus improve the expression ability and robustness of the model to key features.

[0040] In some embodiments, the multi-scale perception branch includes n ESPBlocks connected in series, and the ESPBlock includes a progressive depth separable convolution structure, an eighth splicing layer, a fifth convolutional layer, a PEMA layer, and a ninth splicing layer. This structure gradually extracts and refines the features by stacking multiple 3×3 depth separable convolutions while ensuring computational efficiency. At the same time, the PEMA attention mechanism is innovatively introduced, which can dynamically adjust the importance of different receptive field features, so that the model can focus on the key area features and more comprehensively and accurately capture data information, thereby effectively improving the overall feature expression ability.

[0041] In some embodiments, the road surface scattering detection model can be improved based on the YOLO series model, because the YOLO series model has the following advantages: Both are end-to-end single-stage detection frameworks that can achieve efficient object detection. Compared with two-stage detectors such as Faster R-CNN, YOLO directly regresses bounding boxes and classes through a single forward pass, improving detection speed and making it suitable for real-time applications. Its structure is constantly optimized, such as the introduction of Anchor-Free mechanism, feature pyramid (FPN, PAN), etc., enhancing the detection ability of small and multi-scale targets. In addition, the YOLO series of models are continuously optimized in terms of lightweight, computational efficiency, robustness, etc., making them have good adaptability in embedded devices and cloud inference scenarios. YOLOv8 is a major upgrade of the YOLO series, supporting object detection, image classification, and instance segmentation tasks. Its architecture consists of a backbone network, a neck network, and a detection head. The backbone network uses C2f modules to improve feature extraction efficiency, the neck network uses PANet structure to enhance multi-scale feature fusion, and the detection head introduces Anchor-Free design and DFL loss to improve detection accuracy and flexibility. In addition, YOLOv8 provides n / s / m / l / x five model variants to adapt to different scenarios. In the road spill detection task, YOLOv8s is selected for optimization to balance accuracy and real-time performance and improve detection effect.

[0042] Based on this, when constructing a road spill detection model, YOLOv8s can be preferably selected. Exemplarily, if YOLOv8s is used as the basis network for improvement, through the LMSADet detection head, MSHA module, and C3ESP module, the network structure of the road spill detection model can be referred to Figure 1 .

[0043] The road surface debris detection model adopts modular design, and each sub-module cooperates with each other, giving consideration to lightweight, high precision and robustness. Among them, the LMSADet detection head integrates multi-scale modeling and spatial attention mechanism at the detection end, through the boundary box regression branch and the class prediction branch sharing the feature enhancement structure, only the task is distinguished in the output layer channel number, so that the model can directly optimize the local details and scale differences of the debris target while ensuring the simple structure, effectively improving the detection accuracy and inference efficiency. The MSHA module realizes efficient fusion of cross-scale features by introducing multi-scale convolution and spatial attention mechanism, enhances the model's understanding ability of the semantic relationship between road debris and background, and can better suppress background interference and improve the detection performance under fuzzy boundaries. The C3ESP module cooperates with the multi-scale perception branch and the residual branch in the backbone network, extracts and refines feature information with the help of stacked depth separable convolution structure, and combines residual connection to ensure the integrity of feature fusion and avoid information loss. The three work together to make the model maintain a lightweight structure in complex road surface environment, while having strong feature expression ability and detection robustness, so as to realize fast and accurate identification of road surface debris.

[0044] In some embodiments, in order to ensure that the trained road surface debris detection model can meet the expectations and accurately understand and detect road surface debris of different scales in different scenes, a special data set can be created for training the road surface debris detection model. Specifically, a data set named Road Surface Debris (RSD) is created, which is collected by a vehicle-mounted camera under various environmental conditions, including 10000 images with a resolution of 1280x720. Each image is manually annotated.

[0045] For example, in order to provide a high-quality data set, in addition to ensuring the number of images, images covering different scenes such as urban roads, squares, parks, and areas around buildings can also be collected.

[0046] For example, the images can contain common coverings such as vehicles, people, vegetation, warning signs (warning triangles), etc.

[0047] For example, each scene can include multiple time periods and weather conditions such as daytime, nighttime, sunny, rainy, and foggy, to increase the complexity and comprehensiveness of the scene, and each image can be configured with a corresponding txt label file detailing the location and category of the road surface debris.

[0048] Such a data set can support research and application in the fields of intelligent transportation systems, autonomous vehicles, and road safety monitoring, to improve road safety and traffic efficiency.

[0049] To ensure the effectiveness of model training and the reliability of evaluation results, the data set in the present application is divided into a training set (7000 images), a validation set (1000 images) and a test set (2000 images) in a ratio of 7:1:2.

[0050] Preferably, during the process of dividing the data set, special attention is paid to maintaining the balance of sample distribution between the data sets, ensuring that they have similar statistical characteristics in terms of road surface scattering types, sizes, background environments, etc. This can avoid evaluation bias caused by uneven sample distribution and make the validation results more representative and reliable.

[0051] In some embodiments, in order to comprehensively and accurately measure the performance of each version of the road surface scattering detection model, after at least one version of the road surface scattering detection model converges based on the training set, the converged model can be evaluated by the validation set to avoid model overfitting, verify the generalization of the model, and ensure that the model can run stably after deployment.

[0052] Specifically, the performance of the road surface scattering detection model on the validation set can be evaluated according to preset conditions.

[0053] Exemplarily, these preset conditions can include performance indicators such as the intersection over union (IoU), detection accuracy and recall rate of the road surface scattering detection device.

[0054] IoU is an indicator for evaluating the degree of overlap between the predicted bounding box and the true bounding box, and is the ratio of the intersection to the union of the predicted bounding box and the true bounding box. It plays a key role in determining whether it is correct detection. The calculation formula of IoU can be written as:

[0055] Precision (p) is also called precision, which refers to the proportion of correct predictions to all predictions, as shown in the formula:

[0056] Recall (R) is also called recall, which refers to the proportion of correct predictions to all actual positives, as shown in the formula:

[0057] Average Precision (AP) is calculated from precision and recall. According to the recall value, a line graph of precision is drawn, and the area under the line is calculated, as shown in the formula:

[0058] The mean average precision (mAP) refers to an average value of average precisions AP of C different road surface scattering categories, as shown in the formula:

[0059] That is, after verifying each version of the road surface scattering detection model through the verification set, the road surface scattering detection model of each version can be comprehensively evaluated based on the above several indicators, so as to determine the road surface scattering detection model with the best performance from each version as the trained road surface scattering detection model.

[0060] In some embodiments, the model running environment includes an Intel Xeon Platinum 8255C processor, 314 GB of memory, an NVIDIA Tesla V100 32 GB graphics card, and a CentOS 8.5.2 (64-bit) operating system. The deep neural network is built based on the PyTorch framework, the input image size is [640, 640], and a multi-scale training strategy is used. The batch size is set to 64, the training is performed for 300 epochs, the AdamW optimizer is used, the initial learning rate is 0.01, and the cosine decay strategy is combined for optimization.

[0061] In some embodiments, the road surface scattering detection model training combines classification loss and regression loss for optimization. The binary classification cross-entropy loss (BCE Loss) is used for classification loss to determine the anchor box category; the regression loss is composed of SIoU Loss and DFL Loss to measure the error between the predicted box and the real box. The TAL dynamic matching method is used for positive and negative sample matching strategy, the optimization target is allocated, and the detection accuracy is improved.

[0062] Based on the network structure of the road surface scattering detection model of the foregoing embodiments, the present application provides a road surface scattering detection method based on deep learning.

[0063] The road surface scattering detection method based on deep learning provided by the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of the present application do not make any limitation on the specific type of electronic device.

[0064] To illustrate the technical solutions provided in the present application, the following will take an electronic device as an execution subject to describe various embodiments.

[0065] Figure 2 A schematic flowchart of a deep learning-based road surface spill detection method provided by the present application is shown, which includes: Step 210, the electronic device performs feature extraction on the to-be-detected image based on the backbone network of the pre-trained road surface spill detection model.

[0066] Step 220, the electronic device fuses the image features based on the neck network of the road surface spill detection model to obtain fused features.

[0067] Step 230, the electronic device detects the fused features based on the detection network of the road surface spill detection model to obtain a detection result of the road surface spill in the to-be-detected image.

[0068] In the present embodiment, the model first extracts features of the input image through the backbone network to obtain image features containing road surface information; then fuses the multi-level features through the neck network to form target fused features; finally, the detection network analyzes the target fused features to output a prediction result, including a bounding box of the spill and its category. In this process, the model uses an LMSADet detection head that combines multi-scale fusion and spatial attention mechanism to effectively enhance the expression of target features. Unlike placing the multi-scale mechanism in the backbone or neck network, integrating it into the detection head is closer to the final detection task and can directly optimize the local details and scale differences of the spill. On the one hand, it avoids redundant global calculations at early and middle feature layers, reducing the complexity of the model; on the other hand, it improves the inference efficiency while ensuring high accuracy, thereby realizing fast and accurate detection of road surface spills.

[0069] In some embodiments, the LMSADet detection head includes a bounding box regression detection branch for outputting a predicted bounding box, and a category prediction branch for outputting a predicted spill category, and the branch network structures of the bounding box regression detection branch and the category prediction branch are the same; the branch network structure includes a multi-scale convolution structure, a first concatenation layer, a spatial attention structure, a channel fusion structure, a second concatenation layer, a first DWCBS layer, a second DWCBS layer, and a first convolution layer; wherein the output channel number of the first convolution layer corresponding to the bounding box regression detection branch is set to , and the output channel number of the first convolution layer corresponding to the category prediction branch is set to n c .

[0070] For the target fused features input into the bounding box regression detection branch or the category prediction branch: Step A1, the electronic device performs at least two cascaded convolution operations on the target fusion feature through a multi-scale convolution structure to obtain cascaded convolution features corresponding to each scale.

[0071] Step A2, the electronic device splices the features corresponding to each scale in the channel dimension through a first splicing layer to obtain first spliced features.

[0072] Step A3, the electronic device sequentially performs channel compression, channel expansion and normalization operations on the first spliced features through a spatial attention structure to obtain spatial attention weights corresponding to each scale.

[0073] Step A4, the electronic device, for each scale, performs weighted fusion of the corresponding spatial attention weights and the corresponding mixed convolution features through a channel fusion structure to obtain corresponding spatial enhanced features.

[0074] Step A5, the electronic device splices the spatial enhanced features of each scale through a second splicing layer to obtain attention-aware features.

[0075] Step A6, the electronic device sequentially performs two DWCBS and convolution operations on the attention-aware features through a first DWCBS layer, a second DWCBS layer and a first convolution layer to obtain a predicted bounding box or a predicted scatter category.

[0076] The electronic device utilizes a multi-scale convolution structure to perform cascaded convolution operations at least at two scales to capture feature representations under different receptive fields and obtain multi-scale cascaded convolution features. The cascaded convolution operation corresponding to each scale includes sequentially performed depthwise separable convolution operation and convolution operation. Then the features obtained at each scale are spliced in the channel dimension to form first spliced features containing multi-scale information. Then the spatial attention structure is used to perform channel compression, expansion and normalization on the first spliced features to generate spatial attention weights corresponding to each scale to highlight key information. Further, the channel fusion structure is used to weight and fuse the spatial attention weights of each scale and the corresponding convolution features to obtain spatial enhanced features with higher discriminability. Finally, the attention-aware features are processed by two depthwise separable convolution (DWCBS) and convolution layers to output a predicted bounding box or a scatter category.

[0077] In the embodiments of the present application, the LMSADet detection head obtains different receptive field features through multi-scale convolution, highlights key regions with spatial attention, and realizes fine enhancement by combining channel fusion. Finally, accurate and efficient bounding box regression and category prediction are realized under a lightweight structure, which significantly improves the detection performance and robustness.

[0078] In some embodiments, Figure 3 The network structure of the LMSA Det detection head is shown. From Figure 3 It can be seen that the branch network structure of the bounding box regression detection branch and the category prediction branch is the same, and the target fusion feature B is the batch size, C is the number of channels, H and W are the height and width, respectively. The specific operations are as follows: First, perform serial convolution operations (depth separable convolution operation + ordinary convolution) under different scales (3x3, 5x5, and 7x7) on X respectively, to extract serial convolution features with scale adaptability under different receptive fields:

[0079] Then, the serial convolution features extracted under different scales are spliced in the channel dimension to form a comprehensive feature identifier with better discrimination ability—the first spliced feature.

[0080] Finally, the spatial attention weight corresponding to each scale is calculated according to , and it is multiplied element by element with the corresponding serial convolution feature to enhance the key regions and suppress the non-key regions under different scales, to obtain the spatial enhancement features under each scale. The spatial enhancement features under each scale are spliced to generate the attention-aware feature used for task prediction.

[0081] By sequentially performing two DWCBS layers (depth separable convolution + normalization + SiLU) and the first convolution layer on the attention-aware feature, two DWCBS and convolution operations are performed twice, and the final detection result is obtained.

[0082] Specifically, referring to the following formula, for the bounding box regression detection branch, based on the limitation of the output channel number of the first convolution layer, the predicted bounding box of the target O reg , specifically the bounding box coordinates. For the category prediction branch, the confidence of the corresponding category is output to obtain the predicted spatter category of the target O cls , specifically the confidence score of the corresponding spatter category.

[0083]

[0084]

[0085] ​In the embodiment, the detection head is designed in a unified branch network structure, and the tasks of boundary box regression and category prediction are separated by differentiating the number of output channels of the first convolutional layer at the output end. Specifically, the boundary box regression branch can directly generate the predicted boundary box coordinates of the target under the limitation of the number of output channels, and the category prediction branch outputs the confidence score of the corresponding category, so as to obtain the predicted scattering category of the target. This design can realize the synchronous prediction of position and category while ensuring simple structure and efficient calculation, improve the detection accuracy and inference efficiency, and has high practical value.

[0086] In some embodiments, for different detection scales, corresponding LMSADet detection heads can be configured respectively. As shown in Figure 1 When the detection task is divided into three scales, that is, three decoupled LMSADet detection heads are set, target information is extracted from different scale feature layers correspondingly. Finally, the output results of the three detection heads are spliced into the same image, and non-maximum suppression processing is performed to remove redundant detection boxes, and invalid predictions are filtered by combining the confidence threshold to obtain the final output image. In the output image, not only the predicted boundary boxes of various road surface scatterings are labeled, but also the corresponding category information is attached (if necessary, the confidence score can also be labeled at the same time, so as to accurately detect and visually present the multi-scale and multi-category targets.

[0087] In some embodiments, the MSHA module introduced in the neck network is composed of MSCA and MSSA sub-modules in series, the former realizes the extraction and fusion of multi-scale features in the channel dimension, which can highlight key channel features and suppress redundant information; the latter extracts and fuses multi-scale features in the spatial dimension, which strengthens the model's perception of the spatial distribution of the target region. The two work together to make the model have stronger feature expression and discrimination ability in both channel and space, thereby significantly improving the detection accuracy and robustness of road surface scatterings in complex scenes.

[0088] In some embodiments, the MSCA sub-module includes a first mixed pooling layer, a third splicing layer, a dimension adjustment layer, a multi-scale convolutional layer, a fourth splicing layer, a second convolutional layer, a first activation layer, and a first fusion layer; for the first input feature input into the MSHA module: Step B1, the electronic device performs average pooling and maximum pooling operations on the first input feature through the first mixed pooling layer to obtain first and second pooling results.

[0089] Step B2, the electronic device splices the first and second pooling results through the splicing layer to obtain a first mixed pooling result.

[0090] Step B3, the electronic device performs convolution operations of different scales on the first mixed pooling result after adjusting the dimensions through the multi-scale convolution layer to obtain first convolution features corresponding to each scale.

[0091] The electronic device performs convolution operations of different scales on the first mixed pooling result in the multi-scale convolution layer to extract convolution features under multi-receptive fields.

[0092] Step B4, the electronic device splices the first convolution features of each scale through the third splicing layer to obtain third splicing features.

[0093] Step B5, the electronic device performs an activation operation on the third splicing features convolved through the second convolution layer through the first activation layer to obtain channel weights.

[0094] Step B6, the electronic device fuses the first input features and the channel weights through the fusion layer to obtain multi-scale channel enhanced features.

[0095] The electronic device first performs average pooling and maximum pooling operations on the input features through the first mixed pooling layer, for example, on a 4x4 feature map, the average pooling calculates the mean of each region pixel, and the maximum pooling takes the maximum value of the region, thereby obtaining two complementary pooling results. Then, the two results are spliced in the channel dimension to form the first mixed pooling result containing global statistical information and local significant features. Subsequently, the device performs convolution operations of different scales on the first mixed pooling result through the multi-scale convolution layer, for example, using 3x3, 5x5, and 7x7 convolution kernels, respectively capturing local details, neighborhood relationships, and larger range context features, and obtaining multi-scale convolution features. Then, these features are spliced again to obtain multi-scale fused splicing features. In order to further emphasize key information, the splicing features will be convolved first, and then mapped to the [0,1] interval through an activation function (such as Sigmoid) to generate weight coefficients of each channel. For example, if a certain channel contributes more to target recognition, the weight value will be close to 1. Finally, the device multiplies the weight and the original input features channel by channel, thereby enhancing important channels and obtaining multi-scale channel enhanced features, which not only retains the original information but also highlights the key features.

[0096] In this embodiment, the MSCA sub-module can simultaneously consider global information and local details in the feature extraction process. The mixed pooling operation effectively retains different statistical features, the multi-scale convolution further expands the receptive field range, and the splicing and fusion process ensures the complete expression of multi-scale information; at the same time, the introduction of channel weights enables the model to dynamically highlight key channel features. The multi-scale channel enhanced features obtained ultimately improve the feature expression ability while significantly enhancing the model's discrimination performance and detection accuracy for targets in complex scenes.

[0097] In some embodiments, the MSSA submodule comprises a second mixed pooling layer, a fifth concatenation layer, a multi-scale convolution weighting structure, a sixth concatenation layer, a second activation layer, and a second fusion layer; the multi-scale channel enhanced features input into the MSSA submodule are enhanced as follows: Step C1, the electronic device performs an average pooling operation and a maximum pooling operation on the multi-scale channel enhanced features respectively through the second mixed pooling layer to obtain a third pooling result and a fourth pooling result.

[0098] Step C2, the electronic device splices the third pooling result and the fourth pooling result through the fifth concatenation layer to obtain a second mixed pooling result.

[0099] Step C3, the electronic device performs a convolution operation and a weighting operation corresponding to each scale on the second mixed pooling result in turn through the multi-scale convolution weighting structure to obtain a corresponding weighted feature. The weight corresponding to the weighting operation is obtained through learning; Step C4, the electronic device splices the weighted features of each scale through the sixth concatenation layer to obtain a weighted fusion feature.

[0100] Step C5, the electronic device performs an activation operation on the weighted fusion feature through the second activation layer to obtain a spatial weight.

[0101] Step C6, the electronic device fuses the multi-scale channel enhanced feature and the spatial weight through the fusion layer to obtain a multi-scale channel-spatial enhanced feature.

[0102] In the MSSA submodule, the electronic device first performs an average pooling operation and a maximum pooling operation on the input multi-scale channel enhanced feature through the second mixed pooling layer. For example, on an 8x8 feature map, the average pooling operation obtains the overall mean of the region, reflecting the global background information; the maximum pooling operation obtains the maximum value of the region, highlighting the significant target feature. The third pooling result and the fourth pooling result obtained in this way are channel spliced in the fifth concatenation layer to form a second mixed pooling result with more information. Then, in each scale, the device uses the multi-scale convolution weighting structure to perform convolution and weighting processing on the result in turn, such as extracting features of different receptive fields through 3x3, 5x5, and 7x7 convolution kernels, and then weighting these features by the weight obtained through training and learning, to output the weighted features corresponding to each scale. Subsequently, the weighted features of all scales are spliced into a weighted fusion feature through the sixth concatenation layer.

[0103] To further highlight the key regions, the weighted fusion feature enters the second activation layer, and the weight coefficient of each spatial position is generated through nonlinear mapping, for example, the region close to the target edge may obtain higher weight. Finally, the electronic device fuses these spatial weights with the multi-scale channel enhanced features of the original input (i.e., the output of the MSCA sub-module) position by position through the fusion layer to obtain the final multi-scale channel-spatial enhanced feature, thereby realizing comprehensive strengthening of the features.

[0104] In the present embodiment, the MSSA sub-module not only combines the global and local complementary information brought by average pooling and maximum pooling, but also realizes adaptive modeling of features under different receptive fields through a multi-scale convolution weighting structure, and further dynamically highlights key regions by using a spatial attention mechanism. The final multi-scale channel-spatial enhanced feature can not only fully express semantic information under different scales, but also strengthen the spatial distribution characteristics of the target region, thereby significantly improving the discrimination ability and detection accuracy of the model in a complex background.

[0105] In some embodiments, Figure 4 The network structure diagram of the MSHA module is shown, which is based on Figure 4 the network structure, and the first input feature is first subjected to adaptive global average pooling and global maximum pooling to obtain a first pooling result and a second pooling result , and the two are added to obtain a first mixed pooling result :

[0106]

[0107]

[0108] In the channel dimension, a multi-scale 1D convolution operation is performed on , and the first convolution features corresponding to each scale obtained are spliced to obtain a third spliced feature :

[0109]

[0110] Then, a convolution and a Sigmoid activation function are used to generate an attention-based channel weight , which is applied to to obtain a multi-scale channel enhanced feature .

[0111]

[0112]

[0113] Based on , the global average pooling and the global maximum pooling are calculated to obtain a third pooling result and a fourth pooling result, and the third pooling result and the fourth pooling result are spliced in the channel dimension:

[0114]

[0115] In the spatial dimension, a multi-scale 2D convolution operation is performed on , using learnable weights to perform weighted summation on the multi-scale convolution result to obtain a weighted fusion feature :

[0116]

[0117] Finally, a Sigmoid activation function is used to generate spatial attention weights, which are applied to , so as to obtain an output multi-scale channel-spatial enhanced feature :

[0118] In this embodiment, the MSHA module extracts and fuses multi-scale features in the channel dimension through mixed pooling, multi-scale convolution and channel weighting, which can effectively highlight key channels and suppress redundant information. The latter realizes dynamic fusion of global and local features in the spatial dimension through pooling, convolution weighting and spatial attention mechanism, which can highlight the saliency of the target region and improve the perception ability of the spatial distribution. The two work together to enable the model to enhance feature expression in both channel and spatial dimensions, not only improving the adaptability to complex backgrounds, but also significantly enhancing the recognition ability of different scales and fuzzy boundary targets, thereby comprehensively improving the detection accuracy and robustness.

[0119] In some embodiments, the backbone network is provided with a C3ESP module, and the C3ESP module includes a third convolutional layer, a residual branch, a multi-scale perception branch, a seventh splicing layer and a fourth convolutional layer. For the second input feature input into the C3ESP module: Step D1, the electronic device performs a channel compression operation on the second input feature through the third convolutional layer to obtain a compressed feature.

[0120] Step D2, the electronic device performs a multi-scale deep feature extraction operation on the compressed feature through the multi-scale perception branch to obtain a deep feature.

[0121] Step D3, the electronic device splices the deep features and the compressed features through the seventh splicing layer to obtain seventh splicing features.

[0122] Step D2, the electronic device performs a channel expansion operation on the seventh splicing features through the fourth convolution layer to obtain second output features corresponding to the second input features.

[0123] In the backbone network, the electronic device first performs a channel compression operation on the input second input features through the third convolution layer, for example, compressing a feature map originally having 256 channels to 64 channels, thereby reducing the calculation amount and highlighting the main information. Subsequently, the compressed features are sent into the multi-scale perception branch, in which a plurality of deep separable convolutions are sequentially performed to extract multi-scale features, for example, a plurality of convolutions are connected in series to gradually expand the receptive field, thereby simultaneously capturing local edge details and larger range of context information, to obtain deep features. Then, the electronic device performs channel splicing on the deep features and the initial compressed features in the seventh splicing layer, to realize complementary fusion of shallow and deep information. Finally, the splicing result performs a channel expansion operation through the fourth convolution layer, for example, expanding 64 channels back to 256 channels, to ensure that the output features match the dimensions of the subsequent layers of the network, thereby obtaining corresponding second output features.

[0124] In this embodiment, based on the C3ESP module, the electronic device reduces redundant information and calculation overhead through channel compression, the multi-scale perception branch extracts rich feature representations under different receptive fields, the seventh splicing layer realizes effective fusion of deep and shallow features, and the channel expansion ensures that the information can be completely transmitted after compression and fusion. The overall design improves the feature expression capability while avoiding information loss and redundant calculation, so that the model can capture more detailed and diverse features under the premise of ensuring lightweight, thereby enhancing the adaptability and robustness to complex scenes.

[0125] In some embodiments, the multi-scale perception branch includes n ESPBlocks connected in series, each ESPBlock includes a progressive deep separable convolution structure, an eighth splicing layer, a fifth convolution layer, a PEMA layer, and a ninth splicing layer; for a third input feature input into the ESPBlock: Step E1, the electronic device performs a progressive multi-level deep separable convolution operation on the third input feature through the progressive deep separable convolution structure to obtain deep convolution features of each level. The input corresponding to the first level is the third input feature, and the input of the non-first level is the deep convolution feature corresponding to the previous level. Step E2, the electronic device splices the deep convolution features of each level through the eighth splicing layer to obtain fused deep convolution features.

[0126] Step E3, the electronic device performs a channel fusion operation on the fused deep convolution features through the fifth convolution layer.

[0127] Step E4, the electronic device performs a hybrid attention mechanism on the fused deep convolutional features after channel fusion through the PEMA layer to obtain a hybrid enhanced feature.

[0128] Step E5, the electronic device splices the third input and the hybrid enhanced feature through the ninth splicing layer to obtain a third output feature corresponding to the third input feature.

[0129] In the multi-scale perception branch, the electronic device first performs a multi-level convolution operation on the third input feature through a progressive deep separable convolution structure. For example, the first convolutional layer directly processes the input feature map (such as a feature map of 64×64×12864 \times 64 \times 12864×64×128), to obtain the first layer deep convolutional feature; then each non-first level uses the output of the previous layer as input to continue performing deep separable convolution, and extracts higher-order and more abstract features layer by layer. Then, the electronic device splices the deep convolutional features of each level in the channel dimension through the eighth splicing layer to form a fused deep convolutional feature, thereby integrating multi-scale information of different levels. Then, the fused feature enters the fifth convolutional layer for channel fusion operation, for example, the number of channels after splicing is compressed from 512 back to 256 to maintain the compactness of the feature dimension and integrate information of each layer. After that, the PEMA layer applies a hybrid attention mechanism to the fused feature after channel fusion, highlights key information by dynamically adjusting channel and spatial weights, and obtains a hybrid enhanced feature. Finally, the electronic device splices the third input feature and the hybrid enhanced feature in the ninth splicing layer to obtain a third output feature corresponding to the third input feature, realizing the integration of shallow and enhanced features.

[0130] In this embodiment, through the above processing, the ESPBlock can extract multi-scale features at different levels, integrate information through splicing and channel fusion, and the PEMA layer further strengthens key channels and spatial regions, thereby significantly enhancing the expression ability and discrimination ability of the model for complex targets, while maintaining the balance between lightweight calculation and multi-scale perception.

[0131] In some embodiments, Figure 5 The network structure diagram of the C3ESP module and the ESPBlock is shown. For an ESPBlock, assuming that the third input feature is , the ESPBlock will enhance the extraction of features and the flow of information through the following steps: Initialize a list Y containing the input X :

[0132] For each convolutional layeri (From 1 to n), the output of the last layer is taken as the input of the current layer, and a 3*3 depth separable convolution operation is performed. Here, the input of the first layer is X , and the input of each layer is the output of the previous layer:

[0133] wherein, The convolution result of each convolution layer (i.e. depth convolution feature) will be added to the list Y .

[0134] Connect all levels of depth convolution features with the initial input , and then pass through a 1*1 convolution layer for channel fusion:

[0135] Finally, the importance of features of different receptive field sizes is adjusted through the PEMA attention mechanism.

[0136]

[0137] wherein, the PEMA attention mechanism can be expressed as:

[0138]

[0139]

[0140] By setting the C3ESP module and the series ESPBlock on the basis of the ESP module, the present application can utilize the double-branch feature shunting mechanism and combine the progressive depth separable convolution and the PEMA attention mechanism to realize efficient extraction and fusion of shallow and deep, multi-scale features. The design not only enhances the model's representation ability for fine-grained features and targets in complex backgrounds, but also effectively reduces redundant calculations and improves inference efficiency, thereby realizing lightweight calculation while maintaining high precision and significantly improving overall detection performance.

[0141] In some embodiments, the detection method of each of the above embodiments is implemented based on a trained road surface scattering detection model. The road surface scattering detection model cooperatively applies the LMSADet detection head, the MSHA module, and the C3ESP module, and can realize the advantages of modular design. The LMSADet detection head uses multi-scale feature fusion and spatial attention mechanism, can directly perform bounding box regression and class prediction on targets of different scales, optimizes local detail processing, and at the same time avoids unnecessary global calculation of early and middle feature layers, thereby maintaining high precision and improving inference efficiency. The MSHA module uses the MSCA and MSSA sub-modules in series to perform multi-scale feature extraction and fusion in the channel and spatial dimensions respectively, combines channel and spatial attention mechanisms, dynamically highlights key information, and enhances the model's perception ability for complex backgrounds and targets of different scales. The C3ESP module and its internal ESPBlock structure realize efficient extraction and fusion of shallow and deep, multi-scale features through double-branch feature splitting, progressive depth separable convolution, and PEMA attention mechanism, reduce redundant calculation, and enhance fine-grained feature representation ability. The three work together to enable the model to realize accurate detection of high-precision, multi-scale, and multi-class targets under the premise of lightweight, and significantly improve the robustness and overall performance in complex scenarios.

[0142] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0143] The road surface scattering detection method based on deep learning corresponding to the above embodiments, Figure 6 The structure block diagram of the road surface scattering detection device 6 provided by the embodiments of the present application is shown, and only the parts related to the embodiments of the present application are shown for ease of description.

[0144] Referring to Figure 6 The road surface scattering detection device 6 comprises: The extraction module 61 is configured to perform feature extraction on a to-be-detected image based on a backbone network of a pre-trained road surface scattering detection model, to obtain image features; the to-be-detected image comprises a road surface; The fusion module 62 is configured to fuse the image features based on a neck network of the road surface scattering detection model, to obtain target fusion features; The detection module 63 is configured to detect the target fusion features based on a detection network of the road surface scattering detection model, to obtain a detection result of road surface scattering in the to-be-detected image; The detection network comprises an LMSADet detection head, which enhances the target fusion features based on multi-scale fusion and spatial attention, and detects the enhanced target fusion features to obtain the detection result. In the case that the road surface spilling is included in the image to be detected, the detection result includes a predicted bounding box of the road surface spilling and a predicted spilling category.

[0145] Optionally, the LMSA detection head includes a bounding box regression detection branch for outputting the predicted bounding box, and a category prediction branch for outputting the predicted spilling category, the branch network structures of the bounding box regression detection branch and the category prediction branch are the same; the branch network structure includes a multi-scale convolution structure, a first concatenation layer, a spatial attention structure, a channel fusion structure, a second concatenation layer, a first DWCBS layer, a second DWCBS layer, and a first convolution layer; the detection module 63 includes a detection unit, configured to: For the target fusion feature input into the bounding box regression detection branch or the category prediction branch: Perform at least two series convolution operations on the target fusion feature through the multi-scale convolution structure to obtain series convolution features corresponding to each scale; the series convolution operation includes a depth separable convolution operation and a convolution operation performed in sequence; Concatenate the features corresponding to each scale in the channel dimension through the first concatenation layer to obtain first concatenation features; Perform a channel compression operation, a channel expansion operation, and a normalization operation in sequence on the first concatenation features through the spatial attention structure to obtain spatial attention weights corresponding to each scale; At each scale, weight and fuse the corresponding spatial attention weights and the corresponding mixed convolution features through the channel fusion structure to obtain corresponding spatial enhancement features; Concatenate the spatial enhancement features of each scale through the second concatenation layer to obtain attention-aware features; Perform two DWCBS and convolution operations in sequence on the attention-aware features through the first DWCBS layer, the second DWCBS layer, and the first convolution layer to obtain the predicted bounding box or the predicted spilling category; The first convolution layer corresponding to the bounding box regression detection branch is set to have an output channel number of The first convolution layer corresponding to the category prediction branch is set to have an output channel number of n c .

[0146] Optionally, the neck network is provided with an MSHA module, the MSHA module includes an MSCA submodule and an MSSA submodule, the MSCA submodule and the MSSA submodule are connected in series, the MSCA submodule is configured to perform channel attention-based multi-scale feature extraction and fusion in the channel attention dimension; the MSSA submodule is configured to perform spatial attention-based multi-scale feature extraction and fusion in the spatial dimension.

[0147] Optionally, the MSCA sub-module comprises a first mixed pooling layer, a third splicing layer, a dimension adjustment layer, a multi-scale convolution layer, a fourth splicing layer, a second convolution layer, a first activation layer, and a first fusion layer; the fusion module 62 comprises a fusion unit, and the fusion unit is configured to: for the first input feature input into the MSHA module: performing an average pooling operation and a maximum pooling operation on the first input feature respectively through the first mixed pooling layer to obtain a first pooling result and a second pooling result; splicing the first pooling result and the second pooling result through the splicing layer to obtain a first mixed pooling result; performing a convolution operation of different scales on the first mixed pooling result after the dimension adjustment respectively through the multi-scale convolution layer to obtain first convolution features corresponding to the scales; splicing the first convolution features of the scales through the third splicing layer to obtain third splicing features; performing an activation operation on the third splicing features after the second convolution layer through the first activation layer to obtain channel weights; fusing the first input feature and the channel weights through the fusion layer to obtain multi-scale channel enhanced features.

[0148] Optionally, the MSSA sub-module comprises a second mixed pooling layer, a fifth splicing layer, a multi-scale convolution weighting structure, a sixth splicing layer, a second activation layer, and a second fusion layer; the fusion unit is specifically configured to: for the multi-scale channel enhanced features input into the MSSA sub-module: performing an average pooling operation and a maximum pooling operation on the multi-scale channel enhanced features respectively through the second mixed pooling layer to obtain a third pooling result and a fourth pooling result; splicing the third pooling result and the fourth pooling result through the fifth splicing layer to obtain a second mixed pooling result; at each scale, performing a convolution operation and a weighting operation of the corresponding scale on the second mixed pooling result in sequence through the multi-scale convolution weighting structure to obtain a corresponding weighted feature; the weight corresponding to the weighting operation is obtained through learning; splicing the weighted features of the scales through the sixth splicing layer to obtain a weighted fusion feature; performing an activation operation on the weighted fusion feature through the second activation layer to obtain a spatial weight; fusing the multi-scale channel enhanced features and the spatial weight through the fusion layer to obtain multi-scale channel-spatial enhanced features.

[0149] Optionally, the backbone network is provided with a C3ESP module, and the C3ESP module comprises a third convolution layer, a residual branch, a multi-scale perception branch, a seventh splicing layer, and a fourth convolution layer; the extraction module 61 comprises an extraction unit, and the extraction unit is configured to: For the second input feature of the C3ESP module: The second input feature is compressed by performing channel compression on the third convolutional layer to obtain compressed features; Deep features are obtained by performing multi-scale deep feature extraction operations on compressed features through a multi-scale perceptual branch. The deep features and compressed features are spliced ​​together through the seventh splicing layer to obtain the seventh spliced ​​feature; The fourth convolutional layer performs channel expansion on the seventh concatenated feature to obtain the second output feature corresponding to the second input feature.

[0150] Optionally, the multi-scale sensing branch includes n cascaded ESPBlocks, each ESPBlock comprising a progressively deep separable convolutional structure, an eighth concatenated layer, a fifth convolutional layer, a PEMA layer, and a ninth concatenated layer; the extraction unit is specifically used for: For the third input feature of the input ESPBlock: A progressive multi-level deep separable convolutional operation is performed on the third input feature using a progressively deep separable convolutional structure to obtain the deep convolutional features of each level; the input corresponding to the first level is the third input feature, and the inputs of non-first levels are the deep convolutional features corresponding to the previous level. By concatenating the deep convolutional features of each layer in the eighth concatenation layer, a fused deep convolutional feature is obtained. Channel fusion is performed on the fused deep convolutional features through the fifth convolutional layer; By applying a hybrid attention mechanism to the fused deep convolutional features after channel fusion through a PEMA layer, hybrid enhanced features are obtained. The third input is concatenated with the hybrid enhancement feature through the ninth concatenation layer to obtain the third output feature corresponding to the third input feature.

[0151] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0152] Figure 7 This is a schematic diagram of the physical layer structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 7 of this embodiment includes: at least one processor 70 ( Figure 7 The diagram shows only one processor, memory 71, and a computer program 72 stored in memory 71 that can run on at least one processor 70. When processor 70 executes computer program 72, it implements the steps in any of the above embodiments of the deep learning-based road spill detection method. Figure 2The steps 210-230 are shown.

[0153] The processor 70 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0154] The memory 71 can be an internal storage unit of the electronic device 7 in some embodiments, such as a hard disk or a memory of the electronic device 7. The memory 71 can also be an external storage device of the electronic device 7 in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0155] Further, the memory 71 can include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store operating systems, application programs, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0156] It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0157] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize the steps in the above-mentioned various method embodiments.

[0158] The embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device is caused to execute the steps in the above-mentioned various method embodiments.

[0159] The integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the embodiment of the present application realizes all or part of the processes in the above-mentioned method embodiments, which can be completed by a computer program instructing related hardware. The above-mentioned computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to realize the steps in the above-mentioned various method embodiments. The computer program includes computer program code. The computer program code can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk.

[0160] In the above-mentioned embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0161] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0162] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other manners. For example, the embodiments of the apparatus / network device described above are merely schematic, and the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0163] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0164] The above embodiments are merely used to describe the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for detecting road surface spilling based on deep learning, characterized in that, The method comprises: extracting features of an image to be detected based on a backbone network of a pre-trained road surface scattering detection model, to obtain image features; the image to be detected comprises a road surface; fusing the image features based on a neck network of the road surface scattering detection model, to obtain target fusion features; detecting the target fusion features based on a detection network of the road surface scattering detection model, to obtain a detection result of road surface scattering in the image to be detected; wherein the detection network comprises an LMSADet detection head, which enhances the target fusion features based on multi-scale fusion and spatial attention, and detects the enhanced target fusion features to obtain the detection result; in the case that the image to be detected comprises road surface scattering, the detection result comprises a predicted bounding box of road surface scattering and a predicted scattering category.

2. The pavement spraying detection method of claim 1, wherein, the LMSADet detection head comprises a bounding box regression detection branch for outputting the predicted bounding box, and a category prediction branch for outputting the predicted scattering category, and the branch network structures of the bounding box regression detection branch and the category prediction branch are the same; the branch network structure comprises a multi-scale convolution structure, a first concatenation layer, a spatial attention structure, a channel fusion structure, a second concatenation layer, a first DWCBS layer, a second DWCBS layer, and a first convolution layer; for the target fusion features input into the bounding box regression detection branch or the category prediction branch: perform at least two series convolution operations on the target fusion features in different scales through the multi-scale convolution structure, to obtain series convolution features corresponding to each scale; the series convolution operation comprises a depth separable convolution operation and a convolution operation performed in sequence; concatenate the features corresponding to each scale in the channel dimension through the first concatenation layer, to obtain first concatenation features; perform a channel compression operation, a channel expansion operation, and a normalization operation on the first concatenation features through the spatial attention structure in sequence, to obtain spatial attention weights corresponding to each scale; for each scale, weight fuse the corresponding spatial attention weights and the corresponding mixed convolution features through the channel fusion structure, to obtain spatial enhancement features corresponding to each scale; concatenate the spatial enhancement features of each scale through the second concatenation layer, to obtain attention-aware features; perform DWCBS and convolution operations on the attention-aware features through the first DWCBS layer, the second DWCBS layer, and the first convolution layer in sequence, to obtain the predicted bounding box or the predicted scattering category; The output channel number set by the first convolutional layer corresponding to the bounding box regression detection branch is The output channel number set by the first convolutional layer corresponding to the category prediction branch is n c .

3. The pavement spraying detection method of claim 1, wherein, the neck network is provided with an MSHA module, the MSHA module comprises an MSCA submodule and an MSSA submodule, the MSCA submodule and the MSSA submodule are connected in series, and the MSCA submodule is used for performing multi-scale feature extraction and fusion based on channel attention in the channel attention dimension; the MSSA submodule is used for performing multi-scale feature extraction and fusion based on spatial attention in the spatial dimension.

4. The pavement dispensing detection method of claim 3, wherein, The MSCA submodule comprises a first mixed pooling layer, a third splicing layer, a dimension adjustment layer, a multi-scale convolution layer, a fourth splicing layer, a second convolution layer, a first activation layer, and a first fusion layer; for the first input feature input into the MSHA module: The first mixed pooling layer is used to perform average pooling and maximum pooling on the first input feature respectively, to obtain first and second pooling results; The first and second pooling results are spliced by the splicing layer to obtain a first mixed pooling result; The multi-scale convolution layer is used to perform convolution operations of different scales on the first mixed pooling result after dimension adjustment, to obtain first convolution features corresponding to each scale; The third splicing layer is used to splice the first convolution features of each scale to obtain third splicing features; The first activation layer is used to perform activation on the third splicing features convolved by the second convolution layer, to obtain channel weights; The fusion layer is used to fuse the first input feature and the channel weights to obtain multi-scale channel enhanced features.

5. The pavement spraying detection method as claimed in claim 4, characterized in that, The MSSA submodule comprises a second mixed pooling layer, a fifth splicing layer, a multi-scale convolution weighting structure, a sixth splicing layer, a second activation layer, and a second fusion layer; for the multi-scale channel enhanced features input into the MSSA submodule: The second mixed pooling layer is used to perform average pooling and maximum pooling on the multi-scale channel enhanced features respectively, to obtain third and fourth pooling results; The third and fourth pooling results are spliced by the fifth splicing layer to obtain a second mixed pooling result; For each scale, the multi-scale convolution weighting structure is used to sequentially perform convolution and weighting operations of the corresponding scale on the second mixed pooling result, to obtain corresponding weighted features; the weight corresponding to the weighting operation is obtained by learning; The sixth splicing layer is used to splice the weighted features of each scale to obtain a weighted fusion feature; The second activation layer is used to perform activation on the weighted fusion feature to obtain a spatial weight; The fusion layer is used to fuse the multi-scale channel enhanced features and the spatial weight to obtain multi-scale channel-spatial enhanced features.

6. The pavement spraying detection method of any one of claims 1-5, wherein, The backbone network is provided with a C3ESP module, the C3ESP module comprises a third convolution layer, a residual branch, a multi-scale perception branch, a seventh splicing layer, and a fourth convolution layer; for the second input feature input into the C3ESP module: The third convolution layer is used to perform channel compression on the second input feature to obtain a compressed feature; The multi-scale perception branch is used to perform multi-scale deep feature extraction on the compressed feature to obtain a deep feature; The seventh splicing layer is used to splice the deep feature and the compressed feature to obtain a seventh splicing feature; The fourth convolution layer is used to perform channel expansion on the seventh splicing feature to obtain a second output feature corresponding to the second input feature.

7. The pavement dispensing detection method of claim 6, wherein, The multi-scale perception branch includes n ESPBlocks connected in series, the ESPBlock includes a progressive deep separable convolution structure, an eighth splicing layer, a fifth convolution layer, a PEMA layer, and a ninth splicing layer; for the third input feature input into the ESPBlock: Perform a progressive multi-level deep separable convolution operation on the third input feature through the progressive deep separable convolution structure to obtain a deep convolution feature of each level; The input corresponding to the first level is the third input feature, and the input of the non-first level is the deep convolution feature corresponding to the previous level; Fuse the deep convolution features of each level through the eighth splicing layer to obtain a fused deep convolution feature; Perform a channel fusion operation on the fused deep convolution feature through the fifth convolution layer; Perform a hybrid attention mechanism on the fused deep convolution feature after channel fusion through the PEMA layer to obtain a hybrid enhanced feature; Fuse the third input and the hybrid enhanced feature through the ninth splicing layer to obtain a third output feature corresponding to the third input feature.

8. A pavement spraying detection apparatus, characterized by, Comprise: The extraction module is used for extracting features of a to-be-detected image based on a backbone network of a pre-trained road surface scattering detection model, to obtain image features; The to-be-detected image includes a road surface; The fusion module is used for fusing the image features based on a neck network of the road surface scattering detection model, to obtain fused features; The detection module is used for detecting the fused features based on a detection network of the road surface scattering detection model, to obtain a detection result of road surface scattering in the to-be-detected image; Wherein, the detection network includes an LMSADet detection head, the LMSADet detection head enhances target fusion features based on multi-scale fusion and spatial attention, and detects the enhanced target fusion features to obtain the detection result; In the case that the to-be-detected image includes road surface scattering, the detection result includes a predicted bounding box of road surface scattering and a predicted scattering category.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the deep learning-based road surface scattering detection method of any one of claims 1-7.

10. A computer program product storing a computer program, characterized by, The computer program is executed by the processor to realize the deep learning-based road surface scattering detection method of any one of claims 1-7.

Images