A cloud-edge collaborative light vehicle-mounted pavement disease real-time detection method and system

Abstract

The application discloses a cloud-edge collaborative light vehicle-mounted road disease real-time detection method and system, which comprises collecting road data, creating YOLO-LFE based on a model architecture of YOLOv8n, introducing a lightweight network MobileNetV3 as a backbone network, introducing an enhanced receptive field module, designing a reinforced asymptotic feature pyramid network, and combining YOLO-LFE and a multi-target detection model to avoid multiple detections of the same target. The application introduces the lightweight network MobileNetV3 to reduce the parameters and calculation amount of the model, introduces the enhanced receptive field module to enhance the ability of the model to extract features from small objects, and designs the reinforced asymptotic feature pyramid network to reduce the problem of a large semantic gap during subsequent fusion with primary features. Through improvement between the modules, the problems of a large network structure and the fact that a traditional detection method cannot be applied to embedded devices are solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a cloud-edge collaborative method and system for real-time detection of road surface defects on lightweight vehicles. Background Technology

[0002] As the service life of transportation infrastructure increases, road surface defects become increasingly apparent. Among these defects, road surface defects have a significant impact on the safety of road transport.

[0003] Currently, image-based pavement defect detection methods mainly consist of two types: traditional image detection methods and deep learning detection methods. Traditional image processing techniques typically use manually selected features, such as color, texture, and geometric features, to segment pavement defects and then use machine learning algorithms for classification and matching to achieve defect detection. However, traditional image processing methods have certain limitations when dealing with complex road environments and cannot meet the high requirements for model generalization ability and robustness in practical engineering. In contrast, deep learning-based image processing techniques have made significant progress in the field of target detection, featuring higher accuracy, faster speed, and strong embeddability. These methods can automatically learn features and achieve automated defect detection by training models. However, many advanced detection methods currently suffer from large network structures, making them difficult to adapt to embedded device applications. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a cloud-edge collaborative method for real-time detection of road surface defects on lightweight vehicles, comprising:

[0008] Collect road surface data and build a lightweight feature-enhanced YOLO road surface defect detection network based on the YOLOv8n model architecture;

[0009] In the backbone feature extraction network part of the road surface defect detection network, a lightweight network MobileNetV3 is introduced as the backbone network, and an enhanced receptive field module is introduced to replace the original fast spatial pyramid pooling method in the road surface defect detection network.

[0010] The original enhanced progressive feature fusion network is replaced in the road surface defect detection network, and the road surface defect detection network is combined with a multi-target detection model to achieve real-time detection of vehicle-mounted road surface defects.

[0011] As a preferred embodiment of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects described in this invention, the method of introducing the lightweight network MobileNetV3 as the backbone network includes:

[0012] An attention module and an activation function h-swish(x) are added to the MobileNetV3.

[0013] The non-linear expression and gradual saturation characteristics of the activation function are incorporated into the convolutional and fully connected layers of deep neural networks.

[0014] As a preferred embodiment of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects described in this invention, the enhanced receptive field module includes:

[0015] Replace the original 1×1 convolutional layer, and control the number of output feature channels during the convolution process through the intermediate parameter 'a', i.e., C. out1 =C out / a ;

[0016] A 3×3 serial convolution sequence structure is introduced, and the same receptive field is generated for 3×3, 5×5 and 7×7 convolutions through the convolution sequence structure;

[0017] Increase the Atrous rate from 1, 3, 5 to 3, 5, 7 respectively.

[0018] As a preferred embodiment of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects described in this invention, the method includes: replacing the original enhanced progressive feature fusion network in the road surface defect detection network, comprising:

[0019] Based on the enhanced progressive feature fusion network, an enhanced progressive feature pyramid network is constructed, which is divided into low-level, high-level and top-level features.

[0020] As a preferred embodiment of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects described in this invention, the low-level, high-level, and top-level features include:

[0021] In the low-level features, the high-level features with the smallest semantic information difference are fused;

[0022] In the high-level features, the low-level, high-level and top-level features are fused together;

[0023] In the top-level features, the top-level features are fused with the high-level features, and an adaptive spatial feature fusion module is used to assign different spatial weights to features at different levels during the feature fusion process at all levels.

[0024] As a preferred embodiment of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects described in this invention, it further includes:

[0025] Before the first feature fusion, a convolutional attention module is used to fuse network information;

[0026] Before the second feature fusion, a convolutional attention module is used, and an eSE module is introduced. The eSE module is then fused into the original c2f module.

[0027] As a preferred embodiment of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects described in this invention, the convolutional attention module includes:

[0028] Through the channel attention mechanism, the features of different channels are adaptively weighted, and the weight coefficients are obtained through the weighting to highlight the features with the highest contribution in the channel and suppress the features with the lowest contribution.

[0029] A spatial attention mechanism is introduced, which adaptively adjusts the spatial distribution of feature maps to focus attention on the regions in the image that need to be detected.

[0030] The weighting coefficient refers to the degree of contribution of different channels to the feature.

[0031] Secondly, the present invention provides a cloud-edge collaborative real-time detection system for lightweight vehicle-mounted road surface defects, comprising:

[0032] The lightweight feature-enhanced YOLO pavement distress detection network modeling module is used to collect pavement data and build a lightweight feature-enhanced YOLO pavement distress detection network based on the YOLOv8n model architecture.

[0033] The lightweight feature-enhanced YOLO pavement distress detection network processing module is used to introduce the lightweight network MobileNetV3 as the backbone network in the backbone feature extraction network part of the pavement distress detection network, and introduce an enhanced receptive field module to replace the original fast spatial pyramid pooling method in the pavement distress detection network.

[0034] A lightweight feature-enhanced YOLO pavement distress detection network combined with a detection module is used to replace the original enhanced progressive feature fusion network in the pavement distress detection network, and to combine the pavement distress detection network with a multi-target detection model to achieve real-time detection of vehicle-mounted pavement distress.

[0035] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the above-described method.

[0036] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the above-described method.

[0037] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention collects road surface data and constructs a lightweight feature-enhanced YOLO road surface defect detection network based on the YOLOv8n model architecture; in the backbone feature extraction network part of the road surface defect detection network, a lightweight network MobileNetV3 is introduced as the backbone network, and an enhanced receptive field module is introduced to replace the original fast spatial pyramid pooling method in the road surface defect detection network; the original enhanced progressive feature fusion network is replaced in the road surface defect detection network, and the road surface defect detection network is combined with a multi-object detection model to achieve real-time detection of vehicle-mounted road surface defects; the introduction of the lightweight network MobileNetV3 reduces the model's parameters and computational load; the introduction of the enhanced receptive field module enhances the model's ability to extract features from small objects; the design of the enhanced progressive feature pyramid network reduces the problem of large semantic gaps when fusing with primary features; through improvements between modules, the problems of large existing network structures and the inability of traditional detection methods to be applied to embedded devices are solved. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0039] Figure 1 This is a flowchart illustrating the overall process of a cloud-edge collaborative real-time detection method for road surface defects on lightweight vehicles, as described in one embodiment of the present invention.

[0040] Figure 2 The YOLO-LFE overall network architecture of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects is described in one embodiment of the present invention.

[0041] Figure 3 This is a diagram illustrating the construction of an efficient receptive field module for a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects, as described in one embodiment of the present invention.

[0042] Figure 4 This is a structural diagram of the efficient receptive field module of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects according to an embodiment of the present invention.

[0043] Figure 5 This is a schematic diagram of an enhanced progressive feature fusion network for a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects, as described in an embodiment of the present invention.

[0044] Figure 6 This is a framework diagram of a road surface defect detection and tracking system for a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects according to an embodiment of the present invention.

[0045] Figure 7 This is a diagram of an onboard intelligent detection device for a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects, as described in an embodiment of the present invention.

[0046] Figure 8 This is a visualization of the validation set detection results of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects according to an embodiment of the present invention.

[0047] Figure 9 This is a tracking and visualization diagram of the detection results of the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects according to an embodiment of the present invention;

[0048] Figure 10 This is a detection heatmap of a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects according to an embodiment of the present invention.

[0049] Figure 11 This is a training result diagram of the RDD 2022 dataset for the cloud-edge collaborative real-time detection method for road surface defects on lightweight vehicles, as described in one embodiment of the present invention. Detailed Implementation

[0050] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0053] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0054] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0055] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0056] Example 1

[0057] Reference Figures 1 to 6 This is the first embodiment of the present invention, which provides a cloud-edge collaborative method for real-time detection of road surface defects on lightweight vehicles, including:

[0058] S1. Collect road surface data and construct a lightweight feature-enhanced YOLO road surface defect detection network based on the YOLOv8n model architecture;

[0059] It should be noted that, in order to take into account the limited computing power of edge devices, it is necessary to design a lightweight network with lower computational complexity while maintaining excellent performance. Other models cannot achieve this effect, so the YOLOv8n model architecture is adopted, which is the preferred model architecture of this invention.

[0060] Specifically, YOLOv8n can maintain real-time performance while achieving high detection accuracy and low false negative rate; moreover, YOLOv8 uses multi-scale feature fusion, which can effectively detect target objects of different sizes; and by making predictions at different levels, it captures target information at different scales, thereby improving the accuracy of model detection.

[0061] S2. In the backbone feature extraction network part of the pavement distress detection network, a lightweight network MobileNetV3 is introduced as the backbone network, and an enhanced receptive field module is introduced to replace the original fast spatial pyramid pooling method in the pavement distress detection network.

[0062] Furthermore, an attention (SE) module and an activation function h-swish(x) are added to MobileNetV3;

[0063] It should be noted that the MobileNet series includes three main models: MobileNet V1, MobileNet V2, and MobileNet V3. The MobileNet V1 model is mainly constructed by stacking layers of depthwise separable modules. In the MobileNet V2 model, inverse residual and linear bottleneck layer modules, i.e., bottleneck residual modules, are introduced on the basis of V1. Correspondingly, the MobileNet V3 model inherits the depthwise separable convolution module of the MobileNet V1 model and the bottleneck residual module of the MobileNet V2 model.

[0064] Preferably, the MobileNetV3 used in this invention is the MobileNetV3-Small version;

[0065] Specifically, the SE module, such as Figure 3 As shown, the learning ability of the model is enhanced by learning the relationships between channel features;

[0066] Furthermore, the non-linear expression and gradual saturation characteristics of activation functions are incorporated into the convolutional and fully connected layers of deep neural networks;

[0067] It should be noted that, due to the non-linear expression and gradual saturation characteristics of activation functions, they not only help to provide better gradient flow for the model, but also optimize model performance, thereby significantly improving accuracy.

[0068] Furthermore, an enhanced receptive field module is introduced;

[0069] It should be noted that the architecture of the traditional Enhanced Receptive Field (ESPP) module consists of four main branches. The first branch includes standard 1×1 convolutions and dilated convolutions, aiming to preserve the original receptive field. The second to fourth branches consist of 3×3 convolutional layers and one convolutional layer, designed to quickly extract feature information from different receptive fields. However, the traditional ESPP module typically includes multiple parallel branches, each employing a different pooling scale. These branches capture features from different receptive fields through different pooling operations (such as max pooling or average pooling). These pooling results are then processed by convolutional layers to fuse and adjust features. Finally, these feature maps are concatenated along the channel dimension to form the final enhanced receptive field features. However, due to the pooling operations at different scales and the introduction of convolutional layers, the ESPP module causes a decrease in the spatial resolution of the feature maps. This is because pooling operations divide the feature map into smaller regions and extract the pooled features, thereby losing some of the spatial location information of the original features. Furthermore, the use of convolutional layers further introduces the receptive field of the convolutional kernel, making the features at each location of the feature map affected by the surrounding region.

[0070] To address this issue, this invention replaces the original 1×1 convolutional layer in the enhanced receptive field, controlling the number of output feature channels during the convolution process, i.e., C, through an intermediate parameter 'a'. out1 =C out / a ;

[0071] It should be noted that by replacing the convolutional layers in the original enhanced receptive field and by using intermediate parameters, the problem of reduced feature map spatial resolution caused by redundant convolution operations can be effectively avoided.

[0072] Furthermore, a 3×3 serial convolution sequence structure is introduced, through which the same receptive field is generated for 3×3, 5×5 and 7×7 convolutions;

[0073] It should be noted that introducing a 3×3 serial convolution sequence structure can improve the sampling rate while reducing computational overhead, and its serial structure improves the module's running speed to a certain extent.

[0074] Furthermore, the Atrous rate will be increased from 1, 3, and 5 to 3, 5, and 7 respectively;

[0075] Specifically, increasing the Atrous rate is to improve the detection accuracy of small targets;

[0076] It should be noted that, through the above adjustments to the enhanced receptive field module, it was found that ESPP has stronger feature extraction and detection capabilities than traditional ESPP when processing target images with complex backgrounds and large-scale changes.

[0077] S3. Replace the original enhanced progressive feature fusion network in the road surface defect detection network, and combine the road surface defect detection network with the multi-object detection model to achieve real-time detection of vehicle-mounted road surface defects.

[0078] Furthermore, based on the enhanced progressive feature fusion network, an enhanced progressive feature pyramid network is constructed, which is divided into low-level, high-level and top-level features.

[0079] Specifically, in traditional enhanced progressive feature fusion networks, the Asymptotic Feature Pyramid Network (AFPN) is typically used. However, research has shown that AFPN cannot completely mitigate the large semantic gap when fusing subsequent features with primary features. Therefore, this invention constructs the Enhanced Asymptotic Feature Pyramid Network (EAFPN) based on the enhanced progressive feature fusion network, referencing... Figure 5 The dashed box portion;

[0080] It should be noted that in constructing the enhanced asymptotic feature pyramid network, we found that the semantic gap between non-adjacent level features is greater than the semantic gap between adjacent level features, especially the semantic gap between bottom and top features; this directly leads to poor fusion effect of non-adjacent level features, so it is necessary to fuse the three levels of features.

[0081] Furthermore, among the low-level features, the high-level features with the smallest semantic information gap are fused together;

[0082] Furthermore, in the high-level features, the low-level, high-level, and top-level features are integrated;

[0083] Furthermore, in the top-level features, the top-level features are fused with the high-level features, and an adaptive spatial feature fusion module is used to assign different spatial weights to features at different levels during the feature fusion process at all levels.

[0084] It should be noted that by fusing the features of the three levels, the importance of the key level is enhanced and the influence of contradictory information from different objects is reduced. This improvement makes EAFPN have a stronger feature representation ability in multi-scale object detection tasks.

[0085] Furthermore, before the first feature fusion, a convolutional attention module is used to fuse network information;

[0086] Specifically, channel attention and spatial attention mechanisms are introduced into the convolutional attention module;

[0087] Furthermore, through the channel attention mechanism, the features of different channels are adaptively weighted, and the weight coefficients are obtained through the weighting to highlight the features with the highest contribution in the channel and suppress the features with the lowest contribution.

[0088] Furthermore, a spatial attention mechanism is introduced, which adaptively adjusts the spatial distribution of feature maps to focus attention on the regions in the image that need to be detected.

[0089] Specifically, the region to be detected is the feature with the highest contribution within a channel in the channel attention mechanism;

[0090] The weighting coefficient refers to the degree of contribution of different channels to the feature;

[0091] Furthermore, before the second feature fusion, a convolutional attention module is used, and an eSE module is introduced, which is then fused into the original c2f module.

[0092] It should be noted that by improving the modules in EAFPN, the loss of channel information was avoided while reducing the computational burden and maintaining the performance of the model. At the same time, we found that integrating eSE into the original c2f module reduced the computational overhead while maintaining the model's effective use of channel information.

[0093] Furthermore, to achieve global detection and re-identification of diseases, we extract global feature information for each disease based on the ByteTrack algorithm and assign a unique number to it for disease re-identification. Through the dual constraint of number information and category information, duplicate disease information can be quickly and accurately eliminated. To achieve the above goals, we constructed a two-level detector structure based on YOLO-LFE-ByteTrack, as follows... Figure 6 The (detection and tracking network) is shown in the figure; the effectiveness of the proposed intelligent road damage detection system was verified through field tests.

[0094] Furthermore, this embodiment also provides a cloud-edge collaborative real-time detection system for lightweight vehicle-mounted road surface defects, including:

[0095] The lightweight feature-enhanced YOLO pavement distress detection network modeling module is used to collect pavement data and build a lightweight feature-enhanced YOLO pavement distress detection network based on the YOLOv8n model architecture.

[0096] The lightweight feature-enhanced YOLO pavement distress detection network processing module is used to introduce the lightweight network MobileNetV3 as the backbone network in the backbone feature extraction network part of the pavement distress detection network, and introduce an enhanced receptive field module to replace the original fast spatial pyramid pooling method in the pavement distress detection network.

[0097] A lightweight feature-enhanced YOLO pavement distress detection network combined with a detection module is used to replace the original enhanced progressive feature fusion network in the pavement distress detection network, and to combine the pavement distress detection network with a multi-target detection model to achieve real-time detection of vehicle-mounted pavement distress.

[0098] This embodiment also provides a computer device suitable for a cloud-edge collaborative method for real-time detection of road surface defects on lightweight vehicles, including:

[0099] The system includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects, as proposed in the above embodiments.

[0100] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0101] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects as proposed in the above embodiments.

[0102] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0103] Example 2

[0104] Reference Figure 7 This is the second embodiment of the present invention, which provides a cloud-edge collaborative method for real-time detection of road surface defects on lightweight vehicles, including: collecting road surface data through an on-board intelligent device;

[0105] The in-vehicle intelligent device also includes: a core computing module, a 5G wireless module, a GPS module, an IMU inertial navigation module, and a camera (sensor module);

[0106] Among them, the camera (sensor module) is used to acquire road surface information in real time;

[0107] The core computing module is the brain of the entire device, designed based on the NVIDIA Jetson Orin Nano CLB. It is a modular artificial intelligence supercomputer using the NVIDIA Pascal architecture. It is responsible for processing data acquired from cameras (sensor modules), performing real-time analysis and processing, and then detecting road defects using the YOLO-LFE model (Lightweight Feature-Enhanced YOLO Road Defect Detection Network) in this invention. The core computing module also includes high-performance processors, memory, and storage devices to ensure fast and efficient processing of complex data. The core computing module can also combine information obtained from the GPS module to geolocate road defects, facilitating subsequent viewing and processing by maintenance personnel.

[0108] The GPS module is used to obtain the vehicle's current location information, which can locate the vehicle's coordinates in real time. This is very important for accurately recording the location information of road defects and also helps with subsequent maintenance and management.

[0109] The IMU (Inertial Measurement Unit) is part of the inertial navigation system, used to acquire information about the vehicle's attitude and motion in space. Through the data from the IMU module, the device can more accurately determine the vibration, tilt, and other conditions of the vehicle during driving, which helps to improve the accuracy and reliability of road surface defect detection.

[0110] The 5G wireless module is used to enable high-speed data transmission between the vehicle system and the cloud server. Through 5G technology, the device can achieve stable data transmission while in motion, transmitting real-time collected road information, detection results, etc. to the cloud for further analysis and processing. At the same time, the cloud can also send instructions or update model data to the vehicle system to achieve cloud-edge collaborative work.

[0111] Example 3

[0112] Referring to Tables 1 and 2, the third embodiment of the present invention provides a cloud-edge collaborative method for real-time detection of road surface defects on lightweight vehicles, including: verifying the beneficial effects of the YOLO-LFE model in the present invention using experimental data obtained from experimental results;

[0113] Using self-made datasets and public datasets, the YOLO-LFE model of this invention is compared with existing advanced object detection models (YOLOv8n, YOLOv5s, YOLOv7, RT-DETR, SSD, etc.). The results are shown in Tables 1 and 2.

[0114] Table 1. Detection results of different pavement distress models

[0115]

[0116] Table 2 Ablation experiments of different modules

[0117]

[0118] As shown in Table 1, the YOLO-LFE model outperforms other models in identifying pavement defects; its precision is 91.5%, recall is 73.1%, F1 score is 81.3%, model parameters are 2.03M, model memory usage is 4.24MB, and GFLOPs are 5.1. We tested the two-stage model RT-DETR with different depths and trunks; its detection accuracy is lower than YOLO-LFE, and its model complexity and computational cost are also much higher than YOLO-LFE. Compared with single-stage models SSD, RetinaNet, and EfficientDet-D... While our model's accuracy is lower than that of CenterNet and other models, its recall and F1 score are significantly higher. Compared to other single-stage YOLO variants, our model boasts the highest accuracy and F1 score. ECB-YOLOv5 only slightly surpasses us in recall. Furthermore, our model's memory footprint, parameter count, and GFLOPs are far lower than these single-stage detectors, indicating that our model is more suitable for deployment on embedded devices. Meanwhile, the ablation study in Table 2 shows that, compared to the baseline model, YOLO-LFE effectively improves the model's ability to detect road surface damage in street view images.

[0119] Example 4

[0120] Reference Figures 8 to 11 This is the fourth embodiment of the present invention, which provides a cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects, including: verifying the superiority of the YOLO-LFE model proposed in the present invention in actual detection;

[0121] (1) Demonstrate the superiority of the model in actual detection;

[0122] Based on a self-made dataset, the detection results of YOLOv8n, YOLOv7, EfficientDet, SSD, and YOLO-LFE models were compared. Figure 8 As shown in the figures, our model has a stronger feature extraction capability for road surface defects, resulting in higher target confidence scores compared to the other four models in detecting four types of road surface defects. Furthermore, due to the linear characteristics and spatial continuity of road surface defects, the same defect may be detected multiple times when defect information is incomplete or insufficient. YOLO-LFE ensures the completeness of information extraction for the same defect by improving feature extraction capabilities. Even under shadow or water stain interference, our model still demonstrates superior performance. In the third set of images, YOLOv8 detected the same target multiple times, YOLOv7 incorrectly identified road surface water stains as vertical cracks, SSD had very low confidence, and EfficientDet missed detections. In the fourth set of images, due to shadows caused by building occlusion, the SSD model made false detections, while EfficientDet missed detections. Therefore, it is evident that the YOLO-LFE model's learning and detection capabilities under various conditions are superior to other target detection models.

[0123] We combine YOLO-LFE with the multi-object detection model ByteTrack to avoid multiple detections of the same target; and the detection results are combined with the lightweight feature-enhanced YOLO pavement distress detection network and detection module in this invention to display the pavement detection information and transmit the data to the cloud server, such as... Figure 9 As shown;

[0124] (2) Demonstrate the network's attention in the model;

[0125] We used the Gradient Weighted Class Activation Mapping (Grad-CAM) visualization method to observe which part of the image the algorithm mainly focuses on, where brighter areas indicate that the model is more confident in detecting the target. We present Grad-CAM results for YOLO-LFE, YOLOv8n, YOLOv5s, ECB-YOLO, and RT-DETR-L algorithms; for example... Figure 10 As shown in (a) to (e), YOLO-LFE focuses on the crack region, and its detection performance is better than other algorithms. In our detection task, the background is relatively complex, and the model proposed in this invention has good noise resistance, especially in… Figure 10 In (d), the model is more likely to focus on the truck because it has a slender feature similar to the crack. Although our model also focuses on the truck, it has a brighter color at the location of the defect, indicating that the probability of the defect is highest at this location.

[0126] (3) Verify the performance of the model of this invention;

[0127] We further tested the model on the public dataset—the Road Damage Detection Challenge 2020 (RDD2020). The RDD2020 dataset was collected in Japan, India, and the Czech Republic using smartphones mounted on car windshields, providing a view similar to street view images. It offers 21,041 images with four road damage labels (longitudinal cracks, reverse cracks, complex cracks, and potholes). We randomly selected 2,500 images and divided them into training, testing, and validation sets in an 8:1:1 ratio. We tested different versions of the lightweight YOLO model. The results are as follows: Figure 11 As shown, YOLO-LFE has the highest accuracy while having the fewest model parameters, which fully demonstrates that our invented model YOLO-LFE has superior performance.

[0128] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0129] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0133] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A cloud-edge collaborative method for real-time detection of road surface defects on lightweight vehicles, characterized in that, include: Collect road surface data and build a lightweight feature-enhanced YOLO road surface defect detection network based on the YOLOv8n model architecture; The introduction of the lightweight network MobileNet V3 as the backbone network includes adding an attention module and an activation function h-swish(x) to MobileNet V3; The non-linear expression and gradual saturation characteristics of the activation function are incorporated into the convolutional and fully connected layers of the deep neural network; In the backbone feature extraction network part of the road surface defect detection network, a lightweight network MobileNet V3 is introduced as the backbone network, and an enhanced receptive field module is introduced to replace the original fast spatial pyramid pooling method in the road surface defect detection network. The enhanced receptive field module includes replacing the original 1×1 convolutional layer and controlling the number of output feature channels during the convolution process through an intermediate parameter 'a', i.e., C. out1 =C out / a ; A 3×3 serial convolution sequence structure is introduced, and the same receptive field is generated for 3×3, 5×5 and 7×7 convolutions through the convolution sequence structure; Increase the Atrous rate from 1, 3, 5 to 3, 5, 7 respectively; The original enhanced progressive feature fusion network is replaced in the road surface defect detection network. This includes constructing an enhanced progressive feature pyramid network based on the enhanced progressive feature fusion network, and dividing the enhanced progressive feature pyramid network into low-level, high-level and top-level features. The road surface defect detection network and multi-target detection model are combined to achieve real-time detection of vehicle-mounted road surface defects. Before the first feature fusion, a convolutional attention module is used to fuse network information; Before the second feature fusion, a convolutional attention module is used, and an eSE module is introduced. The eSE module is then fused into the original c2f module.

2. The cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects as described in claim 1, characterized in that, The low-level, high-level, and top-level features include: fusing the high-level features with the smallest semantic information difference among the low-level features; In the high-level features, the low-level, high-level and top-level features are fused together; In the top-level features, the top-level features are fused with the high-level features, and an adaptive spatial feature fusion module is used to assign different spatial weights to features at different levels during the feature fusion process at all levels.

3. The cloud-edge collaborative real-time detection method for lightweight vehicle-mounted road surface defects as described in claim 2, characterized in that, The convolutional attention module includes: Through the channel attention mechanism, the features of different channels are adaptively weighted, and the weight coefficients are obtained through the weighting to highlight the features with the highest contribution in the channel and suppress the features with the lowest contribution. A spatial attention mechanism is introduced, which adaptively adjusts the spatial distribution of feature maps to focus attention on the regions in the image that need to be detected. The weighting coefficient refers to the degree of contribution of different channels to the feature.

4. A cloud-edge collaborative real-time detection system for lightweight vehicle-mounted pavement defects, based on the cloud-edge collaborative real-time detection method for lightweight vehicle-mounted pavement defects according to any one of claims 1 to 3, characterized in that, include: The lightweight feature-enhanced YOLO pavement distress detection network modeling module is used to collect pavement data and build a lightweight feature-enhanced YOLO pavement distress detection network based on the YOLOv8n model architecture. The lightweight feature-enhanced YOLO pavement distress detection network processing module is used to introduce the lightweight network MobileNet V3 as the backbone network in the backbone feature extraction network part of the pavement distress detection network, and introduce an enhanced receptive field module to replace the original fast spatial pyramid pooling method in the pavement distress detection network. A lightweight feature-enhanced YOLO pavement distress detection network combined with a detection module is used to replace the original enhanced progressive feature fusion network in the pavement distress detection network, and to combine the pavement distress detection network with a multi-target detection model to achieve real-time detection of vehicle-mounted pavement distress.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images