A large model-based road inspection method, device, equipment and storage medium

Abstract

The application discloses a road inspection method and device based on a large model, equipment and a storage medium, relates to the field of transportation, is applied to a cloud intelligent layer, and comprises the following steps: obtaining target data sent by a visual perception layer; processing the target data by using a target large model located in the cloud intelligent layer to extract local features to be optimized; optimizing the local features to be optimized based on traffic physical characteristics to detect diseases and predict risks of a target road based on target local features, so that disease detection results and risk prediction results are obtained; generating road maintenance suggestions corresponding to the target road based on the target local features, the disease detection results, the risk prediction results and a target knowledge graph, so as to complete road inspection of the target road; and the cloud intelligent layer, the visual perception layer and an edge computing layer are time-space synchronized based on a precision time protocol and a cubic spline interpolation. The application meets the needs of real-time early warning and whole life cycle management in the road inspection scene.

Description

Technical Field

[0001] This invention relates to the field of transportation, and in particular to a road inspection method, apparatus, equipment, and storage medium based on a large model. Background Technology

[0002] Traditional road inspection relies primarily on manual labor and single automated technologies, resulting in low efficiency, high costs, and significant safety hazards. Existing automated systems often employ single-modal detection (e.g., relying solely on cameras or LiDAR), lacking multi-source data fusion capabilities. Furthermore, general-purpose deep learning models exhibit poor adaptability in complex road scenarios, leading to high false alarm rates. In addition, existing technologies generally suffer from system-level deficiencies such as insufficient edge-cloud collaboration and a lack of dynamic update capabilities, making it difficult to meet the demands of real-time early warning and full lifecycle management. While large models perform well in natural scenes, they have not yet been optimized for the physical characteristics of traffic infrastructure and domain knowledge, resulting in a significant decrease in the accuracy of key feature extraction in road inspection scenarios. Therefore, improving road inspection methods to meet the needs of real-time early warning and full lifecycle management in road inspection scenarios is a pressing issue that needs to be addressed. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a road inspection method, device, equipment, and storage medium based on a large model, to meet the needs of real-time early warning and full lifecycle management in road inspection scenarios. The specific solution is as follows:

[0004] Firstly, this application discloses a road inspection method based on a large model, applied to a cloud-based intelligent layer, including:

[0005] The target data sent by the visual perception layer is obtained by the visual perception layer after processing the multimodal data of the target road collected by the edge computing layer.

[0006] The target data is processed using a target large model located in the cloud intelligent layer to extract local features to be optimized, and the local features to be optimized are optimized based on traffic physical characteristics to obtain target local features;

[0007] Based on the target local features, the target road is subjected to disease detection and risk prediction to obtain corresponding disease detection results and risk prediction results. Based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph, road maintenance suggestions corresponding to the target road are generated to complete the road inspection of the target road.

[0008] The cloud-based intelligent layer, the visual perception layer, and the edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation.

[0009] Optionally, the process by which the visual perception layer acquires target data includes:

[0010] Acquire multimodal data of the target road collected by the edge computing layer;

[0011] The image data in the multimodal data is dynamically segmented based on a preset point cloud denoising algorithm to extract the degraded road image from the image data;

[0012] The degraded road image is enhanced using a target image enhancement network to obtain target data based on the enhanced road image.

[0013] Optionally, the road inspection method based on a large model further includes:

[0014] If the current network latency is less than or equal to a preset network latency threshold, the target large model is used to perform real-time pothole detection on the target road based on the target data to obtain a cloud-based pothole detection result. The target lightweight model is then used through the edge computing layer to perform pothole detection on the target road based on the multimodal data to obtain an edge-based pothole detection result. This allows the target large model to generate a first target pothole detection result based on the target relative entropy, the cloud-based pothole detection result, and the edge-based pothole detection result.

[0015] If the current network latency is greater than a preset network latency threshold, the target lightweight model is used by the edge computing layer to perform real-time pothole detection on the target road based on the multimodal data to obtain a second target pothole detection result.

[0016] Optionally, the training process of the target large model includes:

[0017] Acquire raw data for various road scenes, including raw images and point cloud data and sensor time series data corresponding to each raw image;

[0018] The original image is annotated based on the point cloud data and the sensor time series data to obtain annotated data;

[0019] A target knowledge graph is constructed using the labeled data; the nodes of the target knowledge graph include road defect types, material parameters, and maintenance records.

[0020] The initial large model is pre-trained using the original image to obtain a pre-trained large model;

[0021] The pre-trained large model is fine-tuned by adjusting the weights of each task based on the dynamic gradient equalization algorithm to obtain the target large model.

[0022] The backbone network of the target large model is a hierarchical Transformer, which includes a bottom layer network, a middle layer network, and a top layer network.

[0023] Optionally, the step of processing the target data using a target large model located in the cloud intelligent layer to extract local features to be optimized, and optimizing the local features to be optimized based on traffic physical characteristics to obtain target local features, includes:

[0024] The underlying network of the target large model located in the cloud intelligent layer is used to process the road image in the target data into blocks based on a preset pixel size to obtain the block image. Then, the block image is divided into point clouds based on a preset voxel size to extract the local features to be optimized.

[0025] The intermediate-layer network is used to optimize the local features to be optimized based on the material elastic modulus of the target road and the target traffic flow heatmap to obtain the target local features;

[0026] Accordingly, the process of performing defect detection and risk prediction on the target road based on the target local features to obtain corresponding defect detection results and risk prediction results, and generating road maintenance suggestions for the target road based on the target local features, the defect detection results, the risk prediction results, and the target knowledge graph, to complete the road inspection of the target road, includes:

[0027] The multi-task output head of the top-level network is used to perform defect detection and risk prediction on the target road based on the target local features to obtain the corresponding defect detection results and risk prediction results. Based on the target local features, the defect detection results, the risk prediction results and the target knowledge graph, road maintenance suggestions corresponding to the target road are generated to complete the road inspection of the target road.

[0028] Optionally, the target traffic flow heat map is a traffic flow heat map obtained by predicting the traffic flow of the target road within a preset time using a target long short-term memory network.

[0029] Optionally, the step of processing the target data using the target large model located in the cloud intelligent layer to extract local features to be optimized includes:

[0030] The target large model located in the cloud intelligent layer is used to dynamically adjust the convolution kernel offset using deformable convolution, and the target data is processed based on the adjusted convolution kernel offset and a preset curvature constraint loss function to extract local features to be optimized.

[0031] Secondly, this application discloses a road inspection device based on a large model, applied to a cloud-based intelligent layer, comprising:

[0032] The data acquisition module is used to acquire target data sent by the visual perception layer; the target data is the data obtained by the visual perception layer after processing the multimodal data of the target road collected by the edge computing layer;

[0033] The data processing module is used to process the target data using the target large model located in the cloud intelligent layer to extract the local features to be optimized, and to optimize the local features to be optimized based on traffic physical characteristics to obtain the target local features.

[0034] The feature analysis module is used to perform disease detection and risk prediction on the target road based on the target local features to obtain corresponding disease detection results and risk prediction results, and to generate road maintenance suggestions corresponding to the target road based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph, so as to complete the road inspection of the target road;

[0035] The cloud-based intelligent layer, the visual perception layer, and the edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation.

[0036] Thirdly, this application discloses an electronic device, including:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the aforementioned road inspection method based on a large model.

[0039] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned road inspection method based on a large model.

[0040] In this application, when inspecting a target road, the cloud-based intelligent layer acquires target data sent by the visual perception layer. The target data is obtained by processing the multimodal data of the target road collected by the edge computing layer. The target data is processed using a large target model located in the cloud-based intelligent layer to extract local features to be optimized, and these local features are optimized based on traffic physical characteristics to obtain target local features. Based on the target local features, the target road undergoes defect detection and risk prediction to obtain corresponding defect detection and risk prediction results. Based on the target local features, the defect detection results, the risk prediction results, and the target knowledge graph, road maintenance suggestions corresponding to the target road are generated to complete the road inspection. The cloud-based intelligent layer, the visual perception layer, and the edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation. As can be seen, in this application, after the edge computing layer collects multimodal data such as images, point clouds, and sensor data of the target road, it sends this multimodal data to the visual perception layer. The visual perception layer then processes this multimodal data and sends the processed target data to the cloud intelligence layer. The target big model of the cloud intelligence layer combines traffic physical characteristics (such as material mechanical parameters, load distribution patterns, etc.) to process this target data to perform defect detection and risk prediction for the target road. This significantly improves the accuracy of defect identification while shortening the detection time and reducing the false alarm rate. Finally, based on the defect detection results, risk prediction results, and target knowledge graph, corresponding maintenance suggestions are generated to fulfill the full lifecycle management requirements for inspecting the target road. In addition, the cloud intelligence layer, visual perception layer, and edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation, enabling rapid response from data acquisition to analysis reports and meeting emergency early warning requirements. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a flowchart of a road inspection method based on a large model disclosed in this application;

[0043] Figure 2 This is a schematic diagram of a specific road inspection method based on a large model disclosed in this application;

[0044] Figure 3This is a schematic diagram of a road inspection device based on a large model disclosed in this application;

[0045] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Traditional road inspection relies primarily on manual labor and single automated technologies, resulting in low efficiency, high costs, and significant safety hazards. Existing automated systems often employ single-modal detection (e.g., relying solely on cameras or LiDAR), lacking multi-source data fusion capabilities. Furthermore, general-purpose deep learning models exhibit poor adaptability in complex road scenarios, leading to high false alarm rates. In addition, existing technologies generally suffer from system-level deficiencies such as insufficient edge-cloud collaboration and a lack of dynamic update capabilities, making it difficult to meet the needs of real-time early warning and full lifecycle management. While large-scale models perform well in natural scenes, they have not yet been optimized for the physical characteristics of traffic infrastructure and domain knowledge, resulting in a significant decrease in the accuracy of key feature extraction in road inspection scenarios. To address these technical issues, this application discloses a road inspection method based on a large-scale model, capable of meeting the needs of real-time early warning and full lifecycle management in road inspection scenarios.

[0048] See Figure 1 As shown, this embodiment of the invention discloses a road inspection method based on a large model, applied to the cloud intelligent layer, including:

[0049] Step S11: Obtain target data sent by the visual perception layer; the target data is the data obtained by the visual perception layer after processing the multimodal data of the target road collected by the edge computing layer.

[0050] In this embodiment, the cloud intelligence layer, visual perception layer, and edge computing layer achieve multi-sensor hardware-level clock synchronization based on a precise time protocol. Cubic spline interpolation compensates for data offsets caused by transmission jitter, ensuring frame-level alignment of images, point clouds, and sensor data (spatiotemporal registration error can be less than 0.5 pixels), thus achieving spatiotemporal synchronization. In one specific implementation, the edge devices of the edge computing layer include hardware such as inspection drones and inspection vehicles. The inspection drone is equipped with a high-definition camera supporting a wide dynamic range (e.g., 120dB), ensuring image clarity in strong backlight or nighttime scenes. It also carries a lidar with a detection range sufficient for detection requirements (e.g., 260 meters), allowing the inspection drone to scan the target road using the lidar's laser to generate a high-density point cloud (e.g., 100,000 points / second), and achieve centimeter-level positioning in conjunction with an inertial navigation module. The inspection drone also possesses anti-jamming communication capabilities, employing MIMO (Multiple-Input Multiple-Output) technology and LDPC (Low-Density Parity-Check Code) encoding to maintain a data transmission rate greater than or equal to 50Mbps even in rainy and foggy environments. The inspection vehicle integrates an infrared thermal imager with sufficient thermal sensitivity (e.g., less than 50mK), a millimeter-wave radar supporting four-dimensional imaging (distance, azimuth, altitude, and velocity), and a road vibration sensor with preset sensitivity conditions (e.g., 0.01g). The infrared thermal imager can detect abnormal road surface temperatures (e.g., underground pipe leaks), the millimeter-wave radar can capture dynamic obstacles, and the road vibration sensor can monitor roadbed settlement in real time.

[0051] In this embodiment, after acquiring the raw data, the edge computing layer performs data standardization. Specifically, the edge computing layer uses adaptive histogram equalization to enhance contrast and bicubic interpolation to unify the resolution (e.g., 1920×1080). Simultaneously, when acquiring point cloud data, the edge computing layer uses a preset voxel size (e.g., 5 cubic centimeters) for voxel grid downsampling to reduce data density while preserving as much effective information as possible. In addition, based on PTP (Precision Time Protocol Synchronization), the edge computing layer will also use a sliding window for dynamic calibration of the sensor data stream (the window size can be set to 1 second), fit the time offset using the least squares method, and perform multi-sensor joint calibration based on the calibration board (the detection accuracy of checkerboard corner points can be set to ±0.1 pixels). Furthermore, it will use the improved ORB-SLAM3 (Oriented Fast and Rotated Brief Simultaneous Localization and Mapping 3) algorithm to build a three-dimensional semantic map of the road in real time (including lane lines, signs, and other elements).

[0052] In this embodiment, the computing power of the hardware device of the visual perception layer should meet the preset computing power conditions so that the multimodal data fusion engine deployed on the device can realize real-time processing of the multimodal data sent by the edge computing layer to obtain the target data. In a specific implementation, the process of the visual perception layer acquiring the target data may include: acquiring multimodal data of the target road collected by the edge computing layer; dynamically segmenting the image data in the multimodal data based on a preset point cloud denoising algorithm to extract the degraded road image from the image data; and using a target image enhancement network to enhance the degraded road image to obtain the target data based on the enhanced road image. In other words, the visual perception layer dynamically segments the road surface point cloud and dynamic obstacles such as vehicles and pedestrians based on the improved RANSAC (Random Sample Consensus) algorithm, achieving a point cloud denoising effect with a filtering rate of over 95%. For rainy, foggy, and low-light scenes, the visual perception layer uses the CycleGAN (Cycle-Consistent Generative Adversarial Networks) network to map degraded images to the clear image domain, thereby supporting road inspection in harsh environments such as rain, fog, and nighttime, and significantly improving the accuracy of hazard identification in low-light scenes. At the same time, the cross-modal attention network constructed by the visual perception layer establishes the spatial correspondence between image pixels and point cloud voxels through deformable convolutions, and introduces road surface curvature constraints (calculating the radius of curvature by fitting a plane using RANSAC) to optimize feature matching. The CycleGAN network's generator can contain 8 residual blocks, while the discriminator can use a PatchGAN structure. The resulting image has a PSNR (Peak Signal-to-Noise Ratio) greater than or equal to 32 dB. Figure 2 As shown, after obtaining the target data, the visual perception layer sends the target data to the cloud intelligence layer.

[0053] Step S12: Use the target large model located in the cloud intelligent layer to process the target data to extract the local features to be optimized, and optimize the local features to be optimized based on traffic physical characteristics to obtain the target local features.

[0054] In this embodiment, the target large model of the cloud-based intelligent layer is a hybrid neural network model specifically designed for traffic scenarios, with a parameter scale of 15 bytes or more. The training process of the target large model specifically includes: acquiring raw data for various road scenarios, including raw images and corresponding point cloud data and sensor time-series data; annotating the raw images based on the point cloud data and sensor time-series data to obtain annotated data; constructing a target knowledge graph using the annotated data; the nodes of the target knowledge graph include road defect types, material parameters, and maintenance records; pre-training the initial large model using the raw images to obtain a pre-trained large model; and fine-tuning the pre-trained large model by adjusting the weights of each task based on a dynamic gradient balancing algorithm to obtain the target large model. The backbone network of the target large model is a hierarchical Transformer, comprising a bottom layer network, a middle layer network, and a top layer network.

[0055] Understandably, the raw data used to train this target model covers 20 or more typical road scenarios, including highways, bridge expansion joints, and tunnel seepage. The raw images in this data can be at least 1 million 4K resolution images, at least 5000 kilometers of point cloud data, and at least 100,000 sets of sensor time-series data. When annotating these raw images, a semi-automatic annotation tool can be used, combining SAM (Segment Anything Model) pre-annotation with manual verification to annotate various defects (such as transverse cracks, network cracks, and potholes), resulting in initially annotated images. Then, the point cloud data is used to annotate the 3D defect areas of these initially annotated images, and depth information is associated (accuracy can be set to ±2cm), resulting in secondarily annotated images (i.e., annotated data). Finally, the secondarily annotated images are used to construct a target knowledge graph, which can be built based on the Neo4j graph database. The nodes of the target knowledge graph include road defect types, material parameters, and maintenance records, and the edge relationships can be defined similarly to "crack - leading to - structural strength reduction". When training the initial large model, self-supervised pre-training using MAE (Masked Autoencoder) can be performed on 1 million unlabeled road images. The masking rate is 75%, and the reconstruction loss uses a hybrid L1+L2 loss. The temperature coefficient can be set to 0.07 to construct positive sample pairs (images and point cloud clippings at the same location) and negative sample pairs (randomly sampled data from different locations) for training the initial large model. During the fine-tuning phase, the weights of each task can be automatically adjusted using a dynamic gradient equalization algorithm (GradNorm), and training can be performed in stages according to road complexity. For example, the proportion of flat straight road data in the training set can be 60%, the proportion of curved and conventional road damage data can be 25%, and the proportion of extreme weather and complex road damage data can be 15%. In the fine-tuning phase, training can be performed in stages using flat straight road data, curved and conventional road damage data, and extreme weather and complex road damage data in sequence.

[0056] In one specific implementation, the bottom-layer network can utilize ViT-Base (Vision TransformerBase) to divide the image data into blocks of the target pixel size (e.g., 16×16 pixel patches), and perform point cloud segmentation on the blocks according to a preset voxel size (e.g., 5 cubic centimeters) to obtain the local features to be optimized. Furthermore, the middle-layer network can embed prior knowledge of road material and other traffic physical characteristics into the local features to be optimized. For example, a pre-trained ResNet-18 classification network can weight the material's elastic modulus (e.g., asphalt modulus 1.5 GPa, concrete 30 GPa) in the attention mechanism to emphasize texture continuity when the target road is asphalt and focus on joint features when the target road is concrete. Simultaneously, the LSTM (Long Short-Term Memory) network included in the middle-layer network is used to predict the traffic flow of the target road within a preset time period (e.g., within the next 30 minutes) to obtain a target traffic flow heatmap, which is then overlaid on the feature map. In other words, the target data is processed using a large target model located in the cloud-based intelligent layer to extract local features to be optimized. These local features are then optimized based on traffic physical characteristics to obtain the target local features. Specifically, this can include: using the bottom-layer network of the target large target model in the cloud-based intelligent layer to process the road image in the target data into blocks based on a preset pixel size, obtaining block images; and then performing point cloud segmentation on the block images based on a preset voxel size to extract the local features to be optimized. The middle-layer network optimizes the local features based on the material elastic modulus of the target road and the target traffic flow heatmap to obtain the target local features. This large model design, which integrates road physical characteristics (such as elastic modulus and radius of curvature), improves the accuracy of defect identification for special structures such as bridges and tunnels by 35%.

[0057] In this embodiment, when extracting local features to be optimized, the underlying network can use the Swing Transformer to extract multi-scale image features and capture local to global information through a hierarchical sliding window (the window size can be set to 7×7). Simultaneously, a PointNeXt network (a point cloud network) is used to extract multi-scale point cloud features. Stepwise downsampling is performed through a Set Abstraction layer (the sampling rate can be set to 0.5), and the KNN (K-Nearest Neighbors, an instance-based supervised learning algorithm) algorithm (k can be 16) is used to construct local geometric relationships. For each point cloud cluster, the radius of curvature needs to be calculated to suppress mismatches caused by road surface undulations. Finally, a preset road surface curvature penalty term is used to fuse multi-scale point cloud features with multi-scale image features based on a cross-modal cross-attention mechanism, thereby obtaining the local features to be optimized. In addition, to address the issue of spatial misalignment between the image and the point cloud caused by differences in perspective, a large target model located in the cloud-based intelligent layer is used to process the target data to extract local features to be optimized. This includes: using deformable convolution to dynamically adjust the kernel offset of the target large model located in the cloud-based intelligent layer, and processing the target data based on the adjusted kernel offset and a preset curvature constraint loss function to extract local features to be optimized.

[0058] Step S13: Based on the target local features, perform disease detection and risk prediction on the target road to obtain corresponding disease detection results and risk prediction results, and generate road maintenance suggestions corresponding to the target road based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph to complete the road inspection of the target road.

[0059] In this embodiment, as Figure 2As shown, an optimized YOLOv7-Tiny model (a lightweight object detection model) can run on edge devices. Through FP16 (Half-precision floating-point) quantization, the model size is compressed to 12MB, enabling real-time detection of cracks and potholes (e.g., 30 FPS, latency less than 50ms). Therefore, if the current network latency is less than or equal to a preset network latency threshold, the target large model is used to perform real-time pothole detection on the target road based on target data to obtain cloud-based pothole detection results. The edge computing layer then uses the target lightweight model based on multimodal data to perform pothole detection on the target road to obtain edge-based pothole detection results. This allows the target large model to generate a first target pothole detection result based on the target relative entropy, cloud-based pothole detection results, and edge-based pothole detection results. If the current network latency is greater than the preset network latency threshold, the edge computing layer uses the target lightweight model based on multimodal data to perform real-time pothole detection on the target road to obtain a second target pothole detection result. The optimization of the YOLOv7-Tiny model can be performed using TensorRT (Tensor Runtime, an inference acceleration engine) to convert the model to FP16 precision and fuse convolutional and activation layers, thereby improving inference speed. Simultaneously, dynamic shape optimization is used to enable the optimized lightweight model to support dynamic adjustment of the input resolution (the adjustment range can be from 720p to 4K).

[0060] In other words, this embodiment can run lightweight models (such as MobileViT-XXS, Mobile Vision Transformer Extra Extra Small) at the edge computing layer to handle tasks with high real-time requirements (such as pit detection) and upload critical data via a 5G network (bandwidth usage less than 10Mbps). At the cloud intelligence layer, a target large model with a parameter scale of 15B or more is deployed to perform structural safety assessments (such as finite element stress analysis simulations), and the results are pushed to the management terminal via a WebSocket connection. Simultaneously, to achieve dynamic resource scheduling, when performing high-real-time tasks such as pit detection, the processing node can be adaptively selected based on network conditions (latency, bandwidth). For example, when the network latency is greater than 200ms, local decision-making at the edge computing layer is enabled; when network conditions are good, pit detection results can be generated simultaneously by the edge computing layer and the cloud intelligence layer, and the final pit detection result is determined based on these two results. Specifically, response-based distillation can be used to minimize the KL divergence between the output distributions of the lightweight model and the large model, so as to align the output distributions of the target large model and the lightweight model at the edge based on the KL divergence loss. This collaborative computing between the edge and cloud reduces the number of high-performance computing devices required (only one roadside unit is needed per 50 kilometers). Furthermore, dynamic channel pruning can be used to sparse network channels based on the input scenario, sorting feature map activation values ​​(after ReLU) by L1 norm, and dynamically disabling redundant channels to reduce the computational load of the edge computing layer, thereby adjusting the compression ratio of the lightweight model to between 30% and 70%. Understandably, to further accelerate the computation speed of the lightweight model at the edge, an asynchronous pipeline approach can be adopted, with data acquisition, preprocessing, and transmission executed in parallel, and a circular buffer (buffer size can be set to 500 frames) used to eliminate waiting latency.

[0061] In this embodiment, based on the target's local features, damage detection and risk prediction are performed on the target road to obtain corresponding damage detection results and risk prediction results. Then, based on the target's local features, damage detection results, risk prediction results, and the target knowledge graph, road maintenance suggestions are generated for the target road to complete the road inspection. This includes: using the multi-task output head of the top-level network to perform damage detection and risk prediction on the target road based on the target's local features to obtain corresponding damage detection results and risk prediction results; and then generating road maintenance suggestions for the target road based on the target's local features, damage detection results, risk prediction results, and the target knowledge graph to complete the road inspection. In other words, the multi-task output head of the top-level network of the target large model can process damage detection and risk prediction tasks in parallel based on the target's local features. Then, it can use the target knowledge graph to perform inference based on the damage detection results and risk prediction results to obtain corresponding maintenance suggestions. Specifically, the damage detection task can be executed using an improved Mask R-CNN (Mask Region-based Convolutional Neural Network), and the risk prediction task can be executed using a time-series Transformer. The target large model provided in this embodiment supports online incremental learning, continuously updates the target knowledge graph, and updates model parameters online based on an elastic weight solidification algorithm, enabling prediction of disease evolution and optimization of maintenance decisions. The model update cycle for new disease types (such as salt-frost corrosion) is shortened to 24 hours, and it can also provide an early warning of road collapse risk up to 72 hours in advance (AUC (Area Under Curve) value 0.93), shortening the emergency response time to less than 10 minutes.

[0062] It should be noted that, in this embodiment, when generating road maintenance recommendations for the target road, the target large model can construct a risk matrix and use an improved CVSS (Common Vulnerability Scoring System) to quantify the severity and probability of occurrence (based on historical data and time series predictions). The model then dynamically prioritizes decision feedback based on the quantified scoring results. Furthermore, this target large model is compatible with existing traffic management systems such as BIM (Building Information Modeling) and CMMS (Computerized Maintenance Management System). Data interoperability is achieved through a standard API interface (Application Programming Interface) based on the RESTful protocol. Analysis results are output through a visual interface and a work order system, supporting real-time alarms and maintenance resource scheduling, reducing the deployment cycle by 50%. For example, it can interface with CMMS to automatically generate work orders compliant with ISO 55000 standards, including treatment recommendations (such as crack grouting process), required resources (including materials and manpower), and time windows. It also supports a closed-loop verification system for maintenance decisions, automatically optimizing the model's recommendation strategy by comparing the execution effects (repair rate, cost deviation) of historical work orders. Data interaction with a BIM system allows inspection results to be mapped onto a 3D digital twin of the road, with the evolution trend of road defects marked (time resolution can be set to once per hour). The use of multimodal fusion technology can achieve an F1-score of 98.7% for crack detection, reduce the false alarm rate to 1.2%, and increase the coverage mileage of a single inspection to 200 km / h. From data acquisition to report generation, only 3 minutes are required, achieving a dual improvement in detection accuracy and efficiency. If the automated road inspection method provided in this embodiment replaces more than 80% of manual road inspection tasks, simulation calculations show that annual maintenance budget savings exceed 3 million yuan per 100 km, achieving significant technological improvements and economic benefits in road inspection scenarios.

[0063] As can be seen, in this application, after the edge computing layer collects multimodal data such as images, point clouds, and sensor data of the target road, it sends this multimodal data to the visual perception layer. The visual perception layer then processes this multimodal data and sends the processed target data to the cloud intelligence layer. The target big model of the cloud intelligence layer combines traffic physical characteristics (such as material mechanical parameters, load distribution patterns, etc.) to process this target data to perform defect detection and risk prediction for the target road. This significantly improves the accuracy of defect identification while shortening the detection time and reducing the false alarm rate. Finally, based on the defect detection results, risk prediction results, and target knowledge graph, corresponding maintenance suggestions are generated to fulfill the full lifecycle management requirements for inspecting the target road. In addition, the cloud intelligence layer, visual perception layer, and edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation, enabling rapid response from data acquisition to analysis reports and meeting emergency early warning requirements.

[0064] See Figure 3 As shown, this application discloses a road inspection device based on a large model, applied to the cloud intelligent layer, including:

[0065] The data acquisition module 11 is used to acquire target data sent by the visual perception layer; the target data is the data obtained by the visual perception layer after processing the multimodal data of the target road collected by the edge computing layer.

[0066] The data processing module 12 is used to process the target data using the target large model located in the cloud intelligent layer to extract local features to be optimized, and to optimize the local features to be optimized based on traffic physical characteristics to obtain target local features.

[0067] The feature analysis module 13 is used to perform disease detection and risk prediction on the target road based on the target local features to obtain the corresponding disease detection results and risk prediction results, and to generate road maintenance suggestions corresponding to the target road based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph, so as to complete the road inspection of the target road;

[0068] The cloud-based intelligent layer, the visual perception layer, and the edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation.

[0069] As can be seen, in this application, after the edge computing layer collects multimodal data such as images, point clouds, and sensor data of the target road, it sends this multimodal data to the visual perception layer. The visual perception layer then processes this multimodal data and sends the processed target data to the cloud intelligence layer. The target big model of the cloud intelligence layer combines traffic physical characteristics (such as material mechanical parameters, load distribution patterns, etc.) to process this target data to perform defect detection and risk prediction for the target road. This significantly improves the accuracy of defect identification while shortening the detection time and reducing the false alarm rate. Finally, based on the defect detection results, risk prediction results, and target knowledge graph, corresponding maintenance suggestions are generated to fulfill the full lifecycle management requirements for inspecting the target road. In addition, the cloud intelligence layer, visual perception layer, and edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation, enabling rapid response from data acquisition to analysis reports and meeting emergency early warning requirements.

[0070] In one specific implementation, the visual perception layer may include:

[0071] A multimodal data acquisition module is used to acquire multimodal data of the target road collected by the edge computing layer;

[0072] The image segmentation module is used to dynamically segment the image data in the multimodal data based on a preset point cloud denoising algorithm to extract the degraded road image in the image data;

[0073] An image enhancement module is used to enhance the degraded road image using a target image enhancement network to obtain target data based on the enhanced road image.

[0074] In one specific embodiment, the device may further include:

[0075] The first pothole detection module is used to perform real-time pothole detection on the target road based on the target data using the target large model if the current network latency is less than or equal to a preset network latency threshold, so as to obtain a cloud-based pothole detection result. The module also uses the target lightweight model based on the multimodal data to perform pothole detection on the target road through the edge computing layer, so as to obtain an edge pothole detection result. This allows the target large model to generate a first target pothole detection result based on the target relative entropy, the cloud-based pothole detection result, and the edge pothole detection result.

[0076] The second pothole detection module is used to perform real-time pothole detection on the target road based on the multimodal data through the edge computing layer if the current network latency is greater than a preset network latency threshold, so as to obtain the second target pothole detection result.

[0077] In one specific embodiment, the device may include:

[0078] The raw data acquisition module is used to acquire raw data for various road scenarios. The raw data includes raw images and point cloud data and sensor time series data corresponding to each raw image.

[0079] The data annotation module is used to annotate the original image based on the point cloud data and the sensor time series data to obtain annotated data;

[0080] The knowledge graph construction module is used to construct a target knowledge graph using the labeled data; the nodes of the target knowledge graph include road defect types, material parameters, and maintenance records.

[0081] A large model pre-training module is used to pre-train an initial large model using the original image to obtain a pre-trained large model;

[0082] The large model fine-tuning module is used to adjust the weights of each task based on the dynamic gradient equalization algorithm to fine-tune the pre-trained large model in order to obtain the target large model.

[0083] The backbone network of the target large model is a hierarchical Transformer, which includes a bottom layer network, a middle layer network, and a top layer network.

[0084] In one specific embodiment, the data processing module 12 may include:

[0085] The first feature extraction unit is used to process the road image in the target data into blocks based on a preset pixel size using the underlying network of the target large model located in the cloud intelligent layer to obtain the block image, and to perform point cloud division on the block image based on a preset voxel size to extract the local features to be optimized.

[0086] The feature optimization unit is used to optimize the local features to be optimized by the middle layer network based on the material elastic modulus of the target road and the target traffic flow heatmap to obtain the target local features;

[0087] Accordingly, the feature analysis module 13 may specifically include:

[0088] The feature analysis unit is used to perform defect detection and risk prediction on the target road based on the target local features using the multi-task output head of the top-level network to obtain the corresponding defect detection results and risk prediction results, and to generate road maintenance suggestions corresponding to the target road based on the target local features, the defect detection results, the risk prediction results and the target knowledge graph, so as to complete the road inspection of the target road.

[0089] In one specific embodiment, the data processing module 12 may include:

[0090] The second feature extraction unit is used to dynamically adjust the kernel offset of the target large model located in the cloud intelligent layer using deformable convolution, and to process the target data based on the adjusted kernel offset and the preset curvature constraint loss function to extract local features to be optimized.

[0091] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0092] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the road inspection method based on a large model disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0093] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0094] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0095] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the large-model-based road inspection method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0096] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned road inspection method based on a large model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0100] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A road inspection method based on a large model, characterized in that, Applications in the cloud-based intelligent layer include: The target data sent by the visual perception layer is obtained by the visual perception layer after processing the multimodal data of the target road collected by the edge computing layer. The target data is processed using a target large model located in the cloud intelligent layer to extract local features to be optimized, and the local features to be optimized are optimized based on traffic physical characteristics to obtain target local features; Based on the target local features, the target road is subjected to disease detection and risk prediction to obtain corresponding disease detection results and risk prediction results. Based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph, road maintenance suggestions corresponding to the target road are generated to complete the road inspection of the target road. The cloud-based intelligent layer, the visual perception layer, and the edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation. The backbone network of the target large model is a hierarchical Transformer, which includes a bottom layer network, a middle layer network, and a top layer network. The process of using a target large model located in the cloud intelligent layer to process the target data to extract local features to be optimized, and optimizing the local features to be optimized based on traffic physical characteristics to obtain target local features, includes: The underlying network of the target large model located in the cloud intelligent layer is used to process the road image in the target data into blocks based on a preset pixel size to obtain the block image. Then, the block image is divided into point clouds based on a preset voxel size to extract the local features to be optimized. The intermediate-layer network is used to optimize the local features to be optimized based on the material elastic modulus of the target road and the target traffic flow heatmap to obtain the target local features.

2. The road inspection method based on a large model according to claim 1, characterized in that, The process by which the visual perception layer acquires target data includes: Acquire multimodal data of the target road collected by the edge computing layer; The image data in the multimodal data is dynamically segmented based on a preset point cloud denoising algorithm to extract the degraded road image from the image data; The degraded road image is enhanced using a target image enhancement network to obtain target data based on the enhanced road image.

3. The road inspection method based on a large model according to claim 1, characterized in that, Also includes: If the current network latency is less than or equal to a preset network latency threshold, the target large model is used to perform real-time pothole detection on the target road based on the target data to obtain a cloud-based pothole detection result. The target lightweight model is then used through the edge computing layer to perform pothole detection on the target road based on the multimodal data to obtain an edge-based pothole detection result. This allows the target large model to generate a first target pothole detection result based on the target relative entropy, the cloud-based pothole detection result, and the edge-based pothole detection result. If the current network latency is greater than a preset network latency threshold, the target lightweight model is used by the edge computing layer to perform real-time pothole detection on the target road based on the multimodal data to obtain a second target pothole detection result.

4. The road inspection method based on a large model according to claim 1, characterized in that, The training process of the target large model includes: Acquire raw data for various road scenes, including raw images and point cloud data and sensor time series data corresponding to each raw image; The original image is annotated based on the point cloud data and the sensor time series data to obtain annotated data; A target knowledge graph is constructed using the labeled data; the nodes of the target knowledge graph include road defect types, material parameters, and maintenance records. The initial large model is pre-trained using the original image to obtain a pre-trained large model; The pre-trained large model is fine-tuned by adjusting the weights of each task based on the dynamic gradient balancing algorithm to obtain the target large model.

5. The road inspection method based on a large model according to claim 1, characterized in that, The process of detecting road defects and predicting risks based on the target local features to obtain corresponding defect detection and risk prediction results, and generating road maintenance suggestions for the target road based on the target local features, the defect detection results, the risk prediction results, and the target knowledge graph, to complete the road inspection of the target road, includes: The multi-task output head of the top-level network is used to perform disease detection and risk prediction on the target road based on the target local features to obtain the corresponding disease detection results and risk prediction results. Based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph, road maintenance suggestions corresponding to the target road are generated to complete the road inspection of the target road.

6. The road inspection method based on a large model according to claim 5, characterized in that, The target traffic flow heatmap is a traffic flow heatmap obtained by predicting the traffic flow of the target road within a preset time using a target long short-term memory network.

7. The road inspection method based on a large model according to any one of claims 1 to 6, characterized in that, The process of using a large target model located in the cloud-based intelligent layer to process the target data to extract local features to be optimized includes: The target large model located in the cloud intelligent layer is used to dynamically adjust the convolution kernel offset using deformable convolution, and the target data is processed based on the adjusted convolution kernel offset and a preset curvature constraint loss function to extract local features to be optimized.

8. A road inspection device based on a large model, characterized in that, Applications in the cloud-based intelligent layer include: The data acquisition module is used to acquire target data sent by the visual perception layer; the target data is the data obtained by the visual perception layer after processing the multimodal data of the target road collected by the edge computing layer; The data processing module is used to process the target data using the target large model located in the cloud intelligent layer to extract the local features to be optimized, and to optimize the local features to be optimized based on traffic physical characteristics to obtain the target local features. The feature analysis module is used to perform disease detection and risk prediction on the target road based on the target local features to obtain corresponding disease detection results and risk prediction results, and to generate road maintenance suggestions corresponding to the target road based on the target local features, the disease detection results, the risk prediction results and the target knowledge graph, so as to complete the road inspection of the target road; The cloud-based intelligent layer, the visual perception layer, and the edge computing layer achieve spatiotemporal synchronization based on a precise time protocol and cubic spline interpolation. The backbone network of the target large model is a hierarchical Transformer, which includes a bottom layer network, a middle layer network, and a top layer network. The data processing module includes: The first feature extraction unit is used to process the road image in the target data into blocks based on a preset pixel size using the underlying network of the target large model located in the cloud intelligent layer to obtain the block image, and to perform point cloud division on the block image based on a preset voxel size to extract the local features to be optimized. The feature optimization unit is used to optimize the local features to be optimized by the middle layer network based on the material elastic modulus of the target road and the target traffic flow heatmap to obtain the target local features.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the road inspection method based on a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the road inspection method based on a large model as described in any one of claims 1 to 7.

Images