An artificial intelligence road disease image recognition positioning system

Abstract

The application discloses a road disease image recognition positioning system for artificial intelligence, and relates to the technical field of artificial intelligence image recognition and road maintenance management. A multi-source data acquisition module is used for synchronously acquiring road pavement images and spatial position data. The multi-source data acquisition module comprises a high-resolution image acquisition unit, a satellite positioning unit, an inertial measurement unit and an odometer unit. Each unit realizes hardware-level time synchronization through a time synchronization controller. The application has the advantages that an integrated system architecture comprising an image acquisition module, a positioning sensor module and a coordinate mapping module is constructed, end-to-end mapping of disease detection results from pixel coordinates to geographic coordinates is realized, each detected disease has accurate longitude and latitude position information, and the road maintenance management geographic information system can be directly accessed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence image recognition and road maintenance management technology, specifically to an artificial intelligence-based road defect image recognition and positioning system. Background Technology

[0002] Road defect detection is a crucial aspect of road maintenance and management. With the rapid development of artificial intelligence technology, deep learning-based road defect image recognition methods have gradually replaced traditional manual inspection methods, becoming the mainstream research and application approach.

[0003] In the field of image recognition, convolutional neural networks (CNNs) are currently the most widely used deep learning models. They automatically extract image features through multi-layer convolutional operations, achieving significant results in object detection and image classification tasks. Commonly used object detection frameworks include Faster R-CNN, the YOLO series, and SSD. These frameworks generate candidate regions through region proposal networks or anchor box mechanisms, and then use classification and regression networks to detect and locate objects. Feature Pyramid Networks (FPNs) are a classic multi-scale feature fusion method. Through top-down paths and lateral connections, they fuse high-level semantic information from deep networks with high-resolution detail information from shallow networks, generating feature maps with multi-scale representation capabilities, effectively improving the detection capability for objects at different scales.

[0004] For spatial localization of road defects, a common method is to combine GPS / GNSS satellite positioning systems to obtain vehicle location information and synchronize it with the image acquisition time, thereby associating the defect detection results with geographic coordinates. Inertial navigation systems can provide position compensation in areas where GPS signals are unstable.

[0005] However, existing technologies have the following shortcomings:

[0006] First, existing road defect detection solutions mostly focus on the classification and identification of defect types, lacking a complete technical link to accurately map the identification results to geographic coordinates. This results in the detection results not being directly used for precise dispatching and location-based repair in road maintenance management.

[0007] Second, road defects vary greatly in scale, with small cracks and large potholes coexisting. Existing single feature extraction networks struggle to simultaneously capture the detailed features of small defects and the semantic features of large defects, resulting in a high rate of missed detections for small-scale defects. Furthermore, the sub-region division of fixed grids is difficult to accurately fit the shapes of curved cracks and irregular pothole boundaries, limiting its adaptability to irregularly shaped defects.

[0008] Third, road image acquisition is affected by changes in lighting, shadows, and rainwater. Existing systems lack targeted adaptive image preprocessing methods for the inference stage, resulting in a significant decrease in recognition accuracy under harsh environmental conditions.

[0009] Fourth, high-precision deep learning models typically have a large number of parameters and high computational complexity, making it difficult to achieve real-time inference on vehicle edge devices. Meanwhile, lightweight models may sacrifice detection accuracy. Existing technologies lack collaborative computing capabilities between the edge and the cloud. To address this, we propose an artificial intelligence-based road defect image recognition and positioning system. Summary of the Invention

[0010] The purpose of this invention is to provide an artificial intelligence-based road defect image recognition and positioning system.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a road defect image recognition and positioning system for artificial intelligence, comprising:

[0012] A multi-source data acquisition module is used to simultaneously acquire road surface images and spatial location data. The multi-source data acquisition module includes a high-resolution image acquisition unit, a satellite positioning unit, an inertial measurement unit, and an odometer unit. Each unit achieves hardware-level time synchronization through a time synchronization controller.

[0013] The environmental perception adaptive preprocessing module is used to calculate environmental indicators for the acquired road images, and adaptively select image enhancement strategies for preprocessing based on the environmental indicator analysis results, and output preprocessed images.

[0014] The multi-scale road disease detection module includes a rapid initial screening sub-module deployed on edge devices and a fine detection sub-module deployed on cloud servers. It is used to perform multi-scale road disease detection on preprocessed images and output disease categories, confidence scores, and pixel coordinate bounding boxes.

[0015] The disease spatial positioning module is used to convert the pixel coordinates of disease detection results into geospatial coordinates through a four-level coordinate mapping algorithm;

[0016] The edge-cloud collaborative computing module is used to classify and process the detection results output by the rapid initial screening submodule according to confidence level, and coordinate the allocation of computing tasks between the edge and the cloud.

[0017] The results output and management module is used to annotate the disease detection results with geographic coordinates on geographic information system maps, generate detection reports, and manage maintenance assignments.

[0018] As a further aspect of the present invention: the time synchronization controller receives the GPS second pulse signal as a time reference and marks the acquired data of the high-resolution image acquisition unit, satellite positioning unit, inertial measurement unit and odometer unit with a unified timestamp, and the time alignment error between image data and location data does not exceed 1 millisecond.

[0019] As a further aspect of the present invention: the processing procedure of the environment perception adaptive preprocessing module includes:

[0020] The acquired color road images are converted into grayscale images, and the mean brightness and standard deviation of grayscale images are calculated in parallel as contrast indicators, as well as the noise level estimated by the Laplacian operator.

[0021] When the average brightness is lower than a preset low brightness threshold, histogram equalization is performed on the image; when the contrast is lower than a preset low contrast threshold, adaptive histogram equalization is performed on the image; when the noise level is higher than a preset noise threshold, nonlocal mean denoising is performed on the image, wherein the filter intensity parameter is adaptively adjusted according to the estimated noise level.

[0022] When the average brightness, contrast, and noise levels are all within the preset normal range, standard preprocessing is performed, including image size normalization and pixel value normalization.

[0023] As a further aspect of the present invention: the rapid initial screening submodule uses MobileNetV3-Small as the backbone network, and extracts three feature maps with different downsampling ratios from the three bottleneck modules of the backbone network through depthwise separable convolution. The three feature maps are then fused using a simplified feature pyramid network through a top-down path and element-wise addition. On the fused feature maps at each scale, a lightweight detection head outputs the disease category confidence and bounding box coordinates. After non-maximum suppression processing, a set of candidate regions for initial screening diseases is obtained.

[0024] As a further aspect of the present invention: the fine detection submodule extracts four-level deep feature maps using ResNet-50 as the backbone network, and performs multi-scale feature fusion through an improved feature pyramid network; the improved feature pyramid network sets deformable convolutional layers at each feature fusion node in the top-down path; the deformable convolutional layer includes an offset prediction convolutional layer, the number of output channels of which is 2N, where N is the number of sampling points of the convolutional kernel, and by learning the offset of each sampling position in the horizontal and vertical directions, the sampling position of the convolutional kernel is adaptively adjusted according to the shape of the disease target in the input feature map.

[0025] As a further aspect of the present invention: the fine detection submodule sets up channel and spatial attention modules on the fused feature maps at each scale output by the improved feature pyramid network, and the channel and spatial attention modules sequentially perform channel attention enhancement and spatial attention enhancement:

[0026] The channel attention enhancement process is as follows: global average pooling and global max pooling are performed on the input feature map respectively. The two pooling results are input into a multilayer perceptron with shared parameters and then added element by element. The channel attention weight vector is generated by the Sigmoid activation function. The channel attention weight vector is multiplied with the input feature map channel by channel to obtain the feature map after channel attention enhancement.

[0027] The spatial attention enhancement process is as follows: average pooling and max pooling are performed on the feature map after channel attention enhancement along the channel dimension to obtain two single-channel descriptors. The two single-channel descriptors are concatenated along the channel dimension and then a convolution operation with a kernel size of 7x7 and an output channel of 1 is performed. A spatial attention weight map is generated by passing the sigmoid activation function. The spatial attention weight map is multiplied element-wise with the feature map after channel attention enhancement to obtain the final enhanced feature.

[0028] As a further aspect of the present invention: the detection head of the fine detection submodule performs three prediction tasks in parallel on each enhanced feature map:

[0029] The disease classification and prediction task predicts the probability of each disease category and background class for each anchor frame position, and calculates the classification loss using the focus loss function;

[0030] The bounding box regression prediction task predicts four regression parameters for each anchor box position: center point coordinate offset and width-to-height scaling ratio. The regression loss is calculated using a weighted combination of SmoothL1 loss and IoU loss.

[0031] The severity assessment and prediction task predicts the severity level for each detected disease and calculates the severity assessment loss using the cross-entropy loss function.

[0032] The losses from the three prediction tasks are weighted and fused using set weighting coefficients to obtain the total loss function.

[0033] As a further aspect of the present invention: the processing steps of the four-level coordinate mapping algorithm include:

[0034] The first-level transformation calculates the pixel coordinates of the center point of the disease boundary box;

[0035] The second-level transformation uses a pre-calibrated camera intrinsic parameter matrix to convert the center point pixel coordinates into normalized coordinates in the camera coordinate system.

[0036] The third-level transformation uses pre-calibrated camera extrinsic parameters and camera installation height, combined with the road surface plane assumption, to convert the camera coordinates into longitudinal and lateral distances in the vehicle coordinate system.

[0037] The fourth stage of transformation uses satellite positioning data and heading angle data from the inertial measurement unit that match the timestamp of the detection time to rotate and superimpose the longitudinal and lateral distances in the vehicle coordinate system onto the vehicle's geographic coordinates to obtain the latitude and longitude coordinates of the defect.

[0038] As a further aspect of the present invention: the edge-cloud collaborative computing module classifies the detection results of the rapid initial screening submodule into three levels according to confidence level:

[0039] Candidate regions with a confidence level higher than the preset fine detection threshold are directly used as confirmation detection results;

[0040] Candidate regions with confidence levels between the preset ignore threshold and the fine detection threshold are marked as regions to be finely detected, and the corresponding image regions are uploaded to the cloud server for processing by the fine detection submodule.

[0041] Candidate regions with a confidence level lower than a preset ignore threshold are discarded directly;

[0042] The confirmed detection results at the edge are fused with the detailed detection results returned from the cloud, and the final disease detection results are output after global nonmaximum suppression processing is performed.

[0043] As a further aspect of the present invention, the disease spatial positioning module also includes multi-frame positioning accuracy optimization processing: when the same disease is detected in multiple consecutive frames of images, the dispersion of the positioning results of each frame is calculated. If the dispersion exceeds a preset accuracy threshold, the weighted average of the multi-frame positioning results is used as the final geographic coordinates of the disease, wherein the weighting coefficient is proportional to the detection confidence and satellite positioning accuracy index of the corresponding frame.

[0044] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0045] First, the identification and positioning are integrated. By constructing an integrated system architecture that includes an image acquisition module, a positioning sensor module, and a coordinate mapping module, the end-to-end mapping of the disease detection results from pixel coordinates to geographic coordinates is realized, so that each detected disease has accurate latitude and longitude location information, which can be directly connected to the geographic information system for road maintenance management.

[0046] Second, the accuracy of multi-scale disease detection is significantly improved. By designing a multi-scale feature extraction network that integrates deformable convolution and multi-level attention mechanisms, the feature representation ability of diseases with irregular shapes and different scales is enhanced, effectively reducing the false negative rate of small target diseases.

[0047] Third, robustness enhancement in complex environments: By setting up an environment-aware adaptive preprocessing module, the system can automatically select and execute corresponding image enhancement strategies based on light intensity, image contrast, and noise level indicators during the inference stage, ensuring that high-quality images to be recognized can be obtained under different environmental conditions.

[0048] Fourth, the synergistic optimization of real-time performance and accuracy is achieved by designing an edge-cloud collaborative computing architecture. The rapid initial screening task is deployed at the edge for real-time execution, while the fine identification and positioning computing tasks are executed in the cloud, thus achieving a balance between real-time performance and high accuracy. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall system architecture in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the multi-source data acquisition module in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the environment perception adaptive preprocessing module in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the edge-cloud collaborative computing module in an embodiment of the present invention. Detailed Implementation

[0053] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0054] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] I. System Overall Architecture

[0056] like Figure 1 As shown, the present invention proposes an artificial intelligence-based road defect image recognition and positioning system, the overall architecture of which includes six core modules: a multi-source data acquisition module, an environmental perception adaptive preprocessing module, a multi-scale defect detection module, a defect spatial positioning module, an edge-cloud collaborative computing module, and a result output and management module.

[0057] The multi-source data acquisition module is responsible for synchronously acquiring road images and spatial location data; the environmental perception adaptive preprocessing module performs targeted enhancement processing on the images according to the acquisition environment conditions; the multi-scale disease detection module includes two levels: a rapid initial screening sub-module at the edge and a fine detection sub-module in the cloud; the disease spatial positioning module maps pixel-level detection results to a geographic coordinate system; the edge-cloud collaborative computing module uniformly schedules computing resources at the edge and in the cloud; and the result output and management module outputs the final results in the form of GIS maps, disease reports, and maintenance assignments. The entire system forms a complete closed loop from data acquisition to result application.

[0058] II. System Entire Process

[0059] like Figure 2 As shown, the entire process of this system includes the following processing steps:

[0060] First, multi-source data synchronous acquisition is initiated. High-resolution cameras acquire road images, and GPS / GNSS position coordinates, IMU attitude data, and odometer data are recorded synchronously. The data from each sensor are then time-stamped and synchronized.

[0061] Then, environmental perception analysis is performed on the acquired road images to calculate the average brightness, contrast, and noise level of the images.

[0062] Next, it is determined whether the environmental indicators meet the preset normal range thresholds. If the environmental indicators are abnormal, an enhancement strategy is adaptively selected according to the type of abnormality, including: histogram equalization enhancement for low brightness, CLAHE local enhancement for low contrast, and non-local mean denoising for high noise, to obtain a preprocessed image. If the environmental indicators are normal, standard preprocessing (size normalization, pixel value normalization) is performed to obtain a preprocessed image.

[0063] Next, the preprocessed image is input into the lightweight edge detection model, and features are extracted through the MobileNetV3 backbone network to perform rapid initial screening, outputting the initial screening of disease candidate regions and confidence scores.

[0064] Furthermore, the system determines whether there are candidate regions in the initial screening results with confidence levels below the fine-tuning detection threshold but above the ignore threshold. If candidate regions requiring fine-tuning exist, the corresponding image regions and the initial screening results are uploaded to the cloud server. Fine-tuning is then performed using a multi-scale feature extraction network that integrates deformable convolution and multi-level attention mechanisms, outputting the disease category, confidence level, pixel coordinate bounding box, and severity level. If the confidence levels of all candidate regions are above the fine-tuning detection threshold, the initial screening results at the edge are directly used as the final detection results.

[0065] Subsequently, by integrating the initial screening results from the edge and the detailed detection results from the cloud, non-maximum suppression processing is performed on all detection results to eliminate overlapping detection frames and obtain the final disease detection results.

[0066] Next, based on the timestamp of the disease detection time, the corresponding GPS / GNSS location coordinates and IMU attitude data are extracted. Combined with camera calibration parameters and installation parameters, the pixel coordinates of the disease are converted into geospatial coordinates through a coordinate mapping algorithm.

[0067] Finally, the damage detection results with geographic coordinates are written into the GIS database, the location, type, and severity of the damage are marked on the map, a structured damage detection report is generated, and the maintenance dispatch process is triggered based on the severity of the damage. After the system completes this detection cycle, it continues to acquire the next frame of road image and repeats the above process.

[0068] III. Multi-source data acquisition module

[0069] like Figure 3 As shown, the multi-source data acquisition module consists of four sub-units:

[0070] High-resolution image acquisition unit: Employs an industrial-grade high-resolution camera (resolution no less than 1920×1080 pixels, frame rate no less than 30fps), mounted in front of the inspection vehicle to acquire road surface images from a top-down angle. The camera is equipped with automatic exposure and white balance functions to adapt to different lighting conditions.

[0071] Satellite positioning unit: Employs a GPS / GNSS dual-mode receiver to acquire the longitude, latitude, and elevation information of the detected vehicle in real time, achieving positioning accuracy better than sub-meter level. In areas with signal obstruction, RTK differential positioning technology further enhances accuracy.

[0072] Inertial Measurement Unit (IMU): Integrates a three-axis accelerometer and a three-axis gyroscope to measure the vehicle's attitude angles (pitch angle, roll angle, yaw angle) and acceleration in real time, providing data support for position estimation during GPS signal loss, and providing parameters for camera attitude compensation.

[0073] Odometer unit: Records the distance traveled by the vehicle through wheel encoders, serving as a supplementary verification method for location information.

[0074] Each acquisition unit performs hardware-level time synchronization through a unified time synchronization controller. The time synchronization controller receives the GPS pulse-of-seconds (PPS) signal as a time reference, marking image acquisition and data from each sensor with a unified timestamp, ensuring that the time alignment error between image data and location data does not exceed 1 millisecond. After the data from each sensor is collected through a data buffer queue, it is aligned according to the timestamp through a data synchronization matching operation, outputting a synchronization data packet containing image, location, attitude, and odometer information.

[0075] IV. Environmental Perception Adaptive Preprocessing Module

[0076] like Figure 4 As shown, after receiving the acquired raw road image, the environment perception adaptive preprocessing module performs the following processing steps:

[0077] Environmental perception and analysis phase:

[0078] First, the average brightness of the image is calculated. The original color image is converted to a grayscale image, and the average grayscale value of all pixels is calculated as the brightness index. The grayscale value of each pixel in the grayscale image ranges from 0 to 255. The average brightness is calculated by summing the grayscale values ​​of all pixels in the grayscale image and then dividing by the total number of pixels in the image (i.e., the product of the image width and height). If the average brightness is lower than a preset low brightness threshold (set to 60 in this embodiment), the current environment is determined to be a low brightness environment; if the average brightness is higher than a preset high brightness threshold (set to 200 in this embodiment), the current environment is determined to be an overexposed environment.

[0079] Next, the image contrast is calculated. The standard deviation of the grayscale image is used as the contrast index. The standard deviation is calculated as follows: first, the mean of the grayscale values ​​of all pixels in the grayscale image is calculated; then, the square of the difference between each pixel's grayscale value and the mean is calculated; next, the mean of the squares of all pixel differences is calculated; finally, the square root of this mean is taken. If the contrast is lower than a preset low contrast threshold (set to 30 in this embodiment), the current image is determined to have insufficient contrast.

[0080] Next, the image noise level is estimated. The Laplacian operator estimation method is used to assess the noise level. Specifically, a Laplacian filter operation (using a 3×3 Laplacian convolution kernel) is performed on the grayscale image, and the mean of the absolute values ​​of the filtering results is calculated as the noise level estimate. If the noise level is higher than a preset noise threshold (set to 15 in this embodiment), the current image is determined to have significant noise interference.

[0081] Adaptive reinforcement strategy phase:

[0082] If a low-brightness environment is detected, histogram equalization is performed on the image to stretch the grayscale distribution to the entire grayscale range, thereby improving the overall brightness and contrast.

[0083] If a low-contrast environment is detected, adaptive histogram equalization (CLAHE) is performed on the image. CLAHE divides the image into multiple non-overlapping local regions (8×8 regions in this embodiment), performs histogram equalization on each region separately, and eliminates abrupt changes at the boundaries of the regions through bilinear interpolation, thereby enhancing local contrast. A cropping limit parameter of 2.0 is set in CLAHE to prevent excessive enhancement of local contrast from introducing noise amplification.

[0084] If a high-noise environment is detected, nonlocal means denoising is performed on the image. Nonlocal means denoising works by searching for regions in the image with similar neighborhood structures to the current pixel and performing a weighted average, effectively removing noise while preserving edge and texture details. Its filter strength parameter is adaptively adjusted based on the estimated noise level—the higher the noise level, the stronger the filter.

[0085] The three enhancement strategies described above can be superimposed and executed based on the environmental perception analysis results. If all three environmental indicators are within the normal range, standard preprocessing is performed, including scaling the image size to the uniform size required by the model input (640×640 pixels in this embodiment) and normalizing the pixel values ​​(mapping pixel values ​​from 0 to 255 to the range of 0 to 1). The image after adaptive enhancement or standard preprocessing is then passed as a preprocessed image to the downstream multi-scale disease detection module.

[0086] V. Multi-scale disease detection module

[0087] The multi-scale disease detection module is the core identification engine of this system, which includes two levels: a rapid initial screening sub-module at the edge and a fine detection sub-module in the cloud.

[0088] (a) Edge-end rapid primary screening sub-module

[0089] The edge-end rapid initial screening submodule is deployed on the vehicle-mounted edge computing device and adopts a lightweight network architecture to achieve real-time disease detection.

[0090] For feature extraction: MobileNetV3-Small is used as the backbone network, and image features are extracted through depthwise separable convolution. Depthwise separable convolution decomposes standard convolution into two steps: depthwise convolution (channel-wise convolution) and pointwise convolution (1×1 convolution), which significantly reduces the amount of computation and parameters. Three feature maps of different scales (denoted as C3, C4, and C5) are output from the 3rd, 5th, and 7th bottleneck modules of MobileNetV3-Small, respectively, corresponding to downsampling of the input image by 8x, 16x, and 32x.

[0091] For lightweight feature fusion: a simplified FPN fusion is performed on feature maps of three different scales. First, C5 is upsampled by 2 times to make its spatial size consistent with that of C4. The upsampled result is then fused with C4 element-wise to obtain P4. Then, P4 is upsampled by 2 times to make its spatial size consistent with that of C3. The upsampled result is then fused with C3 element-wise to obtain P3. Finally, fused feature maps of three scales, P3, P4, and P5, are obtained.

[0092] Regarding the fast detection head: On each fused feature map, a lightweight detection head (consisting of two 3×3 depth-separable convolutional layers and one 1×1 convolutional layer) is used to output the disease category confidence and bounding box coordinates respectively. The categories output by the detection head include 7 types of diseases: cracks, potholes, ruts, subsidence, oil seepage, loosening, and network cracking, as well as the background category.

[0093] For fast NMS processing: Non-maximum suppression is performed on the output results of all detection heads, the IoU threshold is set to 0.5, redundant detection boxes are filtered out, and a set of candidate regions for initial screening of diseases is obtained.

[0094] Regarding result grading: the initial screening results are divided into three levels based on confidence level. Candidate regions with confidence levels higher than the fine detection threshold (0.7 in this embodiment) are directly used as confirmed detection results; candidate regions with confidence levels between the ignore threshold (0.3 in this embodiment) and the fine detection threshold are marked as regions to be finely detected; candidate regions with confidence levels lower than the ignore threshold are directly discarded.

[0095] (ii) Cloud-based fine detection submodule

[0096] The cloud-based fine-grained detection submodule is deployed on a cloud server and uses a high-precision network architecture to perform in-depth analysis of the edge-marked areas to be finely detected.

[0097] Regarding the multi-scale feature extraction network architecture: ResNet-50 is used as the backbone network, and four-level feature maps (C2, C3, C4, and C5) are output from the four residual stages of ResNet-50. In the improved FPN feature fusion module, C5 is first upsampled by 2x after 1×1 convolution dimensionality reduction, and then element-wise added to the result of C4 after 1×1 convolution dimensionality reduction. After deformable convolution enhancement, the P4 feature map is output. Then, P4 is upsampled by 2x, and element-wise added to the result of C3 after 1×1 convolution dimensionality reduction. After deformable convolution enhancement, the P3 feature map is output. The fused P3 and P4 feature maps are enhanced by the CBAM attention module and then fed into the detection head. Finally, after non-maximum suppression processing, the disease category, confidence score, pixel coordinate bounding box, and severity level are output.

[0098] Regarding the implementation of deformable convolution enhancement: The deformable convolution module is a key innovation introduced in the feature fusion stage of this invention, which enhances the network's ability to extract features of irregularly shaped road defects (such as curved cracks and irregular pothole boundaries).

[0099] For the input fused feature map, an additional 3×3 convolutional layer (offset prediction convolutional layer) is first used to predict the two-dimensional offset at each sampling location. The output channel number of this convolutional layer is 2N, where N is the number of sampling points in the standard convolutional kernel (N is 9 for a 3×3 convolutional kernel), and the factor of 2 indicates that each sampling point has an offset component in both the horizontal and vertical directions. Therefore, the offset prediction convolutional layer outputs 18 offset values ​​for each spatial location.

[0100] A standard 3×3 convolution kernel has nine sampling points with a fixed offset set relative to its center position. In deformable convolution, the actual sampling position of each sampling point is the fixed offset plus a learned offset. Since the learned offset is usually a non-integer value, the actual sampling position may fall at a non-integer position in the pixel grid, requiring bilinear interpolation to obtain the feature value at that position from the feature map. Bilinear interpolation is calculated by weighting the feature values ​​at four integer coordinate positions surrounding the sampling position, with the weights determined by the distance from the sampling position to the four integer coordinate positions.

[0101] The feature values ​​of each sampling point obtained through bilinear interpolation are multiplied element-wise with the convolution kernel weights and summed to obtain the output feature values ​​of the deformable convolution. This process is repeated for each spatial location of the feature map to obtain the complete deformable convolution output feature map. Through these operations, the receptive field shape of the convolution kernel is no longer limited to a fixed rectangular region, but can adaptively adjust according to the shape of the lesion in the input feature map, thereby more effectively extracting features of curved cracks and irregular pits.

[0102] Regarding the implementation of the CBAM attention mechanism: The CBAM attention module is divided into two sub-modules, channel attention and spatial attention, which enhance the fused feature map in turn.

[0103] The channel attention submodule processes the following: Global average pooling and global max pooling are performed on the input feature map. Global average pooling calculates the mean of all feature values ​​at all spatial locations in each channel, compressing them into a scalar. Global max pooling calculates the maximum value of all feature values ​​at all spatial locations in each channel, compressing them into a scalar. Assuming the input feature map has C channels, both global average pooling and global max pooling output a vector of length C. These two vectors are then input into a shared multilayer perceptron (MLP), which consists of two fully connected layers: the first fully connected layer compresses the C-dimensional vector to C / r dimensions (where r is the dimensionality reduction ratio, taken as 16 in this embodiment) and passes it through the ReLU activation function; the second fully connected layer restores the C / r-dimensional vector to C dimensions. The C-dimensional vectors output by the MLP from the two paths are added element-wise, and then a channel attention weight vector is generated using the Sigmoid activation function. Each element of this weight vector ranges from 0 to 1, representing the importance weight of the corresponding channel. The channel attention weight vector is multiplied channel by channel with the input feature map to obtain the feature map after channel attention enhancement.

[0104] The spatial attention submodule processes the following: For the channel-attention-enhanced feature map, average pooling and max pooling operations are performed along the channel dimension, i.e., the mean and maximum values ​​of the C channel values ​​at each spatial location are calculated, resulting in two single-channel spatial descriptors. These two single-channel descriptors are concatenated along the channel dimension to obtain a 2-channel feature map. A 7×7 convolution operation (outputting 1 channel) is performed on this 2-channel feature map, and a spatial attention weight map is generated using the Sigmoid activation function. Each pixel value in this weight map ranges from 0 to 1, representing the importance weight of the corresponding spatial location. The spatial attention weight map is then element-wise multiplied with the channel-attention-enhanced feature map to obtain the final CBAM-enhanced feature map.

[0105] Regarding the implementation of the multi-task detection head: the detection head performs three parallel prediction tasks on each enhanced feature map.

[0106] Disease classification and prediction task: Using two 3×3 convolutional layers and one 1×1 convolutional layer, the probability of K+1 categories is predicted for each anchor frame position (K is the number of disease categories; in this embodiment, K is 7, namely cracks, potholes, ruts, subsidence, oil seepage, loosening, and network cracking, plus 1 for the background class). The classification loss uses the Focal Loss function, which reduces the loss weight of easily classified samples by adjusting factors, making the model focus more on difficult-to-classify samples, thereby solving the problem of imbalanced positive and negative samples. The focusing parameter gamma in Focal Loss is set to 2.0, and the balance factor alpha is set to 0.25.

[0107] Bounding box regression prediction task: For each anchor box location, four bounding box regression parameters (center point x-offset, center point y-offset, width scaling ratio, and height scaling ratio) are predicted using two 3×3 convolutional layers and one 1×1 convolutional layer. The regression loss is a weighted combination of SmoothL1 Loss and IoU Loss.

[0108] Severity assessment prediction task: For each detected disease, a severity level (mild, moderate, or severe) is predicted using two 3×3 convolutional layers and one 1×1 convolutional layer. The severity assessment loss uses the standard cross-entropy loss function.

[0109] The total loss function for multi-task tasks is calculated as follows:

[0110] ;

[0111] in, This represents the total loss function value across multiple tasks; This represents the classification loss value based on the focus loss function; This represents the bounding box regression loss value of SmoothL1 Loss; The bounding box regression loss value represents the IoU Loss; The cross-entropy loss value represents the severity assessment. The weight coefficient representing the classification loss has a value of 1.0. The weighting coefficient representing the regression loss has a value of 1.0. The weighting coefficient, representing the severity of the loss, has a value of 0.5.

[0112] For model training: The optimization objective was to minimize the total loss function, and the Adam optimizer was used to iteratively update the network parameters. The initial learning rate was set to 0.001, and a cosine annealing learning rate scheduling strategy was employed, gradually decaying the learning rate from the initial value to 0 during training. Data augmentation strategies (random horizontal flipping, random rotation of ±15 degrees, random brightness adjustment from 50% to 150%, and random scaling from 0.8 to 1.2 times) were used to expand the training data. Iterative training continued until the total loss converged, resulting in a well-trained, refined detection model.

[0113] VI. Spatial Location Module for Diseases

[0114] The disease spatial positioning module is responsible for converting the pixel coordinate level disease detection results output by the multi-scale disease detection module into geospatial coordinates (latitude and longitude), and realizing the precise spatial positioning of the disease through a four-level coordinate mapping algorithm.

[0115] Level 1: Extracting pixel coordinates of the detection results. The bounding box pixel coordinates of each detected lesion are obtained from the output of the multi-scale lesion detection module. The center pixel coordinates of the bounding box are calculated as the representative point of the lesion in the image. The pixel coordinates u and v of the bounding box center point are calculated as follows: u equals the sum of the x-coordinates of the top-left and bottom-right corners of the bounding box divided by 2; v equals the sum of the y-coordinates of the top-left and bottom-right corners of the bounding box divided by 2.

[0116] Level 2: Pixel coordinate to camera coordinate conversion. Using a pre-calibrated camera intrinsic parameter matrix, the pixel coordinates of the lesion center point are converted to normalized coordinates in the camera coordinate system. The camera intrinsic parameter matrix contains focal length and principal point coordinate parameters, which are obtained in advance through the camera calibration program. During the conversion, the horizontal offset is obtained by subtracting the principal point coordinate cx from the pixel coordinate u, and then divided by the horizontal focal length fx to obtain the normalized x-coordinate in the camera coordinate system; the vertical offset is obtained by subtracting the principal point coordinate cy from the pixel coordinate v, and then divided by the vertical focal length fy to obtain the normalized y-coordinate in the camera coordinate system.

[0117] Level 3: Transformation from camera coordinates to vehicle coordinates. Using pre-calibrated camera extrinsic parameters (including the camera's rotation matrix and translation vector relative to the vehicle coordinate system) and the camera's mounting height, the normalized camera coordinates are converted into three-dimensional coordinates in the vehicle coordinate system. Since the road surface can be approximated as a plane, this transformation process can be simplified using a plane assumption: assuming the road surface height is a zero-plane, the longitudinal and lateral distances of the defects in the vehicle coordinate system are calculated using the camera mounting height h and the normalized coordinates.

[0118] Level 4: Transformation from vehicle coordinates to geographic coordinates. Using GPS / GNSS location information (longitude, latitude, elevation) and IMU attitude information (heading angle) matched with the detection time timestamp, the defect location in the vehicle coordinate system is converted to geographic coordinates. Based on the vehicle's heading angle, the longitudinal and lateral distances in the vehicle coordinate system are rotated to the east-north direction geographic coordinate increment, and then superimposed onto the vehicle position coordinates provided by GPS / GNSS to obtain the geographic coordinates (longitude and latitude) of the defect.

[0119] Regarding multi-frame positioning accuracy optimization: Odometry data is used to verify the consistency of geographic coordinates of the same lesion in consecutive frames. If the same lesion is detected in multiple consecutive frames, the dispersion of the positioning results in each frame is calculated. If the dispersion exceeds a preset accuracy threshold (2 meters in this embodiment), the weighted average of the multi-frame positioning results is used as the final geographic coordinates. The weighting coefficient is proportional to the detection confidence and GPS positioning accuracy index of the corresponding frame.

[0120] VII. Edge-Cloud Collaborative Computing Module

[0121] The edge-cloud collaborative computing module is responsible for the unified scheduling of computing resources at the edge and in the cloud.

[0122] For real-time processing at the edge: Edge devices continuously run a lightweight detection model to perform rapid initial screening on each frame of road image, ensuring real-time performance (inference latency not exceeding 33 milliseconds / frame, i.e., frame rate not less than 30fps). The edge model is optimized through FP16 half-precision quantization and the TensorRT inference acceleration engine to maximize inference speed with limited computing power.

[0123] In terms of intelligent triage decision-making: triage decisions are made based on the confidence level of each candidate region in the initial screening results. Results with a confidence level higher than the fine-tuning detection threshold are output directly; results with a confidence level between the ignore threshold and the fine-tuning detection threshold are marked as requiring fine-tuning detection and uploaded to the cloud; results with a confidence level lower than the ignore threshold are discarded directly.

[0124] In terms of cloud-based fine processing: The cloud server receives the image regions to be finely detected and their initial screening results uploaded from the edge, performs fine detection through a high-precision multi-scale detection network, and performs coordinate mapping calculations.

[0125] In terms of result fusion: the high-confidence results directly output from the edge and the detailed detection results returned from the cloud are fused together, and global NMS processing is performed on all results to eliminate possible overlapping detection boxes and generate the final disease detection and location results.

[0126] VIII. Results Output and Management Module

[0127] The results output and management module receives the final disease detection and location results and performs the following processing:

[0128] Regarding GIS map annotation: Each detected disease is marked on the GIS map according to its geographic coordinates. The annotation information includes the disease category, severity level, detection time, and location accuracy. Different categories and severity levels of diseases are distinguished and displayed using different icons and colors.

[0129] Regarding the generation of road damage detection reports: Structured road damage detection reports are automatically generated, including information on the inspected road sections, a summary of road damage statistics (quantity of each category and distribution of each severity level), and a list of road damage locations (latitude and longitude, mileage markers).

[0130] Maintenance work assignment triggering: The maintenance work assignment process is automatically triggered based on the severity level of the defects. Severe defects immediately generate emergency maintenance work orders, medium-level defects are included in the regular maintenance plan, and minor defects are included in the inspection record and trend monitoring.

[0131] In terms of historical data management: the detection results are written to the database for persistent storage, and queries and statistical analyses are supported by road segment, time range, disease type and severity, enabling the tracking of disease evolution trends.

[0132] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any variations and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the invention, fall within the protection scope defined by the claims of the present invention.

Claims

1. A road defect image recognition and positioning system for artificial intelligence, characterized in that: The term includes: A multi-source data acquisition module is used to simultaneously acquire road surface images and spatial location data. The multi-source data acquisition module includes a high-resolution image acquisition unit, a satellite positioning unit, an inertial measurement unit, and an odometer unit. Each unit achieves hardware-level time synchronization through a time synchronization controller. The environmental perception adaptive preprocessing module is used to calculate environmental indicators for the acquired road images, and adaptively select image enhancement strategies for preprocessing based on the environmental indicator analysis results, and output preprocessed images. The multi-scale road disease detection module includes a rapid initial screening sub-module deployed on edge devices and a fine detection sub-module deployed on cloud servers. It is used to perform multi-scale road disease detection on preprocessed images and output disease categories, confidence scores, and pixel coordinate bounding boxes. The disease spatial positioning module is used to convert the pixel coordinates of disease detection results into geospatial coordinates through a four-level coordinate mapping algorithm; The edge-cloud collaborative computing module is used to classify and process the detection results output by the rapid initial screening submodule according to confidence level, and coordinate the allocation of computing tasks between the edge and the cloud. The results output and management module is used to annotate the disease detection results with geographic coordinates on geographic information system maps, generate detection reports, and manage maintenance assignments.

2. The road defect image recognition and positioning system for artificial intelligence according to claim 1, characterized in that: The time synchronization controller receives the GPS second pulse signal as a time reference and marks the acquired data of the high-resolution image acquisition unit, satellite positioning unit, inertial measurement unit and odometer unit with a unified timestamp, and the time alignment error between image data and location data does not exceed 1 millisecond.

3. The road defect image recognition and positioning system for artificial intelligence according to claim 1, characterized in that: The processing procedure of the environment-aware adaptive preprocessing module includes: The acquired color road images are converted into grayscale images, and the mean brightness and standard deviation of grayscale images are calculated in parallel as contrast indicators, as well as the noise level estimated by the Laplacian operator. When the average brightness is lower than a preset low brightness threshold, histogram equalization is performed on the image; when the contrast is lower than a preset low contrast threshold, adaptive histogram equalization is performed on the image; when the noise level is higher than a preset noise threshold, nonlocal mean denoising is performed on the image, wherein the filter intensity parameter is adaptively adjusted according to the estimated noise level. When the average brightness, contrast, and noise levels are all within the preset normal range, standard preprocessing is performed, including image size normalization and pixel value normalization.

4. The road defect image recognition and positioning system for artificial intelligence according to claim 1, characterized in that: The rapid initial screening submodule uses MobileNetV3-Small as the backbone network. It extracts three feature maps with different downsampling ratios from the three bottleneck modules of the backbone network through depthwise separable convolution. The three feature maps are then fused using a simplified feature pyramid network through a top-down path and element-wise addition. On the fused feature maps at each scale, a lightweight detection head outputs the disease category confidence and bounding box coordinates. After non-maximum suppression processing, a set of candidate regions for initial screening diseases is obtained.

5. The road defect image recognition and positioning system for artificial intelligence according to claim 1, characterized in that: The fine detection submodule uses ResNet-50 as the backbone network to extract four-level deep feature maps and performs multi-scale feature fusion through an improved feature pyramid network. The improved feature pyramid network sets a deformable convolutional layer at each feature fusion node in the top-down path. The deformable convolutional layer includes an offset prediction convolutional layer with 2N output channels, where N is the number of sampling points of the convolutional kernel. By learning the offset of each sampling position in the horizontal and vertical directions, the sampling position of the convolutional kernel is adaptively adjusted according to the shape of the disease target in the input feature map.

6. The road defect image recognition and positioning system for artificial intelligence according to claim 5, characterized in that: The fine detection submodule sets up channel and spatial attention modules on the fused feature maps at each scale output by the improved feature pyramid network. The channel and spatial attention modules perform channel attention enhancement and spatial attention enhancement in sequence. The channel attention enhancement process is as follows: global average pooling and global max pooling are performed on the input feature map respectively. The two pooling results are input into a multilayer perceptron with shared parameters and then added element by element. The channel attention weight vector is generated by the Sigmoid activation function. The channel attention weight vector is multiplied with the input feature map channel by channel to obtain the feature map after channel attention enhancement. The spatial attention enhancement process is as follows: average pooling and max pooling are performed on the feature map after channel attention enhancement along the channel dimension to obtain two single-channel descriptors. The two single-channel descriptors are concatenated along the channel dimension and then a convolution operation with a kernel size of 7x7 and an output channel of 1 is performed. A spatial attention weight map is generated by passing the sigmoid activation function. The spatial attention weight map is multiplied element-wise with the feature map after channel attention enhancement to obtain the final enhanced feature map.

7. The road defect image recognition and positioning system for artificial intelligence according to claim 5, characterized in that: The detection head of the fine detection submodule performs three prediction tasks in parallel on each enhanced feature map: The disease classification and prediction task predicts the probability of each disease category and background class for each anchor frame position, and calculates the classification loss using the focus loss function; The bounding box regression prediction task predicts four regression parameters for each anchor box position: center point coordinate offset and width-to-height scaling ratio. The regression loss is calculated using a weighted combination of SmoothL1 loss and IoU loss. The severity assessment and prediction task predicts the severity level for each detected disease and calculates the severity assessment loss using the cross-entropy loss function. The losses from the three prediction tasks are weighted and fused using set weighting coefficients to obtain the total loss function.

8. The road defect image recognition and positioning system for artificial intelligence according to claim 1, characterized in that: The processing steps of the four-level coordinate mapping algorithm include: The first-level transformation calculates the pixel coordinates of the center point of the disease boundary box; The second-level transformation uses a pre-calibrated camera intrinsic parameter matrix to convert the center point pixel coordinates into normalized coordinates in the camera coordinate system. The third-level transformation uses pre-calibrated camera extrinsic parameters and camera installation height, combined with the road surface plane assumption, to convert the camera coordinates into longitudinal and lateral distances in the vehicle coordinate system. The fourth stage of transformation uses satellite positioning data and heading angle data from the inertial measurement unit that match the timestamp of the detection time to rotate and superimpose the longitudinal and lateral distances in the vehicle coordinate system onto the vehicle's geographic coordinates to obtain the latitude and longitude coordinates of the defect.

9. The road defect image recognition and positioning system for artificial intelligence according to claim 1, characterized in that: The edge-cloud collaborative computing module categorizes the detection results of the rapid initial screening submodule into three levels based on confidence level: Candidate regions with a confidence level higher than the preset fine detection threshold are directly used as confirmation detection results; Candidate regions with confidence levels between the preset ignore threshold and the fine detection threshold are marked as regions to be finely detected, and the corresponding image regions are uploaded to the cloud server for processing by the fine detection submodule. Candidate regions with a confidence level lower than a preset ignore threshold are discarded directly; The confirmed detection results at the edge are fused with the detailed detection results returned from the cloud, and the final disease detection results are output after global nonmaximum suppression processing is performed.

10. The road defect image recognition and positioning system for artificial intelligence according to claim 8, characterized in that: The disease spatial positioning module also includes multi-frame positioning accuracy optimization processing: when the same disease is detected in multiple consecutive frames of images, the dispersion of the positioning results of each frame is calculated. If the dispersion exceeds a preset accuracy threshold, the weighted average of the multi-frame positioning results is used as the final geographic coordinates of the disease. The weighting coefficient is proportional to the detection confidence and satellite positioning accuracy index of the corresponding frame.

Images