Method for calculating road disease area in weak network environment based on binocular camera and edge AI box

Abstract

The invention discloses a method for calculating a road disease area in a weak network environment based on a binocular camera and an edge AI box, and the method comprises the steps: S1, image preprocessing: carrying out the preprocessing of an input original image, sequentially carrying out the noise processing and image normalization processing, obtaining a processed data set, and dividing the data set into a training set and a verification set; s2, road segmentation: training and optimizing a YOLOv5-seg model to obtain a YOLOv5-seg road segmentation model, and then segmenting the road; and S3, extracting a depth map and calculating a disease area: performing feature extraction on depth map data acquired by the binocular camera in combination with the road boundary, and calculating a disease distance so as to obtain the road disease area. According to the method, the road information in the image can be accurately identified and positioned, the problem of inaccurate or incomplete information caused by errors is avoided, the disease area calculation precision is improved to the greatest extent, and the method is suitable for various road scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of road damage detection, and specifically relates to a method for calculating the road damage area in a weak network environment based on a binocular camera and an edge AI box. Background Art

[0002] With the vigorous development of AI intelligent technology, smart cities have become one of the goals of my country's strategic development. Smart cities are the inevitable product of China's new urbanization development, the continuous integration of modern science and technology into cities and industries, and the continuous innovation and development of society. Among them, AI automatic recognition of road conditions is an important way to control roads. However, there are some problems when traditional edge AI boxes recognize roads. The calculation of the disease area is not accurate enough, there is an error with the actual disease area, and it cannot accurately reflect the degree of harm of the disease. Under complex backgrounds or unclear disease characteristics, traditional edge AI box recognition may have false positives or false negatives, affecting the reliability of detection. It is sensitive to specific environments and lighting conditions, and the recognition accuracy decreases under different conditions (such as rainy days and nights), resulting in large errors in the calculation of road disease area. It is difficult for relevant departments to make decisions when repairing roads. It is difficult for management departments to allocate maintenance resources and cannot focus on the roads that need repair the most, affecting maintenance efficiency.

[0003] At present, the commonly used method for edge AI boxes to identify road damage is usually to train image data on the server side to improve the loss value and accuracy of the model. After the model reaches the index, the trained model is placed on the edge AI box through conversion, and the model reasoning speed is improved through some tools. The recognition results and the calculated damage area data are transmitted to the server side, and most of the steps are completed on the edge side. The disadvantage of this method is that the calculation of the damage area is not accurate enough, the server side is not used to perform secondary accurate identification of the damage, and the road information is not processed accordingly. Some false detections that are not on the road will occur, increasing the false detection rate, and the influence of other objects on the road will lead to large errors in the calculation of the road damage area. In AI road recognition, the damage area is used to determine the severity of road damage and is the most important indicator. Accurate measurement of the damage area can help management departments prioritize serious damage and ensure that limited maintenance resources are used in the best way. Some solutions use the method of uploading the original image and the depth map to recalculate the damage area. Because the depth map is relatively large and cannot be processed by compression, conversion, etc., when there are many road damages in the actual scene, the edge AI upload will be very stressful, which will cause a large number of images to be squeezed, and the practicality is very poor.

[0004] Chinese patent document CN117237925A discloses a computer vision-based intelligent road disease inspection method and system, including: using a camera to collect road disease information and construct a data set, dividing the training set, verification set, and test set in an 8:1:1 ratio; building a yolov5s model; sending the training set to the yolov5s model for training, and after 800 rounds of training, the optimal model is obtained; using a binocular camera to measure the length, width, and area of ​​the disease; using C++ and TensorRT to deploy the optimal model in the NVIDIA Jetson Xavier NX device. This technical solution is applicable to conventional AI disease detection, but the accuracy is low when applied in a weak network environment.

[0005] Chinese patent document CN117745712A discloses a road damage detection method and system using information fusion and cloud-edge deployment, the method comprising: S1: constructing an enhanced image dataset SA and a road edge information dataset E based on the original road image dataset S, combining SA and E for training, and obtaining a trained full-precision road damage detection model; S2: quantifying the trained full-precision road damage detection model to obtain a quantized road damage detection model; inputting the fact image into the quantized road damage detection model for coarse filtering detection, obtaining a road damage coarse filtering detection result and uploading it to the cloud; S3: deploying the trained full-precision road damage detection model in the cloud, processing the coarse filtering detection result to obtain a fine-grained detection result; and analyzing and converging the detection result, selecting samples from the fine-grained detection result to supplement SA and E. This technical solution does not involve depth map processing and road damage area calculation.

[0006] Therefore, it is necessary to provide a method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment, so as to eliminate erroneous data, increase the accuracy, and improve the accuracy of calculating the damage area. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment, so as to eliminate erroneous data, increase the accuracy, and improve the accuracy of calculating the damage area.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: the method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment specifically includes the following steps:

[0009] S1 Image preprocessing: Preprocess the input original image, and perform noise processing and image normalization processing in sequence to obtain the processed data set, and divide the data set into a training set and a validation set;

[0010] S2 Road Segmentation: Train and optimize the YOLOv5-seg model to obtain the YOLOv5-seg road segmentation model, and then use the YOLOv5-seg road segmentation model to segment the road;

[0011] S3 extracts the depth map and calculates the damaged area: The depth map data obtained by the binocular camera is combined with the road boundary to perform feature extraction, and the damage distance is calculated to obtain the road damage area.

[0012] The above technical solution uses advanced image processing and area detection algorithms to accurately identify and locate road information in the image, avoiding inaccurate or incomplete information caused by errors, and maximizing the accuracy of the calculation of the damaged area. The accuracy is improved based on the existing damaged area in combination with the actual edge device processing capabilities and application scenarios. The erroneous data can be eliminated and the accuracy can be increased, which can improve the accuracy of calculating the damaged area.

[0013] Preferably, the specific steps of step S1 are:

[0014] S11 Image border filling: fill the border of the input original image to make the edge of the image complete;

[0015] S12 Noise processing: Use filtering or non-local mean processing to denoise the image;

[0016] S13 Image normalization: Scale the pixel values ​​of an image to the range [0, 1] or [-1, 1] to make the image suitable for the input layer of a convolutional neural network.

[0017] Image preprocessing is used to separate road information and improve the accuracy of disease area calculation. The purpose of image padding is to add extra pixels to the border area of ​​the image in order to solve the problem of border pixel processing in image processing operations. Boundary padding is very important in tasks such as convolution operations, image transformation, and image segmentation to ensure that the algorithm can smoothly process the border part of the image.

[0018] The purpose of margin padding:

[0019] (1) Avoid information loss. During the convolution operation or filtering process, the boundary pixels cannot be completely aligned with the filter window, so the boundary needs to be filled to ensure that the boundary pixels can also be processed normally. If padding is not performed, the image will become smaller after multiple convolutions, and the boundary information may be ignored or lost;

[0020] (2) Reduce edge effects. In filtering operations (such as smoothing filtering and Gaussian filtering), the neighborhood of edge pixels is incomplete, resulting in insufficient edge processing. By filling the boundaries, the edge pixels of the image can have sufficient neighborhood information, avoiding unnatural effects when processing edge areas and reducing the so-called "edge effects";

[0021] (3) Improve the performance of convolutional neural networks. In deep learning, the convolutional layer in a convolutional neural network (CNN) applies the convolution kernel multiple times. Convolutional layers without padding will cause the size of the feature map to gradually decrease and information to be lost. This is especially true for small-sized images. By using an appropriate padding strategy in the convolution operation, it is possible to ensure that the size of the feature map does not decrease after each convolution layer, thereby improving network performance.

[0022] Image normalization is a common preprocessing step that improves the efficiency and accuracy of image processing and machine learning tasks by mapping the pixel values ​​of an image to a specific range (such as [0, 1] or [-1, 1]). Normalization can reduce the impact of factors such as lighting and contrast, ensure faster model convergence, and avoid gradient explosion or vanishing problems.

[0023] Common normalization methods:

[0024] (1) Min-Max normalization: Scale the pixel values ​​of an image to the range of [0, 1] or [-1, 1]. It is usually used to scale the pixel range of an image from [0, 255] to [0, 1]. It is particularly suitable for the input layer of a convolutional neural network. The formula is:

[0025]

[0026] Where: x is the original pixel value; x min and x max are the minimum and maximum pixel values ​​respectively; x′ is the normalized value, usually between [0,1] or [-1,1];

[0027] (2) Z-score normalization (zero mean normalization): normalizes the image pixel values ​​to a standard distribution with a mean of 0 and a standard deviation of 1. It is suitable for scenarios where the data has a large variance and ensures that each pixel value is standardized relative to the overall data distribution. The formula is:

[0028]

[0029] Where: x is the original pixel value; μ is the mean of the pixel value; σ is the standard deviation of the pixel value; x′ is the normalized value;

[0030] (3) Local Contrast Normalization: Local Contrast Normalization (LCN) enhances the contrast in an image by adjusting the mean and standard deviation of each pixel value relative to its neighborhood. It is suitable for tasks such as edge detection. The formula is:

[0031]

[0032] Among them, μ i,j and σ i ,j is at pixel x i , the mean and standard deviation in the neighborhood around j.

[0033] Preferably, the specific steps of using filtering to perform denoising in step S12 are:

[0034] S121 Median filter: Median filter is a commonly used denoising method, especially for salt-and-pepper noise. This method uses the median value in the sliding window instead of the central pixel value, so it can better preserve edge information.

[0035] S122 Frequency domain filtering: Using frequency domain filters, such as low-pass filters (such as Butterworth filters), can be used to remove high-frequency noise from images; the image is first converted to the frequency domain through Fourier transform, then the filter is applied, and then the inverse transform is performed;

[0036] The frequency domain filtering formula is:

[0037]

[0038] Where D0 is the cutoff frequency of the filter; u0 and v0 are both frequency centers; and H(u,v) is the filter function in the frequency domain.

[0039] The purpose of image noise processing is to reduce the noise in the image, enhance the quality and details of the image, and make subsequent image processing or analysis more accurate and effective.

[0040] (1) Median filtering: Median filtering is a commonly used denoising method, especially for salt-and-pepper noise. This method uses the median value in the sliding window instead of the central pixel value, so it can better preserve edge information;

[0041] (2) Non-local mean (NLM) denoising: The non-local mean denoising algorithm uses redundant information in the image to calculate similar regions of pixels, thereby averaging the pixel values ​​of multiple similar regions. NLM is suitable for processing complex noise and can maintain image details.

[0042] (3) Frequency domain filtering: Frequency domain filters, such as low-pass filters (such as Butterworth filters), can be used to remove high-frequency noise in images.

[0043] Preferably, the specific steps of step S2 are:

[0044] S21 builds the model: adds a segmentation head based on the YOLOv5 model to generate segmentation masks;

[0045] S22 model training and optimization: The training set is used for model training. During training, the cosine annealing learning rate, loss function and hyperparameter optimization are performed to generate a road segmentation model.

[0046] S23 Model Reasoning Optimization: The road segmentation model is then quantized, pruned, and optimized using the non-maximum suppression method (NMS) to output the YOLOv5-seg road segmentation model.

[0047] S24 Road segmentation: Use the YOLOv5-seg road segmentation model to segment the road, generate the corresponding segmentation mask, and then output the segmentation result.

[0048] Road segmentation is to reason about images. The structure of the YOLOv5-se model used is to add a segmentation head to the YOLOv5 model to generate segmentation masks. Its working principle is to generate the corresponding segmentation mask while predicting the object category in each detection box. For the road segmentation task, YOLOv5-seg only needs to predict the binary mask of the road during training.

[0049] Key components of the YOLOv5-seg model: Backbone: Extract features, such as using CSPDarknet. Detection Head: Generate bounding boxes and category predictions for objects. Segmentation Head: Additional segmentation branches output segmentation masks for each object. Design efficient and accurate road image processing algorithms to identify and separate road and non-road parts in images, and separate road part boundaries.

[0050] Preferably, the specific steps of optimizing the cosine annealing learning rate and loss function in step S22 are:

[0051] S221 Cosine annealing learning rate optimization: The cosine annealing scheduler is used to optimize the cosine annealing learning rate of the YOLOv5-seg model. The cosine annealing learning rate scheduler is a strategy for gradually reducing the learning rate. It is usually used in the training of deep learning models to improve the convergence and stability of the model. In the training optimization of YOLOv5-seg road segmentation, the cosine annealing scheduler can dynamically adjust the learning rate to prevent the model from falling into a local optimal solution, and gradually reduce the learning rate in the later stage of training to help the model converge better. The core idea of ​​the cosine annealing scheduler is based on the cosine function, which smoothly reduces the learning rate during the training process. The formula is:

[0052]

[0053] Where: η t : learning rate at the current iteration time t; η max : initial learning rate, i.e., maximum learning rate; η min : minimum learning rate, the minimum value that the learning rate approaches in the later stage of training; T cur : current epoch or step; T max : Maximum number of epochs or steps, i.e. the cycle of the entire learning rate scheduling; cos: cosine function;

[0054] S222 loss function optimization: Combine the Dice Loss loss function and the IoU Loss loss function for optimization; in order to combine Dice Loss and IoU Loss, the weighted sum of the Dice Loss loss function and the IoU Loss loss function is used, and the formula is:

[0055] Total Loss=α·Dice Loss+β·IoU Loss;

[0056] Among them: α and β are hyperparameters used to balance the impact of the two loss functions; Dice Loss and IoU Loss are the corresponding Dice coefficient and IoU loss respectively.

[0057] The optimization of the loss function in the segmentation task is very critical, which can help the model converge faster and improve the segmentation accuracy. Common optimization methods are as follows:

[0058] (1) Cross-Entropy Loss: This is the most basic segmentation loss function, but it may not work well for unbalanced classes (such as smaller road areas);

[0059] (2) Dice Loss: focuses on improving the overlap of segmentation and is particularly suitable for dealing with data imbalance problems. Dice Loss performs well for smaller road areas;

[0060] (3) Focal Loss: When dealing with the class imbalance problem in road segmentation tasks, Focal Loss can suppress the loss of simple samples and increase the focus on difficult samples;

[0061] (4) IoU Loss (Intersection over Union Loss): Directly optimizes the IoU between the segmentation result and the true label to improve the accuracy of the segmented area.

[0062] For example, in the road segmentation task, since roads are usually long and winding, segmentation faults are prone to occur. A common practice is to combine Dice Loss and IoU Loss to ensure the continuity and accuracy of the segmented area.

[0063] Preferably, the specific steps of step S23 are:

[0064] S231 Model Quantization: Dynamic quantization and static quantization are used to perform quantization perception during model training;

[0065] S232 Model pruning: Use L1 regularization or sparsity constraints to prune the model; model pruning reduces the model size and inference calculation amount by removing unimportant convolution kernels or neurons, that is, reducing redundant weights;

[0066] S233NMS optimization: According to the IoU value, the confidence of the detection box with an IoU value higher than the threshold is smoothly decayed, and the confidence decay adopts a Gaussian decay function or a linear decay function.

[0067] Model quantization can reduce memory usage and improve inference speed, especially on mobile or embedded devices. YOLOv5 supports the following two quantization methods:

[0068] Dynamic quantization: quantize weights and activation functions to lower precision data types (such as INT8) only during inference, but keep calculations as FP32.

[0069] Static quantization (quantization-aware training): Quantization-aware training is performed during training, and low-precision reasoning is used directly during inference. Quantization-aware training needs to be performed in advance.

[0070] In YOLOv5-seg, Soft-NMS can be used to optimize the selection of bounding boxes in segmentation tasks, especially when the boundaries of multiple roads overlap, Soft-NMS can more smoothly suppress redundant boxes while retaining road targets with fuzzy boundaries. The improvements of Soft-NMS include:

[0071] Soft-NMS does not directly remove detection boxes with IoU higher than the threshold, but smoothly decays the confidence of these boxes according to the IoU value.

[0072] Confidence decay usually adopts a Gaussian decay function or a linear decay function, so that the confidence is gradually reduced instead of being directly set to 0.

[0073] Preferably, the specific steps of step S3 are:

[0074] S31 obtains the depth map: first calibrates the binocular camera to obtain the internal parameters and external parameters of the binocular camera; then performs stereo matching to find the corresponding pixel points in the two images; then calculates the depth value of each pixel according to the disparity map and the internal and external parameters of the camera, and generates a depth map;

[0075] The calculation formula of the depth value Z(x,y) is:

[0076]

[0077] Where Z(x,y) is the depth value corresponding to the pixel point (x,y); f is the focal length of the camera; B is the baseline length between the two cameras; d(x,y) is the disparity between the corresponding points of the pixel point (x,y) in the two images;

[0078] S32 depth map conversion to point cloud: The acquired depth map is mapped to a 3D point through the camera model.

[0079] S33 interpolates obstacle point data: identifies obstacles in combination with the original image, and regenerates the obstacle point data;

[0080] S34 extracts grid data: extracts grid data of the depth map, i.e., distance points of the grid, in combination with the road separation data, verifies the extracted distance data points, interpolates the erroneous depth map data, and uploads the original image and depth map grid data; the distance of a certain pixel point obtained by the depth map is usually within a normal range. The currently used binocular camera supports a ranging range of 1-30 meters. When the obtained distance is more than 30 meters or 0, it is considered that the distance value of the point is wrong;

[0081] S35 Secondary Reasoning: The server side Figure 2The inference is performed again to determine whether the inference data is within the road range. If not, it is discarded and regarded as a false detection and removed. If it is, the nearest distance to the grid data point of the inference point is extracted, and the inverse distance interpolation method is used to calculate the disease distance. The server generates the road disease area based on the detection results and finally outputs the road disease area.

[0082] Camera calibration is a key step in obtaining an accurate depth map. Through camera calibration, the camera's internal parameters (focal length, principal point, etc.) and external parameters (relative position and rotation between cameras) can be obtained. The calibration process usually uses the checkerboard calibration method. Once the camera calibration is completed, the next step is stereo matching. The purpose of stereo matching is to find the corresponding pixels in the two images. Commonly used algorithms are: Block Matching, Semi-Global Matching (SGM), GraphCuts. According to the disparity map and the camera's internal and external parameters, the depth value of each pixel can be calculated and a depth map can be generated. Then the road depth map is combined with the original image to correct the abnormal road values ​​and extract the depth map data. The server calculates the secondary reasoning results of road damage through the uploaded depth map data, and ensures that the calculation of the road damage area is completed quickly and accurately in this scenario under a weak network environment, thereby maximizing the accuracy of the calculation of the damage area.

[0083] Preferably, the depth map is converted into a point cloud, i.e., a collection of points in a three-dimensional space. Each point consists of (x, y, z) coordinates, where (x, y) is the position of the image pixel and z is the depth value; in step S32, assuming that the intrinsic parameter matrix of the camera is K, the depth value of each pixel (u, v) in the depth map is d, and the mapping formula is:

[0084]

[0085] Among them, (c x ,c y ) is the principal point of the camera, f x and f y is the focal length.

[0086] Preferably, the specific steps of step S34 are:

[0087] S341: first, for a disease image, generate distance points of a grid and extract distance data points;

[0088] S342: Verify the extracted distance data points. If erroneous depth map data appears, use the other point data of the depth map on the AI ​​edge box side to interpolate the data using polynomial interpolation or K-nearest neighbor interpolation.

[0089] Preferably, when the inverse distance interpolation method is used to calculate the disease distance in step S35, it is assumed that the value of a point is inversely proportional to the value and distance of the surrounding known points, that is, the closer the known point is to the interpolation point, the greater the influence on its interpolation; the specific steps are:

[0090] S351 calculates weights: For each known point (x i ,y i ) and the interpolation point (x, y) are weighted, and the weight is calculated inversely proportional to the distance. The formula is:

[0091]

[0092] Among them, w i is the weight of the i-th known point; d i is the distance between the point to be interpolated and the i-th known point; p is the power of the weight, usually 1 or 2, which controls the speed of weight decay;

[0093] S352 calculates interpolation: calculates the value z of the point to be interpolated according to the weight and the value of the known point, the formula is:

[0094]

[0095] Among them, z i is the value of the ith known point, and N is the total number of known points.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] (1) Improved accuracy: Advanced image processing and area detection algorithms are used to accurately identify and locate road information in images, avoiding the problem of inaccurate or incomplete information caused by errors, and maximizing the accuracy of disease area calculation;

[0098] (2) High flexibility: Applicable to various road scenes, the binocular camera can automatically calculate the distance in various scenes and maintain high accuracy in scenes with low light, thus improving work efficiency;

[0099] (3) Strong stability: After multiple tests and verifications, the algorithm of the present invention has strong stability and reliability, and can perform well for different types of road images and scenes, ensuring the stable operation of the system;

[0100] (4) Reduced system risk: When the edge AI box transmits a large amount of alarm information through the wireless module, data transmission is the biggest bottleneck. The method of the present invention can reduce the system risk to the lowest level while still achieving a high calculation accuracy for the diseased area, taking into account both accuracy and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 It is a flow chart of a method for calculating road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention;

[0102] Figure 2 It is a flowchart of step S1 image preprocessing of the method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment of the present invention;

[0103] Figure 3 It is a flow chart of step S2 of the method for calculating the road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention;

[0104] Figure 4 It is a flow chart of step S3 of the method for calculating the road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention;

[0105] Figure 5 A schematic diagram of a road original image of a method for calculating road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention;

[0106] Figure 6 A schematic diagram of a captured depth map of a method for calculating a road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention;

[0107] Figure 7 A road separation effect diagram of the method for calculating the road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention;

[0108] Figure 8 A schematic diagram of a method for calculating road damage area in a weak network environment based on a binocular camera and an edge AI box of the present invention, which extracts grid node data in a depth map and uploads the data for secondary reasoning calculation of the area on the server side;

[0109] Fig. 9 It is a schematic diagram of the method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment of the present invention, where the car in the figure is an obstacle and interpolation calculation needs to be performed on the store data;

[0110] Fig.10 It is a schematic diagram of the method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment of the present invention, in which the street lamp is identified as a crack, which is a false detection and can be eliminated through road segmentation information;

[0111] Fig.11 This is a diagram showing the final calculation effect of the method of the present invention for calculating the road damage area in a weak network environment based on a binocular camera and an edge AI box. DETAILED DESCRIPTION

[0112] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0113] Example: Figure 1-Figure 11 As shown, the method for calculating the road damage area based on a binocular camera and an edge AI box in a weak network environment specifically includes the following steps:

[0114] S1 Image preprocessing: Preprocess the input original image, and perform noise processing and image normalization processing in sequence to obtain the processed data set, and divide the data set into a training set and a validation set;

[0115] like Figure 2 As shown, the specific steps of step S1 are:

[0116] S11 Image border filling: fill the border of the input original image to make the edge of the image complete;

[0117] Image preprocessing is used to separate road information and improve the accuracy of disease area calculation. The purpose of image padding is to add extra pixels to the border area of ​​the image in order to solve the problem of border pixel processing in image processing operations. Boundary padding is very important in tasks such as convolution operations, image transformation, and image segmentation to ensure that the algorithm can smoothly process the border part of the image;

[0118] S12 Noise processing: Use filtering or non-local mean processing to denoise the image;

[0119] The specific steps of using filtering to perform denoising in step S12 are:

[0120] S121 Median filter: Median filter is a commonly used denoising method, especially for salt-and-pepper noise. This method uses the median value in the sliding window instead of the central pixel value, so it can better preserve edge information.

[0121] S122 Frequency domain filtering: Using frequency domain filters, such as low-pass filters (such as Butterworth filters), can be used to remove high-frequency noise from images; the image is first converted to the frequency domain through Fourier transform, then the filter is applied, and then the inverse transform is performed;

[0122] The frequency domain filtering formula is:

[0123]

[0124] Where D0 is the cutoff frequency of the filter; u0, v0 are both frequency centers; H(u,v) is the filter function in the frequency domain;

[0125] The purpose of image noise processing is to reduce the noise in the image, enhance the quality and details of the image, and make the subsequent image processing or analysis more accurate and effective;

[0126] (1) Median filtering: Median filtering is a commonly used denoising method, especially for salt-and-pepper noise. This method uses the median value in the sliding window instead of the central pixel value, so it can better preserve edge information;

[0127] (2) Non-local mean (NLM) denoising: The non-local mean denoising algorithm uses redundant information in the image to calculate similar regions of pixels, thereby averaging the pixel values ​​of multiple similar regions. NLM is suitable for processing complex noise and can maintain image details.

[0128] (3) Frequency domain filtering: Frequency domain filters, such as low-pass filters (e.g., Butterworth filters), can be used to remove high-frequency noise from images;

[0129] S13 image normalization: Scale the pixel values ​​of the image to the range of [0, 1] or [-1, 1] to make the image suitable for the input layer of the convolutional neural network;

[0130] Image normalization is a common preprocessing step that improves the efficiency and accuracy of image processing and machine learning tasks by mapping the pixel values ​​of an image to a specific range (such as [0, 1] or [-1, 1]). Normalization can reduce the impact of factors such as lighting and contrast, ensure faster model convergence, and avoid gradient explosion or vanishing problems.

[0131] Common normalization methods:

[0132] (1) Min-Max normalization: Scale the pixel values ​​of an image to the range of [0, 1] or [-1, 1]. It is usually used to scale the pixel range of an image from [0, 255] to [0, 1]. It is particularly suitable for the input layer of a convolutional neural network. The formula is:

[0133]

[0134] Where: x is the original pixel value; x min and x max are the minimum and maximum pixel values ​​respectively; x′ is the normalized value, usually between [0,1] or [-1,1];

[0135] (2) Z-score normalization (zero mean normalization): normalizes the image pixel values ​​to a standard distribution with a mean of 0 and a standard deviation of 1. It is suitable for scenarios where the data has a large variance and ensures that each pixel value is standardized relative to the overall data distribution. The formula is:

[0136]

[0137] Where: x is the original pixel value; μ is the mean of the pixel value; σ is the standard deviation of the pixel value; x′ is the normalized value;

[0138] (3) Local Contrast Normalization: Local Contrast Normalization (LCN) enhances the contrast in an image by adjusting the mean and standard deviation of each pixel value relative to its neighborhood. It is suitable for tasks such as edge detection. The formula is:

[0139]

[0140] Among them, μ i,j and σ i ,j is at pixel x i , the mean and standard deviation of the neighborhood around j;

[0141] S2 Road Segmentation: Train and optimize the YOLOv5-seg model to obtain the YOLOv5-seg road segmentation model, and then use the YOLOv5-seg road segmentation model to segment the road;

[0142] like Figure 3 As shown, the specific steps of step S2 are:

[0143] S21 builds the model: adds a segmentation head based on the YOLOv5 model to generate segmentation masks;

[0144] S22 model training and optimization: The training set is used for model training. During training, the cosine annealing learning rate, loss function and hyperparameter optimization are performed to generate a road segmentation model.

[0145] The specific steps of optimizing the cosine annealing learning rate and loss function in step S22 are:

[0146] S221 Cosine annealing learning rate optimization: The cosine annealing scheduler is used to optimize the cosine annealing learning rate of the YOLOv5-seg model. The cosine annealing learning rate scheduler is a strategy for gradually reducing the learning rate. It is usually used in the training of deep learning models to improve the convergence and stability of the model. In the training optimization of YOLOv5-seg road segmentation, the cosine annealing scheduler can dynamically adjust the learning rate to prevent the model from falling into a local optimal solution, and gradually reduce the learning rate in the later stage of training to help the model converge better. The core idea of ​​the cosine annealing scheduler is based on the cosine function, which smoothly reduces the learning rate during the training process. The formula is:

[0147]

[0148] Where: η t : learning rate at the current iteration time t; η max : initial learning rate, i.e., maximum learning rate; η min : minimum learning rate, the minimum value that the learning rate approaches in the later stage of training; T cur : current epoch or step; T max : Maximum number of epochs or steps, i.e. the cycle of the entire learning rate scheduling; cos: cosine function;

[0149] S222 loss function optimization: Combine the Dice Loss loss function and the IoU Loss loss function for optimization; in order to combine Dice Loss and IoU Loss, the weighted sum of the Dice Loss loss function and the IoU Loss loss function is used, and the formula is:

[0150] Total Loss=α·Dice Loss+β·IoU Loss;

[0151] Among them: α and β are hyperparameters used to balance the impact of the two loss functions; Dice Loss and IoU Loss are the corresponding Dice coefficient and IoU loss respectively;

[0152] The optimization of the loss function in the segmentation task is very critical, which can help the model converge faster and improve the segmentation accuracy. Common optimization methods are as follows:

[0153] (1) Cross-Entropy Loss: This is the most basic segmentation loss function, but it may not work well for unbalanced classes (such as smaller road areas);

[0154] (2) Dice Loss: focuses on improving the overlap of segmentation and is particularly suitable for dealing with data imbalance problems. Dice Loss performs well for smaller road areas;

[0155] (3) Focal Loss: When dealing with the class imbalance problem in road segmentation tasks, Focal Loss can suppress the loss of simple samples and increase the focus on difficult samples;

[0156] (4) IoU Loss (Intersection over Union Loss): Directly optimizes the IoU between the segmentation result and the true label, which can improve the accuracy of the segmented area;

[0157] For example, in the road segmentation task, since roads are usually long and winding, segmentation faults are prone to occur. A common approach is to combine Dice Loss and IoU Loss to ensure the continuity and accuracy of the segmented area.

[0158] S23 Model Reasoning Optimization: The road segmentation model is then quantized, pruned, and optimized using the non-maximum suppression method (NMS) to output the YOLOv5-seg road segmentation model.

[0159] The structure of the YOLOv5-se model adds a segmentation head to the YOLOv5 model to generate segmentation masks. Its working principle is to generate the corresponding segmentation mask while predicting the object category in each detection box. For the road segmentation task, YOLOv5-seg only needs to predict the binary mask of the road during training. The key components of the YOLOv5-seg model: Backbone network: extract features, such as using CSPDarknet. Detection head: Generates bounding boxes and category predictions for objects. Segmentation head: The additional segmentation branch outputs the segmentation mask of each target;

[0160] The specific steps of step S23 are:

[0161] S231 Model Quantization: Dynamic quantization and static quantization are used to perform quantization perception during model training;

[0162] Model quantization can reduce memory usage and improve inference speed, especially on mobile or embedded devices. YOLOv5 supports the following two quantization methods:

[0163] Dynamic quantization: quantize weights and activation functions to low-precision data types (such as INT8) only during inference, but keep calculations as FP32;

[0164] Static quantization (quantization-aware training): quantization-aware training is performed during training, and low-precision reasoning is used directly during inference. Quantization-aware training needs to be performed in advance;

[0165] S232 Model pruning: Use L1 regularization or sparsity constraints to prune the model; model pruning reduces the model size and inference calculation amount by removing unimportant convolution kernels or neurons, that is, reducing redundant weights;

[0166] S233NMS optimization: According to the IoU value, the confidence of the detection box with an IoU value higher than the threshold is smoothly attenuated, and the confidence attenuation adopts Gaussian attenuation function or linear attenuation function;

[0167] In YOLOv5-seg, Soft-NMS can be used to optimize the selection of bounding boxes in segmentation tasks, especially when the boundaries of multiple roads overlap, Soft-NMS can more smoothly suppress redundant boxes while retaining road targets with fuzzy boundaries. The improvements of Soft-NMS include:

[0168] Soft-NMS does not directly remove detection boxes with IoU higher than the threshold, but smoothly attenuates the confidence of these boxes according to the IoU value;

[0169] Confidence decay usually uses a Gaussian decay function or a linear decay function, so that the confidence is gradually reduced instead of being directly set to 0;

[0170] S24 Road segmentation: Use the YOLOv5-seg road segmentation model to segment the road, generate the corresponding segmentation mask, and then output the segmentation result; Figure 7 The following is a road segmentation effect diagram;

[0171] S3 extracts the depth map and calculates the damaged area: extracts features from the depth map data obtained by the binocular camera in combination with the road boundary, and calculates the damage distance to obtain the road damage area;

[0172] like Figure 4 As shown, the specific steps of step S3 are:

[0173] S31 obtains the depth map: first calibrates the binocular camera to obtain the internal parameters and external parameters of the binocular camera; then performs stereo matching to find the corresponding pixel points in the two images; then calculates the depth value of each pixel according to the disparity map and the internal and external parameters of the camera, and generates a depth map;

[0174] The calculation formula of the depth value Z(x,y) is:

[0175]

[0176] Where Z(x,y) is the depth value corresponding to the pixel point (x,y); f is the focal length of the camera; B is the baseline length between the two cameras; d(x,y) is the disparity between the corresponding points of the pixel point (x,y) in the two images;

[0177] Camera calibration is a key step in obtaining an accurate depth map. Through camera calibration, the camera's intrinsic parameters (focal length, principal point, etc.) and extrinsic parameters (relative position and rotation between cameras) can be obtained; the calibration process usually uses the checkerboard calibration method; once the camera calibration is completed, the next step is stereo matching; the purpose of stereo matching is to find the corresponding pixels in the two images. Commonly used algorithms include: Block Matching, Semi-Global Matching (SGM), GraphCuts. Based on the disparity map and the camera's extrinsic parameters, the depth value of each pixel can be calculated and a depth map can be generated; such as Figure 5 The original picture is shown. Figure 6 Shown is a schematic diagram of a depth map;

[0178] S32 depth map conversion to point cloud: The acquired depth map is mapped to a 3D point through the camera model.

[0179] The depth map is converted into a point cloud, which is a collection of points in three-dimensional space. Each point consists of (x, y, z) coordinates, where (x, y) comes from the position of the image pixel and z is the depth value. In step S32, it is assumed that the intrinsic parameter matrix of the camera is K, and the depth value of each pixel (u, v) in the depth map is d. The mapping formula is:

[0180] z = d;

[0181] Among them, (c x ,c y ) is the principal point of the camera, f x and f y is the focal length;

[0182] S33 Interpolation of obstacle point data: Identify obstacles in combination with the original image and regenerate the obstacle point data; in the actual road inspection process, some obstacles are inevitable, such as cars in other lanes, cars in front, pedestrians on the road, battery vehicles, etc., which will affect the distance measurement of the binocular camera and have a great impact on the identified area; by calculating the obstacle coordinates, mark the coordinates of this part in the depth map as obstacles. The depth map distance here needs to be interpolated according to other values ​​to ensure that the data extracted from the depth map is the distance from the binocular camera to the road surface rather than the distance to the obstacle;

[0183] S34 extracts grid data: extracts grid data of the depth map, i.e., distance points of the grid, in combination with the road separation data, verifies the extracted distance data points, interpolates the erroneous depth map data, and uploads the original image and depth map grid data; the distance of a certain pixel point obtained by the depth map is usually within a normal range. The currently used binocular camera supports a ranging range of 1-30 meters. When the obtained distance is more than 30 meters or 0, it is considered that the distance value of the point is wrong;

[0184] The specific steps of step S34 are:

[0185] S341: first, for a disease image, generate distance points of a grid and extract distance data points;

[0186] S342: Verify the extracted distance data points. If erroneous depth map data is found, the AI ​​edge box uses the other depth map point data to interpolate the data using polynomial interpolation or K-nearest neighbor interpolation. Fig. 9 As shown;

[0187] In this embodiment, for a road damage map (for example: 1920*1080 pixels), a 10*10 grid (configurable) of distance points can be generated, the depth map size is about 4MB per map, and the extracted data is about 4KB. If there are 1,000 road damage alarms in one inspection, 4G of data needs to be transmitted, which will directly lead to excessive backlog of transmission files in a weak network environment, and seriously cause the edge box to crash. By extracting key points, only 4MB of data needs to be transmitted, and the pressure on the AI ​​edge box is greatly alleviated. The extracted distance data points are verified, and the problematic points use other point data of the depth map to interpolate the data on the AI ​​edge box side. Commonly used interpolation methods:

[0188] Polynomial interpolation: Polynomial interpolation interpolates by constructing a polynomial that passes through all known points. Common methods include Lagrange interpolation and Newton interpolation. For nnn known points, a polynomial of degree n-1n-1n-1 can be constructed;

[0189] Piecewise Linear Interpolation is a commonly used interpolation method that estimates the value between data by connecting linear segments of adjacent data points. This method can provide simple and effective interpolation results when the data does not change much and is relatively smooth.

[0190] K-Nearest Neighbors Imputation (KNN interpolation) is an interpolation method based on the K-nearest neighbor algorithm, mainly used to fill missing values. It estimates missing values ​​by using the data of neighboring samples and is suitable for multidimensional data sets. KNN interpolation can process numerical and categorical data and is usually used when data is missing or incomplete.

[0191] S35 Secondary reasoning: Figure 8 As shown, the server Figure 2 The second inference is to determine whether the inference data is within the road range. If not, it is discarded and regarded as a false detection and removed. Fig.10 As shown; if in, the nearest distance of the inference point grid data point is extracted, and the inverse distance interpolation method is used to calculate the disease distance. The server generates the road disease area according to the detection results and finally outputs the road disease area;

[0192] Inverse Distance Weighting (IDW) is a distance-based interpolation method commonly used in spatial data analysis. This method assumes that the value of a point is inversely proportional to the value and distance of its surrounding known points, that is, the closer the known points are to the interpolation point, the greater the influence on its interpolation.

[0193] When the inverse distance interpolation method is used to calculate the disease distance in step S35, it is assumed that the value of a point is inversely proportional to the value and distance of the surrounding known points, that is, the closer the known points are to the interpolation point, the greater the influence on the interpolation value; the specific steps are:

[0194] S351 calculates weights: For each known point (x i ,y i ) and the interpolation point (x, y) are weighted, and the weight is calculated inversely proportional to the distance. The formula is:

[0195]

[0196] Among them, w i is the weight of the i-th known point; d i is the distance between the point to be interpolated and the i-th known point; p is the power of the weight, usually 1 or 2, which controls the speed of weight decay;

[0197] S352 calculates interpolation: calculates the value z of the point to be interpolated according to the weight and the value of the known point, the formula is:

[0198]

[0199] Among them, z i is the value of the i-th known point, and N is the total number of known points;

[0200] In this embodiment, the server generates the road damage area according to the detection result as follows:

[0201] The edge AI box can calculate the area of ​​any road damage by uploading information extracted from the road map and depth map, as well as the road segmentation boundaries, even if there is no depth map on the server side. Each road damage map corresponds to a set of data extracted from the depth map. For example, on a 1920*1080 road map, the edge AI box detects that the coordinates of the road damage are (x1, x2, y1, y2) (1200, 600, 1300, 700) respectively. On the server side, by using high-precision model reasoning, it is found that the coordinates of the damage are (x3, x4, y3, y4) (1100, 500, 1400, 800) respectively. We can calculate the distance information of the surrounding points of (x3, y3) based on the depth map grid data, and calculate the distance of the point (x3, y3) by inverse distance interpolation. The distances of the points (x4, y3), (x3, y4), and (x4, y4) are calculated in the same way. Finally, the actual area of ​​the damage is calculated by the distance of the four points, and the severity of the damage is determined based on the size of the area; such as Fig.11 As shown, Fig.11 The blue box in the middle is the crack area detected by the AI ​​edge box, and the green box is the crack area detected by the high-precision model on the server side. The server side uses the grid distance information and the inverse distance interpolation method to recalculate the actual distance of the four points in the green box, and calculates the road disease area based on the actual distance; determine the hazard level of the disease;

[0202] Through one-day inspection in a prefecture-level city in Suzhou, 3215 road disease data (including nine categories: potholes, bumps, damaged joints, strip cracks, subsidence, network cracks, pavement damage, rutting, and pockmarked surfaces) were statistically analyzed and compared with traditional technologies as shown in Table 1.

[0203] Table 1 Comparison results between the traditional AI edge box solution and the method of the present invention

[0204]

[0205] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A method for calculating road damage area in a weak network environment based on a binocular camera and an edge AI box, characterized in that: The specific steps include: S1 Image preprocessing: Preprocess the input original image, and perform noise processing and image normalization processing in sequence to obtain the processed data set, and divide the data set into a training set and a validation set; S2 Road Segmentation: Train and optimize the YOLOv5-seg model to obtain the YOLOv5-seg road segmentation model, and then use the YOLOv5-seg road segmentation model to segment the road; S3 extracts the depth map and calculates the damaged area: The depth map data obtained by the binocular camera is combined with the road boundary to perform feature extraction, and the damage distance is calculated to obtain the road damage area.

2. According to claim 1, the method for calculating road damage area based on binocular camera and edge AI box in a weak network environment is characterized in that: The specific steps of step S1 are: S11 Image boundary filling: fill the boundary of the input original image to make the edge of the image complete; S12 Noise processing: use filtering or non-local mean processing to denoise the image; S13 Image normalization: Scale the pixel values ​​of an image to the range [0, 1] or [-1, 1] to make the image suitable for the input layer of a convolutional neural network.

3. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 2 is characterized in that: The specific steps of using filtering to perform denoising in step S12 are: S121 median filtering: Use the median value in the sliding window to replace the central pixel value; S122 Frequency Domain Filtering: Using frequency domain filters, the image is first converted to the frequency domain through Fourier transform, then the filter is applied, and then the inverse transform is performed; The frequency domain filtering formula is: Where D0 is the cutoff frequency of the filter; u0 and v0 are both frequency centers; and H(u,v) is the filter function in the frequency domain.

4. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 2 is characterized in that: The specific steps of step S2 are: S21 builds the model: adds a segmentation head based on the YOLOv5 model to generate segmentation masks; S22 model training and optimization: The training set is used for model training. During training, the cosine annealing learning rate, loss function and hyperparameter optimization are performed to generate a road segmentation model. S23 Model Reasoning Optimization: The road segmentation model is then quantized, pruned, and optimized using the non-maximum suppression method to output the YOLOv5-seg road segmentation model; S24 Road segmentation: Use the YOLOv5-seg road segmentation model to segment the road, generate the corresponding segmentation mask, and then output the segmentation result.

5. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 4 is characterized in that: The specific steps of optimizing the cosine annealing learning rate and loss function in step S22 are: S221 Cosine annealing learning rate optimization: The cosine annealing scheduler is used to optimize the cosine annealing learning rate of the YOLOv5-seg model. The cosine annealing scheduler is based on the cosine function and smoothly reduces the learning rate during training. The formula is: Where: η t : The learning rate at the current iteration time t; η max : Initial learning rate, i.e. the maximum learning rate; η min : Minimum learning rate, the minimum value that the learning rate approaches in the later stage of training; T cur : current epoch or step; T max : Maximum number of epochs or steps, i.e. the entire learning rate scheduling cycle; cos: cosine function; S222 loss function optimization: Combine the Dice Loss loss function and the IoU Loss loss function for optimization; use the weighted sum of the DiceLoss loss function and the IoU Loss loss function, the formula is: Total Loss=α·Dice Loss+β·IoU Loss; Among them: α and β are hyperparameters used to balance the impact of the two loss functions; Dice Loss and IoU Loss are the corresponding Dice coefficient and IoU loss respectively.

6. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 4 is characterized in that: The specific steps of step S23 are: S231 Model Quantization: Dynamic quantization and static quantization are used to perform quantization perception during model training; S232 Model pruning: Use L1 regularization or sparsity constraints for model pruning; S233NMS optimization: According to the IoU value, the confidence of the detection box with an IoU value higher than the threshold is smoothly decayed, and the confidence decay adopts a Gaussian decay function or a linear decay function.

7. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 4 is characterized in that: The specific steps of step S3 are: S31 obtains the depth map: first calibrates the binocular camera to obtain the internal parameters and external parameters of the binocular camera; then performs stereo matching to find the corresponding pixels in the two images; then calculates the depth value of each pixel based on the disparity map and the internal and external parameters of the camera, and generates a depth map; the calculation formula of the depth value Z(x, y) is: Where Z(x,y) is the depth value corresponding to the pixel point (x,y); f is the focal length of the camera; B is the baseline length between the two cameras; d(x,y) is the disparity between the corresponding points of the pixel point (x,y) in the two images; S32 depth map conversion to point cloud: The acquired depth map is mapped to a 3D point through the camera model. S33 interpolates obstacle point data: identifies obstacles in combination with the original image, and regenerates the obstacle point data; S34 extracts grid data: extracts grid data of the depth map, i.e., distance points of the grid, in combination with the road separation data, verifies the extracted distance data points, interpolates the erroneous depth map data, and uploads the original image and the depth map grid data; S35 Secondary Reasoning: The server performs secondary reasoning on the original image to determine whether the reasoned data is within the road range. If not, it is discarded and regarded as a false detection and removed. If it is, the nearest distance to the grid data point of the reasoning point is extracted, and the inverse distance interpolation method is used to calculate the disease distance. The server generates the road disease area based on the detection results and finally outputs the road disease area.

8. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 7 is characterized in that: In step S32, it is assumed that the intrinsic parameter matrix of the camera is K, the depth value of each pixel (u, v) in the depth map is d, and the mapping formula is: z=d; Among them, (c x ,c y ) is the principal point of the camera, f x and f y is the focal length.

9. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 7 is characterized in that: The specific steps of step S34 are: S341: first, for a disease image, generate distance points of a grid and extract distance data points; S342: Verify the extracted distance data points. If erroneous depth map data appears, use the other point data of the depth map on the AI ​​edge box side to interpolate the data using polynomial interpolation or K-nearest neighbor interpolation.

10. The method for calculating road damage area based on binocular camera and edge AI box in weak network environment according to claim 7, characterized in that: When the inverse distance interpolation method is used to calculate the disease distance in step S35, it is assumed that the value of a point is inversely proportional to the value and distance of the surrounding known points, that is, the closer the known points are to the interpolation point, the greater the influence on the interpolation value; the specific steps are: S351 calculates weights: For each known point (x i ,y i ) and the interpolation point (x, y) are weighted, and the weight is calculated inversely proportional to the distance. The formula is: Among them, w i is the weight of the i-th known point; d i is the distance between the point to be interpolated and the i-th known point; p is the power of the weight, which controls the speed of weight decay; S352 calculates interpolation: calculates the value z of the point to be interpolated according to the weight and the value of the known point, the formula is: Among them, z i is the value of the ith known point, and N is the total number of known points.

Images