AI-based intelligent road defect sorting platform

Abstract

This invention relates to the field of intelligent road maintenance technology, and more particularly to an AI-based intelligent road surface defect sorting platform. The platform uses a multispectral acquisition module and an edge computing module as front-end acquisition and recognition modules, and also includes a defect sorting decision module and a defect interactive display module. It can acquire road surface defect image frames in real time, perform frame extraction preprocessing and single-frame delayed defect area identification to obtain the defect location region within a single road surface image frame; perform 12-dimensional feature analysis of crack defects and maintenance urgency assessment on the defect location region within each single road surface image frame to obtain the maintenance urgency index corresponding to each single road surface image frame; based on the maintenance urgency index corresponding to each single road surface image frame, perform emergency maintenance sorting planning, and issue maintenance disposal commands on the human-computer interaction terminal to execute the corresponding road surface defect maintenance fleet scheduling planning. This invention can improve detection efficiency and sorting accuracy, optimize resources, and reduce unnecessary maintenance investment.

Description

Technical Field

[0001] This invention relates to the field of intelligent maintenance technology, and in particular to an intelligent sorting platform for road surface defects based on AI recognition. Background Technology

[0002] With the acceleration of urbanization, road surface defects are becoming increasingly serious. Damage such as road surface breaks, cracks, and potholes not only affect the service life of roads but also increase traffic safety hazards. In recent years, with the rapid development of artificial intelligence (AI) technology, automated detection technologies based on deep learning and computer vision have been gradually applied to the identification and analysis of road defects. AI algorithm models can automatically extract features of road surface defects from large amounts of image data, achieving automated and intelligent road surface defect identification.

[0003] In addition, similar patents such as CN111126460A disclose an automatic inspection method, medium, equipment, and device for road surface defects based on artificial intelligence. This includes acquiring a road surface image to be detected and its corresponding location information at a preset frequency; inputting the road surface image to be detected into a trained road surface defect recognition model to determine whether the image is a road surface defect; if so, generating an alarm message based on the image and its corresponding location information so that relevant personnel can handle the road surface defect according to the alarm message. This allows for automatic inspection of road surface defects, effectively improving inspection efficiency and saving manpower and resources. While these patents can effectively improve inspection efficiency, they are limited to locating defects and lack a priority sorting mechanism for defects. They fail to perform intelligent grading based on the severity and type of defects, thus affecting subsequent maintenance decision-making. Summary of the Invention

[0004] Therefore, it is necessary for the present invention to provide an AI-based intelligent sorting platform for road surface defects to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above objectives, an AI-based intelligent road surface defect sorting platform is provided, comprising: a pre-processing acquisition and identification module and an intelligent road surface defect sorting platform. The pre-processing acquisition and identification module and the intelligent road surface defect sorting platform are connected to the same network. The pre-processing acquisition and identification module includes a multispectral acquisition module and an edge computing module. The intelligent road surface defect sorting platform includes a defect sorting decision module and a defect interactive display module.

[0006] The multispectral acquisition module is used to acquire road surface distress image frames in different spectral bands in real time at a speed of 60 km / h using a vehicle-mounted array camera that integrates corresponding visible light / infrared sensors; and to perform frame extraction preprocessing on the road surface distress image frames in different spectral bands to generate standard frames of road surface distress in different spectra.

[0007] The edge computing module is used to perform single-frame delay control disease area identification on standard frames of road surface defects under different spectra by deploying the corresponding lightweight YOLOv7 improved model, so as to obtain the corresponding disease location area within a single frame of the road surface image under each spectrum.

[0008] The defect sorting decision module is used to perform 12-dimensional feature analysis of crack defects in the defect location area within a single frame of a pavement image under each spectrum, so as to extract the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image; to perform maintenance emergency assessment on the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image based on fuzzy hierarchical analysis, so as to obtain the maintenance urgency index corresponding to each single frame of the pavement image; and to perform maintenance emergency sorting planning on each single frame of the pavement image based on the maintenance urgency index corresponding to each single frame of the pavement image, so as to generate the pavement maintenance vehicle fleet sorting and scheduling sequence corresponding to the single frame of the pavement image sorting.

[0009] The defect interactive display module is used to display each road surface image sorting frame on the corresponding human-computer interaction terminal to intuitively view the defect location, type and maintenance urgency index corresponding to each road surface image sorting frame, and to issue corresponding maintenance treatment instructions on the human-computer interaction terminal according to the road maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting frame, so as to execute the corresponding road surface defect maintenance fleet scheduling and planning work.

[0010] Furthermore, the multispectral acquisition module includes the following functions:

[0011] A vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and acquire road surface distress image frames in different spectral bands in real time at a speed of 60km / h. The road surface distress image frames in the visible light sensor band are used to capture conventional visible road surface distress, while the road surface distress image frames in the infrared sensor band are used to identify tiny cracks or damage hidden in the road surface based on the corresponding temperature distribution.

[0012] The pavement distress image frames under different spectral bands were processed by time-slot extraction at a frame extraction frequency of 5 seconds / frame to obtain local pavement distress frame sets under different spectral bands.

[0013] The pixel blur level of local frames of road surface defects under different spectral bands is averaged to obtain the average pixel blur level of image frames under different spectral bands.

[0014] Based on the average pixel blurring of image frames under different spectral bands, the corresponding road surface distress image frames under different spectral bands are blurred and denoised to obtain blurred and denoised road surface distress frames under different spectral bands.

[0015] Pixel-level normalization is performed on the blurred and denoised pavement distress frames under different spectral bands to generate standard pavement distress frames under different spectra.

[0016] Furthermore, the edge computing module includes the following functions:

[0017] By inputting standard frames of road surface defects under different spectra to the corresponding edge computing nodes in the local vehicle, the image processing latency of a single frame of road surface defect image under each spectrum can be controlled.

[0018] By performing statistical analysis of road surface region features on the corresponding edge computing nodes in the local vehicle, the edge and texture features of the road surface image region under different spectra are statistically analyzed and a feature matrix is ​​formed to generate the image region feature matrix corresponding to a single frame of road surface defect image under each spectrum.

[0019] By deploying the corresponding lightweight YOLOv7 improved model on the corresponding edge computing node and inputting the image region feature matrix corresponding to the single frame of the road surface distress image under each spectrum, the corresponding single frame of the road surface distress image is used to identify the distress candidate region, so as to segment each single frame of the road surface distress image into various candidate regions, and identify and mark the candidate regions corresponding to the road surface distress, while marking the position and size of the corresponding candidate regions, so as to generate the set of distress candidate regions corresponding to the single frame of the road surface distress image under each spectrum;

[0020] For each candidate region in the candidate region set corresponding to a single frame of pavement distress image under each spectrum, the distress location confidence is calculated to obtain the distress location confidence distribution corresponding to each candidate region in the candidate region set under each spectrum.

[0021] Based on the confidence distribution of the disease location corresponding to each candidate region in the disease candidate region set under each spectrum, confidence optimization screening is performed on each candidate region in the disease candidate region set corresponding to a single frame of the road surface disease image under each spectrum to obtain the corresponding disease location region in a single frame of the road surface image under each spectrum.

[0022] Furthermore, the image processing latency is specifically controlled to be within ≤50ms.

[0023] Furthermore, the confidence optimization screening of each candidate region in the candidate region set corresponding to a single frame of the pavement defect image under each spectrum, based on the confidence distribution of the defect location corresponding to each candidate region in the defect candidate region set under each spectrum, includes:

[0024] The confidence gradient of the candidate area of ​​the road surface disease image for each single frame under each spectrum is calculated based on the confidence distribution of the disease location corresponding to each candidate area in the disease candidate area set under each spectrum.

[0025] Based on the confidence gradient of the candidate regions of road surface defects in a single frame of the image under each spectrum, the confidence distribution of the defect location corresponding to each candidate region in the defect candidate region set under each spectrum is optimized and filtered. If the confidence distribution of the corresponding defect location is greater than or equal to the confidence gradient of the defect candidate region, the corresponding candidate region in its defect candidate region set is selected as the defect location region corresponding to the single frame of the road surface defects image under that spectrum. Candidate regions with confidence distributions less than the confidence gradient of the defect candidate region are then filtered out to obtain the defect location regions corresponding to the single frame of the road surface defects image under each spectrum.

[0026] Furthermore, the disease sorting decision module includes the following functions:

[0027] The image scale corresponding to a single frame of the road surface image under each spectrum is obtained by identifying the location of the damage within a single frame of the road surface image under each spectrum.

[0028] Based on the corresponding image scale, a 12-dimensional feature analysis of cracks and defects is performed on the corresponding defect location regions in a single frame of a pavement image under each spectrum to extract a 12-dimensional feature set of cracks and defects corresponding to a single frame of each pavement image. This set includes the maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, defect area, crack distribution density, defect perimeter, defect shape complexity, defect gray mean, and defect gray standard deviation.

[0029] Based on the fuzzy hierarchical analysis method, the 12-dimensional feature set of cracks and defects corresponding to each single frame of the road surface image is used to conduct an emergency assessment of maintenance, and the maintenance urgency index corresponding to each single frame of the road surface image is obtained.

[0030] Based on the maintenance urgency index corresponding to each road surface image frame, maintenance urgency sorting planning is performed on each road surface image frame to generate a road maintenance vehicle sorting and scheduling sequence corresponding to the road surface image sorting frame.

[0031] Furthermore, the 12-dimensional feature analysis of crack defects within a single frame of the road surface image under each spectrum, based on the corresponding image scale, includes:

[0032] Binarization processing is performed on the corresponding disease location regions within a single frame of the road surface image under each spectrum to generate binarized images of the corresponding disease regions under each spectrum.

[0033] Based on the binarized images of the diseased areas under each spectrum and in combination with the corresponding image scale, the maximum width of the crack, the total length of the crack, the tortuosity of the crack, the area of ​​the diseased area, the crack distribution density, the perimeter of the diseased area, and the shape complexity of the diseased area are calculated.

[0034] By taking points at equal intervals along the crack centerline within the corresponding visible spectral band of the road surface image in each spectrum, dividing the area into several small segments, calculating the vertical width of the crack corresponding to each segment, and calculating the average value, the average crack width is obtained.

[0035] By identifying the location of the damage in the corresponding infrared spectral band within a single frame of the road surface image under each spectrum, the temperature distribution differences between each equally spaced point are determined. Based on the temperature distribution differences between each equally spaced point, the crack depth corresponding to each equally spaced point is estimated. At the same time, the maximum value and average value of the crack depth corresponding to each equally spaced point are determined to obtain the corresponding maximum crack depth and average crack depth.

[0036] The mean and standard deviation of gray levels in the corresponding disease location area within a single frame of the road surface image under each spectrum are calculated to obtain the mean and standard deviation of gray levels of all pixels in the disease location area, thus obtaining the mean and standard deviation of gray levels in the corresponding disease area.

[0037] The above-mentioned maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of ​​the affected area, crack distribution density, perimeter of the affected area, shape complexity of the affected area, mean gray value of the affected area, and standard deviation of gray value of the affected area are combined into a 12-dimensional feature set of cracks and defects corresponding to a single frame of each road image.

[0038] Furthermore, the calculation of the corresponding maximum crack width, total crack length, crack tortuosity, affected area area, crack distribution density, affected area perimeter, and affected area shape complexity based on the binarized images of the affected areas under each spectrum and in conjunction with the corresponding image scale includes:

[0039] By highlighting the corresponding crack area in the binarized image of the disease area under each spectrum, and using morphological operations to remove the corresponding noise and fine branches, the bounding rectangle of the crack area is calculated. The length of the short side of the bounding rectangle is approximated as the corresponding maximum width, and the corresponding maximum crack width is calculated according to the corresponding image scale.

[0040] Skeletonization processing is performed on the binarized images of the diseased areas under each spectrum to simplify the corresponding crack areas into lines with a width of one pixel, and the total number of pixels on the lines is counted. At the same time, the total length of the corresponding crack is calculated by combining the corresponding image scale.

[0041] The corresponding crack start point and crack end point are obtained by binarizing the disease area under each spectrum. The straight distance between the crack start point and the crack end point is calculated based on the crack start point and crack end point. The crack tortuosity is obtained by the ratio of the total crack length to the straight distance between the crack start point and the crack end point.

[0042] By statistically analyzing the total number of pixels in the binarized image of the diseased area under each spectrum, and converting the corresponding diseased area according to the corresponding image scale, the corresponding crack distribution density is calculated based on the ratio between the total crack length and the diseased area.

[0043] Boundary detection is performed on the binarized images of the diseased areas under each spectrum to obtain the boundary of the diseased area after binarization, and the total number of pixels on the boundary of the diseased area is counted. At the same time, the perimeter of the diseased area is calculated according to the corresponding image scale.

[0044] The shape complexity of the diseased area is calculated by the ratio between the square of the perimeter of the diseased area and the area of ​​the diseased area.

[0045] Furthermore, the maintenance urgency assessment of the 12-dimensional feature set of cracks and defects corresponding to each single frame of the pavement image based on the fuzzy hierarchical analysis method is performed to obtain the maintenance urgency index corresponding to each single frame of the pavement image, including:

[0046] Based on the fuzzy hierarchical analysis method, a fuzzy judgment matrix of cracks and defects corresponding to each single frame of road surface image is constructed.

[0047] Based on the crack defect fuzzy judgment matrix corresponding to each single frame of each road surface image, weighted fuzzy inference is performed on each feature factor in the corresponding 12-dimensional feature set of crack defects to generate fuzzy weight values ​​corresponding to each feature factor in each single frame of each road surface image.

[0048] Based on the fuzzy weight values ​​of each feature factor in a single frame of each pavement image, an emergency maintenance assessment is performed on the actual values ​​of each feature factor in the corresponding 12-dimensional feature set of crack defects, resulting in the maintenance urgency index corresponding to each single frame of the pavement image.

[0049] Furthermore, the step of planning the maintenance urgency sorting of each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame includes:

[0050] Based on the maintenance urgency index corresponding to each road surface image frame, the corresponding road surface image frames are sorted and prioritized for emergency maintenance to generate the corresponding road surface maintenance emergency priority sorting image frame sequence.

[0051] The road surface maintenance emergency priority sorting image frame sequence is used to search for the location of defects in each road surface image sorting frame in order to generate a sequence of defect location points corresponding to the road surface image sorting frame.

[0052] By combining the ant colony optimization algorithm of GIS map with the sequence of defect locations corresponding to a single frame of road image sorting, maintenance vehicle route planning is performed. This combines the corresponding defect locations in the sequence with the GIS map and fully considers the corresponding road network topology and traffic conditions to plan the optimal driving route for the corresponding maintenance vehicle, thereby generating the road maintenance vehicle sorting and scheduling sequence corresponding to a single frame of road image sorting.

[0053] The beneficial effects of this invention are:

[0054] The AI-based intelligent road surface defect sorting platform proposed in this invention integrates a multispectral acquisition module and an edge computing module as front-end acquisition and recognition modules, and also includes a defect sorting decision module and a defect interactive display module. Compared with existing technologies, the advantages of this application lie in the use of a vehicle-mounted array camera with integrated visible light / infrared sensors for real-time acquisition of road surface defect images. This plays a significant role in improving the accuracy and efficiency of road surface defect detection. By acquiring images through a vehicle traveling at a speed of 60 km / h, a large area of ​​road surface defect detection can be quickly covered. The high vehicle speed, combined with high-precision imaging... Sensors ensure the acquisition of image data across different spectral bands, enabling accurate capture of various types of pavement defects, such as cracks, potholes, and ruts, under varying environmental and lighting conditions. The addition of infrared sensors ensures clear acquisition of defect images even in low-light conditions at night or under poor lighting conditions, providing useful information different from visible light images. Subsequently, pavement defect image frames from different spectral bands are preprocessed by frame extraction and optimized through image enhancement, noise removal, and other methods to generate standardized pavement defect image frames. This not only improves the consistency of data processing but also significantly enhances the efficiency of pavement defect identification. Secondly, by deploying a lightweight improved YOLOv7 model for single-frame delayed-control identification of pavement distress areas, the accuracy and efficiency of pavement distress detection can be significantly improved. As a highly efficient target detection model, YOLOv7 has strong real-time detection capabilities and low computational resource consumption. It can process large-scale pavement image data quickly while maintaining high accuracy. YOLOv7 is used to identify and locate distress areas in each frame of the image, especially typical distresses such as cracks and potholes. Compared with traditional image processing techniques, YOLOv7, trained by deep learning algorithms, can more accurately distinguish and label different types of distress areas. It can also adapt to changes in different lighting conditions, improve the robustness of distress detection, and ensure the accuracy of distress location areas. Then, by performing 12-dimensional feature analysis on the damaged areas in each frame of the image, it is helpful to comprehensively and accurately assess the nature and severity of pavement damage, especially crack damage. By extracting multi-dimensional features of cracks, including crack length, width, depth, direction, distribution density, etc., the morphological characteristics and evolution patterns of cracks can be comprehensively described. Such multi-dimensional features can provide a more detailed basis for subsequent maintenance decisions and avoid the one-sidedness of single-dimensional analysis.Furthermore, the comprehensive evaluation of these characteristics based on the fuzzy hierarchical analysis method (AHP) further enhances the scientific rigor and objectivity of the severity of road defects. AHP can effectively handle multidimensional and complex problems. By scoring the severity index of each type of defect through a hierarchical evaluation system, it can realize a priority sorting mechanism for road defects and perform intelligent grading based on the severity and type of defects. This provides more accurate sorting and priority determination for maintenance work and provides a more scientific and quantitative guideline for road maintenance. Finally, by displaying sorted single frames of pavement defect images through a human-computer interaction terminal, maintenance personnel can intuitively access information on the location, type, and urgency index of pavement defects, greatly improving the efficiency and decision-making quality of maintenance work. This step not only helps maintenance personnel quickly and accurately understand the specific situation of defects in each frame of the image, but also helps them make the most reasonable maintenance decision allocation by displaying the type and urgency of defects in real time. By displaying the location and type of defects, maintenance personnel can quickly determine which defect areas need to be dealt with immediately and which can be arranged in subsequent work, thereby significantly improving the response speed and work of pavement maintenance, and making maintenance decisions more scientific, fast, and efficient. Attached Figure Description

[0055] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a schematic diagram showing the module connection between the pre-collection and identification module and the intelligent road surface defect sorting platform of the present invention;

[0057] Figure 2 for Figure 1 A functional flowchart of the multispectral acquisition module. Detailed Implementation

[0058] The technical platform of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0059] To achieve the above objectives, please refer to Figures 1 to 2 This invention provides an AI-based intelligent sorting platform for road surface defects. In the embodiments of this invention, please refer to... Figure 1The diagram shown illustrates the module connection between the pre-emergence acquisition and identification module and the intelligent road surface defect sorting platform of this invention. In this example, it includes: a pre-emergence acquisition and identification module and an intelligent road surface defect sorting platform. The pre-emergence acquisition and identification module and the intelligent road surface defect sorting platform are connected to the same network. The pre-emergence acquisition and identification module includes a multispectral acquisition module and an edge computing module. The intelligent road surface defect sorting platform includes a defect sorting decision module and a defect interactive display module.

[0060] The multispectral acquisition module is used to acquire road surface distress image frames in different spectral bands in real time at a speed of 60 km / h using a vehicle-mounted array camera that integrates corresponding visible light / infrared sensors; and to perform frame extraction preprocessing on the road surface distress image frames in different spectral bands to generate standard frames of road surface distress in different spectra.

[0061] In this embodiment of the invention, a vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and acquire road surface defect image frames in different spectral bands in real time at a speed of 60 km / h. A Leopard Imaging LI-V5000 array camera equipped with 12 lenses is selected, of which 6 lenses are equipped with Sony IMX415 visible light sensors (spectral response range 380-780nm) to capture visible defects such as road surface cracks and potholes; the other 6 lenses integrate FLIRBoson 640 infrared sensors (operating band 7.5-13.5μm) to detect abnormal internal road surface temperatures to identify hidden defects. The camera connects to a Xilinx Zynq UltraScale+ system via a gigabit Ethernet interface. Connected to the vehicle-mounted data acquisition system of the MPSoC chip, it synchronously acquires visible and infrared images at a rate of 120 frames per second. The raw data is stored in RAW format on a 4TB vehicle-mounted solid-state drive. When performing frame extraction preprocessing on road surface distress image frames under different spectral bands, a script was written using Python's OpenCV library, setting the frame extraction frequency to 5 seconds per frame, i.e., extracting one frame every 600 frames (120 frames / second × 5 seconds). The extracted images were then subjected to noise reduction, contrast enhancement, and other operations sequentially. A median filtering algorithm (cv2.medianBlur() function, kernel size [missing information]) was used. 3×3) Remove image noise; enhance image contrast through histogram equalization (cv2.equalizeHist() function), adjust the processed image to a fixed resolution of 1920×1080 pixels, convert it to JPEG format and store it. The file name includes information such as spectral type and acquisition timestamp (e.g., "VIS_20241001_103000.jpg" represents an image acquired at 10:30 on October 1, 2024 under the visible light spectrum). Finally, standard frames of road surface defects under different spectra are generated and stored in the "Standard Frames" folder of the vehicle storage system.

[0062] The edge computing module is used to perform single-frame delay control disease area identification on standard frames of road surface defects under different spectra by deploying the corresponding lightweight YOLOv7 improved model, so as to obtain the corresponding disease location area within a single frame of the road surface image under each spectrum.

[0063] In this embodiment of the invention, a corresponding lightweight YOLOv7 improved model is deployed to perform single-frame delay control disease area identification on standard frames of road surface defects under different spectra, using an NVIDIA Jetson AGX onboard edge computing device. On Orin, a lightweight YOLOv7 model, pruned and quantized, was deployed using the PyTorch framework. This reduced the number of model parameters by 40% and improved inference speed by 30%. The trained model was converted to TorchScript format using the `torch.jit.trace()` function to optimize its performance on edge devices. Standard frames of road surface defects under different spectra were input into the model in batches, with each batch containing 16 images. The model outputs the category (cracks, potholes, etc.), confidence score, and location coordinates (top left and bottom right corner coordinates) for each detected defect area. A confidence score threshold of 0.5 was set, and detection results with confidence scores below this threshold were filtered out. For example, if the model detected three defect areas on a certain visible light standard frame, with two of them having confidence scores above 0.5, the coordinate information of these two areas (e.g., [500, 300, 800, 600]) was filtered out. The coordinates of the top left corner (500, 300) and the bottom right corner (800, 600) are recorded in the "Disease Location Table" of the SQLite database. At the same time, the spectral type and image frame number are marked, and finally the corresponding disease location area in a single frame of the road surface image under each spectrum is obtained.

[0064] The defect sorting decision module is used to perform 12-dimensional feature analysis of crack defects in the defect location area within a single frame of a pavement image under each spectrum, so as to extract the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image; to perform maintenance emergency assessment on the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image based on fuzzy hierarchical analysis, so as to obtain the maintenance urgency index corresponding to each single frame of the pavement image; and to perform maintenance emergency sorting planning on each single frame of the pavement image based on the maintenance urgency index corresponding to each single frame of the pavement image, so as to generate the pavement maintenance vehicle fleet sorting and scheduling sequence corresponding to the single frame of the pavement image sorting.

[0065] In this embodiment of the invention, when performing 12-dimensional feature analysis of cracks in the corresponding disease location regions within a single frame of a road surface image under various spectra, the coordinates of the disease location regions are obtained from the SQLite database. Features are extracted using Python's OpenCV and NumPy libraries. For the maximum crack width, morphological opening operations (cv2.morphologyEx() function, with a 3×3 rectangle as the structuring element) are first performed on the disease region image to remove noise. Then, the contour is found using the cv2.findContours() function, and the bounding rectangle is calculated using the cv2.boundingRect() function. The actual width is obtained by multiplying the shorter side length by the image scale (pre-stored in the device parameter configuration file). When calculating the total crack length, the skeletonize() function of the Scikit-Image library is used to convert the binarized image of the disease region into a single-pixel skeleton. The number of skeleton pixels is counted using the np.count_nonzero() function, and then multiplied by the scale to obtain the length. Crack tortuosity is calculated by comparing the straight-line distance between the start and end points of the crack profile (using the `np.linalg.norm()` function) with the total length. A similar method is used to calculate the remaining 12 features, including the area, perimeter, and shape complexity of the affected area. These results form a 12-dimensional feature set stored in a "feature set table." An emergency assessment of maintenance is performed using fuzzy hierarchical analysis. A 12×12 fuzzy judgment matrix is ​​constructed using Python's numpy and pandas libraries. The importance of each feature factor is determined using a 1-9 scaling method (e.g., the maximum crack width is slightly more important than the total crack length, assigned a value of 3). Weights are calculated using the square root method: first, the product of each row of the matrix is ​​calculated, then the 12th root is taken and normalized. The 12-dimensional feature values ​​are then min-max standardized, multiplied by the weights, and summed to obtain the maintenance urgency index, stored in an "urgency table." Finally, the image frames are sorted in descending order based on the urgency index. The maintenance convoy route is planned using a GIS map and an ant colony optimization algorithm, generating a road maintenance convoy scheduling sequence and storing it in a "scheduling table."

[0066] The defect interactive display module is used to display each road surface image sorting frame on the corresponding human-computer interaction terminal to intuitively view the defect location, type and maintenance urgency index corresponding to each road surface image sorting frame, and to issue corresponding maintenance treatment instructions on the human-computer interaction terminal according to the road maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting frame, so as to execute the corresponding road surface defect maintenance fleet scheduling and planning work.

[0067] In this embodiment of the invention, road surface image sorting frames are displayed on a corresponding human-computer interaction terminal. A 15.6-inch industrial-grade touchscreen is selected as the human-computer interaction terminal, and a visual interface is developed based on the Qt framework. Data is read from the "Disease Location Table," "Feature Set Table," "Urgency Table," and "Schedule Table" of the SQLite database. The road surface image sorting frames are displayed on the interface in the form of a map overlay. For each image frame, the defect location area is marked with a different colored box (red box represents cracks, blue box represents potholes). At the same time, information such as defect type and maintenance urgency index is displayed. According to the road maintenance vehicle fleet sorting and scheduling sequence, A visual dispatch route map is generated on the human-machine interaction terminal, matching the coordinates of path nodes with a GIS map and connecting the locations of each road defect with lines to display the driving route. Operators can issue maintenance and treatment instructions through the terminal. The instructions are encapsulated in JSON format (including information such as fleet number, target point coordinates, and estimated departure time) and sent to the vehicle terminal of the maintenance fleet via a 4G / 5G communication module. After receiving the instructions, the vehicle terminal parses the data and imports it into the vehicle navigation system, guiding the fleet to perform road defect maintenance work according to the planned route. At the same time, the real-time location information of the fleet is transmitted back to the human-machine interaction terminal, realizing full-process monitoring and dispatch.

[0068] Furthermore, as an embodiment of the present invention, reference is made to... Figure 2 As shown, Figure 1 A functional flowchart of the multispectral acquisition module is shown in this embodiment. The multispectral acquisition module includes the following functions:

[0069] S11: A vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and collect road surface distress image frames in different spectral bands in real time at a speed of 60km / h. The road surface distress image frames in the visible light sensor band are used to capture conventional visible road surface distress, while the road surface distress image frames in the infrared sensor band are used to determine the tiny cracks or damages hidden in the road surface based on the corresponding temperature distribution.

[0070] In this embodiment of the invention, a vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and acquire road surface distress image frames in different spectral bands in real time at a speed of 60 km / h. The vehicle-mounted array camera is a Leopard Imaging LI-V5000 series equipped with 12 independent lenses, each lens corresponding to a different focal length and viewing angle, which can cover a wide road surface area. The visible light sensor is a Sony IMX415 with a spectral response range of 380-780nm, which can clearly capture common visible road surface distresses such as potholes, cracks, and bulges. The infrared sensor is a FLIR Boson640 with a working wavelength of 7.5-13.5μm, which identifies micro-cracks, voids, and other damages hidden inside the road surface structure by detecting differences in road surface temperature distribution. The camera and sensors are connected through a dedicated vehicle-mounted data acquisition system based on Xilinx Zynq UltraScale+. Built with an MPSoC chip, it has high-speed data processing and storage capabilities. When the vehicle is traveling at 60km / h, the system synchronously acquires corresponding image frames at a rate of 120 frames per second and stores the acquired data in real time in RAW format in the vehicle's solid-state drive. Each hard drive has a capacity of 4TB and can continuously store more than 6 hours of image data.

[0071] S12: Perform time-slot frame extraction on pavement distress image frames in different spectral bands at a frame extraction frequency of 5 seconds / frame to obtain local pavement distress frame sets in different spectral bands.

[0072] In this embodiment of the invention, time-slotted frame extraction is performed on road surface defect image frames under different spectral bands at a frame extraction frequency of 5 seconds / frame. The frame extraction program is written using the OpenCV library in Python. By reading RAW format image files, the images are processed in chronological order. First, the acquired image data is sorted according to the acquisition timestamp to construct a time series index. The frame extraction interval is set to 5 seconds, that is, one frame is extracted every 5 × 120 = 600 frames. For example, starting from the first frame, the 601st frame, the 1201st frame, and so on are extracted sequentially. For each spectral band, an independent local frame set storage path is established. The extracted image frames are saved in PNG format to the corresponding path. Each local frame set contains 1000 images and is stored in folders named "Visible Light_Local Frame Set_01" and "Infrared Light_Local Frame Set_01". During the saving process, metadata such as the original acquisition time and acquisition location (latitude and longitude information obtained through the vehicle GPS module) of each image are recorded synchronously and stored in a JSON format file for easy subsequent traceability and analysis.

[0073] S13: Perform pixel blur averaging quantization on local frame sets of road surface defects under different spectral bands to obtain the average pixel blur averaging of image frames under different spectral bands.

[0074] In this embodiment of the invention, pixel-based blur averaging quantization is performed on local frames of road surface defects under different spectral bands to employ a gradient-based blur assessment method. The relevant calculations are implemented using Python's NumPy and Sci-Py libraries. For each frame in the local frame set, it is first converted to a grayscale image using OpenCV's cvtColor function. Then, the Sobel operator is used to calculate the gradients in the horizontal and vertical directions, obtaining a gradient magnitude image. The variance of the gradient magnitude image is calculated as the blur index of that frame; a smaller variance indicates a more blurred image. This process is repeated for each local frame. For all images in the set, calculate the blur of each frame. Sum the blurs of all images and divide by the total number of images to obtain the average pixel blur of the image frames in the local frame set. For example, a visible light local frame set contains 1000 images. The calculated blurs of each frame are [0.12, 0.15, 0.10, ...]. Sum these values ​​and divide by 1000 to obtain the average pixel blur of the local frame set, which is 0.13. Perform this operation on all local frame sets under each spectral band and store the calculation results in a CSV file containing fields such as band type, local frame set number, and average blur.

[0075] S14: Based on the average pixel blur of the image frame under different spectral bands, perform blur denoising processing on the road surface distress image frame under the corresponding band to obtain the blurred and denoised road surface distress frame under different spectral bands.

[0076] In this embodiment of the invention, the image frames of road surface defects in the corresponding spectral bands are blurred and denoised based on the average pixel blurring of the image frames in different spectral bands. The non-local means denoising algorithm (NL-Means) is adopted and implemented by the fastNlMeansDenoising function in the OpenCV library of Python. The denoising intensity parameter is set according to the average pixel blurring of each local frame set. When the average blurring is greater than 0.1, the denoising intensity parameter is set to 10; when the average blurring is less than or equal to 0.1, it is set to 5. Taking a certain infrared light local frame set as an example, its average blurring is 0.08. The fastNlMeansDenoising function is executed on each frame in this local frame set. The denoised image is saved in the folder "infrared light_blurred denoising frame_01" with a new file name. During the denoising process, the original resolution and color mode of the image are kept unchanged to ensure that the denoised image can effectively remove blurring noise and retain the key feature information of road surface defects, such as the edge details of cracks and the shape of temperature anomaly areas.

[0077] S15: Perform pixel-level normalization on the blurred and denoised pavement distress frames under different spectral bands to generate standard pavement distress frames under different spectra.

[0078] In this embodiment of the invention, pixel-level normalization is performed on the denoised pavement defect frames under different spectral bands. A combination of histogram equalization and normalization is used, employing Python's OpenCV and NumPy libraries. First, histogram equalization is performed on each frame, and the `equalizeHist` function enhances image contrast, making defect features more prominent. Then, the pixel values ​​are normalized to the [0, 1] interval. The specific calculation formula is: `I_norm = I−I_min / I_max−I_min`, where I is the original image pixel value, and I_min and I_max are the minimum and maximum pixel values, respectively. The normalized pixel values ​​are converted into 8-bit unsigned integers and saved in JPEG format to folders such as "Visible Light_Standard Frame_01" and "Infrared Light_Standard Frame_01". This generates standard frames for road surface defects under different spectra. At the same time, a corresponding thumbnail is generated for each standard frame. The thumbnail size is 1 / 10 of the original image and is used for quick preview and indexing. The thumbnails are stored in PNG format in a subfolder with the same name as the standard frame, "_thumb".

[0079] Furthermore, the edge computing module includes the following functions:

[0080] By inputting standard frames of road surface defects under different spectra to the corresponding edge computing nodes in the local vehicle, the image processing latency of a single frame of road surface defect image under each spectrum can be controlled.

[0081] In this embodiment of the invention, the image processing latency for a single frame of road surface defects under different spectra is controlled by inputting standard frames of road surface defects under different spectra to the corresponding edge computing node in the local vehicle. Specifically, the image processing latency is controlled to be within ≤50ms. The local vehicle-mounted edge computing node uses an NVIDIA Jetson AGX Orin device, equipped with a 6-core NVIDIA Carmel ARM 64-bit CPU and 2048 NVIDIA Ampere architecture CUDA cores, providing powerful computing capabilities. Standard frames of road surface defects (visible light and infrared light) under different spectra, stored in the vehicle's solid-state drive, are transmitted to the edge computing node at a rate of 120 frames per second via the vehicle's Ethernet interface. On the edge computing node, the image processing task is set to the highest priority using the Linux system's real-time scheduler (RT-PREEMPT patch). Simultaneously, an asynchronous processing mechanism is adopted, utilizing Python's asyncio library to achieve non-blocking reading and processing of image data. For each image frame, a start timestamp is recorded before data reading and an end timestamp is recorded after processing. The processing latency is monitored in real time by calculating the time difference. If the processing latency of a certain image frame exceeds 50ms, the frame will be automatically skipped and the number of skipped frames will be recorded for subsequent analysis and optimization, ensuring that the overall image processing latency is strictly controlled within the specified range.

[0082] Preferably, by performing statistical analysis of road surface region features on the standard frames of road surface defects under different spectra on the corresponding edge computing nodes in the local vehicle, the edge and texture features corresponding to the road surface image regions under each spectrum are statistically analyzed and a feature matrix is ​​formed, thereby generating the image region feature matrix corresponding to a single frame of road surface defect image under each spectrum.

[0083] In this embodiment of the invention, road surface region feature statistical analysis is performed on the standard frames of road surface defects under different spectra on the corresponding edge computing node in the local vehicle to statistically determine the edge and texture features corresponding to the road surface image regions under each spectrum and form a feature matrix. The OpenCV library and Scikit-Image library are then used to process the image. First, the Canny algorithm of OpenCV is used to perform edge detection on the image. By setting a low threshold of 50 and a high threshold of 150, edge information in the road surface image is extracted to obtain the edge image. Then, the gray co-occurrence matrix (GLCM) of the image is calculated using the graycomatrix and graycoprops functions of the Scikit-Image library, and texture features such as contrast, correlation, energy, and entropy are extracted from it. The edge image and texture feature data are normalized to unify their numerical range to [0, 1]. Finally, the edge information and texture features are concatenated column by column to form a two-dimensional feature matrix, where each row of the matrix represents the feature vector of a region in the image. For example, for a certain frame of visible light road surface damage image, after processing, a feature matrix with shape [1000, 10] is obtained, where 1000 represents the number of image regions divided and 10 represents the feature dimension of each region. This matrix is ​​used as the image region feature matrix of the image frame and stored in the memory of the edge computing node.

[0084] Preferably, by deploying the corresponding lightweight YOLOv7 improved model on the corresponding edge computing node and inputting the image region feature matrix corresponding to the single frame of the road surface distress image under each spectrum, the corresponding single frame of the road surface distress image is used to identify the distress candidate region, so as to divide each single frame of the road surface distress image into various candidate regions, and identify and mark the candidate regions corresponding to the road surface distress, while marking the position and size of the corresponding candidate regions, so as to generate a set of distress candidate regions corresponding to the single frame of the road surface distress image under each spectrum;

[0085] In this embodiment of the invention, a lightweight YOLOv7 improved model is deployed on the corresponding edge computing node, and the image region feature matrix corresponding to a single frame of road surface defects under each spectrum is input to identify the defect candidate region of the corresponding single frame of road surface defects. This process segments each single frame of road surface defects into candidate regions, identifies and marks the candidate regions corresponding to road surface defects, and marks the position and size of the corresponding candidate regions to generate a set of defect candidate regions corresponding to a single frame of road surface defects under each spectrum. The lightweight YOLOv7 improved model is built on the PyTorch framework, and the model parameters are reduced by 40% through pruning and quantization techniques, making it more suitable for running on edge computing nodes. On the edge computing node, the torch.jit.trace function is used to convert the trained model into TorchScript format to improve the model's inference speed, and the image region feature matrix corresponding to a single frame of road surface defects under each spectrum is used as input to the model for inference. The model outputs the predicted category probability, location coordinates (top left and bottom right corner coordinates), and size information for each candidate region. A confidence threshold of 0.5 is set, and candidate regions with a predicted probability greater than 0.5 are identified as areas with pavement defects. Their location coordinates, size, and category information are stored in a list. For example, for a frame of infrared pavement defect image, the model outputs 5 candidate regions. After filtering, 3 are determined as defect candidate regions. The relevant information of these 3 regions is compiled into a list, which serves as the defect candidate region set for that frame of image. This list is stored in the database of the edge computing node using SQLite for easy local storage and retrieval.

[0086] Preferably, the confidence level of the location of the road surface defects is calculated for each candidate region in the defect candidate region set corresponding to a single frame of the road surface defect image under each spectrum, so as to obtain the distribution of the confidence level of the location of the defect corresponding to each candidate region in the defect candidate region set under each spectrum.

[0087] In this embodiment of the invention, the confidence score of the disease location is calculated for each candidate region in the disease candidate region set corresponding to a single frame of the road surface disease image under each spectrum. A pixel-level classification method based on deep learning is employed, using a pre-trained U-Net model on the edge computing nodes. This model is built on the TensorFlow framework. The image of each candidate region is cropped from the original road surface disease image and resized to 256×256 pixels, serving as input to the U-Net model. The model outputs the probability that each pixel belongs to a disease category. These probability values ​​are grouped into a two-dimensional array, which represents the confidence score distribution of the disease location for that candidate region. For example, for a candidate region, the confidence score distribution array has a shape of [256, 256]. Each element in the array represents the probability that a disease exists at the corresponding pixel location. This process is performed on all candidate regions in the disease candidate region set to obtain the confidence score distribution of the disease location for each candidate region. This distribution is then associated with other information about the candidate region (location, size, etc.) and stored in an SQLite database to provide data support for subsequent confidence optimization and screening.

[0088] Preferably, based on the confidence distribution of the disease location corresponding to each candidate region in the disease candidate region set under each spectrum, confidence optimization screening is performed on each candidate region in the disease candidate region set corresponding to a single frame of the road surface disease image under each spectrum, so as to obtain the disease location region corresponding to a single frame of the road surface image under each spectrum.

[0089] In this embodiment of the invention, confidence optimization filtering is performed on each candidate region in the candidate region set corresponding to a single frame of road surface damage image under each spectrum based on the confidence distribution of the damage location corresponding to each candidate region in the candidate region set under each spectrum. The data is processed using Python's NumPy and Pandas libraries. The candidate region set and the damage location confidence distribution data for each candidate region in a single frame of road surface damage image under each spectrum are read from an SQLite database. For each frame of image under each spectrum, each candidate region in its candidate region set is traversed, and a combination of non-maximum suppression (NMS) algorithm and threshold filtering is used. First, a global confidence threshold of 0.3 is set. The confidence of pixels with a probability value less than 0.3 in the confidence distribution of road defects is set to 0. Then, for each candidate region, the mean and standard deviation of its confidence distribution are calculated. If the mean confidence of a candidate region is less than 1.5 times the standard deviation, the confidence of the defect in that region is considered unstable and it is removed from the candidate region set. After filtering, the remaining candidate regions are the defect location regions corresponding to a single frame of the road surface image under this spectrum. The information of these regions (location, size, confidence distribution, etc.) is stored in JSON format on the vehicle's solid-state drive to provide accurate location information for subsequent intelligent sorting of road defects.

[0090] Furthermore, the image processing latency is specifically controlled to be within ≤50ms.

[0091] Furthermore, the confidence optimization screening of each candidate region in the candidate region set corresponding to a single frame of the pavement defect image under each spectrum, based on the confidence distribution of the defect location corresponding to each candidate region in the defect candidate region set under each spectrum, includes:

[0092] The confidence gradient of the candidate area of ​​the road surface disease image for each single frame under each spectrum is calculated based on the confidence distribution of the disease location corresponding to each candidate area in the disease candidate area set under each spectrum.

[0093] In this embodiment of the invention, the confidence gradient of the candidate road surface defects in a single frame of an image under each spectrum is calculated based on the confidence distribution of the defect location corresponding to each candidate region within the defect candidate region set under each spectrum. The data is then processed using Python's NumPy library. The confidence distribution data of the defect location for each candidate region is read from a JSON file storing defect candidate region information. Assuming that the defect candidate region set for a single frame of an image under a certain spectrum contains 10 candidate regions, and the confidence distribution of the defect location for each candidate region is an array of length 100, representing the probability of defect existence at different locations within that region, the confidence gradient is calculated for each candidate region using the finite difference method. Taking the confidence distribution array within the candidate region as an example, the gradient at each location is calculated using the formula gradient = np.gradient(confidence_array). The algorithm generates a gradient array of the same length as `confidence_array`, reflecting the change in confidence within a region. A larger gradient value indicates a more drastic change in confidence. The algorithm calculates the average confidence gradient of all candidate regions in a single frame of the image. It sums the elements of the gradient array corresponding to each candidate region and divides the sum by the number of candidate regions to obtain the confidence gradient of the disease candidate regions in that single frame of the image under that spectrum. For example, if the gradient values ​​of 10 candidate regions at the 10th position are [0.1, 0.2, 0.15, ...], summing these values ​​and dividing by 10 gives the average gradient value at that position. This process is repeated for all positions, resulting in an average confidence gradient array of length 100, representing the confidence gradient of the disease candidate regions in that single frame of the image under that spectrum. This array is then stored in a CSV file containing fields such as spectrum type, image frame number, and confidence gradient values ​​at each position.

[0094] Preferably, based on the confidence gradient of the candidate regions corresponding to the road surface distress images in a single frame under each spectrum, the confidence distribution of the distress location corresponding to each candidate region in the set of candidate regions under each spectrum is optimized and filtered. If the confidence distribution of the corresponding distress location is greater than or equal to the confidence gradient of the candidate regions, the corresponding candidate region in the set of candidate regions is selected as the distress location region corresponding to the single frame of the road surface distress image under that spectrum. Candidate regions with confidence distributions less than the confidence gradient of the candidate regions are then filtered out to obtain the distress location regions corresponding to the single frame of the road surface distress images under each spectrum.

[0095] In this embodiment of the invention, the confidence distribution of the disease location corresponding to each candidate region in the disease candidate region set under each spectrum is optimized and filtered based on the confidence gradient of the candidate region corresponding to a single frame of the road surface disease image under each spectrum. This is achieved by using the pandas library in Python to read the confidence gradient data from a CSV file and the confidence distribution data of the candidate regions in a JSON file. For each frame of an image under each spectrum, each candidate region in its disease candidate region set is traversed. Taking a frame of an image under a certain visible light spectrum as an example, assuming that the frame has 8 candidate regions, for one candidate region, its disease location confidence distribution array is compared element-wise with the corresponding frame's candidate region confidence gradient array. If the element value at a certain position in the confidence distribution array is greater than or equal to the element value at the corresponding position in the gradient array, then the disease confidence at that position is considered high. The process involves several steps: first, retaining the information of the candidate region at that location; second, removing the information from the candidate region if the number of valid locations remaining is greater than 0; and third, removing the candidate region from the candidate region if the number of valid locations remaining is less than 0. This process is repeated for all candidate regions of the same image frame to obtain the disease location region of that image frame under that spectrum. The same filtering process is repeated for all image frames under all spectra to obtain the corresponding disease location regions within each single frame of the road surface image under each spectrum. The filtering results are then stored in a new JSON file format, containing detailed information such as spectrum type, image frame number, disease location region coordinates, and the distribution of retained confidence levels. This provides accurate data support for the subsequent accurate identification and sorting of road surface defects.

[0096] Furthermore, the disease sorting decision module includes the following functions:

[0097] The image scale corresponding to a single frame of the road surface image under each spectrum is obtained by identifying the location of the damage within a single frame of the road surface image under each spectrum.

[0098] In this embodiment of the invention, the image scale corresponding to a single frame of a road surface image under each spectrum is obtained by identifying the corresponding defect location region within that frame. In the vehicle-mounted detection system, the image acquisition equipment has undergone precise calibration during installation, and the image scale information is pre-stored in a device parameter configuration file. The file is in JSON format and contains the correspondence between the spectral type (visible light, infrared light, etc.), image frame number, and scale. The configuration file is read using Python's json library. For example, when processing a road surface image under a certain infrared spectrum, the statement `with open('image_scale_config.json', 'r') as f: config =` is used. The `json.load(f)` function reads the file content and then extracts the corresponding scale from the configuration data based on the image frame number. Assuming the image frame number is IR_005, its scale obtained from the configuration file is that 1 pixel represents an actual length of 0.05 meters. The obtained image scale information, along with other metadata of the image frame (such as acquisition time and acquisition location coordinates), is stored in the "Image Metadata Table" of the SQLite database. The table structure includes fields such as image frame number, spectral type, image scale, acquisition time, and acquisition location latitude and longitude, so that it can be called later for feature analysis.

[0099] Preferably, based on the corresponding image scale, a 12-dimensional feature analysis of cracks is performed on the corresponding disease location area in a single frame of the pavement image under each spectrum to extract a 12-dimensional feature set of cracks for each single frame of the pavement image. This set includes the maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, disease area, crack distribution density, perimeter of the disease area, shape complexity of the disease area, mean gray value of the disease area, and standard deviation of gray value of the disease area.

[0100] In this embodiment of the invention, 12-dimensional feature analysis of crack defects is performed on the corresponding defect location regions within a single frame of a pavement image under each spectrum based on the corresponding image scale. This extracts the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image. The image scale is read from the "Image Metadata Table" of the SQLite database, and the coordinate information of the defect location regions is obtained from the "Defect Features" table. The data is then processed using Python's OpenCV and NumPy libraries. For the maximum crack width, morphological opening operations are first performed on the defect region image to remove noise. The cv2.morphologyEx() function is used, employing 3×3 rectangular structural elements, and then cv2.findCo... The `ntours()` function finds the outline of the crack region, and then the `cv2.boundingRect()` function calculates the bounding rectangle. Multiplying the shorter side of the bounding rectangle by the image scale yields the actual maximum crack width. When calculating the total crack length, the `skeletonize()` function from the Scikit-Image library is used to skeletonize the binarized image of the affected area, converting it into single-pixel-wide lines. The `np.count_nonzero()` function counts the number of pixels on the lines, and this count is multiplied by the image scale to obtain the actual total crack length. For crack tortuosity, the coordinates of the starting and ending pixels of the crack outline are first obtained, and then the Euclidean distance formula (np.lina) is used. The `lg.norm()` function calculates the straight-line distance between two points, and combines this with the total crack length to calculate the ratio. When calculating the area of ​​the diseased region, the `np.count_nonzero()` function counts the number of pixels with a value of 255 in the binarized image of the diseased region, multiplying this number by the square of the image scale to obtain the actual area. The crack distribution density is calculated by dividing the total crack length by the area of ​​the diseased region. Boundary detection is performed using the `cv2.Canny()` function, and the `cv2.findContours()` function obtains the contour. The `len()` function counts the number of contour pixels and multiplies this number by the image scale to obtain the perimeter of the diseased region. Finally, the formula "shape complexity = perimeter of diseased region² / area of ​​diseased region" is applied. "To calculate the shape complexity, the image of the diseased area is converted to a grayscale image using the cv2.cvtColor() function. The mean and standard deviation of the grayscale are calculated using the np.mean() and np.std() functions, respectively. The calculated 12-dimensional feature data (maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of ​​the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, mean grayscale of the diseased area, and standard deviation of the grayscale of the diseased area) are compiled into a list and stored in the "12-dimensional feature set table" in the SQLite database. The table structure includes the image frame number, spectral type, and 12 feature fields."

[0101] Preferably, the maintenance urgency index of each pavement image frame is obtained by performing a maintenance emergency assessment on the 12-dimensional feature set of cracks and defects corresponding to each pavement image frame based on the fuzzy hierarchical analysis method.

[0102] In this embodiment of the invention, an emergency maintenance assessment of the 12-dimensional feature set of cracks corresponding to a single frame of each pavement image is performed based on fuzzy hierarchical analysis. A fuzzy judgment matrix is ​​constructed using Python's NumPy and Pandas libraries. First, the importance comparison relationship of each feature factor in the 12-dimensional feature set is determined using a 1-9 scale, where 1 indicates that two feature factors are equally important, and 9 indicates that one feature factor is extremely more important than the other. For example, for the maximum crack width and total crack length, if the maximum crack width is considered to have a slightly greater impact on the severity of pavement damage than the total crack length, it can be assigned a value of 3. Through pairwise comparisons, a 12×12 fuzzy judgment matrix is ​​constructed. OK Column elements Indicates the first The first characteristic factor affects the second The relative importance of each characteristic factor, and satisfying =1 / , =1, the weight of each characteristic factor is calculated using the root method. First, calculate... And then Find the 12th root to get Finally, normalization is performed, and the calculation is performed. The fuzzy weight values ​​of each feature factor are obtained. The actual 12-dimensional feature values ​​of each road surface image frame are read from the "12-dimensional feature set table" in the SQLite database and then subjected to min-max normalization. The formula is as follows: ,in These are the original values. and These are the minimum and maximum values ​​of the feature factor across all image frames. The standardized feature value array is multiplied by the corresponding elements of the fuzzy weight value array and then summed. The maintenance urgency index corresponding to each road surface image frame is obtained and stored in the "Maintenance Urgency Table" of the SQLite database. The table structure includes fields such as image frame number, spectral type, and maintenance urgency index.

[0103] Preferably, maintenance urgency sorting planning is performed on each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame, so as to generate a road maintenance vehicle sorting and scheduling sequence corresponding to the road surface image sorting frame.

[0104] In this embodiment of the invention, maintenance urgency sorting is planned for each pavement image frame based on its corresponding maintenance urgency index. The maintenance urgency index, spectral type, and image frame number information for each pavement image frame are read from the "Maintenance Urgency Table" in the SQLite database. The data is read into a data frame using Python's pandas library, and sorted in descending order by the maintenance urgency index column using the `sort_values()` function (e.g., `df.sort_values(by='Maintenance Urgency Index, ascending=False, inplace=True)`). This ensures that images with higher urgency indices are listed first. After sorting, the data from the image frame number column is extracted to form a list. The coordinate information of the disease location area corresponding to each image frame is retrieved from the "Disease Features" table. The disease location area is divided into corresponding grids using Python's numpy library, and the coordinates of the center point of each grid are calculated as the disease location point. The sequence of disease location points for each image frame is stored in the "Disease Location Point Table," and the Shapefile format is read using Python's geopandas library. GIS map data, including a road network layer and a traffic condition layer, is used. The sequence of disease location points for each image frame is read from the "Disease Location Point Table." An ant colony optimization algorithm is implemented using the Python-based ant-colony-optimization library. The algorithm sets the number of ants to 50, the pheromone evaporation coefficient to 0.1, the heuristic factor to 2, the expected heuristic factor to 3, and the maximum number of iterations to 100. Disease location points are used as the starting and passing nodes for the ants, and road connections in the road network topology are used as feasible paths. The algorithm is then used to determine the feasible paths based on real-time traffic conditions in the traffic condition layer. By analyzing traffic flow data, the path length of congested road sections is multiplied by a congestion coefficient (e.g., 2 for severe congestion). Through iterative calculation, the optimal driving path corresponding to each point sequence is obtained. The coordinates of nodes in the path, the names of the roads traversed, and other information are compiled into a list, which is the road maintenance fleet sorting and scheduling sequence corresponding to a single frame of the road image. This sequence is stored in the "Maintenance Fleet Scheduling Table" in the SQLite database. The table structure includes fields such as image frame number, scheduling sequence information (path node data, etc., stored in JSON format), and estimated travel time, providing an accurate planning scheme for the actual scheduling of maintenance fleets.

[0105] Furthermore, the 12-dimensional feature analysis of crack defects within a single frame of the road surface image under each spectrum, based on the corresponding image scale, includes:

[0106] Binarization processing is performed on the corresponding disease location regions within a single frame of the road surface image under each spectrum to generate binarized images of the corresponding disease regions under each spectrum.

[0107] In this embodiment of the invention, the corresponding defect location regions within a single frame of the road surface image under each spectrum are binarized. This is then performed using Python's OpenCV library to read the coordinate information of the defect location regions within a single frame of the road surface image under each spectrum from the SQLite database. The corresponding regions are then cropped from the original road surface defect image. Assuming the coordinates of the defect location region under a certain visible light spectrum frame are (100, 100) at the top left and (300, 300) at the bottom right, the cv2.rectangle() function is used to locate this region on the original image, and then the cv2.cvtCo... The `lor()` function converts the cropped image to a grayscale image and performs binarization using a fixed threshold method. The `cv2.threshold()` function sets the threshold to 127, setting pixels with grayscale values ​​greater than 127 to 255 (white, representing the diseased area) and pixels with grayscale values ​​less than or equal to 127 to 0 (black, representing the background area). This generates a binarized image of the diseased area. The processed binarized image is saved as a PNG file in a local folder, with the filename including information such as the spectral type and image frame number, such as "Visible Light_001_Disease Binarization.png", for easy retrieval in subsequent steps.

[0108] Preferably, the maximum crack width, total crack length, crack tortuosity, area of ​​the diseased region, crack distribution density, perimeter of the diseased region, and shape complexity of the diseased region are calculated based on the binarized image of the diseased region under each spectrum and in combination with the corresponding image scale.

[0109] In this embodiment of the invention, the maximum crack width, total crack length, crack tortuosity, area of ​​the affected region, crack distribution density, perimeter of the affected region, and shape complexity are calculated based on the binarized images of the affected region under each spectrum and the corresponding image scale. Using Python's OpenCV and NumPy libraries, for the maximum crack width, morphological opening operations are performed to remove noise. The `cv2.findContours()` function is used to find the crack region contour, and the `cv2.boundingRect()` function is used to calculate the bounding rectangle of the contour. The length of the shorter side of the bounding rectangle is used as an approximation of the maximum crack width, converted using the image scale (assuming 1 pixel represents 0.1 cm in actual length). When calculating the total crack length, the `skeletonize()` function of the Scikit-Image library is used to skeletonize the binarized image, and the `np.count_nonzero()` function is used to count the number of pixels in the skeleton lines. Then, based on the scale... To calculate the crack tortuosity, the following steps are performed: First, the `cv2.findContours()` function is used to obtain the start and end pixels of the crack contour. Then, the Euclidean distance formula (using the `np.linalg.norm()` function) is used to calculate the straight-line distance between the two points. This distance is then combined with the calculated total crack length to calculate the ratio. When counting the area of ​​the affected region, the `np.count_nonzero()` function is used to count the number of pixels in the affected region in the binarized image. The actual area is then converted according to the scale, and the crack distribution density is calculated. The total crack length is divided by the area of ​​the affected region to perform boundary detection using the `cv2.Canny()` function. Next, the `cv2.findContours()` function is used to obtain the contour. The `len()` function is used to count the number of pixels in the contour and convert the perimeter of the affected region. Finally, the shape complexity is calculated using the formula "shape complexity = perimeter of affected region² / area of ​​affected region". All calculation results are recorded in the "Disease Features" table of the SQLite database, corresponding to each spectrum and image frame number.

[0110] Preferably, by taking points at equal intervals on the crack centerline of the corresponding visible spectral band within a single frame of the road surface image under each spectrum, dividing it into several small segments, calculating the vertical width of the crack corresponding to each of the several small segments, and calculating the average value, the corresponding average crack width is obtained.

[0111] In this embodiment of the invention, by dividing the crack centerline of the corresponding visible spectral band within a single frame of the road surface image under each spectrum into several small segments at equal intervals, and calculating the vertical width of the crack corresponding to each small segment, and simultaneously calculating the average value, the corresponding average crack width is obtained. For the binarized image of the disease area in the visible spectral band, using the OpenCV and SciPy libraries of Python, the crack area is first thinned using the cv2.ximgproc.thinning() function to obtain a crack centerline with a width of one pixel. Assuming the length of the crack centerline is 100 pixels, and the equal interval is set to 5 pixels, 20 small segments can be divided. For each For each small segment, the vertical width is calculated by scanning pixels perpendicular to the centerline and counting the number of white pixels (representing cracks). For example, if 10 consecutive white pixels are scanned vertically in a segment, the vertical width of that segment is 10 pixels. After calculating the vertical width of all segments, the np.mean() function is used to calculate the average of these width values ​​to obtain the average crack width. Assuming the vertical widths of the 20 segments are [8, 10, 9, ...], the average value is calculated to be 9.2 pixels. Combined with the image scale (1 pixel represents an actual length of 0.1 cm), the actual average crack width is calculated to be 0.92 cm. The result is recorded in the record of the corresponding image frame in the "Disease Characteristics" table.

[0112] Preferably, the temperature distribution difference between each equally spaced point is determined by the location of the disease in the corresponding infrared spectral band within a single frame of the road surface image under each spectrum, and the crack depth corresponding to each equally spaced point is estimated based on the temperature distribution difference between each equally spaced point. At the same time, the maximum value is determined based on the crack depth corresponding to each equally spaced point and the average value is calculated to obtain the corresponding maximum crack depth and average crack depth.

[0113] In this embodiment of the invention, the temperature distribution differences between equally spaced points are determined by identifying the location of the damage within a single frame of a road surface image in each infrared spectral band. The crack depth at each equally spaced point is then estimated based on these temperature distribution differences. Simultaneously, the maximum and average crack depths are determined based on the crack depths at each equally spaced point to obtain the maximum and average crack depths. Temperature data for the damage location area is read from a file storing infrared spectral image data. Assuming the temperature data is stored in matrix form, with each element corresponding to the temperature value of a pixel, points are taken along the crack path within the crack area using the same equally spaced method as before. For two adjacent equally spaced points, the temperature difference between them is calculated. Using a pre-established temperature-depth relationship model (which is derived from fitting a large amount of experimental data, for example, the temperature difference and crack depth satisfy a linear relationship: depth = coefficient × temperature difference + constant, and the coefficient and constant are determined experimentally), the crack depth corresponding to each equidistant point is estimated. Assuming that the temperature difference between two adjacent points is 5℃, the corresponding crack depth is calculated to be 2 cm based on the corresponding model and through experiments (e.g., depth = 0.3 × temperature difference + 0.5). After calculating the crack depth of all equidistant points in turn, the maximum value is found using the np.max() function, and the average value is calculated using the np.mean() function to obtain the maximum crack depth and the average crack depth. The results are recorded in the "Disease Features" table of the SQLite database to complete the disease feature data of this image frame.

[0114] Preferably, the mean and standard deviation of the gray level of the corresponding disease location area in a single frame of the road surface image under each spectrum are calculated to obtain the mean gray level and standard deviation of gray level of all pixels in the disease location area, and thus obtain the mean gray level and standard deviation of gray level of the corresponding disease area.

[0115] In this embodiment of the invention, the mean and standard deviation of the grayscale values ​​of the corresponding damage locations within a single frame of a road surface image under various spectra are calculated to obtain the average grayscale value and standard deviation of all pixels within the damage location area. Using Python's OpenCV and NumPy libraries, the damage location area image is cropped from the original road surface damage image, converted to a grayscale image using the cv2.cvtColor() function, and the average grayscale value of all pixels in the grayscale image is calculated using the np.mean() function. The mean grayscale value of the affected area is calculated. For example, if a grayscale image of a affected area has 1000 pixels and the total grayscale value is 50000, then the mean grayscale value is 50000 / 1000 = 50. The standard deviation of the grayscale values ​​is calculated using the np.std() function to reflect the dispersion of pixel grayscale values. Assuming the calculated standard deviation is 10, the mean grayscale value and the standard deviation of the grayscale value of the affected area are recorded in the "Disease Characteristics" table of the SQLite database. This provides data support for analyzing the grayscale characteristics of the affected area, and this data can be used to distinguish different types of pavement defects.

[0116] Preferably, the above-mentioned maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of ​​the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, mean gray value of the diseased area, and standard deviation of gray value of the diseased area are combined into a 12-dimensional feature set of cracks and diseases corresponding to a single frame of each road surface image.

[0117] In this embodiment of the invention, the corresponding maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, affected area area, crack distribution density, affected area perimeter, affected area shape complexity, affected area grayscale mean, and affected area grayscale standard deviation are combined into a 12-dimensional feature set of crack defects corresponding to each single frame of the road image. Using Python's NumPy library, the above 12 feature data items for each spectrum of the corresponding image frame are read from the "Disease Features" table of the SQLite database. These data are arranged in a fixed order to form a one-dimensional array of length 12, for example, [maximum crack width, average crack width, maximum crack depth, ..., affected area grayscale standard deviation]. This one-dimensional array is stored as a sample in a local NumPy array file. The filename includes the spectrum type and image frame number, such as "visible light_001_12". The system generates a "12-dimensional feature set.npy" file and creates a "12-dimensional feature set index table" in the database to record information such as the feature set file path, spectral type, and image frame number corresponding to each image frame. This facilitates intelligent sorting, classification, and analysis of pavement defects based on these feature sets, providing comprehensive data support for pavement maintenance decisions.

[0118] Furthermore, the calculation of the corresponding maximum crack width, total crack length, crack tortuosity, affected area area, crack distribution density, affected area perimeter, and affected area shape complexity based on the binarized images of the affected areas under each spectrum and in conjunction with the corresponding image scale includes:

[0119] By highlighting the corresponding crack area in the binarized image of the disease area under each spectrum, and using morphological operations to remove the corresponding noise and fine branches, the bounding rectangle of the crack area is calculated. The length of the short side of the bounding rectangle is approximated as the corresponding maximum width, and the corresponding maximum crack width is calculated according to the corresponding image scale.

[0120] In this embodiment of the invention, the corresponding crack region is highlighted by binarizing the disease area image under each spectrum, and morphological operations are used to remove the corresponding noise and fine branches. Then, the bounding rectangle of the crack region is calculated, and the length of the short side of the bounding rectangle is approximated as the corresponding maximum width. The maximum width of the crack is calculated according to the corresponding image scale. This operation is performed using the OpenCV library of Python. First, the stored binarized images of the disease area under each spectrum are read. In the image, the pixel value of the crack region is 255, and the background region is 0. Morphological opening operation is used to remove noise and fine branches, specifically through the cv2.morphologyEx() function, using a rectangular structuring element of size 3×3. The image is opened to eliminate small areas that are not tightly connected to the main body of the crack. Then, the cv2.findContours() function is used to find the contours in the image and obtain the contour information of the crack area. The cv2.boundingRect() function is then used to calculate the bounding rectangle of the contour, obtaining the coordinates of the upper left corner, width, and height of the bounding rectangle. The length of the shorter side of the bounding rectangle is used as an approximation of the maximum width of the crack. Assuming that the image scale is 1 pixel representing an actual length of 0.1 cm, if the length of the shorter side of the bounding rectangle is 5 pixels, then the converted maximum width of the crack is 5 × 0.1 = 0.5 cm. The result is recorded in the corresponding CSV file, which contains fields such as spectral type, image frame number, and maximum crack width.

[0121] Preferably, the binarized images of the diseased areas under each spectrum are processed by skeletonization to simplify the corresponding crack areas into lines with a width of one pixel, and the total number of pixels on the lines is counted. At the same time, the total length of the corresponding crack is calculated by combining the corresponding image scale.

[0122] In this embodiment of the invention, the corresponding binarized images of the diseased areas under each spectrum are skeletonized to simplify the corresponding crack areas into lines of single-pixel width. The total number of pixels on the lines is counted, and the total length of the crack is calculated by combining the corresponding image scale. The skeletonization function in the Scikit-Image library is used to perform skeletonization on the binarized images of the diseased areas, converting the crack areas into single-pixel-width skeleton images. The np.count_nonzero() function is used to count the total number of pixels with a pixel value of 1 (representing skeleton lines) in the skeleton image to obtain the pixel count value. Assuming that 1 pixel in the image scale represents an actual length of 0.1 cm, if the total number of pixels is counted to be 200, then the converted total length of the crack is 200 × 0.1 = 20 cm. The total crack length information, along with data such as spectral type and image frame number, is stored in the "Disease Features" table of the SQLite database for subsequent querying and analysis.

[0123] Preferably, the corresponding crack start pixel and crack end pixel are obtained by binarizing the disease area under each spectrum, so as to calculate the straight distance between the crack start point and the crack end point, and the corresponding crack tortuosity is obtained by the ratio of the total crack length to the straight distance between the crack start point and the crack end point.

[0124] In this embodiment of the invention, the corresponding crack start-point and crack end-point pixels are obtained from the binarized images of the diseased areas under each spectrum. The straight-line distance between the crack start-point and end-point is calculated based on these pixels. The crack tortuosity is then determined by the ratio of the total crack length to the straight-line distance between the start and end points. This operation is implemented using Python's NumPy and OpenCV libraries. First, the contour of the crack region is obtained using the corresponding `cv2.findContours()` function. For each contour, its start and end points are used as the crack start-point and end-point pixels. The straight-line distance between the start and end pixels is calculated using the Euclidean distance formula and the `np.linalg.norm()` function. Assuming the start-point coordinates are (10, 10) and the end-point coordinates are (50, 50), the straight-line distance is... The pixel is read from the "Disease Characteristics" table in the SQLite database, corresponding to the total crack length of the image frame. Assuming it's 100 pixels, it's calculated using the formula "Crack Tortuosity = Total Crack Length / Straight-Line Distance", i.e., 100 / ( The value is approximately 1.77. The crack tortuosity result is updated to the corresponding record in the "Disease Characteristics" table to provide a quantitative indicator for assessing crack morphology.

[0125] Preferably, the total number of pixels in the binarized image of the diseased area under each spectrum is counted, and the area of ​​the corresponding diseased area is calculated according to the corresponding image scale. At the same time, the corresponding crack distribution density is calculated according to the ratio between the total length of the corresponding crack and the area of ​​the diseased area.

[0126] In this embodiment of the invention, the total number of pixels in the binarized image of the diseased area under each spectrum is counted, and the area of ​​the corresponding diseased area is calculated according to the corresponding image scale. At the same time, the crack distribution density is calculated based on the ratio between the total crack length and the area of ​​the diseased area. The np.count_nonzero() function is used to count the total number of pixels with a pixel value of 255 (representing the diseased area) in the binarized image of the diseased area. Assuming that the image scale is 1 pixel representing an actual area of ​​0.01 square centimeters, if the total number of pixels is counted to be 1000, then the converted area of ​​the diseased area is 1000 × 0.01 = 10 square centimeters. The total crack length corresponding to the image frame is read from the "Disease Features" table of the SQLite database. Assuming it is 20 centimeters, the crack distribution density is calculated using the formula "Crack Distribution Density = Total Crack Length / Diseased Area", i.e., 20 / 10 = 2 centimeters / square centimeter. The crack distribution density result is recorded in the "Disease Features" table to provide data support for analyzing the severity of the disease.

[0127] Preferably, boundary detection is performed on the binarized images of the diseased areas under each spectrum to obtain the boundary of the diseased area after binarization, and the total number of pixels on the boundary of the diseased area is counted. At the same time, the perimeter of the corresponding diseased area is calculated according to the corresponding image scale.

[0128] In this embodiment of the invention, boundary detection is performed on the binarized images of the diseased areas under each spectrum to obtain the boundary of the corresponding diseased area after binarization, and the total number of pixels on the boundary of the diseased area is counted. At the same time, the perimeter of the corresponding diseased area is calculated according to the corresponding image scale. The boundary detection is performed using the cv2.Canny() function of the OpenCV library, with a low threshold of 50 and a high threshold of 150, to obtain the edge image of the diseased area. The cv2.findContours() function is used to find the contour in the edge image to obtain the contour information of the boundary of the diseased area. The len() function is used to count the total number of pixels on the contour to obtain the boundary pixel count value. Assuming that the image scale is 1 pixel representing an actual length of 0.1 cm, if the total number of pixels is counted to be 150, the perimeter of the diseased area after conversion is 150 × 0.1 = 15 cm. The perimeter information of the diseased area is added to the corresponding record in the "Disease Features" table of the SQLite database to improve the disease feature data.

[0129] Preferably, the shape complexity of the diseased area is calculated based on the ratio between the square of the perimeter of the diseased area and the area of ​​the diseased area.

[0130] In this embodiment of the invention, the shape complexity of the affected area is calculated based on the ratio between the square of the perimeter of the affected area and the area of ​​the affected area. The perimeter and area data of the affected area for each spectrum are read from the "Disease Features" table of the SQLite database. Assuming that the perimeter of the affected area in a certain image frame is 15 cm and the area of ​​the affected area is 10 square centimeters, the shape complexity is calculated using the formula "shape complexity = perimeter of affected area² / area of ​​affected area", i.e., 15² / 10 = 22.5. The calculated shape complexity result is recorded in the "Disease Features" table. The larger the shape complexity value, the more complex the shape of the affected area, providing an important quantitative basis for the classification and assessment of pavement defects.

[0131] Furthermore, the maintenance urgency assessment of the 12-dimensional feature set of cracks and defects corresponding to each single frame of the pavement image based on the fuzzy hierarchical analysis method is performed to obtain the maintenance urgency index corresponding to each single frame of the pavement image, including:

[0132] Based on the fuzzy hierarchical analysis method, a fuzzy judgment matrix of cracks and defects corresponding to each single frame of road surface image is constructed.

[0133] In this embodiment of the invention, a fuzzy judgment matrix for crack defects corresponding to each single frame of a pavement image is constructed based on the fuzzy hierarchical analysis method using the 12-dimensional feature set of crack defects for each pavement image. This is achieved using Python's NumPy and Pandas libraries. First, each feature factor within the 12-dimensional feature set is determined, including the maximum crack width and average crack width, etc. A fuzzy scaling matrix using a 1-9 scaling method is established, where 1 indicates that two feature factors are equally important, 9 indicates that one feature factor is more important than the other, and intermediate values ​​represent different degrees of importance. For a given pavement image frame with a 12-dimensional feature set, each feature factor is compared in importance with other feature factors in turn. For example, when comparing the maximum crack width with the total crack length, based on the actual pavement defect situation and expert experience, if the maximum crack width is considered slightly more important than the total crack length, it is assigned a value of 3 according to the scaling method. Through pairwise comparisons, a 12×12 matrix is ​​constructed, where the matrix contains the first feature factor and the second feature factor. OK Column elements Indicates the first The first characteristic factor affects the second The relative importance of each characteristic factor, and satisfying =1 / , =1, assuming all comparisons have been completed, a fuzzy judgment matrix is ​​obtained as follows (partial example): The matrix is ​​stored as a NumPy array and then saved as a CSV file using the pandas library. The filename includes the spectral type and image frame number, such as "Visible Light_001_Fuzzy Judgment Matrix.csv", which is convenient for subsequent calling and analysis.

[0134] Preferably, based on the crack defect fuzzy judgment matrix corresponding to each single frame of each road surface image, weighted fuzzy inference is performed on each feature factor in the corresponding 12-dimensional feature set of crack defects to generate fuzzy weight values ​​corresponding to each feature factor in each single frame of each road surface image.

[0135] In this embodiment of the invention, weighted fuzzy inference is performed on each feature factor in the corresponding 12-dimensional feature set of crack defects based on the fuzzy judgment matrix of crack defects corresponding to each single frame of each road surface image. The weights are calculated using the square root method, and the calculation process is implemented using Python's NumPy library. First, the previously saved fuzzy judgment matrix CSV file is read, converted into a NumPy array, and the product of each row element of the matrix is ​​calculated. For example, for the matrix... Okay, calculate This results in an array of length 12. For arrays Calculate the 12th root of each element in the array, i.e. , get array , array Perform normalization and calculation This yields an array of fuzzy weight values ​​for each feature factor. Suppose that the fuzzy judgment matrix of a single frame of a road surface image is calculated to obtain an array of fuzzy weight values. =[0.15, 0.12, 0.10, ...], which correspond to the weights of 12 characteristic factors such as maximum crack width and average crack width. These weight values, along with information such as spectral type and image frame number, are stored in the "Feature Weight Table" of the SQLite database for easy use in subsequent emergency maintenance assessments.

[0136] Preferably, based on the fuzzy weight values ​​corresponding to each feature factor within a single frame of each pavement image, an emergency maintenance assessment is performed between the actual values ​​of each feature factor within the corresponding 12-dimensional feature set of crack defects, to obtain the maintenance urgency index corresponding to each single frame of the pavement image.

[0137] In this embodiment of the invention, an emergency maintenance assessment is performed based on the fuzzy weight values ​​corresponding to each feature factor within a single frame of each pavement image, compared to the actual values ​​of each feature factor within the corresponding 12-dimensional feature set of crack defects. This is achieved using the numpy library in Python. The fuzzy weight values ​​of each feature factor within a single frame of each pavement image are read from the "Feature Weight Table" in the SQLite database, while the actual values ​​of the 12-dimensional feature set of the corresponding image frame are read from the "Disease Features" table. The actual values ​​of each feature factor are then standardized using a min-max standardization method, as shown in the formula: ,in These are the original values. and The minimum and maximum values ​​of this feature factor across all image frames are given, resulting in a standardized feature value array. To calculate the maintenance urgency index, multiply the standardized feature value array by the corresponding elements of the fuzzy weight value array and then sum them. Suppose that the maintenance urgency index of a single frame of a road surface image is calculated to be 0.75. This index, along with information such as spectral type and image frame number, is recorded in the "Maintenance Urgency Table" of the SQLite database. By comparing the maintenance urgency indices of different image frames, the priority of road surface maintenance can be determined, providing a quantitative basis for road surface maintenance decisions.

[0138] Furthermore, the step of planning the maintenance urgency sorting of each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame includes:

[0139] Based on the maintenance urgency index corresponding to each road surface image frame, the corresponding road surface image frames are sorted and prioritized for emergency maintenance to generate the corresponding road surface maintenance emergency priority sorting image frame sequence.

[0140] In this embodiment of the invention, the corresponding pavement image frames are prioritized and sorted based on their maintenance urgency index. The maintenance urgency index, spectral type, and image frame number of each pavement image frame are read from the "Maintenance Urgency Table" in the SQLite database. The data is then read into a data frame using Python's pandas library. The data frame's `sort_values()` function is used to sort the data in descending order by the maintenance urgency index column, for example, `df.sort_values(by='Maintenance Urgency Index', ascending=False, inplace=True)`. Road surface images with high urgency indices are placed at the beginning of the sequence, while those with low urgency indices are placed at the end. After sorting, the data from the image frame number column is extracted and compiled into a list. This list is the road maintenance emergency priority sorting image frame sequence. Assuming the image frame number column in the sorted data frame is ['001', '003', '002', ...], this list is stored in a Python pickle file named "Road Maintenance Emergency Priority Sorting Image Frame Sequence.pkl". At the same time, a "Sorting Sequence Index Table" is created in the SQLite database to record information such as the storage path and generation time of the sequence file, facilitating quick access and management of the sequence data later.

[0141] Preferably, a defect location search is performed on each single-frame of the road image sorting within the emergency priority sorting image frame sequence for road maintenance to generate a sequence of defect location points corresponding to the single-frame of the road image sorting;

[0142] In this embodiment of the invention, by performing a defect location search on each single pavement image sorting frame within the emergency priority sorting image frame sequence for pavement maintenance, the image frame sequence list is read from the file "Emergency Priority Sorting Image Frame Sequence for Pavement Maintenance.pkl". Each image frame number in the list is traversed, and for each image frame number, the defect location coordinates corresponding to that image frame are queried from the "Defect Features" table in the SQLite database. Assuming an image frame number of 001 has defect location coordinates of upper left (100, 100) and lower right (300, 300), the defect location within this area is further refined using... Python's NumPy library divides the area into a 10×10 grid and calculates the center point coordinates of each grid as the location of the defects. For example, the center point coordinates of the first grid are (110, 110). The center point coordinates of all grids are calculated sequentially to form a list. This list is the sequence of defect location points corresponding to a single frame of the road image. The sequence of defect location points corresponding to each image frame number is stored in a newly created "Defect Location Point Table" in the SQLite database. The table structure includes fields such as image frame number, point number, and point coordinates (x-coordinate, y-coordinate) for subsequent path planning.

[0143] Preferably, the maintenance fleet route is planned by combining the ant colony optimization algorithm of the GIS map with the sequence of defect locations corresponding to the single frame of the road image sorting. This combines the corresponding defect locations in the sequence with the GIS map and fully considers the corresponding road network topology and traffic conditions to plan the optimal driving route for the corresponding maintenance fleet, thereby generating the road maintenance fleet sorting and scheduling sequence corresponding to the single frame of the road image sorting.

[0144] In this embodiment of the invention, the maintenance vehicle route is planned by combining the ant colony optimization algorithm of GIS map with the sequence of fault locations corresponding to a single frame of road image sorting. This combines the corresponding fault locations within the sequence with the GIS map and, taking into full account the corresponding road network topology and traffic conditions, plans the optimal driving route for the corresponding maintenance vehicle. The GIS map data is read using the geopandas library in Python, and the map data is in Shapefile format. The format includes information such as road network layers and traffic condition layers. It reads the "Disease Location Table" to sort the sequence of disease location points corresponding to each frame of each road image. For each point sequence, an ant colony optimization algorithm is implemented using the Python-based ant-colony-optimization library. The algorithm sets the number of ants to 50, the pheromone evaporation coefficient to 0.1, the heuristic factor to 2, the expected heuristic factor to 3, and the maximum number of iterations to 100. Disease location points are used as the starting and passing nodes of the ants, and road connections in the road network topology are used as feasible paths for the ants. Simultaneously, based on real-time traffic flow data in the traffic condition layer, the paths of congested road sections are optimized. The length is multiplied by a congestion coefficient (e.g., 2 when congestion is severe) to simulate the impact of traffic conditions on route selection. Through iterative calculations using an ant colony optimization algorithm, the optimal driving path corresponding to each point sequence is obtained. The node coordinates, road names, and other information in the optimal driving path are compiled into a list. This list is the road maintenance fleet sorting and scheduling sequence corresponding to a single frame of road image sorting. The sorting and scheduling sequence of each single frame of road image sorting is stored in the "Maintenance Fleet Scheduling Table" in the SQLite database. The table contains fields such as image frame number, scheduling sequence information (path node data, etc., stored in JSON format), and estimated travel time, providing accurate route planning schemes and time references for the actual scheduling of maintenance fleets.

[0145] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A road surface defect intelligent sorting platform based on AI recognition, characterized in that, Includes the following modules: The multispectral acquisition module is used to acquire road surface damage image frames in different spectral bands in real time at a speed of 60km / h by integrating corresponding visible light / infrared sensors with a vehicle-mounted array camera. Frame extraction preprocessing is performed on pavement distress image frames under different spectral bands to generate standard pavement distress frames under different spectra. The edge computing module is used to perform single-frame delay control disease area identification on standard frames of road surface defects under different spectra by deploying the corresponding lightweight YOLOv7 improved model, so as to obtain the corresponding disease location area within a single frame of the road surface image under each spectrum. The defect sorting decision module is used to perform 12-dimensional feature analysis of crack defects in the corresponding defect location area in a single frame of a pavement image under each spectrum, so as to extract the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image; and to perform maintenance emergency assessment on the 12-dimensional feature set of crack defects corresponding to each single frame of the pavement image based on the fuzzy hierarchical analysis method, so as to obtain the maintenance urgency index corresponding to each single frame of the pavement image. Based on the maintenance urgency index corresponding to each individual pavement image frame, maintenance urgency sorting planning is performed on each individual pavement image frame to generate a pavement maintenance vehicle convoy sorting and scheduling sequence corresponding to each individual pavement image frame; this includes the following functions: The image scale corresponding to a single frame of the road surface image under each spectrum is obtained by identifying the location of the damage within a single frame of the road surface image under each spectrum. Based on the corresponding image scale, a 12-dimensional feature analysis of crack defects was performed on the corresponding defect location regions within a single frame of pavement images under each spectrum to extract a 12-dimensional feature set of crack defects for each single frame of pavement images. This set includes the maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, defect area, crack distribution density, defect perimeter, defect shape complexity, mean gray value, and standard deviation of gray value. Binarization processing is performed on the corresponding disease location regions within a single frame of the road surface image under each spectrum to generate binarized images of the corresponding disease regions under each spectrum. Based on the binarized images of the affected areas under each spectrum and combined with the corresponding image scale, the maximum crack width, total crack length, crack tortuosity, affected area area, crack distribution density, affected area perimeter, and affected area shape complexity are calculated; including: By highlighting the corresponding crack area in the binarized image of the disease area under each spectrum, and using morphological operations to remove the corresponding noise and fine branches, the bounding rectangle of the crack area is calculated. The length of the short side of the bounding rectangle is approximated as the corresponding maximum width, and the corresponding maximum crack width is calculated according to the corresponding image scale. Skeletonization processing is performed on the binarized images of the diseased areas under each spectrum to simplify the corresponding crack areas into lines with a width of one pixel, and the total number of pixels on the lines is counted. At the same time, the total length of the corresponding crack is calculated by combining the corresponding image scale. The corresponding crack start point and crack end point are obtained by binarizing the disease area under each spectrum. The straight distance between the crack start point and the crack end point is calculated based on the crack start point and crack end point. The crack tortuosity is obtained by the ratio of the total crack length to the straight distance between the crack start point and the crack end point. By statistically analyzing the total number of pixels in the binarized image of the diseased area under each spectrum, and converting the corresponding diseased area according to the corresponding image scale, the corresponding crack distribution density is calculated based on the ratio between the total crack length and the diseased area. Boundary detection is performed on the binarized images of the diseased areas under each spectrum to obtain the boundary of the diseased area after binarization, and the total number of pixels on the boundary of the diseased area is counted. At the same time, the perimeter of the diseased area is calculated according to the corresponding image scale. The shape complexity of the diseased area is calculated based on the ratio between the square of the perimeter of the diseased area and the area of ​​the diseased area. By taking points at equal intervals along the crack centerline within the corresponding visible spectral band of the road surface image in each spectrum, dividing the area into several small segments, calculating the vertical width of the crack corresponding to each segment, and calculating the average value, the average crack width is obtained. By identifying the location of the damage in the corresponding infrared spectral band within a single frame of the road surface image under each spectrum, the temperature distribution differences between each equally spaced point are determined. Based on the temperature distribution differences between each equally spaced point, the crack depth corresponding to each equally spaced point is estimated. At the same time, the maximum value and average value of the crack depth corresponding to each equally spaced point are determined to obtain the corresponding maximum crack depth and average crack depth. The mean and standard deviation of gray levels in the corresponding disease location area within a single frame of the road surface image under each spectrum are calculated to obtain the mean and standard deviation of gray levels of all pixels in the disease location area, thus obtaining the mean and standard deviation of gray levels in the corresponding disease area. The above-mentioned maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of ​​the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, mean gray value of the diseased area, and standard deviation of gray value of the diseased area are combined into a 12-dimensional feature set of cracks and diseases corresponding to a single frame of each road image. Based on the fuzzy hierarchical analysis method, an emergency maintenance assessment of the 12-dimensional feature set of cracks corresponding to single frames of road surface images is performed. A 12×12 fuzzy judgment matrix is ​​constructed through pairwise comparisons. The matrix contains the first... OK Column elements Indicates the first The first characteristic factor affects the second The relative importance of each characteristic factor, and satisfying The weights of each characteristic factor are calculated using the root method. First, the weights are calculated... Again Find the 12th root to get Finally, normalization is performed, and the calculation is performed. The fuzzy weight values ​​of each feature factor are obtained, and the actual 12-dimensional feature values ​​of each road surface image frame are read and subjected to min-max normalization. in These are the original values. and Let the minimum and maximum values ​​of this feature factor be the values ​​across all image frames. Then, multiply the standardized feature value array by the corresponding elements of the fuzzy weight value array and sum the results. The maintenance urgency index corresponding to each single frame of the road surface image is obtained; Based on the maintenance urgency index corresponding to each road surface image frame, maintenance urgency sorting planning is carried out for each road surface image frame to generate a road maintenance vehicle sorting and scheduling sequence corresponding to the road surface image sorting frame. The road damage interactive display module is used to display individual frames of road image sorting on the corresponding human-computer interaction terminal, so as to intuitively view the location, type and maintenance urgency index of the road damage corresponding to each individual frame of road image sorting. Based on the road maintenance fleet sorting and scheduling sequence corresponding to the individual frames of road image sorting, the module issues corresponding maintenance and treatment instructions on the human-computer interaction terminal to execute the corresponding road damage maintenance fleet scheduling and planning work.

2. The AI-based intelligent road surface defect sorting platform according to claim 1, characterized in that, The multispectral acquisition module includes the following functions: A vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and collect road surface distress image frames in different spectral bands in real time at a speed of 60km / h. The road surface distress image frames in the visible light band are used to capture conventional visible road surface distress, while the road surface distress image frames in the infrared light band are used to identify tiny cracks or damage hidden in the road surface based on the corresponding temperature distribution. The pavement distress image frames under different spectral bands were processed by time-slot extraction at a frame extraction frequency of 5 seconds / frame to obtain local pavement distress frame sets under different spectral bands. The pixel blur level of local frames of road surface defects under different spectral bands is averaged to obtain the average pixel blur level of image frames under different spectral bands. Based on the average pixel blurring of image frames under different spectral bands, the corresponding road surface distress image frames under different spectral bands are blurred and denoised to obtain blurred and denoised road surface distress frames under different spectral bands. Pixel-level normalization is performed on the blurred and denoised pavement distress frames under different spectral bands to generate standard pavement distress frames under different spectra.

3. The AI-based intelligent road surface defect sorting platform according to claim 1, characterized in that, The edge computing module includes the following functions: By inputting standard frames of road surface defects under different spectra to the corresponding edge computing nodes in the local vehicle, the image processing latency of a single frame of road surface defect image under each spectrum can be controlled. By performing statistical analysis of road surface region features on the corresponding edge computing nodes in the local vehicle, the edge and texture features of the road surface image region under different spectra are statistically analyzed and a feature matrix is ​​formed to generate the image region feature matrix corresponding to a single frame of road surface defect image under each spectrum. By deploying the corresponding lightweight YOLOv7 improved model on the corresponding edge computing node and inputting the image region feature matrix corresponding to the single frame of the road surface distress image under each spectrum, the corresponding single frame of the road surface distress image is used to identify the distress candidate region, so as to segment each single frame of the road surface distress image into various candidate regions, and identify and mark the candidate regions corresponding to the road surface distress, while marking the position and size of the corresponding candidate regions, so as to generate the set of distress candidate regions corresponding to the single frame of the road surface distress image under each spectrum; For each candidate region in the candidate region set corresponding to a single frame of pavement distress image under each spectrum, the distress location confidence is calculated to obtain the distress location confidence distribution corresponding to each candidate region in the candidate region set under each spectrum. Based on the confidence distribution of the disease location corresponding to each candidate region in the disease candidate region set under each spectrum, confidence optimization screening is performed on each candidate region in the disease candidate region set corresponding to a single frame of the road surface disease image under each spectrum to obtain the corresponding disease location region in a single frame of the road surface image under each spectrum.

4. The AI-based intelligent road surface defect sorting platform according to claim 3, characterized in that, The image processing latency is specifically controlled to be within ≤50ms.

5. The AI-based intelligent road defect sorting platform according to claim 3, characterized in that, The confidence optimization screening of each candidate region in the candidate region set corresponding to a single frame of road surface defect image under each spectrum, based on the confidence distribution of the defect location corresponding to each candidate region in the defect candidate region set under each spectrum, includes: The confidence gradient of the candidate area of ​​the road surface disease image for each single frame under each spectrum is calculated based on the confidence distribution of the disease location corresponding to each candidate area in the disease candidate area set under each spectrum. Based on the confidence gradient of the candidate regions of road surface defects in a single frame of the image under each spectrum, the confidence distribution of the defect location corresponding to each candidate region in the defect candidate region set under each spectrum is optimized and filtered. If the confidence distribution of the corresponding defect location is greater than or equal to the confidence gradient of the defect candidate region, the corresponding candidate region in its defect candidate region set is selected as the defect location region corresponding to the single frame of the road surface defects image under that spectrum. Candidate regions with confidence distributions less than the confidence gradient of the defect candidate region are then filtered out to obtain the defect location regions corresponding to the single frame of the road surface defects image under each spectrum.

6. The AI-based intelligent road surface defect sorting platform according to claim 1, characterized in that, The method of planning for maintenance urgency sorting of each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame includes: Based on the maintenance urgency index corresponding to each road surface image frame, the corresponding road surface image frames are sorted and prioritized for emergency maintenance to generate the corresponding road surface maintenance emergency priority sorting image frame sequence. The road surface maintenance emergency priority sorting image frame sequence is used to search for the location of defects in each road surface image sorting frame in order to generate a sequence of defect location points corresponding to the road surface image sorting frame. By combining the ant colony optimization algorithm of GIS map with the sequence of defect locations corresponding to a single frame of road image sorting, maintenance vehicle route planning is performed. This combines the corresponding defect locations in the sequence with the GIS map and fully considers the corresponding road network topology and traffic conditions to plan the optimal driving route for the corresponding maintenance vehicle, thereby generating the road maintenance vehicle sorting and scheduling sequence corresponding to a single frame of road image sorting.

Images