Pavement disease intelligent sorting platform based on AI identification

Abstract

The invention relates to the technical field of intelligent maintenance, in particular to an intelligent pavement disease sorting platform based on AI recognition. The platform takes a multispectral acquisition module and an edge calculation module as front acquisition and identification modules, and further comprises a disease sorting decision module and a disease interactive display module, and can acquire a pavement disease image frame in real time to perform frame extraction preprocessing and single-frame delay control disease area identification so as to obtain a disease position area in a pavement image single frame; performing crack disease 12-dimensional feature analysis and maintenance emergency evaluation on the disease position area in the single frame of the pavement image to obtain a maintenance emergency degree index corresponding to each single frame of the pavement image; and carrying out maintenance emergency sorting planning based on the maintenance emergency degree index corresponding to each pavement image single frame, and issuing a maintenance disposal instruction on the man-machine interaction terminal to execute corresponding pavement disease maintenance motorcade scheduling planning work. According to the method, the detection efficiency and the sorting accuracy can be improved, resources are optimized, and the investment of unnecessary maintenance is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent maintenance technology, and in particular to an intelligent pavement disease sorting platform based on AI recognition. Background Art

[0002] With the acceleration of urbanization, pavement damage is becoming increasingly serious. Damage, cracks, potholes, and other road conditions not only impact road lifespan but also increase traffic safety risks. In recent years, with the rapid development of artificial intelligence (AI), automated detection technologies based on deep learning and computer vision have been increasingly applied to the identification and analysis of road defects. AI algorithm models can automatically extract the characteristics of road defects from large amounts of image data, enabling automated and intelligent identification of road defects.

[0003] In addition, similar patents such as CN111126460A disclose an artificial intelligence-based automatic inspection method for pavement defects, a medium, equipment and a device, which include obtaining a road surface image to be detected and positioning information corresponding to the road surface image to be detected at a preset frequency; inputting the road surface image to be detected into a trained road surface defect recognition model to determine whether the road surface image to be detected is a road surface defect image through the road surface defect recognition model; if so, generating an alarm message based on the road surface image to be detected and the positioning information corresponding to the road surface image to be detected, so that relevant personnel can deal with the road surface defect according to the alarm information; it can automatically inspect road surface defects, effectively improve the efficiency of road surface defect inspection, and save manpower and material resources required for road surface defect inspection. Although the above patents can effectively improve the efficiency of defect inspection, they are limited to completing defect positioning, lack a priority sorting mechanism for defects, and fail to perform intelligent grading processing based on the severity and type of defects, thereby affecting subsequent maintenance decision allocation. Summary of the Invention

[0004] Based on this, it is necessary for the present invention to provide an intelligent pavement disease sorting platform based on AI identification to solve at least one of the above technical problems.

[0005] To achieve the above objectives, an AI-based pavement defect intelligent sorting platform is provided, comprising: a front-end acquisition and identification module and a pavement defect intelligent sorting platform, wherein the front-end acquisition and identification module and the pavement defect intelligent sorting platform are connected to the same network, wherein the front-end acquisition and identification module includes a multispectral acquisition module and an edge computing module, and the pavement defect intelligent sorting platform includes a disease sorting decision module and a disease interactive display module, wherein: The multispectral acquisition module is used to use a vehicle-mounted array camera to integrate corresponding visible light / infrared sensors and collect pavement disease image frames in different spectral bands in real time at a speed of 60 km / h; perform frame extraction preprocessing on the pavement disease image frames in different spectral bands to generate standard pavement disease frames in different spectrums; The edge computing module is used to perform single-frame delay control disease area identification on the road surface disease standard frames under different spectra by deploying the corresponding lightweight YOLOv7 improved model to obtain the corresponding disease location area in the single frame of the road surface image under each spectrum; The disease sorting decision module is used to perform 12-dimensional crack disease feature analysis on the disease location area corresponding to the single frame of the pavement image under each spectrum to extract the 12-dimensional crack disease feature set corresponding to each single frame of the pavement image; perform maintenance urgency assessment on the 12-dimensional crack disease feature set corresponding to each single frame of the pavement image based on the fuzzy hierarchical analysis method to obtain the maintenance urgency index corresponding to each single frame of the pavement image; and perform maintenance emergency sorting planning on each single frame of the pavement image based on the maintenance urgency index corresponding to each single frame of the pavement image to generate a pavement maintenance fleet sorting and scheduling sequence corresponding to the sorted single frame of the pavement image; The disease interactive display module is used to display each road surface image sorting single frame on the corresponding human-computer interaction terminal to intuitively view the disease location, type and maintenance urgency index corresponding to each road surface image sorting single frame, and to issue corresponding maintenance disposal instructions on the human-computer interaction terminal according to the road surface maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting single frame, so as to execute the corresponding road surface disease maintenance fleet scheduling and planning work.

[0006] Furthermore, the multispectral acquisition module includes the following functions: A vehicle-mounted array camera integrates corresponding visible light / infrared sensors and captures pavement damage image frames in different spectral bands in real time at a speed of 60 km / h. The pavement damage image frames in the visible light sensor band capture conventional visible pavement damage, while the pavement damage image frames in the infrared sensor band identify tiny cracks or damage hidden in the pavement based on the corresponding temperature distribution. Perform time slot frame extraction on the pavement disease image frames in different spectral bands at a frame extraction frequency of 5 seconds per frame to obtain a set of local pavement disease frames in different spectral bands; The pixel fuzziness of the local frame set of pavement defects in different spectral bands is averaged and quantified to obtain the average pixel fuzziness of the image frame in different spectral bands; Based on the average value of pixel fuzziness of image frames in different spectral bands, the pavement disease image frames in the corresponding bands are subjected to fuzzy denoising processing to obtain fuzzy denoised frames of pavement disease in different spectral bands; The fuzzy denoised frames of pavement damage in different spectral bands are normalized at the pixel level to generate standard frames of pavement damage in different spectra.

[0007] Furthermore, the edge computing module includes the following functions: By inputting the standard frames of road surface damage under different spectra into the corresponding edge computing nodes in the local vehicle, the image processing delay corresponding to a single frame of road surface damage images under each spectrum can be controlled. By performing statistical analysis of road surface regional features on the corresponding edge computing node in the local vehicle, the edge and texture features corresponding to the road surface image area under different spectra are statistically analyzed and formed into a feature matrix. The image region feature matrix corresponding to a single frame of the road surface disease image under each spectrum is generated. By deploying the corresponding lightweight YOLOv7 improved model on the corresponding edge computing node and inputting the image region feature matrix corresponding to the single frame of the pavement disease image under each spectrum, the corresponding single frame of the pavement disease image is identified for the disease candidate area. Each single frame of the pavement disease image is segmented into candidate areas, and the candidate areas corresponding to the pavement disease are identified and calibrated. The location and size of the corresponding candidate areas are also marked to generate a set of disease candidate areas corresponding to the single frame of the pavement disease image under each spectrum. Calculate the confidence of the defect position of each candidate area in the defect candidate area set corresponding to a single frame of the pavement defect image under each spectrum to obtain the distribution of the confidence of the defect position of each candidate area in the defect candidate area set under each spectrum; Based on the confidence distribution of the defect location corresponding to each candidate area in the defect candidate area set under each spectrum, each candidate area in the defect candidate area set corresponding to a single frame of the pavement defect image under each spectrum is subjected to confidence optimization screening to obtain the corresponding defect location area in the single frame of the pavement defect image under each spectrum.

[0008] Furthermore, the image processing delay is specifically controlled within ≤50ms.

[0009] Furthermore, the confidence optimization screening of each candidate area in the set of defect candidate areas corresponding to a single frame of the pavement defect image under each spectrum based on the defect position confidence distribution corresponding to each candidate area in the set of defect candidate areas under each spectrum includes: The confidence gradient of the candidate disease area corresponding to a single frame of the pavement disease image under each spectrum is calculated based on the confidence distribution of the disease position corresponding to each candidate area in the candidate disease area set under each spectrum; Based on the confidence gradient of the defect candidate area corresponding to a single frame of the pavement defect image under each spectrum, the confidence distribution of the defect position corresponding to each candidate area in the defect candidate area set under each spectrum is optimized and screened. If the corresponding defect position confidence distribution is greater than or equal to the confidence gradient of the defect candidate area, the corresponding candidate area in the defect candidate area set is screened out as the defect position area corresponding to the single frame of the pavement defect image under that spectrum, and the candidate areas with a confidence distribution less than the confidence gradient of the defect candidate area are screened out to obtain the defect position area corresponding to the single frame of the pavement defect image under each spectrum.

[0010] Furthermore, the disease sorting decision module includes the following functions: The image scale corresponding to the single frame of the pavement image under each spectrum is obtained through the corresponding disease position area in the single frame of the pavement image under each spectrum; Based on the corresponding image scale, a 12-dimensional feature analysis of crack disease is performed on the corresponding disease location area in a single frame of the pavement image under each spectrum to extract the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image, including the maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, grayscale mean of the diseased area, and grayscale standard deviation of the diseased area. Based on the fuzzy analytic hierarchy process, the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image is evaluated for maintenance urgency, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained; Based on the maintenance urgency index corresponding to each road surface image frame, maintenance emergency sorting planning is performed on each road surface image frame to generate a road maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting single frame.

[0011] Furthermore, the 12-dimensional feature analysis of crack damage in the corresponding damage location area in a single frame of the pavement image under each spectrum based on the corresponding image scale includes: Binarization is performed on the corresponding diseased location area in a single frame of the road surface image under each spectrum to generate a binary image of the corresponding diseased area under each spectrum; Based on the corresponding binary image of the damaged area under each spectrum and combined with the corresponding image scale, the corresponding maximum crack width, total crack length, crack tortuosity, damaged area, crack distribution density, perimeter of the damaged area, and shape complexity of the damaged area are calculated; The crack centerline in the defect location area corresponding to the visible spectrum band in a single frame of the pavement image under each spectrum is divided into several small segments by taking points at equal intervals. The vertical widths of the cracks corresponding to the several small segments are calculated respectively, and the average value is calculated at the same time to obtain the corresponding average crack width. The temperature distribution difference between each equally spaced point is determined by the defect location area of the corresponding infrared spectral band in a single frame of the pavement image under each spectrum. The crack depth corresponding to each equally spaced point is estimated based on the temperature distribution difference between each equally spaced point. At the same time, the maximum value of the crack depth corresponding to each equally spaced point is determined and the average value is calculated to obtain the corresponding maximum crack depth and average crack depth. The regional grayscale mean and standard deviation of the corresponding defect location area in a single frame of the road surface image under each spectrum are calculated to calculate the grayscale mean and grayscale standard deviation corresponding to all pixels in the defect location area, and obtain the corresponding grayscale mean and grayscale standard deviation of the defect area; The corresponding maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, grayscale mean of the diseased area, and grayscale standard deviation of the diseased area are merged into a 12-dimensional feature set of crack diseases corresponding to each single frame of the pavement image.

[0012] Furthermore, the calculation of the corresponding maximum crack width, total crack length, crack tortuosity, area of the damaged area, crack distribution density, perimeter of the damaged area, and shape complexity of the damaged area based on the corresponding binary image of the damaged area under each spectrum and in combination with the corresponding image scale includes: The corresponding crack area is highlighted by binarizing the image of the diseased area under each spectrum and the corresponding noise and small branches are removed by morphological operations. Then, the circumscribed rectangle of the crack area is calculated, and the length of the short side of the circumscribed rectangle is approximately regarded as the corresponding maximum width. The corresponding maximum width of the crack is converted according to the corresponding image scale. The binary images of the diseased areas under each spectrum were skeletonized to simplify the corresponding crack areas into lines with a single pixel width. The total number of pixels on the lines was counted, and the total length of the cracks was calculated based on the corresponding image scale. The corresponding crack starting pixel point and crack ending pixel point are obtained by binarizing the corresponding diseased area under each spectrum, and the straight-line distance between the corresponding crack starting pixel point and the crack ending pixel point is calculated based on the crack starting pixel point and the crack ending pixel point, and the corresponding crack tortuosity is calculated based on the ratio of the total crack length to the straight-line distance between the crack starting point and the crack ending point; By counting the total number of pixels in the binary image of the corresponding diseased area under each spectrum, and converting the corresponding diseased area according to the corresponding image scale, the corresponding crack distribution density is calculated based on the ratio between the total length of the corresponding crack and the area of the diseased area; Perform boundary detection on the binarized image of the diseased area under each spectrum to obtain the boundary of the corresponding diseased area after binarization, and count the total number of pixels corresponding to the boundary of the diseased area. At the same time, calculate the corresponding perimeter of the diseased area according to the corresponding image scale; The shape complexity of the corresponding diseased area is calculated based on the ratio between the square of the perimeter of the diseased area and the area of the diseased area.

[0013] Furthermore, the maintenance urgency assessment is performed on the 12-dimensional feature set of crack damage corresponding to each single frame of the pavement image based on the fuzzy analytic hierarchy process, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained, including: Based on the fuzzy analytic hierarchy process, the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image is used to construct the fuzzy judgment matrix of crack disease corresponding to each single frame of the pavement image; Based on the crack disease fuzzy judgment matrix corresponding to each single frame of the pavement image, weighted fuzzy reasoning is performed on each characteristic factor in the corresponding 12-dimensional feature set of the crack disease to generate the fuzzy weight value corresponding to each characteristic factor in each single frame of the pavement image; Based on the fuzzy weight values corresponding to each characteristic factor in each single frame of the pavement image, a maintenance urgency assessment is performed between the actual values of each characteristic factor in the 12-dimensional feature set of the corresponding crack disease, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained.

[0014] Furthermore, performing maintenance emergency sorting planning on each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame includes: Based on the maintenance urgency index corresponding to each road surface image single frame, the corresponding road surface image single frame is sorted and sorted according to emergency priority to generate a corresponding road surface maintenance emergency priority sorting image frame sequence; Performing a disease location search on each pavement image sorting single frame within the pavement maintenance emergency priority sorting image frame sequence to generate a disease location point sequence corresponding to the pavement image sorting single frame; The maintenance fleet path planning is carried out by combining the ant colony optimization algorithm with the GIS map for the sequence of disease location points corresponding to the single frame of road surface image sorting. The corresponding disease location points in the sequence are combined with the GIS map, and the optimal driving path is planned for the corresponding maintenance fleet while fully considering the corresponding road network topology and traffic conditions. In this way, the road maintenance fleet sorting and scheduling sequence corresponding to the single frame of road surface image sorting is generated.

[0015] Beneficial effects of the present invention: The AI-based intelligent pavement defect sorting platform proposed in the present invention generally uses a multispectral acquisition module and an edge computing module as front-end acquisition and identification modules, and also includes a disease sorting decision module and a disease interactive display module. Compared with the existing technology, the beneficial effect of this application is that it uses a vehicle-mounted array camera and integrates visible light / infrared sensors to perform real-time acquisition of pavement disease images, which plays an important role in improving the accuracy and efficiency of pavement disease detection. By collecting images while the vehicle is traveling at a speed of 60km / h, a larger range of pavement disease detection areas can be quickly covered. While the vehicle is traveling at a faster speed, high-precision Sensors can ensure the collection of image data in different spectral bands, so that various types of road defects, such as cracks, potholes, and ruts, can be accurately captured in different environments and lighting conditions. The addition of infrared sensors ensures the clear collection of defect images even at night when there is insufficient light or in poor lighting conditions, and provides useful information different from visible light images. Subsequently, the road defect image frames in different spectral bands are pre-processed by frame extraction and optimized through image enhancement, noise removal and other means to generate standardized road defect image frames. This not only improves the consistency of data processing, but also greatly improves the efficiency of road defect identification. Secondly, by deploying a lightweight YOLOv7 improved model to perform single-frame delayed control disease area identification in pavement disease areas, the accuracy and efficiency of pavement disease detection can be significantly improved. As an efficient target detection model, YOLOv7 has strong real-time detection capabilities and low computing resource consumption. It can quickly process large-scale pavement image data while maintaining high accuracy. YOLOv7 is used to identify and locate diseased areas in each frame of the image, especially typical diseases such as cracks and potholes. Compared with traditional image processing technology, YOLOv7 can more accurately distinguish and mark different types of diseased areas through training of deep learning algorithms. It can also adapt to changes in different lighting conditions, improve the robustness of disease detection, and ensure the accuracy of disease location areas. Then, by performing a 12-dimensional feature analysis on the damaged area in each frame of the image, it is helpful to comprehensively and accurately assess the nature and severity of pavement diseases, especially crack-type diseases. By extracting multi-dimensional features of cracks, including their length, width, depth, direction, distribution density, etc., the morphological characteristics and evolution patterns of the cracks can be fully described. Such multi-dimensional features can provide a more detailed basis for subsequent maintenance decisions, avoiding the one-sidedness of single-dimensional analysis.In addition, a comprehensive assessment of these characteristics based on the fuzzy analytic hierarchy process (AHP) further deepens the scientific nature and objectivity of the urgency of the disease. AHP can effectively handle multidimensional and complex problems. It scores the urgency index of each disease through a hierarchical evaluation system, and can realize a priority sorting mechanism for pavement diseases. It can also perform intelligent grading processing based on the severity and type of the disease, thereby providing more accurate sorting and priority determination for maintenance work, and providing a more scientific and quantitative guidance basis for pavement maintenance. Finally, by displaying the sorted single frames of pavement disease images on the human-computer interaction terminal, maintenance personnel can be intuitively provided with the location information, type, and maintenance urgency index of the pavement disease, greatly improving the efficiency of maintenance work and the quality of decision-making. This step not only helps maintenance personnel quickly and accurately understand the specific situation of the disease in each frame of the image, but also helps staff make the most reasonable maintenance decision allocation by displaying the type and urgency of the disease in real time. By displaying the location and type of the disease, maintenance personnel can quickly determine which disease areas need to be addressed immediately and which can be arranged in subsequent work, thereby significantly improving the response speed and work of pavement maintenance, making maintenance decisions more scientific, rapid, and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments thereof made with reference to the following drawings: Figure 1 This is a schematic diagram of the module connection between the front-end collection and identification module and the pavement disease intelligent sorting platform of the present invention; Figure 2 for Figure 1 Schematic diagram of the functional flow of the multispectral acquisition module. DETAILED DESCRIPTION

[0017] The following is a clear and complete description of the technical platform of the present invention in conjunction with the accompanying drawings. It is obvious that the embodiments described are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

[0018] To achieve this, please refer to Figures 1 to 2 The present invention provides an intelligent sorting platform for road surface defects based on AI recognition. In the embodiments of the present invention, please refer to Figure 1FIG. 1 is a schematic diagram showing the module connection between the pre-collection and identification module and the pavement defect intelligent sorting platform of the present invention. In this example, the pre-collection and identification module and the pavement defect intelligent sorting platform are connected to the same network. The pre-collection and identification module includes a multispectral acquisition module and an edge computing module. The pavement defect intelligent sorting platform includes a defect sorting decision module and a defect interactive display module. The multispectral acquisition module is used to use a vehicle-mounted array camera to integrate corresponding visible light / infrared sensors and collect pavement disease image frames in different spectral bands in real time at a speed of 60 km / h; perform frame extraction preprocessing on the pavement disease image frames in different spectral bands to generate standard pavement disease frames in different spectrums; In an embodiment of the present invention, a vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and collect road surface disease image frames in different spectral bands in real time at a speed of 60 km / h. A Leopard Imaging LI-V5000 array camera equipped with 12 lens groups is selected, of which 6 lens groups are equipped with Sony IMX415 visible light sensors (spectral response range 380-780nm) to capture visible diseases such as road cracks and potholes; the other 6 lens groups are integrated with FLIR Boson 640 infrared sensors (working band 7.5-13.5μm) to detect abnormal internal road surface temperature to identify hidden diseases. The camera is connected to a Xilinx Zynq UltraScale+ system via a Gigabit Ethernet interface. The MPSoC chip is connected to the vehicle data acquisition system, and visible light and infrared light images are synchronously collected at a rate of 120 frames per second. The original data is stored in the 4TB vehicle solid-state drive in RAW format. When the pavement disease image frames in different spectral bands are pre-processed by extracting frames, a script is written using the Python OpenCV library, and the frame extraction frequency is set to 5 seconds / frame, that is, one frame is extracted every 600 frames (120 frames / second × 5 seconds). The extracted images are subjected to noise reduction, contrast enhancement and other operations in sequence: the median filter algorithm (cv2.medianBlur() function with a kernel size of 3×3) to remove image noise; histogram equalization (cv2.equalizeHist() function) is used to enhance image contrast. The processed image is resized to a fixed resolution of 1920×1080 pixels and converted to JPEG format for storage. The file name contains information such as the spectrum type and acquisition timestamp (for example, "VIS_20241001_103000.jpg" indicates an image acquired at 10:30 on October 1, 2024, using the visible light spectrum). Finally, standard frames of pavement defects under different spectra are generated and stored in the "standard frames" folder of the on-board storage system.

[0019] The edge computing module is used to perform single-frame delay control disease area identification on the road surface disease standard frames under different spectra by deploying the corresponding lightweight YOLOv7 improved model to obtain the corresponding disease location area in the single frame of the road surface image under each spectrum; In the embodiment of the present invention, the corresponding lightweight YOLOv7 improved model is deployed to perform single-frame delay control disease area recognition on the road disease standard frames under different spectra, and the vehicle-mounted edge computing device NVIDIA Jetson AGX On Orin, a pruned and quantized lightweight YOLOv7 model was deployed based on the PyTorch framework, reducing the number of model parameters by 40% and increasing the inference speed by 30%. The torch.jit.trace() function was used to convert the trained model into TorchScript format to optimize the running efficiency on edge devices. Standard frames of road surface defects under different spectra were input into the model in batches, with each batch containing 16 frames of images. The model outputs the category (cracks, potholes, etc.), confidence level, and location coordinates (upper left and lower right corner coordinates) of each detected defect area. The confidence threshold was set to 0.5, and detection results with confidence levels below the threshold were filtered out. For example, a standard frame of visible light was detected by the model and three defect areas were identified, two of which had confidence levels above 0.5. The coordinate information of these two areas (such as [500, 300, 800, 600]) was filtered out. The coordinates of the upper left corner are (500, 300) and the coordinates of the lower right corner are (800, 600) in the "Disease Location Table" of the SQLite database. The spectrum type and image frame number are also marked. Finally, the corresponding disease location area in a single frame of the road surface image under each spectrum is obtained.

[0020] The disease sorting decision module is used to perform 12-dimensional crack disease feature analysis on the disease location area corresponding to the single frame of the pavement image under each spectrum to extract the 12-dimensional crack disease feature set corresponding to each single frame of the pavement image; perform maintenance urgency assessment on the 12-dimensional crack disease feature set corresponding to each single frame of the pavement image based on the fuzzy hierarchical analysis method to obtain the maintenance urgency index corresponding to each single frame of the pavement image; and perform maintenance emergency sorting planning on each single frame of the pavement image based on the maintenance urgency index corresponding to each single frame of the pavement image to generate a pavement maintenance fleet sorting and scheduling sequence corresponding to the sorted single frame of the pavement image; In an embodiment of the present invention, when performing a 12-dimensional feature analysis of crack disease on the corresponding disease location area within a single frame of a pavement image under each spectrum, the coordinates of the disease location area are obtained from the SQLite database, and features are extracted using Python's OpenCV library and NumPy library. For the maximum width of the crack, a morphological opening operation (cv2.morphologyEx() function, with a 3×3 rectangle as the structural element) is first performed on the diseased area image to remove noise, and then the contour is found using the cv2.findContours() function. The bounding rectangle is calculated using the cv2.boundingRect() function, and the actual width is obtained by multiplying the length of the short side by the image scale (pre-stored in the device parameter configuration file). When calculating the total length of the crack, the skeletonize() function of the Scikit-Image library is used to convert the binary image of the diseased area into a single-pixel skeleton, and the number of skeleton pixels is counted using the np.count_nonzero() function, and then multiplied by the scale to obtain the length. The crack tortuosity was calculated by calculating the straight-line distance between the start and end points of the crack contour (np.linalg.norm() function) and comparing it to the total length to obtain a ratio. A similar method was used to calculate the remaining 12 features, such as the area, perimeter, and shape complexity of the diseased area. The results were organized into a 12-dimensional feature set and stored in the "feature set table." Maintenance urgency assessment was performed based on the fuzzy analytic hierarchy process. A 12×12 fuzzy judgment matrix was constructed using Python's numpy and pandas libraries. The importance of each feature factor was determined using a 1-9 scaling method (e.g., the maximum crack width was slightly more important than the total crack length, so it was assigned a value of 3). Weights were calculated using the square root method: the product of each row of the matrix was first calculated, the 12th root was taken, and normalization was performed. The 12-dimensional feature values were normalized to the minimum and maximum values, and the sum was multiplied by the weights to obtain the maintenance urgency index, which was stored in the "urgency table." Finally, the image frames were sorted in descending order according to the urgency index. The maintenance fleet routes were planned using GIS maps and the ant colony optimization algorithm. A pavement maintenance fleet scheduling sequence was generated and stored in the "scheduling table."

[0021] The disease interactive display module is used to display each road surface image sorting single frame on the corresponding human-computer interaction terminal to intuitively view the disease location, type and maintenance urgency index corresponding to each road surface image sorting single frame, and to issue corresponding maintenance disposal instructions on the human-computer interaction terminal according to the road surface maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting single frame, so as to execute the corresponding road surface disease maintenance fleet scheduling and planning work.

[0022] In the embodiment of the present invention, by displaying each pavement image sorting single frame on the corresponding human-computer interaction terminal, a 15.6-inch industrial-grade touch screen is selected as the human-computer interaction terminal, and a visual interface is developed based on the Qt framework. Data is read from the "disease location table", "feature set table", "urgency table" and "scheduling table" of the SQLite database, and the pavement image sorting single frame is displayed on the interface in the form of a map overlay. For each frame of the image, the disease location area is marked with a different color frame (a red frame indicates a crack, and a blue frame indicates a pothole). At the same time, information such as the disease type and the maintenance urgency index is displayed. According to the pavement maintenance fleet sorting scheduling sequence, A visual dispatch route map is generated on the human-computer interaction terminal, the coordinates of the path nodes are matched with the GIS map, and the driving route is displayed by connecting the various disease location points with lines. The operator can issue maintenance and disposal instructions through the terminal. The instructions are encapsulated in JSON format (including fleet number, target point coordinates, estimated departure time, etc.) and sent to the maintenance fleet's on-board terminal through the 4G / 5G communication module. After receiving the instructions, the on-board terminal parses the data and imports it into the on-board navigation system, guiding the fleet to perform road disease maintenance work according to the planned route. At the same time, the fleet's real-time location information is transmitted back to the human-computer interaction terminal to realize full-process monitoring and dispatching.

[0023] Further, as an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 The multispectral acquisition module in this embodiment includes the following functions: S11: A vehicle-mounted array camera is integrated with corresponding visible light / infrared sensors and collects pavement disease image frames in different spectral bands in real time at a speed of 60 km / h. The pavement disease image frames in the visible light sensor band capture conventional visible pavement diseases, while the pavement disease image frames in the infrared sensor band identify tiny cracks or damage hidden in the pavement based on the corresponding temperature distribution. In an embodiment of the present invention, a vehicle-mounted array camera is used to integrate corresponding visible light / infrared sensors and collect image frames corresponding to road surface defects in different spectral bands in real time at a speed of 60 km / h. The vehicle-mounted array camera uses the Leopard Imaging LI-V5000 series equipped with 12 independent lenses, each of which corresponds to a different focal length and viewing angle, which can cover a wide road area. The visible light sensor uses Sony IMX415 with a spectral response range of 380-780nm, which can clearly capture common visible defects such as potholes, cracks, and bumps on the road surface; the infrared sensor uses FLIR Boson640 with an operating band of 7.5-13.5μm. By detecting differences in road surface temperature distribution, it can identify damage defects such as tiny cracks and voids hidden inside the road surface structure. The camera and sensor are connected through a dedicated vehicle-mounted data acquisition system based on the Xilinx Zynq UltraScale+ It is built with an MPSoC chip and has high-speed data processing and storage capabilities. When the vehicle is traveling at a speed of 60km / h, the system synchronously captures corresponding image frames at a rate of 120 frames per second, and stores the collected data in real-time in the on-board solid-state hard drive in RAW format. Each hard drive has a capacity of 4TB and can continuously store more than 6 hours of image data.

[0024] S12: performing time slot frame extraction processing on the pavement disease image frames in different spectral bands at a frame extraction frequency of 5 seconds / frame to obtain a pavement disease local frame set in different spectral bands; In an embodiment of the present invention, pavement disease image frames in different spectral bands are time-slotted and processed at a frame rate of 5 seconds per frame. A frame extraction program is written using the Python OpenCV library. The RAW image files are read and processed in chronological order. First, the acquired image data is sorted according to the acquisition timestamp to construct a time series index. The frame extraction interval is set to 5 seconds, that is, an image frame is extracted every 5×120=600 frames. For example, starting from frame 1, the 601st frame, the 1201st frame, and so on are extracted in sequence. For each spectral band, an independent local frame set storage path is established, and the extracted image frames are saved in PNG format to the corresponding path. Each local frame set contains 1000 image frames and is stored in folders named "Visible Light_Local Frame Set_01" and "Infrared Light_Local Frame Set_01". During the storage process, metadata such as the original acquisition time and acquisition location (latitude and longitude information obtained by the on-board GPS module) of each image frame are synchronously recorded and stored in a JSON format file to facilitate subsequent tracing and analysis.

[0025] S13: performing pixel fuzziness average quantization on the pavement disease local frame set in different spectral bands to obtain the average pixel fuzziness of the image frame in different spectral bands; In the embodiment of the present invention, pixel blurriness of a local frame set of pavement defects in different spectral bands is averaged and quantified to adopt a gradient-based blur assessment method. The relevant calculations are implemented using Python's NumPy library and Sci-Py library. For each frame image in the local frame set, it is first converted into a grayscale image using the cvtColor function of OpenCV. Then, the Sobel operator is used to calculate the gradient of the image in the horizontal and vertical directions to obtain a gradient amplitude image. The variance of the gradient amplitude image is calculated as the blur index of the frame image. The smaller the variance, the more blurred the image. The local frame is traversed. For all images in the set, calculate the blurriness of each frame, add up the blurriness of all images and divide it by the total number of images to get the average pixel blurriness of the image frame in the local frame set. For example, a visible light local frame set contains 1000 frames of images. After calculation, the blurriness of each frame is [0.12, 0.15, 0.10, ...]. Sum these values and divide them by 1000 to get the average pixel blurriness of the local frame set as 0.13. This operation is performed for all local frame sets under each spectral band. The calculation results are stored in a CSV file, which contains fields such as band type, local frame set number, and average blurriness.

[0026] S14: performing fuzzy denoising on the pavement disease image frames in the corresponding bands based on the average values of pixel fuzziness of the image frames in the different spectral bands, thereby obtaining fuzzy denoised pavement disease frames in the different spectral bands; In an embodiment of the present invention, blur denoising is performed on pavement damage image frames in different spectral bands based on the average pixel blur values of the image frames in the corresponding bands. The non-local means denoising algorithm (NL-Means) is used and implemented using the fastNlMeansDenoising function in the Python OpenCV library. A denoising intensity parameter is set based on the average pixel blur value of each local frame set. When the average blur value is greater than 0.1, the denoising intensity parameter is set to 10; when the average blur value is less than or equal to 0.1, the denoising intensity parameter is set to 5. For example, a local infrared frame set with an average blur value of 0.08 is used. The fastNlMeansDenoising function is executed on each frame in the local frame set, and the denoised image is saved with a new file name in the "Infrared_Blurred Denoised Frame_01" folder. During the denoising process, the original resolution and color mode of the image are maintained unchanged, ensuring that the denoised image can effectively remove blur noise while retaining key characteristic information of the pavement damage, such as the edge details of the cracks and the shape of the temperature anomaly area.

[0027] S15: performing pixel-level standardization processing on the pavement disease fuzzy denoised frames in different spectral bands to generate pavement disease standard frames in different spectra.

[0028] In the embodiment of the present invention, pixel-level normalization processing is performed on the blurred and denoised frames of pavement defects in different spectral bands. A method combining histogram equalization and normalization is adopted, and the operation is completed using Python's OpenCV library and NumPy library. First, histogram equalization processing is performed on each frame of the image, and the image contrast is enhanced by the equalizeHist function to make the disease features in the image more obvious. Then, the pixel values of the image are normalized to the range of [0, 1]. The specific calculation formula is: I_norm= I−I_min / I_max−I_min, where I is the original image pixel value, I_min and I_max are the minimum and maximum values of the image pixel values, respectively, and I_ norm is the normalized pixel value. The standardized image is converted to an 8-bit unsigned integer type and saved in JPEG format in folders such as "Visible Light_Standard Frame_01" and "Infrared Light_Standard Frame_01". Standard frames of pavement defects under different spectra are generated. At the same time, a corresponding thumbnail is generated for each standard frame. The thumbnail size is 1 / 10 of the original image for quick preview and indexing. The thumbnail is stored in PNG format in the "_thumb" subfolder with the same name as the standard frame.

[0029] Furthermore, the edge computing module includes the following functions: By inputting the standard frames of road surface damage under different spectra into the corresponding edge computing nodes in the local vehicle, the image processing delay corresponding to a single frame of road surface damage images under each spectrum can be controlled. In this embodiment of the present invention, standard frames of pavement damage under different spectra are input to the corresponding edge computing node on a local vehicle to control the image processing latency of a single frame of pavement damage images under each spectrum. Specifically, the image processing latency is controlled to ≤50ms. The local on-vehicle edge computing node uses an NVIDIA Jetson AGX Orin device, equipped with a 6-core NVIDIA Carmel ARM 64-bit CPU and 2048 NVIDIA Ampere architecture CUDA cores, providing powerful computing power. Standard frames of pavement damage under different spectra (visible and infrared) stored on the vehicle's solid-state drive are transmitted to the edge computing node via the vehicle's Ethernet interface at a rate of 120 frames per second. On the edge computing node, the Linux system's real-time scheduler (RT-PREEMPT patch) is used to set the image processing task to the highest priority. Furthermore, an asynchronous processing mechanism is employed, utilizing the Python asyncio library to enable non-blocking reading and processing of image data. For each frame, the start timestamp is recorded before data reading and the end timestamp is recorded after processing. The processing latency is monitored in real time by calculating the time difference. If the processing delay of a frame of image exceeds 50ms, the frame will be automatically skipped and the number of skipped frames will be recorded for subsequent analysis and optimization to ensure that the overall image processing delay is strictly controlled within the specified range.

[0030] Preferably, a statistical analysis of road surface area characteristics is performed on the corresponding edge computing node in the local vehicle for the standard frames of road surface defects under different spectra, so as to obtain the edge and texture characteristics corresponding to the road surface image region under each spectrum and form a feature matrix, thereby generating an image region feature matrix corresponding to a single frame of the road surface defect image under each spectrum; In an embodiment of the present invention, a statistical analysis of pavement area features is performed on standard frames of pavement defects under different spectra on a corresponding edge computing node in a local vehicle to statistically determine the edge and texture features corresponding to the pavement image area under each spectrum and form a feature matrix. The image is then processed using the OpenCV library and the Scikit-Image library. First, edge detection is performed on the image using the Canny algorithm of OpenCV. By setting a low threshold of 50 and a high threshold of 150, edge information in the pavement image is extracted to obtain an edge image. Then, the gray-level co-occurrence matrix (GLCM) of the image is calculated using the graycomatrix and graycoprops functions of the Scikit-Image library, and texture features such as contrast, correlation, energy, and entropy are extracted from it. The edge image and texture feature data are normalized so that their numerical range is unified to [0, 1]. Finally, the edge information and texture features are concatenated column by column to form a two-dimensional feature matrix, in which each row of the matrix represents a feature vector of a region in the image. For example, for a frame of visible light pavement damage image, after processing, a feature matrix with a shape of [1000, 10] is obtained, where 1000 represents the number of divided image regions and 10 represents the feature dimension of each region. This matrix is used as the image region feature matrix of the frame image and stored in the memory of the edge computing node.

[0031] Preferably, the corresponding lightweight YOLOv7 improved model is deployed on the corresponding edge computing node and the image region feature matrix corresponding to the single frame of the pavement disease image under each spectrum is input to identify the candidate areas of the corresponding single frame of the pavement disease image, so as to segment each single frame of the pavement disease image into candidate areas, and identify and calibrate the candidate areas corresponding to the pavement disease, and mark the positions and sizes of the corresponding candidate areas, so as to generate a set of candidate areas corresponding to the single frame of the pavement disease image under each spectrum; In an embodiment of the present invention, a corresponding lightweight YOLOv7 improved model is deployed on a corresponding edge computing node and an image region feature matrix corresponding to a single frame of a pavement disease image under each spectrum is input to identify candidate disease areas for the corresponding single frame of the pavement disease image, so as to segment each single frame of the pavement disease image into candidate areas, identify and calibrate the candidate areas corresponding to the pavement disease, and mark the positions and sizes of the corresponding candidate areas to generate a set of candidate disease areas corresponding to the single frame of the pavement disease image under each spectrum. The lightweight YOLOv7 improved model is built based on the PyTorch framework, and the model parameters are reduced by 40% through pruning and quantization technology, making it more suitable for running on edge computing nodes. On the edge computing node, the torch.jit.trace function is used to convert the trained model into TorchScript format to improve the inference speed of the model, and the image region feature matrix corresponding to the single frame of the pavement disease image under each spectrum is used as input and passed into the model for inference. The model outputs the category prediction probability, location coordinates (upper left and lower right corner coordinates), and size information of each candidate area. The confidence threshold is set to 0.5, and candidate areas with a prediction probability greater than 0.5 are determined to be areas with pavement defects. Their location coordinates, size, and category information are stored in a list. For example, for a frame of infrared pavement defect image, the model outputs 5 candidate areas. After screening, 3 of them are determined to be candidate defect areas. The relevant information of these 3 areas is combined into a list as the candidate defect area set for the frame image and stored in the database of the edge computing node. The database uses SQLite to facilitate local fast storage and query.

[0032] Preferably, a disease position confidence is calculated for each candidate region in the disease candidate region set corresponding to a single frame of the pavement disease image under each spectrum, so as to obtain a disease position confidence distribution corresponding to each candidate region in the disease candidate region set under each spectrum; In this embodiment of the present invention, a deep learning-based pixel-level classification method is employed to calculate the defect location confidence for each candidate region within a defect candidate region set corresponding to a single frame of a pavement defect image under various spectra. A pre-trained U-Net model, built on the TensorFlow framework, is used on edge computing nodes. Each candidate region image is cropped from the original pavement defect image and resized to 256×256 pixels. This serves as input to the U-Net model, which outputs the probability that each pixel belongs to a defect category. These probabilities are organized into a two-dimensional array, representing the defect location confidence distribution for that candidate region. For example, for a candidate region, the confidence distribution array has a shape of [256, 256], where each element in the array represents the probability of a defect at the corresponding pixel location. This process is repeated for all candidate regions within the defect candidate region set, resulting in a defect location confidence distribution for each candidate region. This distribution is then associated with other candidate region information (such as location and size) and stored in a SQLite database to provide data support for subsequent confidence optimization screening.

[0033] Preferably, confidence optimization screening is performed on each candidate area in the defect candidate area set corresponding to a single frame of a pavement defect image under each spectrum based on the defect location confidence distribution corresponding to each candidate area in the defect candidate area set under each spectrum, so as to obtain the defect location area corresponding to the single frame of the pavement defect image under each spectrum.

[0034] In an embodiment of the present invention, confidence optimization screening is performed on each candidate area in the disease candidate area set corresponding to a single frame of a pavement disease image under each spectrum based on the disease location confidence distribution corresponding to each candidate area in the disease candidate area set under each spectrum. The data is processed using Python's NumPy library and Pandas library. The disease candidate area set corresponding to a single frame of a pavement disease image under each spectrum and the disease location confidence distribution data of each candidate area are read from a SQLite database. For each frame of image under each spectrum, each candidate area in the disease candidate area set is traversed, and a combination of a non-maximum suppression (NMS) algorithm and threshold screening is used. First, a global confidence threshold is set to 0.3, and the confidence corresponding to the pixel position with a probability value less than 0.3 in the disease location confidence distribution is set to 0. Then, for each candidate area, the mean and standard deviation of its confidence distribution are calculated. If the confidence mean of a candidate area is less than 1.5 times the standard deviation, the disease confidence of this area is considered unstable and it is removed from the candidate area set. After screening, the remaining candidate areas are the corresponding disease location areas in a single frame of the road surface image under this spectrum. The information of these areas (location, size, confidence distribution, etc.) is stored in the vehicle solid-state drive in JSON format, providing accurate location information for subsequent intelligent sorting of road diseases.

[0035] Furthermore, the image processing delay is specifically controlled within ≤50ms.

[0036] Furthermore, the confidence optimization screening of each candidate area in the set of defect candidate areas corresponding to a single frame of the pavement defect image under each spectrum based on the defect position confidence distribution corresponding to each candidate area in the set of defect candidate areas under each spectrum includes: The confidence gradient of the candidate disease area corresponding to a single frame of the pavement disease image under each spectrum is calculated based on the confidence distribution of the disease position corresponding to each candidate area in the candidate disease area set under each spectrum; In an embodiment of the present invention, the confidence gradient of the candidate disease area corresponding to a single frame of the pavement disease image under each spectrum is calculated based on the confidence distribution of the disease position corresponding to each candidate area in the candidate disease area set under each spectrum, and the data is processed using the NumPy library of Python. The confidence distribution data of the disease position of each candidate area is read from the JSON file storing the information of the candidate disease area. It is assumed that the candidate disease area set of a single frame of the pavement disease image under a certain spectrum contains 10 candidate areas, and the confidence distribution of the disease position of each candidate area is an array of length 100, which represents the possibility of the existence of diseases at different positions in the area. For each candidate area, the confidence gradient is calculated using the finite difference method. Taking the confidence distribution array in the candidate area as an example, the gradient of each position is calculated by the formula gradient=np.gradient(confidence_array) Value, get a gradient array of the same length as confidence_array, which reflects the change of confidence within the region. The larger the gradient value, the more drastic the change of confidence. Calculate the average confidence gradient of all candidate regions in the single frame of the image, add the elements of the gradient array corresponding to each candidate region and divide it by the number of candidate regions to get the confidence gradient of the disease candidate region of the single frame image under the spectrum. For example, the gradient values of the 10th position of 10 candidate regions are [0.1, 0.2, 0.15, ...] respectively. Add these values and divide them by 10 to get the average gradient value of the position. Repeat this operation for all positions to finally get an average confidence gradient array of length 100, which represents the confidence gradient of the disease candidate region of the single frame image under the spectrum and stores it in a CSV file. The file contains fields such as spectrum type, image frame number, and confidence gradient value of each position.

[0037] Preferably, confidence optimization screening is performed on the defect position confidence distribution corresponding to each candidate area in the defect candidate area set under each spectrum based on the confidence gradient of the defect candidate area corresponding to the single frame of the pavement defect image under each spectrum. If the corresponding defect position confidence distribution is greater than or equal to the confidence gradient of the defect candidate area, the corresponding candidate area in the defect candidate area set is screened out as the defect position area corresponding to the single frame of the pavement defect image under the spectrum, and the candidate area whose confidence distribution is less than the confidence gradient of the defect candidate area is screened out to obtain the defect position area corresponding to the single frame of the pavement defect image under each spectrum.

[0038] In an embodiment of the present invention, confidence optimization screening is performed on the disease position confidence distribution corresponding to each candidate area in the disease candidate area set under each spectrum based on the disease candidate area confidence gradient corresponding to a single frame of a pavement disease image under each spectrum, so as to read the confidence gradient data in the CSV file and the disease candidate area confidence distribution data in the JSON file by using Python's pandas library. For each frame image under each spectrum, each candidate area in its disease candidate area set is traversed. Taking a frame image under a certain visible light spectrum as an example, assuming that the frame image has 8 candidate areas, for one of the candidate areas, its disease position confidence distribution array is compared element by element with the disease candidate area confidence gradient array of the corresponding frame. If the element value at a certain position in the confidence distribution array is greater than or equal to the element value at the corresponding position in the gradient array, it is considered that the disease confidence at this position is higher. , retain the information of the candidate area at that position; conversely, if it is less than, remove the information of that position from the candidate area. After traversing all positions of a candidate area, if the number of remaining valid positions of the candidate area is greater than 0, filter it out as the disease location area corresponding to the single frame of the pavement disease image under the spectrum; if the number of valid positions is 0, delete the candidate area from the candidate area set. Perform this operation on all candidate areas of the frame image to obtain the disease location area of the frame image under the spectrum. Repeat the above screening process for all image frames under all spectra, and finally obtain the disease location area corresponding to the single frame of the pavement image under each spectrum. The screening results are stored in a new JSON file format. The file contains detailed information such as spectrum type, image frame number, disease location area coordinates, and retained confidence distribution, providing accurate data support for the subsequent accurate identification and sorting of pavement diseases.

[0039] Furthermore, the disease sorting decision module includes the following functions: The image scale corresponding to the single frame of the pavement image under each spectrum is obtained through the corresponding disease position area in the single frame of the pavement image under each spectrum; In an embodiment of the present invention, the image scale corresponding to a single frame of the road surface image under each spectrum is obtained by using the corresponding defect position area in a single frame of the road surface image under each spectrum. In the vehicle-mounted detection system, the image acquisition device has been accurately calibrated when installed, and the image scale information is pre-stored in the device parameter configuration file. The file adopts JSON format and contains the correspondence between the spectrum type (visible light, infrared light, etc.), the image frame number and the scale. The configuration file is read by using the Python json library. For example, when processing a frame of road surface image under an infrared spectrum, the statement with open('image_scale_config.json', 'r') as f: config = json.load(f) reads the file contents and extracts the corresponding scale from the configuration data based on the image frame number. Assuming the image frame number is IR_005, the configuration file shows that 1 pixel represents an actual length of 0.05 meters. The obtained image scale information is stored together with other metadata of the frame (such as acquisition time and acquisition location coordinates) in the "image metadata table" of the SQLite database. The table structure contains fields such as image frame number, spectral type, image scale, acquisition time, and acquisition location latitude and longitude, so that it can be called for subsequent feature analysis.

[0040] Preferably, a 12-dimensional feature analysis of crack damage is performed on the corresponding defect location area within a single frame of the pavement image under each spectrum based on the corresponding image scale to extract a 12-dimensional feature set of crack damage corresponding to each single frame of the pavement image, including the maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the defect area, crack distribution density, perimeter of the defect area, shape complexity of the defect area, grayscale mean of the defect area, and grayscale standard deviation of the defect area; In an embodiment of the present invention, a 12-dimensional feature analysis of crack disease is performed on the corresponding disease location area in a single frame of a pavement image under each spectrum based on the corresponding image scale to extract a 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image. The image scale is read from the "image metadata table" of the SQLite database, and the coordinate information of the disease location area is obtained from the "disease feature" table. The coordinate information is processed by using the OpenCV library and NumPy library of Python. For the maximum width of the crack, a morphological opening operation is first performed on the diseased area image to remove noise. The cv2.morphologyEx() function is used to adopt a 3×3 rectangular structure element, and then the cv2.findCo is used. The ntours() function is used to find the outline of the crack area, and the bounding rectangle is calculated by the cv2.boundingRect() function. The length of the short side of the bounding rectangle is multiplied by the image scale to obtain the actual maximum width of the crack. When calculating the total length of the crack, the skeletonize() function of the Scikit-Image library is used to skeletonize the binary image of the diseased area and convert it into a single-pixel wide line. The number of pixels on the line is counted by the np.count_nonzero() function, and then multiplied by the image scale to obtain the actual total length of the crack. For the crack tortuosity, the coordinates of the starting and ending pixels of the crack outline are first obtained, and the Euclidean distance formula (np.lina The lg.norm() function is used to calculate the straight-line distance between two points, and the ratio is calculated based on the total length of the cracks. When counting the area of the diseased area, the np.count_nonzero() function is used to count the number of pixels with a pixel value of 255 in the binary image of the diseased area, and the actual area is obtained by multiplying it by the square of the image scale. The crack distribution density is calculated by dividing the total length of the cracks by the area of the diseased area. The boundary is detected by the cv2.Canny() function, and the contour is obtained by the cv2.findContours() function. The len() function is used to count the number of contour pixels and multiply it by the image scale to obtain the perimeter of the diseased area. The shape complexity is calculated according to the formula "shape complexity = perimeter of diseased area² / area of diseased area". "Calculate the shape complexity by using the cv2.cvtColor() function to convert the diseased area image into a grayscale image, and use the np.mean() function and np.std() function to calculate the grayscale mean and grayscale standard deviation respectively. The calculated 12-dimensional feature data (maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, diseased area area, crack distribution density, diseased area perimeter, diseased area shape complexity, diseased area grayscale mean, diseased area grayscale standard deviation) are combined into a list and stored in the "12-dimensional feature set table" of the SQLite database. The table structure contains the image frame number, spectrum type and 12 feature fields.

[0041] Preferably, a maintenance urgency assessment is performed on the 12-dimensional feature set of crack damage corresponding to each single frame of the pavement image based on the fuzzy analytic hierarchy process to obtain a maintenance urgency index corresponding to each single frame of the pavement image; In an embodiment of the present invention, a maintenance emergency assessment is performed on the 12-dimensional feature set of crack damage corresponding to each single frame of each pavement image based on the fuzzy hierarchical analysis method, so as to construct a fuzzy judgment matrix by using the numpy library and pandas library of Python. First, the importance comparison relationship of each feature factor in the 12-dimensional feature set is determined, and a 1-9 scaling method is adopted, where 1 indicates that two feature factors are equally important, and 9 indicates that one feature factor is extremely more important than the other. For example, for the maximum width of the crack and the total length of the crack, if it is believed that the maximum width of the crack has a slightly greater impact on the severity of the pavement disease than the total length of the crack, a value of 3 can be assigned. By pairwise comparison, a 12×12 fuzzy judgment matrix is constructed, and the first factor in the matrix is 9. OK Elements of a column Indicates the The characteristic factors The relative importance of the characteristic factors, and meet the =1 / , =1, use the square root method to calculate the weight of each characteristic factor, first calculate , then Find the 12th root and we get , and finally normalize and calculate , obtain the fuzzy weight value of each feature factor, read the actual value of the 12-dimensional feature of each road image frame from the "12-dimensional feature set table" in the SQLite database, and perform minimum-maximum normalization processing on it. The formula is ,in is the original value, and are the minimum and maximum values of the feature factor in all image frames respectively, and the normalized feature value array is multiplied by the corresponding elements of the fuzzy weight value array and then summed, that is, , the maintenance urgency index corresponding to each single frame of the pavement image is obtained and stored in the "maintenance urgency table" of the SQLite database. The table structure contains fields such as image frame number, spectrum type, and maintenance urgency index.

[0042] Preferably, maintenance urgency sorting planning is performed on each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame, so as to generate a road maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting single frame.

[0043] In an embodiment of the present invention, maintenance urgency sorting planning is performed on each pavement image frame based on the maintenance urgency index corresponding to each pavement image frame, the maintenance urgency index, spectral type and image frame number information of each pavement image frame are read from the "maintenance urgency table" of the SQLite database, the data is read as a data frame using the Python pandas library, and the sort_values() function is used to sort in descending order according to the maintenance urgency index column, for example, df.sort_values(by='maintenance urgency index, ascending=False, inplace=True), so that the image frame with a high urgency index is placed in the front, and after the sorting is completed, the data composition list of the image frame number column is extracted, the coordinate information of the disease location area corresponding to each image frame is queried from the "disease feature" table, and the disease location area is divided into corresponding grids by using the Python numpy library, the coordinates of the center point of each grid are calculated as the disease location point, the disease location point composition sequence of each image frame is stored in the "disease location point table", and the Shapefile format is read by using the Python geopandas library. GIS map data, including road network layer and traffic condition layer, reads the disease location point sequence of each image frame from the "disease location point table", uses the Python-based ant-colony-optimization library to implement the ant colony optimization algorithm, sets the number of ants to 50, the pheromone evaporation coefficient to 0.1, the heuristic factor to 2, the expected heuristic factor to 3, the maximum number of iterations to 100, takes the disease location point as the starting and passing node of the ant, and the road connection relationship in the road network topology structure as the feasible path. According to the real-time traffic in the traffic condition layer, the ant colony optimization algorithm is used. Based on traffic flow data, the path length of the congested road section is multiplied by the congestion coefficient (for example, the coefficient is 2 when congestion is severe). After iterative calculation, the optimal driving path corresponding to each point sequence is obtained. The node coordinates in the path, the names of the roads passed, and other information are compiled into a list. This is the road maintenance fleet sorting and scheduling sequence corresponding to the single frame of road image sorting. This list is stored in the "Maintenance Fleet Scheduling Table" in the SQLite database. The table structure contains fields such as image frame number, scheduling sequence information (data such as path nodes are stored in JSON format), and estimated travel time, providing accurate planning solutions for the actual scheduling of the maintenance fleet.

[0044] Furthermore, the 12-dimensional feature analysis of crack damage in the corresponding damage location area in a single frame of the pavement image under each spectrum based on the corresponding image scale includes: Binarization is performed on the corresponding diseased location area in a single frame of the road surface image under each spectrum to generate a binary image of the corresponding diseased area under each spectrum; In the embodiment of the present invention, the corresponding defect location area in a single frame of the road surface image under each spectrum is binarized, and the operation is performed using the Python OpenCV library. The coordinate information of the defect location area in a single frame of the road surface image under each spectrum is read from the SQLite database, and the corresponding area in the original road surface disease image is cropped out. Assuming that the coordinates of the defect location area under a certain frame of visible light spectrum are (100, 100) in the upper left corner and (300, 300) in the lower right corner, the cv2.rectangle() function is used to locate the area on the original image, and then cv2.cvtCo The lor() function converts the cropped image into a grayscale image and binarizes it using a fixed threshold method. The cv2.threshold() function is used to set the threshold to 127, and pixels with grayscale values greater than 127 are set to 255 (white, representing the diseased area), and pixels with grayscale values less than or equal to 127 are set to 0 (black, representing the background area). This generates a binary image of the diseased area and saves the processed binary image in PNG format in a local folder. The file name contains information such as the spectral type and image frame number, such as "visible light_001_disease binarization.png", which is convenient for subsequent calls.

[0045] Preferably, the corresponding maximum crack width, total crack length, crack tortuosity, area of the damaged area, crack distribution density, perimeter of the damaged area, and shape complexity of the damaged area are calculated based on the corresponding binary image of the damaged area under each spectrum and combined with the corresponding image scale; In an embodiment of the present invention, the corresponding maximum crack width, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, and shape complexity of the diseased area are calculated according to the binary image of the diseased area corresponding to each spectrum and combined with the corresponding image scale. By using Python's OpenCV library and NumPy library, for the maximum crack width, morphological opening operation is used to remove noise, and then the cv2.findContours() function is used to find the outline of the crack area, and the cv2.boundingRect() function is used to calculate the outline circumscribed rectangle, and the length of the short side of the circumscribed rectangle is used as the approximate value of the maximum crack width. It is converted in combination with the image scale (assuming that 1 pixel represents an actual length of 0.1 cm). When calculating the total crack length, the skeletonize() function of the Scikit-Image library is used to skeletonize the binary image, and the np.count_nonzero() function is used to count the number of skeleton line pixels. The actual length is converted to the scale. To calculate the tortuosity of the crack, the starting and ending pixels of the crack contour are first obtained using the cv2.findContours() function. The straight-line distance between the two points is calculated using the Euclidean distance formula (np.linalg.norm() function). The ratio is calculated based on the calculated total length of the crack. When counting the area of the diseased area, the np.count_nonzero() function is used to count the number of pixels in the diseased area in the binary image. The actual area is converted according to the scale. The crack distribution density is calculated and the total crack length is divided by the area of the diseased area. The boundary is detected using the cv2.Canny() function. The contour is then obtained using the cv2.findContours() function. The len() function is used to count the number of contour pixels and convert it to the perimeter of the diseased area. Finally, the shape complexity is calculated according to the formula "shape complexity = perimeter of diseased area² / area of diseased area". All calculation results are recorded in the "Disease Characteristics" table of the SQLite database, corresponding to each spectrum and image frame number.

[0046] Preferably, the crack centerline within the defect location area corresponding to the visible spectrum band in a single frame of the road surface image under each spectrum is divided into several small segments at equal intervals, and the vertical widths of the cracks corresponding to the several small segments are calculated respectively, and the average value is calculated at the same time to obtain the corresponding average crack width; In an embodiment of the present invention, the crack center line in the defect position area corresponding to the visible spectrum band in a single frame of the pavement image under each spectrum is divided into several small segments by taking points at equal intervals, and the vertical widths of the cracks corresponding to the several small segments are calculated respectively, and the average value is calculated at the same time to obtain the corresponding average crack width. For the binary image of the defect area in the visible spectrum band, Python's OpenCV library and SciPy library are used. First, the crack area is thinned by the cv2.ximgproc.thinning() function to obtain a crack center line with a single pixel width. Assuming that the length of the crack center line is 100 pixels and the points are taken at equal intervals of 5 pixels, 20 small segments can be divided. For each For a small segment, the number of white pixels (representing cracks) is counted by scanning pixels in the direction perpendicular to the center line as the vertical width of the small segment. For example, if 10 consecutive white pixels are scanned in the vertical direction of a small segment, the vertical width of the small segment is 10 pixels. After calculating the vertical widths of all small segments in turn, the np.mean() function is used to calculate the average of these width values to obtain the average crack width. Assuming that the vertical widths of the 20 small segments are [8, 10, 9, ...] respectively, the average value is calculated to be 9.2 pixels. Combined with the image scale (1 pixel represents an actual length of 0.1 cm), the actual average crack width is converted to 0.92 cm. The result is recorded in the record of the corresponding image frame in the "Disease Characteristics" table.

[0047] Preferably, the temperature distribution difference between each equally spaced point is determined by the defect location area of the corresponding infrared spectral band in a single frame of the road surface image under each spectrum, and the crack depth corresponding to each equally spaced point is estimated based on the temperature distribution difference between each equally spaced point. At the same time, the maximum value is determined based on the crack depth corresponding to each equally spaced point and the average value is calculated to obtain the corresponding maximum crack depth and average crack depth; In an embodiment of the present invention, the temperature distribution difference between each equally spaced point is determined by the defect location area corresponding to the infrared spectral band in a single frame of the road surface image under each spectrum, and the crack depth corresponding to each equally spaced point is estimated based on the temperature distribution difference between each equally spaced point. At the same time, the maximum value is determined and the average value is calculated based on the crack depth corresponding to each equally spaced point to obtain the corresponding maximum crack depth and average crack depth. The temperature data of the defect location area is read from the file storing the infrared spectral image data. It is assumed that the temperature data is stored in a matrix form, and each element corresponds to the temperature value of a pixel point. In the crack area, points are taken on the crack path in the same equally spaced manner as before. For two adjacent equally spaced points, the temperature difference between them is calculated. The pre-established temperature-depth relationship model (the model is fitted based on a large amount of experimental data, for example, the temperature difference and crack depth satisfy a linear relationship: depth = coefficient × temperature difference + constant, and the coefficient and constant are determined through experiments) is used to estimate the crack depth corresponding to each equally spaced point. Assuming that the temperature difference between two adjacent points is 5°C, the corresponding crack depth is calculated to be 2 cm according to the corresponding model through experiments (for example, depth = 0.3 × temperature difference + 0.5). After calculating the crack depths of all equally spaced points in turn, the np.max() function is used to find the maximum value, and the np.mean() function is used to calculate the average value to obtain the maximum crack depth and the average crack depth. The results are recorded in the "Disease Characteristics" table of the SQLite database to improve the disease feature data of the image frame.

[0048] Preferably, the regional grayscale mean and standard deviation of the corresponding defect location area in a single frame of the road surface image under each spectrum are calculated to calculate the grayscale mean and grayscale standard deviation corresponding to all pixels in the defect location area, and obtain the corresponding grayscale mean and grayscale standard deviation of the defect area; In the embodiment of the present invention, the regional grayscale mean and standard deviation of the corresponding disease location area in a single frame of the road surface image under each spectrum are calculated to calculate the grayscale mean and grayscale standard deviation corresponding to all pixels in the disease location area. By using Python's OpenCV library and NumPy library, the disease location area image is cropped from the original road disease image, converted into a grayscale image by the cv2.cvtColor() function, and the average grayscale value of all pixels in the grayscale image is calculated using the np.mean() function. To the grayscale mean of the diseased area, for example, a grayscale image of a diseased area has a total of 1000 pixels, and the total grayscale value is 50000, then the grayscale mean is 50000 / 1000=50, and the standard deviation of the grayscale value is calculated by using the np.std() function to reflect the discrete degree of the pixel grayscale value. Assuming that the calculated standard deviation is 10, the grayscale mean and grayscale standard deviation of the diseased area are recorded in the "Disease Characteristics" table of the SQLite database to provide data support for analyzing the grayscale characteristics of the diseased area. These data can be used to distinguish different types of pavement diseases.

[0049] Preferably, the above corresponding maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, grayscale mean of the diseased area and grayscale standard deviation of the diseased area are merged into a 12-dimensional feature set of crack diseases corresponding to each single frame of the pavement image.

[0050] In an embodiment of the present invention, the corresponding maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, grayscale mean of the diseased area and grayscale standard deviation of the diseased area are merged into a 12-dimensional feature set of crack diseases corresponding to each single frame of the pavement image, and the above 12 feature data of the corresponding image frame under each spectrum are read from the "Disease Features" table of the SQLite database by using the Python NumPy library. These data are arranged in a fixed order to form a one-dimensional array with a length of 12, such as [maximum crack width, average crack width, maximum crack depth, ..., grayscale standard deviation of the diseased area]. The one-dimensional array is taken as a sample and stored in a local NumPy array file. The file name contains the spectrum type and image frame number, such as "visible light_001_12 At the same time, a "12-dimensional feature set index table" is created in the database to record the feature set file path, spectral type, image frame number and other information corresponding to each image frame, so as to facilitate the subsequent intelligent sorting, classification and analysis of pavement diseases based on these feature sets, and provide a comprehensive data basis for pavement maintenance decisions.

[0051] Furthermore, the calculation of the corresponding maximum crack width, total crack length, crack tortuosity, area of the damaged area, crack distribution density, perimeter of the damaged area, and shape complexity of the damaged area based on the corresponding binary image of the damaged area under each spectrum and in combination with the corresponding image scale includes: The corresponding crack area is highlighted by binarizing the image of the diseased area under each spectrum and the corresponding noise and small branches are removed by morphological operations. Then, the circumscribed rectangle of the crack area is calculated, and the length of the short side of the circumscribed rectangle is approximately regarded as the corresponding maximum width. The corresponding maximum width of the crack is converted according to the corresponding image scale. In an embodiment of the present invention, the corresponding crack area portion is highlighted by binarizing the corresponding diseased area under each spectrum and using morphological operations to remove the corresponding noise and small branches. Then, the circumscribed rectangle of the crack area portion is calculated, and the length of the short side corresponding to the circumscribed rectangle is approximately regarded as the corresponding maximum width. The corresponding maximum width of the crack is converted according to the corresponding image scale. The operation is performed by using Python's OpenCV library. First, the stored binary image of the diseased area under each spectrum is read. The pixel value of the crack area in the image is 255, and the pixel value of the background area is 0. The morphological opening operation is used to remove noise and small branches. Specifically, the cv2.morphologyEx() function is used to use a rectangular structure element of size 3×3. , perform an opening operation to eliminate small areas that are not tightly connected to the main body of the crack. Then, use the cv2.findContours() function to find the contours in the image and obtain the contour information of the crack area. Then use the cv2.boundingRect() function to calculate the circumscribed rectangle of the contour, and obtain the upper left corner coordinates, width and height of the circumscribed rectangle. The length of the short side of the circumscribed rectangle is used as the approximate value of the maximum width of the crack. Assuming that the image scale is 1 pixel, the actual length is 0.1 cm. If the length of the short side of the circumscribed rectangle is 5 pixels, the converted maximum width of the crack is 5×0.1=0.5 cm. The result is recorded in the corresponding CSV file, which contains fields such as spectrum type, image frame number, and maximum width of the crack.

[0052] Preferably, skeletonization is performed on the binary image of the diseased area corresponding to each spectrum to simplify the corresponding crack area into a line with a single pixel width, and the total number of pixels on the line is counted, and the total length of the corresponding crack is calculated in combination with the corresponding image scale; In an embodiment of the present invention, skeletonization is performed on the binary image of the diseased area corresponding to each spectrum to simplify the corresponding crack area into a line with a single pixel width, and the total number of pixels on the line is counted. At the same time, the corresponding total crack length is converted in combination with the corresponding image scale. The skeletonization process is performed on the binary image of the diseased area using the skeletonize() function in the Scikit-Image library to convert the crack area into a skeleton image with a single pixel width. The total number of pixels with a pixel value of 1 (representing skeleton lines) in the skeleton image is counted using the np.count_nonzero() function to obtain the pixel number value. Assuming that the image scale is 1 pixel representing an actual length of 0.1 cm, if the total number of pixels counted is 200, the converted total crack length is 200×0.1=20 cm. The total crack length information is stored together with data such as the spectrum type and image frame number in the "Disease Characteristics" table of the SQLite database for subsequent query and analysis.

[0053] Preferably, the corresponding crack starting pixel point and crack ending pixel point are obtained by binarizing the corresponding diseased area under each spectrum, so as to calculate the corresponding straight-line distance between the crack starting pixel point and the crack ending pixel point, and the corresponding crack tortuosity is calculated according to the ratio of the total crack length to the straight-line distance between the crack starting point and the crack ending point; In an embodiment of the present invention, the corresponding crack starting pixel point and crack ending pixel point are obtained by binarizing the corresponding diseased area image under each spectrum, so as to calculate the corresponding straight-line distance between the crack starting pixel point and the crack ending pixel point, and the corresponding crack tortuosity is obtained according to the ratio of the total crack length to the straight-line distance between the crack starting point and the end point. This operation is implemented by using Python's NumPy library and OpenCV library. First, the contour of the crack area is obtained by using the corresponding cv2.findContours() function. For each contour, its starting point and end point are used as the crack starting pixel point and the crack ending pixel point. The Euclidean distance formula is used to calculate the straight-line distance between the starting pixel point and the ending pixel point by the np.linalg.norm() function. Assuming that the coordinates of the starting pixel point are (10, 10) and the coordinates of the ending pixel point are (50, 50), the straight-line distance is Pixels, read the total length of the crack corresponding to the image frame from the "Disease Characteristics" table in the SQLite database. Assuming it is 100 pixels, calculate it using the formula "crack tortuosity = total crack length / straight-line distance", that is, 100 / ( )≈1.77, and update the crack tortuosity results to the corresponding records in the "Disease Characteristics" table to provide a quantitative indicator for evaluating crack morphology.

[0054] Preferably, the total number of pixels in the binary image of the diseased area corresponding to each spectrum is counted, and the area of the corresponding diseased area is converted according to the corresponding image scale. At the same time, the corresponding crack distribution density is calculated according to the ratio between the total length of the corresponding cracks and the area of the diseased area; In an embodiment of the present invention, the total number of pixels in the binary image of the damaged area corresponding to each spectrum is counted, and the corresponding damaged area is converted according to the corresponding image scale. At the same time, the corresponding crack distribution density is calculated according to the ratio between the corresponding total crack length and the area of the damaged area. The np.count_nonzero() function is used to count the total number of pixels with a pixel value of 255 (representing the damaged area) in the binary image of the damaged area. Assuming that the image scale is 1 pixel represents an actual area of 0.01 square centimeters, if the total number of pixels counted is 1000, the converted damaged area is 1000×0.01=10 square centimeters. The total crack length corresponding to the image frame is read from the "Damage Features" table in the SQLite database, assuming it is 20 centimeters. The total crack length is calculated using the formula "crack distribution density = total crack length / damaged area", that is, 20 / 10=2 centimeters / square centimeter. The crack distribution density result is recorded in the "Damage Features" table to provide data support for analyzing the severity of the disease.

[0055] Preferably, boundary detection is performed on the binarized image of the diseased area corresponding to each spectrum to obtain the boundary of the diseased area after binarization, and the total number of pixels corresponding to the boundary of the diseased area is counted, and the corresponding perimeter of the diseased area is calculated according to the corresponding image scale; In an embodiment of the present invention, boundary detection is performed on the binarized image of the diseased area corresponding to each spectrum to obtain the boundary of the diseased area after binarization, and the total number of pixels on the boundary of the diseased area is counted. At the same time, the corresponding perimeter of the diseased area is converted according to the corresponding image scale. Boundary detection is performed by using the cv2.Canny() function of the OpenCV library, and the low threshold is set to 50 and the high threshold is set to 150 to obtain the edge image of the diseased area. The contour in the edge image is found by the cv2.findContours() function to obtain the contour information of the boundary of the diseased area. The total number of pixels on the contour is counted using the len() function to obtain the number of boundary pixels. Assuming that the image scale is 1 pixel represents an actual length of 0.1 cm, if the total number of pixels counted is 150, the converted perimeter of the diseased area is 150×0.1=15 cm, and the perimeter information of the diseased area is added to the corresponding record in the "Disease Characteristics" table of the SQLite database to improve the disease feature data.

[0056] Preferably, the shape complexity of the corresponding diseased area is calculated based on the ratio between the square of the perimeter of the diseased area and the area of the diseased area.

[0057] In an embodiment of the present invention, the shape complexity of the corresponding damaged area is calculated based on the ratio between the square of the perimeter of the damaged area and the area of the damaged area. The perimeter and area data of the damaged area of the corresponding image frame under each spectrum are read from the "Damage Characteristics" table of the SQLite database. Assuming that the perimeter of the damaged area of a certain image frame is 15 cm and the area of the damaged area is 10 square centimeters, the shape complexity result obtained by calculation is recorded in the "Damage Characteristics" table by the formula "Shape Complexity = Perimeter of Damaged Area² / Area of Damaged Area", that is, 15² / 10=22.5. The larger the shape complexity value, the more complex the shape of the damaged area, which provides an important quantitative basis for the classification and evaluation of pavement diseases.

[0058] Furthermore, the maintenance urgency assessment is performed on the 12-dimensional feature set of crack damage corresponding to each single frame of the pavement image based on the fuzzy analytic hierarchy process, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained, including: Based on the fuzzy analytic hierarchy process, the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image is used to construct the fuzzy judgment matrix of crack disease corresponding to each single frame of the pavement image; In an embodiment of the present invention, a fuzzy judgment matrix of crack disease corresponding to each single frame of a pavement image is constructed based on a fuzzy hierarchical analysis method for a 12-dimensional feature set of crack disease corresponding to each single frame of a pavement image. The operation is performed by using the numpy library and the pandas library of Python. First, each characteristic factor in the 12-dimensional feature set is determined, including the maximum width of the crack, the average width of the crack, etc., and a fuzzy scaling matrix of the 1-9 scaling method is established, where 1 indicates that two characteristic factors are equally important, 9 indicates that one characteristic factor is extremely more important than the other, and the intermediate values represent different degrees of importance. For the 12-dimensional feature set of a single frame of a pavement image, each characteristic factor is compared with the other characteristic factors in order. For example, when comparing the maximum width of the crack with the total length of the crack, according to the actual situation of the pavement disease and expert experience, if it is considered that the maximum width of the crack is slightly more important than the total length of the crack, it is assigned a value of 3 according to the scaling method. Through pairwise comparison, a 12×12 matrix is constructed. The first characteristic factor in the matrix is 9. OK Elements of a column Indicates the The characteristic factors The relative importance of the characteristic factors, and meet the =1 / , =1, assuming that all comparisons have been completed, a fuzzy judgment matrix is obtained as follows (partial example): , store the matrix as a numpy array and save it as a CSV file using the pandas library. The file name contains the spectrum type and image frame number, such as "visible light_001_fuzzy judgment matrix.csv", which is convenient for subsequent calls and analysis.

[0059] Preferably, based on the crack disease fuzzy judgment matrix corresponding to each single frame of the pavement image, weighted fuzzy reasoning is performed on each characteristic factor in the 12-dimensional feature set corresponding to the crack disease, so as to generate a fuzzy weight value corresponding to each characteristic factor in each single frame of the pavement image; In the embodiment of the present invention, the weighted fuzzy reasoning is performed on each characteristic factor in the 12-dimensional feature set of the corresponding crack disease based on the fuzzy judgment matrix of the crack disease corresponding to each single frame of the pavement image, and the weight is calculated by using the square root method. The calculation process is implemented using the Python numpy library. First, the previously saved fuzzy judgment matrix CSV file is read and converted into a numpy array. The product of the elements of each row of the matrix is calculated. For example, for the matrix Row, calculate , get an array of length 12 , for the array Find the 12th root of each element in , get the array , the array Perform normalization and calculate , get the fuzzy weight value array of each feature factor , assuming that the fuzzy judgment matrix of a single frame of a road image is calculated, the fuzzy weight value array obtained is =[0.15, 0.12, 0.10, ⋯], which correspond to the weights of 12 characteristic factors such as the maximum crack width and the average crack width, respectively. These weight values are stored in the “feature weight table” of the SQLite database together with information such as the spectrum type and image frame number, so as to facilitate the use in subsequent maintenance emergency assessment.

[0060] Preferably, a maintenance urgency assessment is performed on the actual values of each characteristic factor in the 12-dimensional feature set of the corresponding crack disease based on the fuzzy weight value corresponding to each characteristic factor in each single frame of the pavement image, so as to obtain the maintenance urgency index corresponding to each single frame of the pavement image.

[0061] In an embodiment of the present invention, a maintenance emergency assessment is performed based on the actual values of each characteristic factor in the 12-dimensional feature set of the corresponding crack disease based on the fuzzy weight value corresponding to each characteristic factor in each single frame of the pavement image. The fuzzy weight value of each characteristic factor in each single frame of the pavement image is read from the "feature weight table" of the SQLite database by using the Python numpy library for calculation. At the same time, the actual value of the 12-dimensional feature set of the corresponding image frame is read from the "disease feature" table. The actual value of each characteristic factor is standardized using the minimum-maximum standardization method. The formula is: ,in is the original value, and The minimum and maximum values of the feature factor in all image frames are obtained respectively, and the standardized feature value array is obtained. , calculate the maintenance urgency index, multiply the standardized feature value array with the corresponding fuzzy weight value array and sum them up, that is, Assume that after calculation, the maintenance urgency index of a single pavement image frame is 0.75. This index, along with information such as spectral type and image frame number, is recorded in the "Maintenance Urgency Table" of the SQLite database. By comparing the maintenance urgency indexes of different image frames, the priority of pavement disease maintenance can be determined, providing a quantitative basis for pavement maintenance decisions.

[0062] Furthermore, performing maintenance emergency sorting planning on each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame includes: Based on the maintenance urgency index corresponding to each road surface image single frame, the corresponding road surface image single frame is sorted and sorted according to emergency priority to generate a corresponding road surface maintenance emergency priority sorting image frame sequence; In the embodiment of the present invention, the maintenance urgency index corresponding to each road surface image frame is used to sort the corresponding road surface image frame in an emergency priority manner. The maintenance urgency index, spectral type and image frame number information of each road surface image frame are read from the "maintenance urgency table" of the SQLite database. The data is read as a data frame using the Python pandas library. The sort_values() function of the data frame is used to sort the data in descending order according to the maintenance urgency index column, for example, df.sort_values(by='maintenance urgency index', ascending=False, inplace=True). The pavement image frames with high urgency index are arranged at the front of the sequence, and those with low urgency index are arranged at the back. After the sorting is completed, the data of the image frame number column is extracted to form a list. This list is the pavement maintenance emergency priority sorting image frame sequence. Assume that the image frame number column in the sorted data frame is ['001', '003', '002', ...]. The list is stored in a Python pickle file named "Pavement Maintenance Emergency Priority Sorting Image Frame Sequence.pkl". At the same time, a "sorting sequence index table" is created in the SQLite database to record the storage path, generation time and other information of the sequence file to facilitate subsequent rapid call and management of the sequence data.

[0063] Preferably, a disease location search is performed on each pavement image sorting single frame in the pavement maintenance emergency priority sorting image frame sequence to generate a disease location point sequence corresponding to the pavement image sorting single frame; In the embodiment of the present invention, a disease location search is performed on each pavement image sorting single frame in the pavement maintenance emergency priority sorting image frame sequence, and an image frame sequence list is read from the "pavement maintenance emergency priority sorting image frame sequence.pkl" file. Each image frame number in the list is traversed, and for each image frame number, the coordinate information of the disease location area corresponding to the image frame is queried from the "disease characteristics" table of the SQLite database. Assuming that an image frame number is 001, its disease location area coordinates are (100, 100) in the upper left corner and (300, 300) in the lower right corner, the disease location in the area is further refined, and the image frame number is used. Python's numpy library divides the area into 10×10 grids and calculates the coordinates of the center point of each grid as the defect location point. For example, the coordinates of the center point of the first grid are (110, 110). The coordinates of the center points of all grids are calculated in sequence to form a list. This list is the sequence of defect location points corresponding to the single frame of the road surface image sorting. The sequence of defect location points corresponding to each image frame number is stored in a newly created "Disease Location Point Table" in the SQLite database. The table structure contains fields such as the image frame number, point sequence number, and point coordinates (x coordinate, y coordinate) for subsequent call-in during path planning.

[0064] Preferably, maintenance fleet path planning is performed on the sequence of disease location points corresponding to a single frame of road surface image sorting by using an ant colony optimization algorithm combined with a GIS map, so as to combine the corresponding disease location points in the sequence with the GIS map, and plan the optimal driving path for the corresponding maintenance fleet while fully considering the corresponding road network topology and traffic conditions, so as to generate a road maintenance fleet sorting and scheduling sequence corresponding to the single frame of road surface image sorting.

[0065] In the embodiment of the present invention, the maintenance team path planning is performed on the sequence of disease location points corresponding to the single frame of road surface image sorting by combining the ant colony optimization algorithm with the GIS map, so as to combine the corresponding disease location points in the sequence with the GIS map, and fully consider the corresponding road network topology and traffic conditions to plan the optimal driving path for the corresponding maintenance team, so as to read the GIS map data by using the Python geopandas library, and the map data adopts Shapefile The format contains information such as the road network layer and the traffic condition layer. The sequence of disease location points corresponding to each road surface image sorting single frame is read from the "Disease Location Point Table". For each point sequence, the ant colony optimization algorithm is implemented using the Python-based ant-colony-optimization library. The number of ants is set to 50, the pheromone evaporation coefficient is 0.1, the heuristic factor is 2, the expected heuristic factor is 3, and the maximum number of iterations is 100. The disease location points are used as the starting and passing nodes of the ants, and the road connection relationship in the road network topology is used as the feasible path of the ants. At the same time, according to the real-time traffic flow data in the traffic condition layer, the path of the traffic congested section is The length is multiplied by the congestion coefficient (for example, the coefficient is 2 when congestion is severe) to simulate the impact of traffic conditions on path selection. After iterative calculations using the ant colony optimization algorithm, the optimal driving path corresponding to each point sequence is obtained. The node coordinates, road names, and other information in the optimal driving path are compiled into a list. This list is the road maintenance fleet sorting and scheduling sequence corresponding to the road image sorting single frame. The sorting and scheduling sequence of each road image sorting single frame is stored in the "Maintenance Fleet Scheduling Table" in the SQLite database. The table contains fields such as the image frame number, scheduling sequence information (data such as path nodes are stored in JSON format), and estimated travel time. This provides accurate path planning solutions and time references for the actual scheduling of the maintenance fleet.

[0066] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. An AI-based intelligent pavement disease sorting platform, characterized by: include: A front-end acquisition and identification module and a pavement disease intelligent sorting platform are connected to the same network. The front-end acquisition and identification module includes a multispectral acquisition module and an edge computing module. The pavement disease intelligent sorting platform includes a disease sorting decision module and a disease interactive display module. The multispectral acquisition module is used to use a vehicle-mounted array camera to integrate corresponding visible light / infrared sensors and collect pavement disease image frames in different spectral bands in real time at a speed of 60 km / h; perform frame extraction preprocessing on the pavement disease image frames in different spectral bands to generate standard pavement disease frames in different spectrums; The edge computing module is used to perform single-frame delay control disease area identification on the road surface disease standard frames under different spectra by deploying the corresponding lightweight YOLOv7 improved model to obtain the corresponding disease location area in the single frame of the road surface image under each spectrum; The disease sorting decision module is used to perform 12-dimensional crack disease feature analysis on the disease location area corresponding to the single frame of the pavement image under each spectrum to extract the 12-dimensional crack disease feature set corresponding to each single frame of the pavement image; perform maintenance urgency assessment on the 12-dimensional crack disease feature set corresponding to each single frame of the pavement image based on the fuzzy hierarchical analysis method to obtain the maintenance urgency index corresponding to each single frame of the pavement image; and perform maintenance emergency sorting planning on each single frame of the pavement image based on the maintenance urgency index corresponding to each single frame of the pavement image to generate a pavement maintenance fleet sorting and scheduling sequence corresponding to the sorted single frame of the pavement image; The disease interactive display module is used to display each road surface image sorting single frame on the corresponding human-computer interaction terminal to intuitively view the disease location, type and maintenance urgency index corresponding to each road surface image sorting single frame, and to issue corresponding maintenance disposal instructions on the human-computer interaction terminal according to the road surface maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting single frame, so as to execute the corresponding road surface disease maintenance fleet scheduling and planning work.

2. The AI-based intelligent pavement disease sorting platform according to claim 1 is characterized in that: The multispectral acquisition module includes the following functions: A vehicle-mounted array camera integrates corresponding visible light / infrared sensors and captures pavement damage image frames in different spectral bands in real time at a speed of 60 km / h. The pavement damage image frames in the visible light sensor band capture conventional visible pavement damage, while the pavement damage image frames in the infrared sensor band identify tiny cracks or damage hidden in the pavement based on the corresponding temperature distribution. Perform time slot frame extraction on the pavement disease image frames in different spectral bands at a frame extraction frequency of 5 seconds per frame to obtain a set of local pavement disease frames in different spectral bands. The pixel fuzziness of the local frame set of pavement defects in different spectral bands is averaged and quantified to obtain the average pixel fuzziness of the image frame in different spectral bands; Based on the average value of pixel fuzziness of image frames in different spectral bands, the pavement disease image frames in the corresponding bands are subjected to fuzzy denoising processing to obtain fuzzy denoised frames of pavement disease in different spectral bands; The fuzzy denoised frames of pavement damage in different spectral bands are normalized at the pixel level to generate standard frames of pavement damage in different spectra.

3. The AI-based intelligent pavement disease sorting platform according to claim 1 is characterized in that: The edge computing module includes the following functions: By inputting the standard frames of road surface damage under different spectra into the corresponding edge computing nodes in the local vehicle, the image processing delay corresponding to a single frame of road surface damage images under each spectrum can be controlled. By performing statistical analysis of road surface regional features on the corresponding edge computing node in the local vehicle, the edge and texture features corresponding to the road surface image area under different spectra are statistically analyzed and formed into a feature matrix. The image region feature matrix corresponding to a single frame of the road surface disease image under each spectrum is generated. By deploying the corresponding lightweight YOLOv7 improved model on the corresponding edge computing node and inputting the image region feature matrix corresponding to the single frame of the pavement disease image under each spectrum, the corresponding single frame of the pavement disease image is identified for the disease candidate area. Each single frame of the pavement disease image is segmented into candidate areas, and the candidate areas corresponding to the pavement disease are identified and calibrated. The location and size of the corresponding candidate areas are also marked to generate a set of disease candidate areas corresponding to the single frame of the pavement disease image under each spectrum. Calculate the confidence of the defect position of each candidate area in the defect candidate area set corresponding to a single frame of the pavement defect image under each spectrum to obtain the distribution of the confidence of the defect position of each candidate area in the defect candidate area set under each spectrum; Based on the confidence distribution of the defect location corresponding to each candidate area in the defect candidate area set under each spectrum, each candidate area in the defect candidate area set corresponding to a single frame of the pavement defect image under each spectrum is subjected to confidence optimization screening to obtain the corresponding defect location area in the single frame of the pavement defect image under each spectrum.

4. The AI-based intelligent pavement disease sorting platform according to claim 3 is characterized in that: The image processing delay is specifically controlled within ≤50ms.

5. The AI-based intelligent pavement disease sorting platform according to claim 3 is characterized in that: The confidence optimization screening of each candidate area in the disease candidate area set corresponding to a single frame of the pavement disease image under each spectrum based on the disease position confidence distribution corresponding to each candidate area in the disease candidate area set under each spectrum includes: The confidence gradient of the candidate disease area corresponding to a single frame of the pavement disease image under each spectrum is calculated based on the confidence distribution of the disease position corresponding to each candidate area in the candidate disease area set under each spectrum; Based on the confidence gradient of the defect candidate area corresponding to a single frame of the pavement defect image under each spectrum, the confidence distribution of the defect position corresponding to each candidate area in the defect candidate area set under each spectrum is optimized and screened. If the corresponding defect position confidence distribution is greater than or equal to the confidence gradient of the defect candidate area, the corresponding candidate area in the defect candidate area set is screened out as the defect position area corresponding to the single frame of the pavement defect image under that spectrum, and the candidate areas with a confidence distribution less than the confidence gradient of the defect candidate area are screened out to obtain the defect position area corresponding to the single frame of the pavement defect image under each spectrum.

6. The AI-based intelligent pavement disease sorting platform according to claim 1 is characterized in that: The disease sorting decision module includes the following functions: The image scale corresponding to the single frame of the road surface image under each spectrum is obtained through the corresponding disease position area in the single frame of the road surface image under each spectrum; Based on the corresponding image scale, a 12-dimensional feature analysis of crack disease is performed on the corresponding disease location area in a single frame of the pavement image under each spectrum to extract the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image, including the maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, grayscale mean of the diseased area, and grayscale standard deviation of the diseased area. Based on the fuzzy analytic hierarchy process, the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image is evaluated for maintenance urgency, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained; Based on the maintenance urgency index corresponding to each road surface image frame, maintenance emergency sorting planning is performed on each road surface image frame to generate a road maintenance fleet sorting and scheduling sequence corresponding to the road surface image sorting single frame.

7. The AI-based intelligent pavement disease sorting platform according to claim 6 is characterized in that: The 12-dimensional feature analysis of crack disease in the corresponding disease location area in the single frame of the road surface image under each spectrum based on the corresponding image scale includes: Binarization is performed on the corresponding diseased location area in a single frame of the road surface image under each spectrum to generate a binary image of the corresponding diseased area under each spectrum; Based on the corresponding binary image of the damaged area under each spectrum and combined with the corresponding image scale, the corresponding maximum crack width, total crack length, crack tortuosity, damaged area, crack distribution density, perimeter of the damaged area, and shape complexity of the damaged area are calculated; The crack centerline in the defect location area corresponding to the visible spectrum band in a single frame of the pavement image under each spectrum is divided into several small segments by taking points at equal intervals. The vertical widths of the cracks corresponding to the several small segments are calculated respectively, and the average value is calculated at the same time to obtain the corresponding average crack width. The temperature distribution difference between each equally spaced point is determined by the defect location area of the corresponding infrared spectral band in a single frame of the pavement image under each spectrum. The crack depth corresponding to each equally spaced point is estimated based on the temperature distribution difference between each equally spaced point. At the same time, the maximum value of the crack depth corresponding to each equally spaced point is determined and the average value is calculated to obtain the corresponding maximum crack depth and average crack depth. The regional grayscale mean and standard deviation of the corresponding defect location area in a single frame of the road surface image under each spectrum are calculated to calculate the grayscale mean and grayscale standard deviation corresponding to all pixels in the defect location area, and obtain the corresponding grayscale mean and grayscale standard deviation of the defect area; The corresponding maximum crack width, average crack width, maximum crack depth, average crack depth, total crack length, crack tortuosity, area of the diseased area, crack distribution density, perimeter of the diseased area, shape complexity of the diseased area, grayscale mean of the diseased area, and grayscale standard deviation of the diseased area are merged into a 12-dimensional feature set of crack diseases corresponding to each single frame of the pavement image.

8. The AI-based intelligent pavement disease sorting platform according to claim 7 is characterized in that: The calculation of the corresponding maximum crack width, total crack length, crack tortuosity, area of the damaged area, crack distribution density, perimeter of the damaged area, and shape complexity of the damaged area based on the corresponding binary image of the damaged area under each spectrum and in combination with the corresponding image scale includes: The corresponding crack area is highlighted by binarizing the image of the diseased area under each spectrum and the corresponding noise and small branches are removed by morphological operations. Then, the circumscribed rectangle of the crack area is calculated, and the length of the short side of the circumscribed rectangle is approximately regarded as the corresponding maximum width. The corresponding maximum width of the crack is converted according to the corresponding image scale. The binary images of the diseased areas under each spectrum were skeletonized to simplify the corresponding crack areas into lines with a single pixel width. The total number of pixels on the lines was counted, and the total length of the cracks was calculated based on the corresponding image scale. The corresponding crack starting pixel point and crack ending pixel point are obtained by binarizing the corresponding diseased area under each spectrum, and the straight-line distance between the corresponding crack starting pixel point and the crack ending pixel point is calculated based on the crack starting pixel point and the crack ending pixel point, and the corresponding crack tortuosity is calculated based on the ratio of the total crack length to the straight-line distance between the crack starting point and the crack ending point; By counting the total number of pixels in the binary image of the corresponding diseased area under each spectrum, and converting the corresponding diseased area according to the corresponding image scale, the corresponding crack distribution density is calculated based on the ratio between the total length of the corresponding crack and the area of the diseased area; Perform boundary detection on the binarized image of the diseased area under each spectrum to obtain the boundary of the corresponding diseased area after binarization, and count the total number of pixels corresponding to the boundary of the diseased area. At the same time, calculate the corresponding perimeter of the diseased area according to the corresponding image scale; The shape complexity of the corresponding diseased area is calculated based on the ratio between the square of the perimeter of the diseased area and the area of the diseased area.

9. The AI-based intelligent pavement disease sorting platform according to claim 6 is characterized in that: The maintenance urgency assessment is performed on the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image based on the fuzzy analytic hierarchy process, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained, which includes: Based on the fuzzy analytic hierarchy process, the 12-dimensional feature set of crack disease corresponding to each single frame of the pavement image is used to construct the fuzzy judgment matrix of crack disease corresponding to each single frame of the pavement image; Based on the crack disease fuzzy judgment matrix corresponding to each single frame of the pavement image, weighted fuzzy reasoning is performed on each characteristic factor in the corresponding 12-dimensional feature set of the crack disease to generate the fuzzy weight value corresponding to each characteristic factor in each single frame of the pavement image; Based on the fuzzy weight values corresponding to each characteristic factor in each single frame of the pavement image, a maintenance urgency assessment is performed between the actual values of each characteristic factor in the 12-dimensional feature set of the corresponding crack disease, and the maintenance urgency index corresponding to each single frame of the pavement image is obtained.

10. The AI-based intelligent pavement disease sorting platform according to claim 6 is characterized in that: The maintenance emergency sorting planning for each road surface image frame based on the maintenance urgency index corresponding to each road surface image frame includes: Based on the maintenance urgency index corresponding to each road surface image single frame, the corresponding road surface image single frame is sorted and sorted according to emergency priority to generate a corresponding road surface maintenance emergency priority sorting image frame sequence; Performing a disease location search on each pavement image sorting single frame within the pavement maintenance emergency priority sorting image frame sequence to generate a disease location point sequence corresponding to the pavement image sorting single frame; The maintenance fleet path planning is carried out by combining the ant colony optimization algorithm with the GIS map for the sequence of disease location points corresponding to the single frame of road surface image sorting. The corresponding disease location points in the sequence are combined with the GIS map, and the optimal driving path is planned for the corresponding maintenance fleet while fully considering the corresponding road network topology and traffic conditions. In this way, the road maintenance fleet sorting and scheduling sequence corresponding to the single frame of road surface image sorting is generated.

Images