Road disease intelligent identification method and system based on image identification technology

Through the intelligent road disease recognition method based on image recognition technology, the shortcomings of artificial visual detection are solved, automated, high-speed and accurate road disease recognition are realized, detection efficiency and safety are improved, and cost and environmental impact are reduced.

CN119068268AActive Publication Date: 2024-12-03COMM DESIGN INST CO LTD OF JIANGXI PROV

Patent Information

Application Number
CN202411562370.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-05
Publication Date
2024-12-03
Estimated Expiration
2044-11-05

AI Technical Summary

Technical Problem

In the prior art, road disease detection relies on artificial vision, and has problems such as high cost, low accuracy, unsafe, affecting traffic and wasting time, making it difficult to meet current detection needs.

Method used

Using an intelligent road disease recognition method based on image recognition technology, the road image is captured for preprocessing, edge features, texture features and linear features are extracted, deep learning models are used for classification training and disease recognition, three-dimensional point cloud data are generated, and serious diseases are evaluated. and generate a test report.

Benefits of technology

It realizes automatic processing of road image data, quickly identifying diseased areas, reduce manual patrol time, improve detection accuracy and reliability, reduce labor costs, detect potential problems early, optimize maintenance resource use, and reduce traffic interference and environmental damage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068268A_ABST
    Figure CN119068268A_ABST
Patent Text Reader

Abstract

The invention discloses a road disease intelligent identification method and system based on an image identification technology, and the method comprises the steps: capturing a road image, carrying out the preprocessing of the road image, extracting edge features, texture features and linear features from the preprocessed road image, carrying out the classification training of the extracted edge features, texture features and linear features, and obtaining a road disease identification result. The method comprises the following steps: identifying different types of road diseases in a road image, positioning a road disease area, generating three-dimensional point cloud data from the road image at different visual angles of the positioned road disease area, reconstructing a road surface image from the three-dimensional point cloud data, and evaluating the road disease severity of the reconstructed road surface image. And predicting the severity of the road disease, obtaining a road surface image prediction disease identification result, generating the road surface image prediction disease identification result into a detection report, and storing the detection report. A large amount of road image data can be automatically processed by using an image recognition technology, and a disease area can be quickly recognized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of road damage recognition, and in particular to a method and system for intelligently recognizing road damage based on image recognition technology. Background Art

[0002] At present, pavement defect detection has always adopted an on-site investigation method based on artificial vision, that is, a staff member with detection experience walks on the shoulder of the road or sits in a slow-moving car, relying on visual observation to record the condition of the road surface. The recorded information includes the location, type, severity and number of pavement defects, and then returns to the room to calculate the degree of pavement damage. This method has many shortcomings such as high cost, low accuracy, traffic impact, unsafe, and waste of time, and can no longer meet the current requirements of pavement defect detection. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for intelligently identifying road damage based on image recognition technology to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for intelligently identifying road hazards based on image recognition technology, comprising: Step S1: capturing a road image and preprocessing the road image; Step S2: extracting edge features, texture features and straight line features from the preprocessed road image; Step S3: classify and train the extracted edge features, texture features, and straight line features, identify different types of road damage in the road image, and locate the road damage area, generate three-dimensional point cloud data from the road images of different viewpoints in which the road damage area is located, and reconstruct the road surface image from the three-dimensional point cloud data; Step S4: evaluating the severity of road damage in the reconstructed road surface image, and predicting the severity of road damage to obtain a road surface image prediction damage recognition result; Step S5: Generate the road surface image prediction disease recognition result into a detection report and store it.

[0005] Further, in step S1, the road image is preprocessed, and the specific process is as follows: Step S11: Fix the camera on the detection vehicle or drone, set the sampling frequency, and use the camera to capture the road image. The laser radar sensor and the camera work synchronously to obtain the height difference data of the road surface; Step S12: synchronizing the timestamps of the camera and the lidar sensor using a synchronization interface board to ensure the consistency of the height difference data obtained on the road surface; Step S13: using a median filter algorithm to remove road image noise; Step S14: using a histogram equalization algorithm to improve the quality and contrast of the road image; converting the captured road image into a grayscale road image, and using a weighted average method to fuse different modalities, the different modalities include the road image and the height difference data of the road surface obtained by the laser radar sensor, and using the weighted average method to assign different weights according to the importance of different modalities, and combining different modalities to achieve more accurate disease identification; Step S15: Use a bilinear interpolation algorithm to resize the road image to a fixed size, and use a bilateral filtering algorithm to perform edge-preserving smoothing on the road image resized to the fixed size, thereby reducing noise and details in the road image while maintaining edge and important clarity.

[0006] Further, in step S2, edge features, texture features and straight line features are extracted from the preprocessed road image. The specific process is as follows: Step S21: using the Sobel operator to calculate the gradient amplitude and direction of the road image, thinning the edge, retaining the local maximum value in the road image, using high and low thresholds to detect the edge of the road image, connecting the strong edge and the weak edge of the road image, and obtaining the edge features in the road image; converting the road image after edge detection into a binary image; The local binary pattern algorithm is used to analyze the texture features in the binary image, which is expressed as: LBP(x,y)= (I p -I(x,y))2 p (1); Wherein, LBP (x, y) represents the local binary pattern analysis road image texture feature calculated at the coordinate (x, y); Indicates the number of pixels around the central pixel; represents a symbolic function; I p Represents the pixel values ​​around the central pixel I (x, y); I (x, y) represents the grayscale value of the image at the coordinate (x, y); 2 p Represents the pixel values ​​around the central pixel 2 (x, y); x and y both represent the pixel coordinates in the image; Detect the straight line features of road cracks in binary images using the standard Hough transform algorithm, expressed as: [ρ = xcosa + ysina] (2); Where ρ represents the vertical distance from the image origin to the detected line; xcosa represents the cosine value; ysina represents the sine value; Step S22: by mapping the edge points in the binary image to the parameter space, the edge points in the binary image form a curve in the parameter space, the parameters of the straight line are found from the formed curve by the accumulator, and the straight line with a high cumulative value is extracted from the parameters of the straight line found by the accumulator; Step S23: Line segment detection: An improved RANSAC method is used to detect straight lines with high cumulative values ​​from the accumulator to obtain the straight line features of the road cracks in the binary image; dilation and erosion operations are applied to process the straight line features of the road cracks to make the edges of the cracks more continuous and remove unnecessary noise points.

[0007] Further, in step S3, the road surface image is reconstructed, and the specific process is as follows: Step S31: inputting edge features, texture features and straight line features into a deep learning model, and using the deep learning model to perform classification training on the edge features, texture features and straight line features to identify different types of road damage in the road image; Step S32: Use a cross-validation method to evaluate the performance of the deep learning model, use a U-shaped network to identify different types of road damage in the road image for segmentation, accurately locate the road damage area, and output the identified and located road damage area in the form of annotations to the initially captured road image, displaying the road image damage type and location; Step S33: Use motion structure recovery technology to generate three-dimensional point cloud data from road images of different perspectives to locate the diseased area, use SIFT algorithm to obtain road surface height difference data in road images of different perspectives for matching, then use particle swarm algorithm to estimate the posture of camera and lidar sensor, optimize the three-dimensional point cloud data through bundling adjustment algorithm, and use Poisson surface reconstruction algorithm to reconstruct the road surface image from the three-dimensional point cloud data.

[0008] Further, in step S4, the road surface image prediction disease recognition result is obtained, and the specific process is as follows: The area of ​​the disease in the road image is calculated by segmenting the road image, which is expressed as: [A=∑ n i=1 S(qi)](3); Where A represents the area of ​​the disease in the road image; n represents the number of pixels in the diseased area in the segmented image; S(qi) represents the area of ​​the diseased area; qi represents the diseased area; Combined with the reconstructed road surface image, the defect volume in the road image is calculated and expressed as: [V=∑ n i=1 V(qi)](4); Where V represents the defect volume in the road image; V(qi) represents the volume of the defect area; Based on the area and volume of the road damage in the road image, the neural network model is used to evaluate the severity of the road damage in the reconstructed road surface image, which is expressed as: [B = f (A, V)] (5); Where, B represents the severity of road damage in the reconstructed road surface image; f represents the neural network model; By using the random forest machine learning algorithm to calculate the severity of road damage in the reconstructed road surface image, the remaining life and maintenance requirements of the road surface are obtained as the road surface image prediction disease recognition result.

[0009] Further, in step S5, the road surface image prediction disease recognition result is generated into a detection report, and the specific process is as follows: Step S51: storing the captured road image and road surface image prediction disease recognition results in a relational database, regularly backing up the captured road image and road surface image prediction disease recognition results, using scheduled task scheduling to store the backup files on a remote server, and realizing rapid retrieval and query of the captured road image and road surface image prediction disease recognition results through a search engine solution; Step S52: Generate a test report based on the road surface image prediction and disease identification results through a typesetting system, use a drawing library based on a computer programming language to visualize the test report generated as the road surface image prediction and disease identification results, design interactive charts, use a drawing library based on a computer programming language to visualize the three-dimensional point cloud data, generate an interactive three-dimensional model, send the generated test report to relevant personnel via email, and provide an online access link.

[0010] Further, in step S4, before evaluating the severity of road damage in the reconstructed road surface image; The road images and road surface images predicted by the disease recognition results in the new environment are collected regularly as the sample number. The sample number is used to expand the road disease information data in the relational database. The transfer learning technology is used to fine-tune the parameters of the neural network model to adapt to the data changes in the new environment. It is expressed as: (6); In the formula, represents the average loss function; It means to average the loss values ​​of all N samples, that is, to add the loss values ​​of each sample and divide it by the number of samples N to get the average loss function; N represents the number of samples; L ( , ) is the main loss function part, which represents the error between the predicted value of the neural network model and the true label; L represents the loss function; Represents the loss function of the neural network model; Represents the input edge features, texture features and straight line features of the i-th sample number; Represents the neural network model parameters; represents the true value of the i-th sample number; λΦ(θ) represents the regularization term to control the complexity of the neural network model and prevent overfitting; A hyperparameter representing the strength of regularization, which determines the weight of regularization; During the operation of evaluating the road damage severity of the reconstructed road surface image, the stochastic gradient descent algorithm is used to update the neural network model parameters in real time.

[0011] Further, in step S12, before synchronizing the timestamps of the camera and the lidar sensor using the synchronization interface board; The data is synchronized by timestamps, and the road surface height difference data from the camera and the lidar sensor are collected synchronously to ensure the consistency of the road surface height difference data. The road surface height difference data is fused using filters, and the synchronized data of the camera and the lidar sensor are aligned using the iterative closest point algorithm, which is expressed as: ∑ i=1 N b Z i - ( R Q i+t ) 2] (7); In the formula, represents the Euclidean distance; Represents a point in the target point cloud; represents the rotation matrix; represents a point in the source point cloud; represents the translation vector; Combine road surface images and three-dimensional point cloud data, use the weighted average method to fuse road surface images and three-dimensional point cloud data, use deep learning models to identify road surface image diseases or defects, and fuse cameras and lidar sensors to improve the accuracy of deep learning models in identifying road surface image diseases. The effect of improving the accuracy of deep learning models in identifying road surface image diseases is verified through actual road data.

[0012] Furthermore, the road damage intelligent recognition system based on image recognition technology is used in any road damage intelligent recognition method based on image recognition technology, including: An image acquisition and preprocessing module, used for capturing road images and preprocessing road images; A feature extraction module is used to extract edge features, texture features and straight line features from the preprocessed road image; The disease recognition and 3D reconstruction module is used to classify and train the extracted edge features, texture features and straight line features, identify different types of road diseases in road images, locate road disease areas, generate 3D point cloud data from road images of different viewing angles in which road disease areas are located, and reconstruct road surface images from the 3D point cloud data; A damage assessment module is used to assess the severity of road damage in the reconstructed road surface image, predict the severity of road damage, and obtain a road surface image prediction damage recognition result; The data storage and management and report generation and visualization module is used to generate and store the road surface image prediction disease identification results into a detection report.

[0013] Compared with the existing technology, the present invention has the following beneficial effects: (1) The present invention utilizes image recognition technology to automatically process a large amount of road image data, quickly identify damaged areas, and significantly reduce the time and labor intensity of manual inspections. By processing road image data in real time, it can immediately discover and report road damage, take maintenance measures in a timely manner, and reduce the impact of damage on traffic safety.

[0014] (2) With the help of advanced road image recognition and deep learning models, the present invention can achieve high-precision disease identification and avoid errors and missed detections in manual detection of road images. Systematic road image recognition can ensure the consistency of detection results, is not affected by manual operations, and improves the reliability of disease detection.

[0015] (3) The high-resolution camera and lidar sensor system of the present invention can accurately obtain road surface and environmental data, provide high-quality data support for subsequent disease identification, reduce reliance on manual inspections, and reduce labor costs and training costs. Through accurate disease location and assessment, it optimizes the use of maintenance resources and avoids unnecessary repeated inspections and repairs.

[0016] (4) Through regular testing and real-time monitoring, the present invention can detect potential disease problems early, carry out preventive maintenance in advance, and reduce the accident rate. The system can cover a large area of ​​roads to ensure that all diseases can be discovered and handled in time, thereby improving the overall road safety level.

[0017] (5) The system of the present invention can accumulate and analyze a large amount of historical disease data, help managers understand the development trends and patterns of diseases, provide data support for decision-making, use deep learning models to classify diseases, and use U-shaped networks for image segmentation. The system can accurately identify and locate various types of road diseases, perform intelligent analysis of disease data, predict the development of diseases and the service life of roads, and optimize maintenance strategies.

[0018] (6) The present invention reduces the number of repeated constructions, reduces environmental damage and waste of resources through accurate disease detection and maintenance planning. Efficient detection and maintenance reduce the frequency of use of vehicles and related monitoring equipment, saves energy consumption, and reduces carbon emissions.

[0019] (7) The rapid and accurate disease detection and repair of the present invention reduces road closures and traffic disruptions, improves road traffic efficiency, and enhances the public travel experience. Through the detailed reports and data provided by the intelligent system, the public can understand the road maintenance status and enhance their trust and satisfaction with road management. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below with reference to implementation cases and accompanying drawings.

[0022] like Figure 1 As shown, the road disease intelligent identification method based on image recognition technology includes: Step S1: capturing a road image and preprocessing the road image; Step S2: extracting edge features, texture features and straight line features from the preprocessed road image; Step S3: classify and train the extracted edge features, texture features, and straight line features, identify different types of road damage in the road image, and locate the road damage area, generate three-dimensional point cloud data from the road images of different viewpoints in which the road damage area is located, and reconstruct the road surface image from the three-dimensional point cloud data; Step S4: evaluating the severity of road damage in the reconstructed road surface image, and predicting the severity of road damage to obtain a road surface image prediction damage recognition result; Step S5: Generate the road surface image prediction disease recognition result into a detection report and store it.

[0023] In step S1, the road image is preprocessed, and the specific process is as follows: Step S11: Fix the camera (FLIR Blackfly S USB3) on the detection vehicle or drone, set the sampling frequency, and use the camera to capture road images. The lidar sensor and the camera work synchronously to obtain the height difference data of the road surface. Step S12: using the synchronization interface board to synchronize the timestamps of the camera and the lidar sensor to ensure the consistency of the height difference data of the road surface, performing preliminary processing and storage on the height difference data of the road surface through the edge computing device to ensure real-time performance, and monitoring the data collection process in real time through the camera unit to ensure data integrity and accuracy; Step S13: using a median filter algorithm to remove road image noise; Step S14: using a histogram equalization algorithm to improve the quality and contrast of the road image; converting the captured road image into a grayscale road image (black and white image), and using a weighted average method to fuse different modalities, the different modalities include road images and height difference data of the road surface obtained by a laser radar sensor, and using the weighted average method to assign different weights according to the importance of different modalities, and combining different modalities to achieve more accurate disease identification; Step S15: Use a bilinear interpolation algorithm to resize the road image to a fixed size (such as 512x512 pixels), and use a bilateral filtering algorithm to perform edge-preserving smoothing on the road image resized to the fixed size, thereby reducing noise and details in the road image while maintaining edge and important clarity.

[0024] Among them, in step S2, edge features, texture features and straight line features are extracted from the preprocessed road image, and the specific process is as follows: Step S21: using the Sobel operator to calculate the gradient amplitude and direction of the road image, thinning the edge, retaining the local maximum value in the road image, using high and low thresholds to detect the edge of the road image, connecting the strong edge and the weak edge of the road image, and obtaining the edge features in the road image; converting the road image after edge detection into a binary image; The local binary pattern algorithm is used to analyze the texture features in the binary image, which is expressed as: LBP(x,y)= (I p -I(x,y))2 p (1); Wherein, LBP (x, y) represents the local binary pattern analysis road image texture feature calculated at the coordinate (x, y); Indicates the number of pixels around the central pixel; represents a symbolic function; I pRepresents the pixel values ​​around the central pixel I (x, y) (usually an 8-neighborhood or a 16-neighborhood); I (x, y) represents the grayscale value of the image at the coordinate (x, y); 2 p Represents the pixel values ​​around the central pixel 2 (x, y); x and y both represent the pixel coordinates in the image; Detect the straight line features of road cracks in binary images using the standard Hough transform algorithm, expressed as: [ρ = xcosa + ysina] (2); Where ρ represents the vertical distance from the image origin (0,0) to the detected line; xcos represents the cosine value; ysin represents the sine value; Step S22: by mapping the edge points in the binary image to the parameter space, the edge points in the binary image form a curve in the parameter space, the parameters of the straight line are found from the formed curve by the accumulator, and the straight line with a high cumulative value is extracted from the parameters of the straight line found by the accumulator; Step S23: Line segment detection: In crack detection, PROSAC (Progressive Sample Consensus) is an improved RANSAC (Random Sampling Consensus) method. It uses an improved RANSAC method to detect straight lines with high cumulative values ​​extracted from the accumulator for detection, which can more accurately detect the straight line features of road cracks in binary images; dilation and erosion operations are applied to further process the straight line features of road cracks, making the edges of cracks more continuous and removing unnecessary noise points; Step S24: Remove short or irregularly shaped straight lines in the binary image, retain longer and straight cracks, group the detected straight lines to form a complete crack area, and extract the geometric features and positional features (length, width, direction) of the cracks for further analysis or reporting.

[0025] The road surface image is reconstructed in step S3, and the specific process is as follows: Step S31: input edge features, texture features and straight line features into a deep learning model, and use the deep learning model in the TensorFlow framework to classify and train the edge features, texture features and straight line features to identify different types of road damage in the road image; Step S32: Use a cross-validation method to evaluate the performance of the deep learning model, use a U-shaped network to identify different types of road damage in the road image for segmentation, accurately locate the road damage area, and output the identified and located road damage area in the form of annotations to the initially captured road image, displaying the road image damage type and location; Step S33: Use motion structure recovery technology to generate three-dimensional point cloud data from road images of different perspectives for locating the damaged area, use SIFT algorithm to obtain road surface height difference data in road images of different perspectives for matching, and then use particle swarm algorithm to estimate the posture (position and direction) of the camera and lidar sensor, optimize the three-dimensional point cloud data through bundling adjustment algorithm, and use Poisson surface reconstruction algorithm to reconstruct the road surface image from the three-dimensional point cloud data.

[0026] Among them, in step S4, the road surface image prediction disease recognition result is obtained, and the specific process is: The area of ​​the disease in the road image is calculated by segmenting the road image, which is expressed as: [A=∑ n i=1 S(qi)](3); Where A represents the area of ​​the disease in the road image; n represents the number of pixels in the diseased area in the segmented image; S(qi) represents the area of ​​the diseased area; qi represents the diseased area; Combined with the reconstructed road surface image, the disease volume in the road image is calculated and expressed as: [V=∑ n i=1 V(qi)](4); Where V represents the defect volume in the road image; V(qi) represents the volume of the defect area; Based on the area and volume of the road damage in the road image, the neural network model is used to evaluate the severity of the road damage in the reconstructed road surface image, which is expressed as: [B = f (A, V)] (5); Where B represents the severity of road damage in the reconstructed road surface image; f represents the deep learning model (neural network model); The road surface disease severity of the reconstructed road surface image is calculated by using a random forest machine learning algorithm to obtain the remaining life and maintenance requirements of the road surface as the road surface image prediction disease recognition result; The evaluation results are recorded in a relational database and an evaluation report is generated. The evaluation report is automatically generated using Python scripts.

[0027] Among them, in step S5, the road surface image prediction disease recognition result is generated into a detection report, and the specific process is: Step S51: using a relational database to design a table structure, the relational database design table structure includes an image data table, a disease recognition result table and an evaluation result table, storing the captured road images and road surface image prediction disease recognition results in the relational database, regularly backing up the captured road images and road surface image prediction disease recognition results, using scheduled task scheduling, storing the backup files on a remote server, realizing fast retrieval and query of big data such as the captured road images and road surface image prediction disease recognition results through a search engine solution, and designing a search engine solution to optimize query efficiency; Step S52: Use a typesetting system to design a disease detection report template, the disease detection report template includes a data summary, test results, evaluation results and a three-dimensional reconstruction diagram, and generate a test report from the road surface image prediction disease recognition results through the typesetting system; the test report is in a PDF format, including charts and text descriptions, and a drawing library based on a computer programming language is used to visualize the generation of the road surface image prediction disease recognition results as a test report, and interactive charts are designed, including line charts, bar charts and heat maps, and a drawing library based on a computer programming language is used to visualize the three-dimensional point cloud data, generate an interactive three-dimensional model, and send the generated test report to relevant personnel via email, and provide an online access link.

[0028] Wherein, in step S4, before evaluating the severity of road damage in the reconstructed road surface image; The road images and road surface images predicted by the disease recognition results in the new environment are collected regularly as the sample number. The sample number is used to expand the road disease information data in the relational database. The transfer learning technology is used to fine-tune the parameters of the neural network model to adapt to the data changes in the new environment. It is expressed as: (6); In the formula, represents the average loss function; It means to average the loss values ​​of all N samples, that is, to add the loss values ​​of each sample and divide it by the number of samples N to get the average loss function; N represents the number of samples; L ( , ) is the main loss function part, which represents the error between the predicted value of the neural network model and the true label; L represents the loss function; Represents the loss function of the neural network model; Represents the input edge features, texture features and straight line features of the i-th sample number; Represents the neural network model parameters; represents the true value of the i-th sample number; λΦ(θ) represents the regularization term to control the complexity of the neural network model and prevent overfitting; A hyperparameter representing the strength of regularization, which determines the weight of regularization; In the process of evaluating the severity of road damage in reconstructed road surface images, the stochastic gradient descent algorithm is used to update the neural network model parameters in real time. By real-time monitoring the performance of the neural network model, the loss function of the neural network model and the changes in the accuracy of the neural network model in evaluating the severity of road damage in reconstructed road surface images over time or training cycles are analyzed, user feedback is collected, and the recognition accuracy of the neural network model is improved.

[0029] Wherein, before acquiring the road image; Transplant the deep learning model to the inspection vehicle or drone, use tensor flow to lightweight optimize the deep learning model, use mobile edge computing devices to calculate the operating efficiency of the deep learning model, process, extract features and identify defects on the captured road images on the inspection vehicle or drone, reduce the amount of data transmission and reduce latency, store temporary data on the inspection vehicle or drone, use efficient storage technology to manage data and ensure data persistence, process, extract features and identify defects on the captured road images and transmit them to the relational database via the wireless network to ensure data transmission and reduce latency.

[0030] Wherein, in step S12, before synchronizing the timestamps of the camera and the lidar sensor using the synchronization interface board; The data is synchronized by timestamps, and the road surface height difference data from the camera and the lidar sensor are collected synchronously to ensure the consistency of the road surface height difference data. The road surface height difference data is fused using filters, and the synchronized data of the camera and the lidar sensor are aligned using the iterative closest point algorithm, which is expressed as: ∑ i=1 N b Z i - ( R Q i+t ) 2] (7); In the formula, represents the Euclidean distance; Represents a point in the target point cloud; represents the rotation matrix; represents a point in the source point cloud; Represents the translation vector, used for translation; Combine road surface images and three-dimensional point cloud data, use the weighted average method to fuse road surface images and three-dimensional point cloud data, use deep learning models to identify road surface image diseases or defects, and fuse cameras and lidar sensors to improve the accuracy of deep learning models in identifying road surface image diseases. Verify the multimodal fusion effect through actual road data. A single modality uses one type of sensor data for identification and analysis, and compares the multimodal fusion recognition effect. The performance of multimodal fusion recognition effect analysis is improved.

[0031] The road damage intelligent recognition system based on image recognition technology is used in any road damage intelligent recognition method based on image recognition technology, including: An image acquisition and preprocessing module, used for capturing road images and preprocessing road images; A feature extraction module is used to extract edge features, texture features and straight line features from the preprocessed road image; The disease recognition and 3D reconstruction module is used to classify and train the extracted edge features, texture features and straight line features, identify different types of road diseases in road images, locate road disease areas, generate 3D point cloud data from road images of different viewing angles in which road disease areas are located, and reconstruct road surface images from the 3D point cloud data; A damage assessment module is used to assess the severity of road damage in the reconstructed road surface image, predict the severity of road damage, and obtain a road surface image prediction damage recognition result; The data storage and management and report generation and visualization module is used to generate and store the road surface image prediction disease identification results into a detection report.

[0032] In summary, the use of image recognition technology can automatically process large amounts of road image data, quickly identify damaged areas, and significantly reduce the time and labor intensity of manual inspections. By processing image data in real time, road defects can be discovered and reported immediately, and maintenance measures can be taken in time to reduce the impact of defects on traffic safety. Advanced image recognition algorithms and deep learning models can achieve high-precision defect identification and avoid errors and missed detections in manual inspections. Systematic image recognition can ensure the consistency of detection results, unaffected by manual operations, and improve the reliability of defect detection.

[0033] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. The intelligent road disease identification method based on image recognition technology is characterized by: include: Step S1: capturing a road image and preprocessing the road image; Step S2: extracting edge features, texture features and straight line features from the preprocessed road image; Step S3: classify and train the extracted edge features, texture features, and straight line features, identify different types of road damage in the road image, and locate the road damage area, generate three-dimensional point cloud data from the road images of different viewpoints in which the road damage area is located, and reconstruct the road surface image from the three-dimensional point cloud data; Step S4: evaluating the severity of road damage in the reconstructed road surface image, and predicting the severity of road damage to obtain a road surface image prediction damage recognition result; Step S5: Generate the road surface image prediction disease recognition result into a detection report and store it.

2. The method for intelligently identifying road damage based on image recognition technology according to claim 1 is characterized in that: In step S1, the road image is preprocessed, and the specific process is as follows: Step S11: Fix the camera on the detection vehicle or drone, set the sampling frequency, and use the camera to capture the road image. The laser radar sensor and the camera work synchronously to obtain the height difference data of the road surface; Step S12: synchronizing the timestamps of the camera and the lidar sensor using a synchronization interface board to ensure the consistency of the height difference data obtained on the road surface; Step S13: using a median filter algorithm to remove road image noise; Step S14: using a histogram equalization algorithm to improve the quality and contrast of the road image; converting the captured road image into a grayscale road image, and using a weighted average method to fuse different modalities, the different modalities include the road image and the height difference data of the road surface obtained by the laser radar sensor, and using the weighted average method to assign different weights according to the importance of different modalities, and combining different modalities to achieve more accurate disease identification; Step S15: Use a bilinear interpolation algorithm to resize the road image to a fixed size, and use a bilateral filtering algorithm to perform edge-preserving smoothing on the road image resized to the fixed size, thereby reducing noise and details in the road image while maintaining edge and important clarity.

3. The method for intelligently identifying road damage based on image recognition technology according to claim 2 is characterized in that: In step S2, edge features, texture features and straight line features are extracted from the preprocessed road image. The specific process is as follows: Step S21: using the Sobel operator to calculate the gradient amplitude and direction of the road image, thinning the edge, retaining the local maximum value in the road image, using high and low thresholds to detect the edge of the road image, connecting the strong edge and the weak edge of the road image, and obtaining the edge features in the road image; converting the road image after edge detection into a binary image; The local binary pattern algorithm is used to analyze the texture features in the binary image, which is expressed as: LBP(x,y)= (I p -I(x,y))2 p (1); Wherein, LBP (x, y) represents the local binary pattern analysis road image texture feature calculated at the coordinate (x, y); Indicates the number of pixels around the central pixel; represents a symbolic function; I p Represents the pixel values ​​around the central pixel I (x, y); I(x,y) represents the gray value of the image at the coordinate (x,y); 2 p Represents the pixel values ​​around the central pixel 2 (x, y); x and y both represent the pixel coordinates in the image; Detect the straight line features of road cracks in binary images using the standard Hough transform algorithm, expressed as: [ρ = xcosa + ysina] (2); Where ρ represents the vertical distance from the image origin to the detected line; xcosa represents the cosine value; ysina represents the sine value; Step S22: by mapping the edge points in the binary image to the parameter space, the edge points in the binary image form a curve in the parameter space, the parameters of the straight line are found from the formed curve by the accumulator, and the straight line with a high cumulative value is extracted from the parameters of the straight line found by the accumulator; Step S23: line segment detection: extracting straight lines with high cumulative values ​​from the accumulator through an improved RANSAC method for detection, and obtaining straight line features of road cracks in the binary image; Dilation and erosion operations are applied to process the straight line features of road cracks, making the edges of cracks more continuous and removing unnecessary noise.

4. The method for intelligently identifying road damage based on image recognition technology according to claim 3 is characterized in that: In step S3, the road surface image is reconstructed, and the specific process is as follows: Step S31: inputting edge features, texture features and straight line features into a deep learning model, and using the deep learning model to perform classification training on the edge features, texture features and straight line features to identify different types of road damage in the road image; Step S32: Use a cross-validation method to evaluate the performance of the deep learning model, use a U-shaped network to identify different types of road damage in the road image for segmentation, accurately locate the road damage area, and output the identified and located road damage area in the form of annotations to the initially captured road image, displaying the road image damage type and location; Step S33: Use motion structure recovery technology to generate three-dimensional point cloud data from road images of different perspectives to locate the diseased area, use SIFT algorithm to obtain road surface height difference data in road images of different perspectives for matching, then use particle swarm algorithm to estimate the posture of camera and lidar sensor, optimize the three-dimensional point cloud data through bundling adjustment algorithm, and use Poisson surface reconstruction algorithm to reconstruct the road surface image from the three-dimensional point cloud data.

5. The method for intelligently identifying road damage based on image recognition technology according to claim 4 is characterized in that: In step S4, the road surface image prediction disease recognition result is obtained, and the specific process is as follows: The area of ​​the disease in the road image is calculated by segmenting the road image, which is expressed as: [A=∑ n i=1 S(qi)](3); Where A represents the area of ​​the disease in the road image; n represents the number of pixels in the diseased area in the segmented image; S(qi) represents the area of ​​the diseased area; qi represents the diseased area; Combined with the reconstructed road surface image, the disease volume in the road image is calculated and expressed as: [V=∑ n i=1 V(qi)](4); Where V represents the defect volume in the road image; V(qi) represents the volume of the defect area; Based on the area and volume of the road damage in the road image, the neural network model is used to evaluate the severity of the road damage in the reconstructed road surface image, which is expressed as: [B = f (A, V)] (5); Where, B represents the severity of road damage in the reconstructed road surface image; f represents the neural network model; By using the random forest machine learning algorithm to calculate the severity of road damage in the reconstructed road surface image, the remaining life and maintenance requirements of the road surface are obtained as the road surface image prediction disease recognition result.

6. The method for intelligently identifying road damage based on image recognition technology according to claim 5 is characterized in that: In step S5, the road surface image prediction disease recognition result is generated into a detection report, and the specific process is as follows: Step S51: storing the captured road image and road surface image prediction disease recognition results in a relational database, regularly backing up the captured road image and road surface image prediction disease recognition results, using scheduled task scheduling to store the backup files on a remote server, and realizing rapid retrieval and query of the captured road image and road surface image prediction disease recognition results through a search engine solution; Step S52: Generate a test report based on the road surface image prediction and disease identification results through a typesetting system, use a drawing library based on a computer programming language to visualize the test report generated as the road surface image prediction and disease identification results, design interactive charts, use a drawing library based on a computer programming language to visualize the three-dimensional point cloud data, generate an interactive three-dimensional model, send the generated test report to relevant personnel via email, and provide an online access link.

7. The method for intelligently identifying road damage based on image recognition technology according to claim 6 is characterized in that: In step S4, before evaluating the road damage severity of the reconstructed road surface image; The road images and road surface images predicted by the disease recognition results in the new environment are collected regularly as the sample number. The sample number is used to expand the road disease information data in the relational database. The transfer learning technology is used to fine-tune the parameters of the neural network model to adapt to the data changes in the new environment. It is expressed as: (6); In the formula, represents the average loss function; It means to average the loss values ​​of all N samples, that is, to add the loss values ​​of each sample and divide it by the number of samples N to get the average loss function; N represents the number of samples; L ( , ) is the main loss function part, which represents the error between the predicted value of the neural network model and the true label; L represents the loss function; Represents the loss function of the neural network model; Represents the input edge features, texture features and straight line features of the i-th sample number; Represents the neural network model parameters; represents the true value of the i-th sample number; λΦ(θ) represents the regularization term to control the complexity of the neural network model and prevent overfitting; A hyperparameter representing the strength of regularization, which determines the weight of regularization; During the operation of evaluating the road damage severity of the reconstructed road surface image, the stochastic gradient descent algorithm is used to update the neural network model parameters in real time.

8. The method for intelligently identifying road damage based on image recognition technology according to claim 7 is characterized in that: In step S12, before synchronizing the timestamps of the camera and the lidar sensor using the synchronization interface board; The data is synchronized by timestamps, and the road surface height difference data from the camera and the lidar sensor are collected synchronously to ensure the consistency of the road surface height difference data. The road surface height difference data is fused using filters, and the synchronized data of the camera and the lidar sensor are aligned using the iterative closest point algorithm, which is expressed as: (7); In the formula, represents the Euclidean distance; Represents a point in the target point cloud; represents the rotation matrix; represents a point in the source point cloud; represents the translation vector; Combine road surface images and three-dimensional point cloud data, use the weighted average method to fuse road surface images and three-dimensional point cloud data, use deep learning models to identify road surface image diseases or defects, and fuse cameras and lidar sensors to improve the accuracy of deep learning models in identifying road surface image diseases. The effect of improving the accuracy of deep learning models in identifying road surface image diseases is verified through actual road data.

9. A road damage intelligent recognition system based on image recognition technology, used in the road damage intelligent recognition method based on image recognition technology as claimed in any one of claims 1 to 8, characterized in that: include: An image acquisition and preprocessing module, used for capturing road images and preprocessing road images; A feature extraction module is used to extract edge features, texture features and straight line features from the preprocessed road image; The disease recognition and 3D reconstruction module is used to classify and train the extracted edge features, texture features and straight line features, identify different types of road diseases in road images, locate road disease areas, generate 3D point cloud data from road images of different viewing angles in which road disease areas are located, and reconstruct road surface images from the 3D point cloud data; A damage assessment module is used to assess the severity of road damage in the reconstructed road surface image, predict the severity of road damage, and obtain a road surface image prediction damage recognition result; The data storage and management and report generation and visualization module is used to generate and store the road surface image prediction disease identification results into a detection report.

Citation Information

Patent Citations

  • Asphalt pavement upheaval multi-dimensional feature index identifying method

    CN107883891A

  • Road marking line intelligent recognition method and system

    CN111950409A

  • Structural disease detection method and device

    CN116416371A

  • Pavement disease recognition method, model training method, electronic equipment and medium

    CN116843983A

  • Method of obtaining a three-dimensional image of an object containing metal inclusions in computed tomography

    RU2801336C1

Cited By

  • Unmanned aerial vehicle-oriented emergency material air transportation method based on road network characteristics

    CN119536354A

  • Road disease detection method and system based on deep learning and ground-air cooperation

    CN119723290A

  • RTK and image reconstruction combined road scene space rapid investigation system

    CN120107504A

  • A road scene spatial rapid survey system combining RTK and image reconstruction

    CN120107504B

  • Tunnel disease target intelligent identification and extraction method based on machine vision

    CN120374607A