A road surface disease identification method and system based on multi-source data deep learning
Through the multi-source data deep learning method, combined with ground penetrating radar and industrial camera data, the convolutional neural network model is trained, which solves the problems of data integration lag and insufficient reporting in road road disease detection, and achieves efficient and accurate disease identification and positioning .
Patent Information
- Application Number
- CN202111566144.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-12-20
AI Technical Summary
The prior art has problems in the detection of road pavement disease, such as lagging data integration, insufficient comprehensive reporting, and lack of accurate pavement structure disease depth and dimensional measurement methods.
Through the multi-source data deep learning method, vehicle-mounted ground penetrating radar and industrial cameras are used to obtain pavement data, perform heterogeneous fusion of RGB images and radar images, train convolutional neural network models, extract disease characteristics, and realize disease identification, positioning and size calculations. .
It realizes automatic identification of different types of road surface cracks and calculation and positioning of disease sizes based on machine learning, improves the efficiency of highway patrols and the accuracy of disease identification, and generates detailed disease identification reports.
Smart Images

Figure CN114266892B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a road pavement defect recognition technology, and in particular to a road pavement defect recognition method and system based on multi-source data deep learning. Background Art
[0002] Road surfaces are extremely important for highways. Due to construction conditions, geographical environment, climate, vehicle driving and other reasons, roads have various diseases. Highway surface and shallow diseases are easy to observe and detect, but diseases at the roadbed are not easy to be found. If they are not handled promptly and effectively, the use of the highway will be affected and the lives of drivers will be seriously threatened.
[0003] In recent years, the detection technology of road pavement defects has gradually developed from the original manual detection to two-dimensional detection technology, including the use of cameras, array scanning, structured light scanning and other technologies. However, the overall pavement data analysis is lagging behind, and the data is usually analyzed offline after generation. The pavement defect report is not comprehensive enough. At the same time, the use of cameras alone can only find the image information of the defect surface, and lacks accurate analysis of the depth and size measurement method of the pavement structure defect. As a non-destructive, highly accurate, efficient and adaptable detection technology, ground penetrating radar replaces the original destructive and non-destructive detection methods and is widely used in many fields such as archaeology, mineral exploration, disaster geological survey, geotechnical engineering survey, engineering quality inspection, building structure inspection and military target detection.
[0004] With the development of machine learning technology in recent years, the use of machine learning to automatically identify road pavement defects has become a hot topic. Although there is no need for manual extraction of target recognition and detection, it is still difficult to accurately detect road pavement defects due to improper algorithm design and selection. Therefore, efficient and accurate identification and positioning of road pavement defects in complex environments are of great significance to highway maintenance. Summary of the invention
[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art, and further provide a technical solution, which trains a highway pavement disease sample library through heterogeneous fusion of multi-source data, extracts effective disease features, and constructs a convolutional neural network model for machine learning, and finally generates a detailed disease identification report including disease location, disease type, road signal, and disease size; it can realize automatic identification of different types of pavement cracks and calculation and positioning of disease size based on machine learning, meet the needs of pavement detection and maintenance, and improve the efficiency of highway inspections and the accuracy of disease identification.
[0006] The present invention provides a road surface disease identification method and system based on multi-source data deep learning, comprising:
[0007] The vehicle-mounted road inspection vehicle starts at a certain point on the highway and cruises at a fixed speed;
[0008] The vehicle-mounted road inspection vehicle is equipped with a ground-penetrating radar and an industrial camera, which acquires and numbers the road inspection RGB image and the radar image in real time, and establishes a one-to-many correspondence between the road inspection RGB image number and the radar image number;
[0009] The acquired road inspection RGB images are processed and the Yolo algorithm is used to perform disease analysis and detection;
[0010] When a certain disease is determined, the radar image corresponding to the road inspection GRB image and the radar image number is queried, and the crack depth is calculated based on the Cascade R-CNN algorithm;
[0011] Generate a pavement damage report containing the damage type, damage depth and damage location.
[0012] Preferably, the processing of the acquired road inspection RGB image further includes preprocessing the image, including image denoising, image sharpening and image segmentation.
[0013] Preferably, the disease analysis and detection using the yolo algorithm specifically includes: inputting each frame of the road inspection RGB image; dividing the road inspection RGB image into a 3*3 grid using the yolo algorithm; predicting 2 borders for each grid, including the confidence that each border is a target and the probability of each border area in multiple categories, predicting 3*3*2 target windows based on the previous step, and removing target windows with relatively low probability based on a set threshold; applying image classification and positioning processing to each grid; obtaining the bounding box coordinates of the predicted object and the corresponding category probability.
[0014] Preferably, each grid is set as a label y, and the label y is defined as an eight-dimensional vector, y = {pc, bx, by, bh, bw, c1, c2, c3}, where pc is the probability of the target object existing in the grid, bx, by, bh, bw are specified bounding boxes; c1, c2, c3 are the crack type category values at the position.
[0015] Preferably, the radar image is first subjected to PCA filtering, the image is reconstructed, the gradient amplitude of the reconstructed image is calculated, and the background clutter is removed by using a threshold method, so as to finally obtain an image after clutter removal.
[0016] Preferably, the crack depth calculation is to calculate the depth of the crack hanging from the pavement layer by regression, further comprising:
[0017] Normalize the radar image after removing clutter;
[0018] Set up the SVM classifier to preliminarily classify the radar images according to image quality, and input images of different qualities into different convolutional neural network models.
[0019] Preferably, the convolutional neural network model includes a pooling layer, a convolution layer, a fully connected layer, and a regression layer.
[0020] Preferably, the loss function of the regression layer is:
[0021] Among them, y' n and n They represent the predicted value obtained by the regression layer based on the radar image and the measured value of the crack size respectively. N is the number of images used in one iterative settlement. m represents the number of output parameters. m=2 is set, and m includes the vertical crack type value and the crack depth size value.
[0022] The present invention also provides a road disease identification system based on multi-source data deep learning, comprising: a vehicle-mounted road inspection vehicle, a remote control center and a client;
[0023] The vehicle-mounted road inspection vehicle is equipped with a ground-penetrating radar, an industrial camera, a main controller, a storage module, a wireless network device and a GPS positioning module; the main controller is connected to the ground-penetrating radar, the industrial camera, the storage module, the wireless network device and the GPS module respectively; the ground-penetrating radar is used to transmit or receive ground-penetrating radar detection signals, obtain radar images along the inspection route, and send the radar images to the main controller; the industrial camera is used to obtain road inspection RGB images in real time; the main controller is used to form and drive detection signals, digital sampling and A / D conversion processing of detection and reception echo signals, numbering and relationship mapping of road inspection RGB images and radar images, data processing, and machine learning; the storage module is used to store road inspection RGB images and radar images and their numbers, crack types in road diseases and various calculation data; the wireless network device is used for wireless transmission of data; the GPS module is used for position positioning;
[0024] The remote control center is used to issue instructions to the vehicle-mounted road inspection vehicle and receive data, and finally send a road surface disease report to the client;
[0025] The client is used to receive the pavement disease identification result fed back by the remote control center, and display the pavement disease report including the disease type, disease depth and disease location.
[0026] Preferably, the ground penetrating radar includes a radar body, a transmitter, a transmitting antenna connected to the transmitter, a receiver, and a receiving antenna connected to the receiver. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0028] Figure 1 This is a flow chart of a method for identifying road defects by deep learning of multi-source data according to an embodiment of the present invention;
[0029] Figure 2 This is a structural schematic diagram of a road surface disease identification system based on deep learning of multi-source data according to another embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "one", "said" and "the" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used here may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0031] See also Figure 1As shown, a schematic diagram of a method for identifying road defects by deep learning of multi-source data according to an embodiment of the present invention is provided. The method specifically comprises:
[0032] Step S11, the vehicle-mounted road inspection vehicle starts from a certain point on a highway at a fixed speed;
[0033] Step S12: The vehicle-mounted road inspection vehicle is equipped with a ground-penetrating radar and an industrial camera, and obtains and numbers the road inspection RGB image and the radar image in real time, and performs a one-to-many correspondence between the road inspection RGB image number and the radar image number;
[0034] Step S13, processing the acquired road inspection RGB image, and performing disease analysis and detection using the YOLO algorithm;
[0035] Step S14: when it is determined to be a certain disease, query the road inspection GRB image and the radar image corresponding to the radar image number, and calculate the crack depth based on the CascadeR-CNN algorithm;
[0036] Step S15: Generate a pavement disease report including the disease type, disease depth and disease location.
[0037] In an embodiment of the present invention, the road inspection RGB image is acquired by real-time acquisition by a high-precision camera of a calibrated industrial camera, and the image resolution is uniformly set to 1920*1080 pixels, wherein the acquired road inspection RGB image is further preprocessed, including image denoising, image sharpening and image segmentation.
[0038] Furthermore, the use of the yolo algorithm for disease analysis and detection specifically includes: inputting each frame of the road inspection RGB image; dividing the road inspection RGB image into a 3*3 grid using the yolo algorithm; predicting 2 borders for each grid, including the confidence that each border is a target and the probability of each border area in multiple categories, and predicting 3*3*2 target windows based on the previous step; removing target windows with relatively low probability based on a set threshold; applying image classification and positioning processing to each grid; and obtaining the bounding box of the predicted object and the corresponding class probability.
[0039] Each grid is set as a label y, and the label y is defined as an eight-dimensional vector, y = {pc, bx, by, bh, bw, c1, c2, c3}, where pc is the probability of the target object existing in the grid, bx, by, bh, bw are the specified bounding boxes; c1, c2, c3 are the crack type category values at the location.
[0040] The pavement disease described in the embodiment of the present invention is mainly aimed at the detection of crack diseases, and the classification of crack diseases includes: network cracks, transverse cracks and longitudinal cracks. If the detection object is the transverse crack in the crack disease, set c2=1, then c1=0, c3=0.
[0041] Ground Penetrating Radar (GPR) is a geophysical method that uses antennas to transmit and receive high-frequency electromagnetic waves to detect the material properties and distribution patterns inside a medium. The GPR method can draw on the technical achievements of seismic exploration methods from data acquisition, data processing (including processing software) to data interpretation. In recent years, with the deepening of theoretical research on electromagnetic waves, some electromagnetic characteristics such as polarization characteristics have been studied more deeply, and have been developed and applied in radar equipment, acquisition technology and data processing methods. Due to the high precision, high efficiency and non-destructive characteristics of GPR detection, it is currently mainly used in many fields such as archaeology, mineral exploration, disaster geological surveys, geotechnical engineering surveys, engineering quality inspections, building structure inspections and military target detection. GPR is a method of transmitting high-frequency electromagnetic waves (1MHz-1GHz) generated by a pulse source to a multi-layer structure through a transmitting antenna. Since high-frequency electromagnetic waves will be emitted when they encounter non-uniform dielectric surfaces on their propagation path, the receiving antenna located on the ground can receive reflected echoes from the surface and interior of the structure. The processing system of the ground penetrating radar performs signal shaping and amplification on the reflected echoes and then records them. These processed reflected echoes can be used to inversely calculate parameters such as the location and size of internal defects in multi-layer structures and to distinguish the type of defects.
[0042] Furthermore, the radar image is first subjected to PCA filtering, the image is reconstructed, the gradient amplitude of the reconstructed image is calculated, and the background clutter is removed using the threshold method. Finally, the image after clutter removal is obtained, and the crack depth is further calculated.
[0043] Among them, the crack depth calculation is to calculate the depth size of the crack hanging on the pavement layer through regression, including: normalizing the radar image after removing clutter; setting up an SVM classifier to preliminarily classify the radar image according to image quality, and inputting images of different qualities into different convolutional neural network models.
[0044] Among them, normalization processing mainly refers to scaling the image, normalizing it to the same size, and recording the scaling ratio.
[0045] Furthermore, the convolutional neural network model includes a pooling layer, a convolution layer, a fully connected layer, and a regression layer, wherein the loss function of the regression layer is: Among them, y' n and nThey represent the predicted value obtained by the regression layer based on the radar image and the measured value of the crack size respectively. N is the number of images used in one iterative settlement. m represents the number of output parameters. m=2 is set, and m includes the vertical crack type value and the crack depth size value.
[0046] See also Figure 2 As shown, a schematic diagram of the structure of a road surface disease identification system based on multi-source data deep learning according to another embodiment of the present invention includes: a vehicle-mounted road inspection vehicle 1, a remote control center 2 and a client 3;
[0047] The vehicle-mounted road inspection vehicle 1 is equipped with a ground penetrating radar 11, an industrial camera 12, a main controller 13, a storage module 14, a wireless network device 15 and a GPS positioning module 16; the main controller 13 is connected to the ground penetrating radar 11, the industrial camera 12, the storage module 14, the wireless network device 15 and the GPS module 16 respectively; the ground penetrating radar 11 is used to transmit or receive ground penetrating radar detection signals, obtain radar images along the inspection route, and send the radar images to the main controller 13; the industrial camera 12 is used to obtain road inspection RGB images in real time; the main controller 13 is used for the formation and driving of detection signals, the digital sampling processing and A / D conversion processing of detection and reception echo signals, the numbering and relationship mapping of road inspection RGB images and radar images, data processing, and machine learning; the storage module 14 is used to store road inspection RGB images and radar images and their numbers, crack types in road diseases and various calculation data; the wireless network device 15 is used for wireless transmission of data; the GPS module 16 is used for position positioning;
[0048] The remote control center 2 is used to send instructions to the vehicle-mounted road inspection vehicle 1 and receive data, and finally send a road surface disease report to the client 3;
[0049] The client 3 is used to receive the pavement disease identification result fed back by the remote control center 2, and display the pavement disease report including the disease type, disease depth and disease location.
[0050] The ground penetrating radar 11 includes a radar body, a transmitter, a transmitting antenna connected to the transmitter, a receiver, and a receiving antenna connected to the receiver.
[0051] In the embodiment of the present invention, the speed of the vehicle-mounted road inspection vehicle is set to 80km / h, the time sampling rate of the ground penetrating radar is 1024 scans / s, and the ground penetrating radar uses a shielded antenna with a size of 30cm*30cm*14cm. The scanning coverage of this antenna is consistent with the antenna size, and the receiving sky is 20cm from the ground. The main controller can also be equipped with a GPU of NVIDA GeForce GTX 1060 8GB specification for convolutional neural network training. It takes an average of about 0.2 seconds to train a 512*512 pixel ground penetrating radar image.
[0052] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the method flow steps of the embodiment of the above-mentioned method.
[0053] The present invention provides a road disease identification method and system based on multi-source data deep learning. By installing a ground-penetrating radar and an industrial camera on a vehicle-mounted road inspection vehicle, the radar image generated by the ground-penetrating radar signal and the high-definition road inspection RGB image are heterogeneously fused, and effective disease features are extracted through the construction of a road disease sample library, and a convolutional neural network model is constructed to identify road diseases, and finally a disease identification report including disease location, disease type and disease size is generated.
[0054] Compared with the prior art, the beneficial effects of the present invention are as follows: it can realize automatic identification of different types of pavement cracks and calculation and positioning of disease sizes based on machine learning, meet the needs of pavement detection and maintenance, and improve the efficiency of highway inspections and the accuracy of disease identification.
[0055] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A road surface disease identification method based on multi-source data deep learning, characterized in that: include: The vehicle-mounted road inspection vehicle starts at a certain point on the highway and cruises at a fixed speed; The vehicle-mounted road inspection vehicle is equipped with a ground-penetrating radar and an industrial camera, which acquires and numbers the road inspection RGB image and the radar image in real time, and establishes a one-to-many correspondence between the road inspection RGB image number and the radar image number; The acquired road inspection RGB images are processed and the YOLO algorithm is used to perform disease analysis and detection; When a certain disease is determined, the radar image corresponding to the road inspection GRB image and the radar image number is queried, and the crack depth is calculated based on the Cascade R-CNN algorithm; Generate a pavement damage report containing the damage type, damage depth and damage location.
2. The method for identifying road damage based on multi-source data deep learning according to claim 1, characterized in that: The processing of the acquired road inspection RGB image further includes preprocessing the image, including image denoising, image sharpening and image segmentation.
3. The method for identifying road damage based on multi-source data deep learning according to claim 1 is characterized in that: The disease analysis and detection using the YOLO algorithm specifically includes: inputting each frame of the road inspection RGB image; dividing the road inspection RGB image into a 3*3 grid form using the YOLO algorithm; predicting two borders for each grid, including the confidence that each border is a target and the probability that each border area is in multiple categories, predicting 3*3*2 target windows based on the previous step, and removing target windows with relatively low probability based on a set threshold; applying image classification and positioning processing to each grid; and obtaining the bounding box coordinates of the predicted object and the corresponding category probability.
4. The method for identifying road damage by deep learning of multi-source data according to claim 3 is characterized in that: Each grid is set to label y, and the label y is defined as an eight-dimensional vector. y={pc,bx,by,bh,bw,c1,c2,c3}, where pc is the category probability of the target object existing in the grid, bx, by, bh, bw are the specified bounding boxes; c1, c2, c3 are the crack type category values at the location.
5. The method for identifying road damage by deep learning of multi-source data according to claim 1, characterized in that: The radar image is first subjected to PCA filtering, the image is reconstructed, the gradient amplitude of the reconstructed image is calculated, and the background clutter is removed by using a threshold method, and finally an image after clutter removal is obtained.
6. The method for identifying road damage by deep learning of multi-source data according to claim 1, characterized in that: The crack depth calculation is to calculate the depth of the crack hanging down from the pavement layer by regression, further comprising: Normalize the radar image after removing clutter; Set up the SVM classifier to preliminarily classify the radar images according to image quality, and input images of different qualities into different convolutional neural network models.
7. The method for identifying road damage by deep learning of multi-source data according to claim 6, characterized in that: The convolutional neural network model includes a pooling layer, a convolution layer, a fully connected layer, and a regression layer.
8. The method for identifying road damage by deep learning of multi-source data according to claim 7, characterized in that: The loss function of the regression layer is: Among them, y' in represents the predicted value of the i-th parameter of the n-th sample crack obtained by the regression layer based on the radar image; y in represents the measured value of the i-th parameter of the n-th sample crack obtained by the regression layer based on the radar image; The value of n ranges from 1 to N, where N is the number of images used in one iterative settlement; m represents the number of output parameters, m=2 is set, and m includes the vertical crack type value and the crack depth size value; i represents the i-th parameter output, and a total of 2 parameters are output.
9. A road surface disease identification system based on multi-source data deep learning, characterized in that: include: Vehicle-mounted road inspection vehicle, remote control center and client; The vehicle-mounted road inspection vehicle is equipped with a ground-penetrating radar, an industrial camera, a main controller, a storage module, a wireless network device and a GPS positioning module; the main controller is connected to the ground-penetrating radar, the industrial camera, the storage module, the wireless network device and the GPS positioning module respectively; the ground-penetrating radar is used to transmit or receive ground-penetrating radar detection signals, obtain radar images along the inspection route, and send the radar images to the main controller; the industrial camera is used to obtain road inspection RGB images in real time; the main controller is used to form and drive detection signals, detect digital sampling processing and A / D conversion processing of received echo signals, number road inspection RGB images and radar images, and perform relationship mapping, data processing and machine learning; the storage module is used to store road inspection RGB images and radar images and their numbers, crack types in road diseases and various calculation data; the wireless network device is used for wireless transmission of data; the GPS positioning module is used for position positioning; The remote control center is used to issue instructions to the vehicle-mounted road inspection vehicle and receive data, and finally send a road surface disease report to the client; The client is used to receive the pavement disease identification result fed back by the remote control center, and display the pavement disease report including the disease type, disease depth and disease location.
10. The road surface disease identification system based on multi-source data deep learning according to claim 9 is characterized in that: The ground penetrating radar comprises a radar body, a transmitter, a transmitting antenna connected to the transmitter, a receiver and a receiving antenna connected to the receiver.
Citation Information
Patent Citations
Railway roadbed disease radar map real-time detection method based on convolutional neural network
CN110706211A
Road surface disease target detection and instance segmentation method based on road foreground image
CN111553236A
Cited By
Automatic positioning and traction device for assisting in clogging detection of tunnel drainage pipe
CN121025320A
