Road surface state evaluation system and method based on vehicle-mounted mobile vision camera

Abstract

The invention discloses a road surface state evaluation system and method based on a vehicle-mounted mobile visual camera, and the vehicle-mounted mobile camera (1) belongs to a vehicle and moves along with the vehicle to shoot a road surface image; the pavement disease database (2) is used for storing defect data; the disease identification subsystem (3) is used for identifying various types of roads and various diseases; the disease quantification subsystem (4) carries out three-dimensional accurate measurement on pavement diseases, and the disease evaluation subsystem (5) carries out accurate evaluation on pavement conditions according to specifications. According to the method, pavement disease images can be shot through a vehicle-mounted mobile camera, disease data classification is carried out in combination with a pavement disease database, support is provided for subsequent deep learning, meanwhile, pavement diseases are identified by adopting a disease identification subsystem, and then the diseases are subjected to three-dimensional accurate measurement through a disease quantification subsystem, so that the accuracy of the measurement result is improved. And finally, evaluating pavement diseases, carrying out hidden danger early warning on pavement with secondary and below evaluation results, checking pavement traffic conditions in combination with a Beidou technology, and carrying out pavement maintenance according to the pavement traffic conditions and road importance degrees.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image processing, and in particular to a road condition assessment technology based on a vehicle-mounted mobile visual camera. Background Art

[0002] Although there are various methods for detecting road defects in the prior art, they are still mainly focused on the two traditional methods of manual inspection and road inspection vehicles. These two methods have their own advantages and disadvantages, but both have certain limitations in practical applications. First, the manual inspection method relies on the workers' on-site observation and judgment. Although this method can directly observe the actual situation of the road surface, it obviously has many shortcomings. On the one hand, manual inspection requires workers to walk directly on the road, which will not only interfere with normal traffic and affect driving safety, but also may endanger the personal safety of inspectors. On the other hand, since the detection results mainly rely on the personal experience and subjective judgment of the inspectors, their accuracy and consistency are difficult to guarantee and are easily affected by human factors. In contrast, road inspection vehicles can achieve fast and efficient detection of the road surface by carrying various sensors and cameras. This method significantly improves the detection efficiency and reduces interference with traffic. However, road inspection vehicles also have some obvious disadvantages. First, the operation is complicated, requiring professional personnel to drive and operate, requiring high technical level of personnel, and road inspection vehicles are expensive. Secondly, although road inspection vehicles have achieved a certain degree of automation, the overall level of automation is still low, making it difficult to cope with complex and changing road environments and types of defects. In addition, when the inspection vehicle is driving too slowly while photographing road defects, it will also cause certain obstructions to traffic. Traditional road inspection methods can no longer meet the growing demand for road maintenance inspection tasks.

[0003] In recent years, with the rapid development of computer technology and image processing technology, computer vision technology has stood out in structural health monitoring with its advantages of low cost, non-contact and non-destructiveness. Many scholars have applied computer vision technology to the structural health monitoring of civil engineering. More and more researchers have begun to explore new ways to detect pavement defects, among which the automatic detection method of pavement defects based on machine vision and deep learning has attracted much attention. This method can quickly and accurately detect various pavement defects by training a deep learning model to realize automatic recognition and classification of pavement images. Compared with traditional target detection algorithms, deep learning algorithms show strong robustness and generalization capabilities in target detection, especially when dealing with multi-scale target detection in complex backgrounds. Summary of the invention

[0004] The purpose of the present invention is to achieve accurate, efficient and long-term comprehensive pavement disease detection and pavement evaluation.

[0005] The present invention is a road condition assessment system and method based on a vehicle-mounted mobile visual camera. The road condition assessment system based on the vehicle-mounted mobile visual camera includes a road disease assessment system including a vehicle-mounted mobile camera 1, a road disease database 2, a disease recognition subsystem 3, a disease quantification subsystem 4, and a disease assessment subsystem 5. The vehicle-mounted mobile camera 1 is assigned to a vehicle and moves with the vehicle to shoot road images; the road disease database 2 is used to store defect data, classify defect data and subsequently supplement defect data, and provide a large amount of training data for a deep learning model; the disease recognition subsystem 3 is used to identify various types of roads and various diseases; the disease quantification subsystem 4 provides a basis for judging road maintenance and repair by performing three-dimensional and precise measurement of road diseases; the disease assessment subsystem 5 accurately assesses road conditions in accordance with specifications, and carries out hidden danger warnings and repairs on roads with evaluation levels of secondary and below.

[0006] According to the above-mentioned road condition assessment system based on a vehicle-mounted mobile visual camera, the road condition assessment method based on a vehicle-mounted mobile visual camera of the present invention comprises the following steps: Step (1) First, the field collected data and the existing data set are classified into road damage types such as cracks, potholes, subsidence, rutting, looseness, manholes, patches, color seepage, rolling, sinking, and pushing through the ViT image classification module 2-1. Then, the new damage image is output to the unsupervised self-update module 2-2 through the disease recognition subsystem 3 to update the existing data set. Finally, all data sets form a complete database to provide support for deep learning model training.

[0007] In step (2), the vehicle-mounted mobile visual camera 1 captures the original image, and then uses the image super-resolution module 3-1 to convert the low-resolution original image into a high-resolution image or a super-resolution image. After that, the data trained by the pavement disease database 2 and the pre-trained weights, transfer learning strategy, attention mechanism, defect detection network, and semi-supervised training weights are input into the disease recognition module 3-2 for feature extraction and analysis, and the disease and its type are determined, and finally the segmented image is output.

[0008] Step (3) pre-classifies the segmented image output by the defect recognition subsystem 3, and then uses the defect quantity detection module 4-1 to output the defect quantity. Then, the defect stereo quantification module 4-2 is used to detect the defect quantity, defect length, defect width, defect depth, defect area, defect ratio and overall road surface flatness in combination with the binocular photography principle.

[0009] Finally, step (4) compares the quantification result output by the disease quantification subsystem 4 with the specification and inputs it into the multi-category grade assessment module 5-1 for road grade assessment. If it meets the specification, an assessment report is output. If it does not meet the specification, the hidden danger warning module 5-2 is used to issue a hidden danger warning and the road traffic conditions are checked in combination with Beidou technology. Road maintenance is carried out according to the road traffic conditions and the importance of the road.

[0010] The present invention has the following advantages over the prior art: (1) The present invention uses a vehicle-mounted mobile camera to directly photograph road surface defects, thereby avoiding road blockage or traffic congestion caused by traditional road surface disease detection. At the same time, combined with visual methods, the collected road surface images can be quickly and accurately analyzed, achieving low-cost, non-contact, high-efficiency, all-round, non-destructive, and high-precision road surface disease assessment.

[0011] (2) The present invention organizes and classifies existing pavement defect datasets and field-collected datasets, and combines unsupervised modules to achieve self-update of disease images, ultimately forming a pavement disease database to provide support for deep learning training.

[0012] (3) The defect identification subsystem and defect quantification subsystem proposed in the present invention can classify and quantify actual road defects, and use the binocular principle combined with a vehicle-mounted mobile camera to perform stereoscopic quantification of actual defects.

[0013] (4) The present invention evaluates the actual road surface condition by comparing the disease quantification results with existing specifications. For roads with evaluation results of poor or below, the early warning module performs road surface early warning and combines Beidou technology to check the road traffic conditions. The road surface is repaired according to the road traffic conditions and the importance of the road. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a structural block diagram of a road condition assessment system and method based on a vehicle-mounted mobile visual camera in an embodiment of the present invention, Figure 2 is a flowchart of a pavement damage database in an embodiment of the present invention. Figure 3 is a flowchart of the disease identification subsystem in an embodiment of the present invention, Figure 4 is a flowchart of the disease quantification subsystem in an embodiment of the present invention, Figure 5 This is a flowchart of the disease assessment subsystem in an embodiment of the present invention. Figure 6 is a flowchart of the ViT image classification module in an embodiment of the present invention, Figure 7 is a flowchart of an image super-resolution module in an embodiment of the present invention, Figure 8 is a flowchart of a disease identification module in an embodiment of the present invention, Fig. 9 is a flowchart of a disease stereo quantification module in an embodiment of the present invention, Fig.10This is a flowchart of the multi-class level assessment module in the embodiment of the present invention. Among them, 1: vehicle-mounted camera, 2: road surface disease database, 3: disease identification subsystem, 4: disease quantification subsystem, 5: disease assessment subsystem, 2-1: ViT image classification module, 2-2: unsupervised self-update module, 3-1: image super-resolution module, 3-2: disease identification module, 4-1: disease quantity detection module, 4-2: disease stereo quantification module, 5-1: multi-class level assessment module, 5-2: hidden danger warning module. DETAILED DESCRIPTION

[0015] The present invention is a road condition assessment system and method based on a vehicle-mounted mobile visual camera, and provides a road disease database 2, which is characterized in that various disease types and road types are classified, and new data are updated to the existing data set to expand the data set, including a ViT image classification module 2-1, an unsupervised self-update module 2-2 and a disease recognition system 3, and the data set is expanded by the unsupervised self-update module 2-2. The ViT image classification module 2-1 is used to classify the road type and disease type of the existing data set and the field collected data, and the disease recognition subsystem 3 outputs a new disease image, and then the new data is updated to the existing data set through the unsupervised self-update module 2-2 to expand the data set. The disease recognition subsystem 3 is characterized in that the collected image data is identified and a segmented image is output, including an image super-resolution module 3-1 and a disease recognition module 3-2, and the image is taken by the vehicle-mounted mobile visual camera 1, and the super-resolution image is output through the image super-resolution module 3-1, and the image trained by the road disease database 2 is input into the disease recognition module 3-2 to output the segmented image. The characteristic of the defect quantification subsystem 4 is that the specific road surface defects are mainly detected by the defect stereo quantification module 4-2. It includes a defect quantity detection module 4-1 and a defect stereo quantification module 4-2. The segmented image is output by the defect recognition subsystem 3 and predefined classification is performed. The defect quantity detection module 4-1 can detect the number of defects and detect the depth of defects in combination with the binocular principle. The defect assessment subsystem 5 is characterized by comparing the quantification results output by the defect quantification subsystem 4 with the specifications, and judging whether it meets the specifications in the multi-class level assessment module 5-1. If it meets the specifications, the road surface defect detection continues and the assessment report is output. If it does not meet the specifications, the hidden danger warning module 5-2 will issue a warning message, and then the practitioners will judge whether to repair the road surface.

[0016] The following is a further detailed description of a road condition assessment system and method based on a vehicle-mounted mobile visual camera according to the present invention in conjunction with the accompanying drawings. Figure 1As shown, the steps of the method are as follows: First, the road disease types are classified by the ViT image classification module using the field collected data and the existing data set, and then the new disease image is output to the unsupervised self-update module 2-2 through the disease recognition subsystem 3 to update the existing data set. Finally, all data sets form a complete database to provide support for deep learning model training. Second, the original image is captured by the vehicle-mounted mobile visual camera 1, and then the low-resolution original image is formed into a high-resolution image or a super-resolution image using the image super-resolution module 3-1, and then the data trained in the road disease database 2 is input into the disease recognition module 3-2 for feature extraction and analysis and to determine the disease and its type. Third, the segmented image output by the disease recognition subsystem 3 is pre-classified, and then the disease quantity detection module 4-1 is used to output the disease quantity, and then the disease is quantified in the disease stereo quantification module 4-2. Fourth, the quantification results of the disease quantification subsystem 4 are input into the multi-category grade assessment module 5-1 for road grade assessment, and hidden danger warning is carried out through the hidden danger warning module 5-2. In addition, the road traffic conditions are checked in combination with Beidou technology, and road maintenance is carried out according to the road traffic conditions and the importance of the road.

[0017] like Figure 2 As shown, the pavement disease database 2 in the embodiment of the present invention includes: field collected data and existing data sets. The road type classification and disease type classification are performed through the ViT image classification module 2-1, and when the disease recognition subsystem 3 outputs a new disease image, the unsupervised self-update module 2-2 can be used to update the new data to the existing data set to expand the data set, and finally form a pavement disease database with existing data classification and real-time data update.

[0018] like Figure 3 As shown, in the embodiment of the present invention, the defect recognition subsystem 3 first uses the vehicle-mounted mobile visual camera 1 to capture the original image, and then uses the image super-resolution module 3-1 to output the low-resolution original image into a super-resolution image or a high-resolution image, and then combines the data trained by the pavement defect database 2 to input it into the defect recognition module 3-2, and uses pre-trained weights, transfer learning strategies, attention mechanisms, defect detection networks and semi-supervised training to perform feature extraction and analysis and determine the defects and their types, and finally outputs a segmented image.

[0019] like Figure 4As shown, the defect quantification subsystem 4 in the embodiment of the present invention first pre-classifies the segmented image output by the defect recognition subsystem 3, and then the defect quantity detection module 4-1 outputs the defect quantity, and finally, combined with binocular shooting, the defect stereo quantification module 4-2 detects the defect quantity, defect length, defect width, defect depth, defect area, defect ratio and overall road surface smoothness. Among them, the number of defects is counted directly through the segmented image output, the length and width of the defect are calculated by the vehicle-mounted mobile visual camera and the road surface angle and the pixel length of the segmented image defect, the depth of the defect is calculated by the color value of the defect shown by the vehicle-mounted mobile visual camera at different angles, the area and ratio of the defect are directly output by the color represented by different defects in the segmented image, and the overall road surface smoothness is judged by the relative distance between the road surface image posture taken by the vehicle-mounted mobile visual camera and the overall road surface and the specific defect quantification result.

[0020] like Figure 5 As shown, in the embodiment of the present invention, the damage assessment subsystem 5 outputs the quantification result from the damage quantification subsystem 4 and compares it with the specification, and evaluates whether the quantification result meets the specification in the multi-category level assessment module 5-1. If it meets the specification, an assessment report is output. If it does not meet the specification, the hidden danger warning module 5-2 is used to issue a hidden danger warning, and the Beidou technology is combined to check the road traffic conditions, and perform road maintenance according to the road traffic conditions and the importance of the road.

[0021] like Figure 6 As shown, the ViT image classification module 2-1 in the embodiment of the present invention adopts the ViT image classification algorithm, firstly divides the original image into proportional sub-images, and then combines the block + position embedding module to process the divided sub-images into blocks and form a sequence to input into the Transformer encoder, and at the same time combines the multi-head attention mechanism to output to the multi-layer perceptron head to perform accurate disease classification of various road surfaces.

[0022] like Figure 7 As shown, the image super-resolution module 3-1 in the embodiment of the present invention utilizes the SRGAN image super-resolution reconstruction algorithm of deep learning, wherein the original image is input into the generation network to output the super-resolution image through multiple residual blocks and convolution operations, and the various types of super-resolution images generated are input into the discrimination network, through various types of convolution, full connection, and activation function layer operations, and finally the required super-resolution image is discriminated and output.

[0023] like Figure 8 As shown, the disease recognition module 3-2 in the embodiment of the present invention adopts the FCN semantic segmentation deep learning model, and the disease image is processed through a variety of fully connected layers, pooling layers and convolution layers. The operation is used to extract the disease features, and finally the Softmax classification function is used to output the disease segmentation image.

[0024] like Fig. 9As shown, in the embodiment of the present invention, the defect stereo quantification module 4-2 first outputs multi-angle pictures from the vehicle-mounted mobile visual camera to the defect recognition module 3-2, and outputs a segmented image. The multi-angle defect image is combined with the binocular principle to output the number of defects, defect length, defect width, defect depth, defect area, defect ratio and the overall road surface flatness detection.

[0025] like Fig.10 As shown, in the embodiment of the present invention, the multi-category grade assessment module 5-1 first outputs the quantification result of the defect quantification subsystem 4, and at the same time, the defect identification subsystem 3 outputs the road surface type and the type of defect and compares them with the highway technical condition assessment and evaluation standard specifications, and evaluates them as excellent, good, medium, poor, and poor. If the assessment grade is poor or below, the hidden danger warning module 5-2 is performed, and the Beidou technology is combined to divide the traffic grade according to the traffic conditions, and finally the road surface is repaired in order of grade.

[0026] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A road condition assessment system based on a vehicle-mounted mobile visual camera, comprising a road condition assessment system comprising a vehicle-mounted mobile camera (1), a road condition database (2), a condition identification subsystem (3), a condition quantification subsystem (4), and a condition assessment subsystem (5), characterized in that: The vehicle-mounted mobile camera (1) is attached to the vehicle and moves with the vehicle to capture road images; the road surface disease database (2) is used to store defect data, classify defect data and subsequently supplement defect data, providing a large amount of training data for the deep learning model; the disease identification subsystem (3) is used to identify various types of roads and various diseases; the disease quantification subsystem (4) provides a basis for judging road maintenance and repair by performing three-dimensional and precise measurement of road diseases; the disease assessment subsystem (5) accurately assesses road conditions in accordance with specifications, and carries out hidden danger warnings and repairs on roads with an assessment grade of secondary or below.

2. The road condition assessment system based on a vehicle-mounted mobile visual camera according to claim 1 is characterized in that: The pavement disease database (2) is composed of field collected data and existing data sets using a ViT image classification module (2-1) to classify road types and disease types, and when the disease recognition subsystem (3) outputs a new disease image, the unsupervised self-updating module (2-2) is used to update the new data to the existing data set to expand the data set, and finally form a disease database that classifies existing data and updates data in real time; wherein the ViT image classification module (2-1) uses a ViT image classification algorithm, combined with a Transformer encoder output to a multi-layer perceptron head, to accurately classify diseases of various road surfaces.

3. The road condition assessment system based on a vehicle-mounted mobile visual camera according to claim 1 is characterized in that: The disease recognition subsystem (3) first uses a vehicle-mounted mobile visual camera (1) to capture an original image, uses an image super-resolution module (3-1) to output a low-resolution original image into a super-resolution image or a high-resolution image, and then combines the road disease database (2) to input the training data into the disease recognition module (3-2), uses pre-trained weights, transfer learning strategy, attention mechanism, defect detection network and semi-supervised training to extract and analyze features and judge the disease and its type, and finally outputs a segmented image; wherein the image super-resolution module (3-1) uses the deep learning SRGAN image super-resolution reconstruction algorithm, and through the adversarial process of the generation network and the discrimination network, it can finally generate a super-resolution image with high-quality details; the disease recognition module (3-2) uses the FCN semantic segmentation deep learning model to output images of different diseases as segmented images of different colors.

4. The road surface damage assessment system based on a vehicle-mounted mobile visual camera according to claim 1 is characterized in that: The defect quantification subsystem (4) first performs a predefined classification on the segmented image output by the defect recognition subsystem (3), and then the defect quantity detection module (4-1) outputs the defect quantity. Finally, the defect stereo quantification module (4-2) detects the defect quantity, defect length, defect width, defect depth, defect area, defect ratio and overall road surface smoothness by combining the binocular principle; wherein the defect quantity is directly counted by the segmented image output, the defect length and width are calculated by the vehicle-mounted mobile visual camera and the road surface angle and the segmented image defect pixel length, the defect depth is calculated by the defect color value displayed by the vehicle-mounted mobile visual camera at different angles, the defect area and ratio are directly output by the color represented by different defects in the segmented image, and the overall road surface smoothness is judged by the relative distance between the road surface image posture taken by the vehicle-mounted mobile visual camera and the overall road surface and the specific defect quantification result.

5. The road condition assessment system based on a vehicle-mounted mobile visual camera according to claim 1 is characterized in that: The damage assessment subsystem (5) outputs the quantitative results of the damage quantification subsystem (4) and the damage identification subsystem (3) in the multi-class level assessment module (5-1), and compares the damage type and road surface type with the highway technical condition assessment standard specification to form an overall road surface assessment result of excellent, good, medium, poor, and poor. If the assessment level is poor or below, a hidden danger warning module (5-2) is used to issue a hidden danger warning, and the traffic level is divided according to the traffic conditions in combination with Beidou technology. Finally, the road surface is repaired in order of the level.

6. The road surface condition assessment method of the road surface condition assessment system based on a vehicle-mounted mobile visual camera according to claim 1, characterized in that: The steps are: Step (1) first classifies the road damage types using the ViT image classification module using the field collected data and the existing data set. Then, the new damage image is output to the unsupervised self-update module (2-2) through the damage recognition subsystem (3) to update the existing data set. Finally, all data sets form a complete database to provide support for deep learning model training. Step (2) The vehicle-mounted mobile visual camera (1) captures the original image, and then uses the image super-resolution module (3-1) to reconstruct the low-resolution original image into a high-resolution image or a super-resolution image. Then, the data trained by the pavement disease database 2 is input into the disease recognition module 3-2, and the pre-trained weights, transfer learning strategy, attention mechanism, defect detection network, and semi-supervised training weights are combined to extract and analyze features and judge the disease and its type, and finally the segmented image is output; Step (3) pre-classifies the segmented image output by the defect recognition subsystem (3), and then uses the defect quantity detection module (4-1) to output the defect quantity, and then uses the defect stereo quantification module (4-2) in combination with the binocular photography principle to detect the defect quantity, defect length, defect width, defect depth, defect area, defect ratio and overall road surface flatness; Finally, step (4) compares the quantification result output by the defect quantification subsystem (4) with the specification and inputs it into the multi-class grade assessment module (5-1) for road grade assessment. If it meets the specification, an assessment report is output. If it does not meet the specification, a hazard warning module (5-2) is used to issue a hazard warning. In addition, Beidou technology is used to check the road traffic conditions. Road maintenance is performed based on the road traffic conditions and the importance of the road. Practitioners directly use the road condition assessment system based on the vehicle-mounted mobile visual camera to carry out visual inspection on the road to be inspected by the driving vehicle. Finally, a road assessment report is directly generated and delivered to the owner, who then determines whether the road surface should be maintained.

Images