Visual image and radar image fused road disease inspection method

Abstract

The invention belongs to the technical field of road disease detection, and particularly relates to a visual image and radar image fused road disease detection method, which comprises the following steps: S1, a camera and a laser radar collect road data; s2, preprocessing the collected data of the laser radar; s3, fusing the image information and the radar information; s4, redundant results are removed; and S5, uploading the road disease information. Redundant data in a radar image are effectively reduced by optimizing a laser radar data processing flow, so that the storage pressure of mobile equipment is relieved, and by means of an innovative mode of deep fusion of visual image data and laser radar point cloud data, not only can the road disease type be accurately identified, but also the depth information of the disease can be acquired, and the accuracy of the road disease is improved. And the accuracy and efficiency of road disease detection are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of road damage detection, and in particular to a road damage inspection method that fuses visual images and radar images. Background Art

[0002] During the use of roads, due to the long-term effects of vehicle loads and environmental factors, aging and disease problems will occur. These diseases not only affect driving comfort, but may also cause traffic accidents and threaten driving safety. Human inspection and maintenance require a large amount of manpower and funds every year. Realizing automated detection of road diseases is of great significance to improving road maintenance efficiency and reducing traffic accidents caused by road diseases.

[0003] At present, the road disease detection schemes are mainly divided into two types: radar images and machine vision. Although the solution using LiDAR can have a good detection effect on small gaps in the road and can construct 3D images, the detection speed is slow and will affect normal traffic. The detection speed of machine vision is fast, but it cannot extract the height and depth information of bulges, subsidence, etc., and due to the high complexity of road images, the complex background of the image to be detected, and the different sizes of road diseases, there is a problem of poor detection effect of small targets.

[0004] In the prior art related to the present invention, "CN112817006A A vehicle-mounted intelligent road disease detection method and system" uses laser radar to determine suspected road disease areas and store them in the data cache area, and then uses deep learning algorithms to identify them. However, in sections with poor road conditions, there are problems such as large data cache volume, low detection speed, and limited generalization ability for unknown types or non-standard road diseases. "CN118015243A A lightweight road disease detection method based on YOLOV5" lightweights the network model by changing the network depth and width. Although it reduces the computational complexity, it may limit the model's ability to extract information on the feature map and affect the detection performance of small-sized road diseases. Therefore, we propose a road disease inspection method that fuses visual images and radar images to solve the above problems. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies of the prior art, the present invention provides a road damage inspection method by fusing visual images and radar images, which solves the problems raised in the above-mentioned background technology.

[0007] (II) Technical solution

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A road disease inspection method using a fusion of visual images and radar images comprises the following steps:

[0010] S1: Cameras and LiDAR collect road data, which includes the installation of collection equipment and synchronization of collected data;

[0011] S2: LiDAR data preprocessing operations, which include scanning measurement and information generation, point cloud downsampling, and ground data filtering;

[0012] S3: Fusion of image information and radar information, which includes feature input, splicing of input features, feature conversion, feature fusion generation, image feature post-processing and fusion result judgment;

[0013] S4: Redundant result removal, which includes IOU value calculation, IOU threshold setting and preliminary judgment, detection box category judgment and confidence comparison judgment;

[0014] S5: Upload of road damage information, which includes information addition and integration, human-computer interactive display and mobile module uploading to the terminal server.

[0015] Furthermore, the specific contents of the acquisition equipment construction in S1 are: to identify road defects such as longitudinal cracks, transverse cracks, cracks and collapses, use an area array camera and a 64-line laser radar, use the camera position as the origin of the local coordinate system, and calibrate the camera and laser radar through a chessboard and a calibration plate;

[0016] The specific content of data acquisition synchronization is: use an external synchronization signal generator to simultaneously trigger the acquisition of the camera and lidar, set the acquisition frequency of the two to be consistent, and add a unique identifier and timestamp to each data file.

[0017] Furthermore, the specific contents of the scanning measurement and information generation in S2 are: using a laser scanner to perform high-speed scanning measurement of road defects, recording information such as the three-dimensional coordinates of the measured target, and combining it with data from a global integrated navigation satellite system (GNSS) and an inertial measurement system (IMU) to generate a spatial point with position information and depth information;

[0018] The specific content of point cloud downsampling is as follows: using the downsampling method based on the voxel grid method, establishing an axial bounding box, dividing the bounding box into n equal parts along the coordinate axis, and then for each voxel (i.e., the small space unit after division), calculating the centroid of all points in it as the sampling value;

[0019] The specific content of filtering out ground data is: using a slope-based filtering algorithm, calculating the elevation difference between the current point and its neighboring points, setting an elevation difference threshold to determine whether the point is a ground point, traversing the point cloud data, retaining non-ground points, and discarding ground points.

[0020] Furthermore, the specific content of the feature input in S3 is: the feature input is an RGB image with a size of 648×648×3 and a dense depth map with a size of 648×648×2 generated by the point cloud and preprocessed, and then the RGB image and the point cloud depth map are input in square slices using a sliding window method;

[0021] The specific content of splicing the input features is as follows: the feature map input in the previous stage is used to fuse the two feature maps through the feature fusion module (CRC), and then a 1×1 convolution kernel is used to keep the size of the feature map unchanged, increase the nonlinear characteristics of the feature map, and promote the feature map to achieve cross-channel information interaction, thereby fully fusing the data features of the two feature maps;

[0022] The specific content of feature conversion is: using the linear transformation method, the transformation parameters x1, x2, y1, y2 are generated through the component T. In the component T, the merged features are received as input through the transformation module (TFN), and these transformation parameters are output, and finally these parameters are used to generate the fused features;

[0023] The specific content of feature fusion generation is as follows: Fusion feature generation (G) combines the original features with the transformed features and outputs the fused feature map. The output feature maps are RGB fusion map F rgb ' and PL fusion graph F pl ';

[0024] The specific contents of image feature post-processing are as follows: in the post-processing part, the output feature features of different stage levels are sent to the pyramid pooling module (PPM) so that the features at different scales will not be lost;

[0025] The specific content of the fusion result judgment is as follows: According to the different situations of laser radar and visual detection, the output of the detection result can be divided into the following situations: Ⅰ: When the matching degree between the depth information of the laser radar and the detection result in the image exceeds the set threshold, the detection algorithm will add the category information of the image detection to the target and output the category and depth information of the target;

[0026] II: If the 3D depth information of the LiDAR is projected on the image, but does not overlap with the 2D detection box in the image or the overlap is lower than the set threshold, the algorithm will not perform further fusion processing and only output the 2D detection box and classification information of the target;

[0027] III: When only the image recognition algorithm detects the target and the lidar does not detect the corresponding depth information, the algorithm will retain the two-dimensional detection box in the image.

[0028] Furthermore, the specific content of the IOU value calculation in S4 is: for a pair of boxes A and B to be compared, the intersection area (Intersection) and the union area (Union) are calculated: the IOU value is calculated by the formula IOU=Intersection / Union;

[0029] The specific contents of IOU threshold setting and preliminary judgment are as follows: the IOU threshold is set according to factors such as the type of road damage, image resolution, and actual detection requirements, and the IOU value of the detection frame A and frame B to be compared is calculated. If the IOU value is greater than the set threshold, the next step of comprehensive judgment is entered; if it is less than the threshold, it can be preliminarily judged that frame A and frame B do not represent the same target, and other detection frames are continued to be compared;

[0030] The specific content of the detection frame category judgment is: determine the categories of the detection frame A and frame B, and judge whether they are the same. Only when the categories of the two are the same, indicating that they are the same type of road disease objects, can they be further judged as the same target;

[0031] The specific content of the confidence comparison judgment is: when the detection box category is the same and the IOU value meets the conditions, obtain the confidence of box A and box B, and calculate the confidence difference. If the confidence of one box is higher than the other box to a set degree, the box is considered more reliable and retained, and the other box is considered redundant. If the confidence difference does not reach the set degree, proceed to the next step of other rule judgment.

[0032] Furthermore, the specific content of the information addition and integration in S5 is: after completing the disease identification, the system automatically obtains the GPS coordinate information of the current location, and matches it with the geographic information database to determine the road name; the specific content of the human-computer interaction display is: the human-computer interaction module uses a high-resolution display screen to display the disease information with a clear image interface; the specific content of the mobile module uploading to the terminal server is: the mobile module is connected to the terminal server through the 4G / 5G network.

[0033] (III) Beneficial effects

[0034] Compared with the prior art, the present invention provides a road damage inspection method by fusing visual images and radar images, which has the following beneficial effects:

[0035] 1. The combination of LiDAR and industrial cameras is used as road data collection equipment, which can not only identify the type of road damage, but also obtain the depth information of the damage. The three-dimensional point cloud data of LiDAR can assist the binocular camera to more accurately determine the location and scope of the damage, thereby improving the detection accuracy and efficiency.

[0036] 2. To address the problem of excessive LiDAR point cloud data, point cloud downsampling and ground point cloud filtering methods are used to retain the depth information of the defect without increasing the system's computational complexity.

[0037] 3. Applying the IOU deduplication algorithm in the video image detection process can eliminate repeated road disease detection results in consecutive frames, reduce redundant data, and alleviate storage pressure on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2 It is a schematic diagram of the process of segmenting ground point cloud and non-ground point cloud of the present invention;

[0040] Figure 3 This is a schematic diagram of the feature fusion network structure of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the CTG feature fusion module of the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the CRC feature extraction module of the present invention;

[0043] Figure 6 This is a schematic diagram of the structure of the TFN feature extraction module of the present invention;

[0044] Figure 7 It is a schematic diagram of the fusion result structure of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Example

[0047] like Figure 1-7 As shown, a road damage inspection method for fusing visual images and radar images is proposed in one embodiment of the present invention, comprising the following steps:

[0048] S1: Cameras and LiDAR collect road data, which includes the construction of collection equipment and synchronization of collected data;

[0049] The specific content of the acquisition equipment construction is: to identify road defects such as longitudinal cracks, transverse cracks, cracks and collapses, using area array cameras and 64-line laser radars. A camera is installed at the center line of the top of the test vehicle, located at the junction of the roof and the windshield, facing straight ahead and tilted downward by 5°. The camera position is used as the origin of the local coordinate system, and the camera and laser radar are calibrated using a chessboard and calibration plate.

[0050] The specific content of data acquisition synchronization is: use an external synchronization signal generator to simultaneously trigger the acquisition of the camera and lidar, set the acquisition frequency of the two to be consistent, and add a unique identifier and timestamp to each data file.

[0051] S2: preprocessing of lidar data, which includes scanning measurement and information generation, point cloud downsampling and filtering of ground data;

[0052] The specific contents of scanning measurement and information generation are: using laser scanners to perform high-speed scanning measurement of road defects, recording information such as the three-dimensional coordinates of the measured targets, and combining them with the data of the Global Integrated Navigation Satellite System (GNSS) and Inertial Measurement System (IMU) to generate spatial points with position information and depth information;

[0053] The specific content of point cloud downsampling is: using the downsampling method based on the voxel grid method, establishing an axial bounding box, dividing the bounding box into n equal parts along the coordinate axis, and calculating the centroid of all points in each voxel as the sampling value to simplify the point cloud data and reduce the storage capacity of mobile devices. center ,y center , z center are the centroid of each point after calculation, x i ,y i , z i The coordinates of the sampling point on the xyz axis are expressed as:

[0054]

[0055]

[0056] The specific content of filtering out ground data is: using a slope-based filtering algorithm, calculating the difference in elevation between the current point and its neighboring points, setting an elevation difference threshold to determine whether the point is a ground point, traversing the point cloud data, retaining non-ground points, and discarding ground points. According to the elevation value of the traversed point cloud data, determine the lowest point in the moving window and mark it as 0, set the maximum slope of the terrain, and calculate the elevation difference threshold to determine the remaining points. The height difference function calculation formula is:

[0057]

[0058] Where s represents the maximum terrain slope percentage, d represents the horizontal distance between two points, and δ z is the standard deviation.

[0059] According to the elevation value of the traversal point cloud data, V i is the lowest point in the moving window and is marked as 0. The maximum slope s of the terrain is set for the ground point. The height difference threshold h is obtained by calculating the product of the horizontal distance between the two points and s. max , determine whether the remaining points are ground points. If point V i+1 With point V i The height difference is less than h max , so point V i+1 The mark is the ground point; and point V i+2 With point V i The height difference is greater than h max , so point V i+2 Mark as non-ground points. Continue to traverse the grid point cloud data until the traversal is completed. All the points marked as non-ground points are retained, and the rest are discarded as ground points. Figure 2 shown.

[0060] S3: Fusion of image information and radar information, which includes feature input, splicing of input features, feature conversion, feature fusion generation, image feature post-processing and fusion result judgment;

[0061] The feature extraction and feature map fusion of RGB images and radar images (PL) are completed using the CSPnet backbone network of YOLOV5. The backbone network is divided into two branches, one for RGB image feature extraction and the other for radar image (PL image) feature extraction. The extracted feature maps are divided into four stages: Z1, Z2, Z3, and Z4. The two sets of feature maps generated in each stage are spliced ​​and output as the feature map of this stage. The network structure is as follows: Figure 3 As shown in Figure 2, the proposed information fusion process CTG consists of three processes: concatenation (C), transformation (T) and generation (G).

[0062] The specific content of the image feature input is as follows: the input is an RGB image of size 648×648×3 and a dense depth map of size 648×648×2 generated from the point cloud and preprocessed. Then the RGB image and point cloud depth map are input in square slices using a sliding window method.

[0063] The specific content of feature splicing (C) is as follows: In the splicing process (C), the feature map input in the previous stage is used to fuse the two feature maps through the CRC module. The module CRC used for feature fusion is composed of the convolution layer Conv, the activation function ReLu and the fully connected layer Concat. First, the convolution layer is used to perform a convolution operation on the input feature map. The convolution layer can extract the features of the local area. Next, the activation function ReLU is used to increase the nonlinear expression ability of the model and help the network learn more complex features. After the activation function, a fully connected layer is used to splice the feature maps of different branches in the depth direction, that is, Concatenation. This allows the feature information of different branches to be processed together in subsequent networks. After the construction, a 1×1 convolution kernel is used to keep the size of the feature map unchanged, increase the nonlinear features of the feature map, and enable the feature map to achieve cross-channel information interaction, thereby fully fusing the data features of the two feature maps. The CRC structure diagram is as follows Figure 5 shown.

[0064] The specific content of feature transformation (T) is as follows: after feature fusion, the RGB and PL features are transformed into each other's data space through transformation (T). In feature transformation T, there are differences between RGB information and PL information, and the features extracted from the RGB image are inconsistent with the features extracted from the PL image. In order to solve this problem of inconsistent feature space, we use the linear transformation method to generate transformation parameters x1, x2, y1, and y2 through component T. In component T, the merged features are received as input by the transformation module (TFN) and these transformation parameters are output. Finally, these parameters are used to generate the fused features. The weights of the convolutional layer in the TFN module are obtained through the back-propagation algorithm during the training process. The weights of the convolutional layer are initialized with uniform distribution. During the training process, these weights are continuously updated by the gradient descent method to minimize the loss function, thereby generating linear parameters. The TFN structure diagram is shown in the figure below. Figure 6 shown.

[0065] The specific content of the fusion feature generation (G) is as follows: after the fusion feature generation (G) merges the original features with the transformed features, it outputs the fused feature map. The output feature maps are RGB fusion map F rgb ' and PL fusion graph F pl '. The calculation formula is as follows:

[0066] F rgb '=F rgb +(x2F pl +y2)

[0067] F pl '=F pl +(x1F rgb +x2)

[0068] Among them, F rgb is the image before fusion, F pl is the radar image before fusion, x1, x2, y1, y2 are the linear parameters generated by the TFN module. The CTG structure diagram is shown in Figure 4 shown.

[0069] The specific content of image feature post-processing is as follows: in the post-processing part, the output feature features of different stages and levels are sent to the pyramid pooling module (PPM) so that features at different scales will not be lost.

[0070] The specific content of the fusion result judgment is: According to the different situations of laser radar and visual detection, the output of the detection results can be divided into the following situations:

[0071] Ⅰ: When the depth information of the LiDAR matches the detection results in the image to a certain degree, the detection algorithm adds the category information detected in the image to the target and outputs the category and depth information of the target.

[0072] Ⅱ: If the 3D depth information of the LiDAR is projected on the image, but does not overlap with the 2D detection box in the image or the overlap is lower than the set threshold, the algorithm will not perform further fusion processing and only output the 2D detection box and classification information of the target.

[0073] III: When only the image recognition algorithm detects the target and the lidar does not detect the corresponding depth information, the algorithm will retain the two-dimensional detection box in the image.

[0074] The fusion result diagram is as follows: Figure 7 shown.

[0075] S4: Redundant result filtering, which includes IOU value calculation, IOU threshold setting and preliminary judgment, detection box category judgment and confidence comparison judgment;

[0076] In multiple consecutive frames, the same road damage object has the following characteristics: the same category, that is, the category of the same road damage object in different frames is consistent. After coordinate unification, the position between adjacent images does not change much, which makes the detection box of the same object have a large IOU (intersection-over-union ratio).

[0077] The specific content of the IOU value calculation method is as follows: For a pair of boxes A and B to be compared, the intersection area is calculated as follows: First, determine the coordinate range of box A and box B on the two-dimensional plane. Assume that the coordinate of the upper left corner of box A is (x a1 ,y a1 ), the coordinate of the lower right corner is (x a2 ,ya2 ); the coordinates of the upper left corner of box B are (x b1 ,y b1 ), the coordinate of the lower right corner is (x b2 ,y b2 );

[0078] Calculate the coordinates of the upper left corner of the intersection area (x i1 ,y i1 ) and the lower right corner coordinate (x i2 ,y i2 );

[0079] x i1 =max(x a1 ,x b1 )

[0080] y i1 =max(y a1 ,y b1 )

[0081] x i2 =min(x a2 ,x b2 )

[0082] y i2 =min(y a2 ,y b2 )

[0083] If x i2 >x i1 And y i2 >y i1 , then the intersection region exists, and its area is (x i2 -x i1 )×(y i2 -y i1 ); otherwise the area of ​​the intersection is 0.

[0084] Calculation of the area of ​​the union region:

[0085] The area of ​​box A is: (x a2 -x a1 )×(y a2 -y a1 )

[0086] The area of ​​box B is: (x b2 -x b1 )×(y b2 -y b1 )

[0087] The area of ​​the union region is the sum of the area of ​​box A and the area of ​​box B minus the area of ​​the intersection region, that is:

[0088] Union=(xa2 -x a1 )×(y a2 -y a1 )+(x b2 -x b1 )×(y b2 -y b1 )-Intersection

[0089] Calculate the IOU value:

[0090] According to the intersection area and union area calculated above, the IOU value is calculated by the formula IOU=Intersection / Union.

[0091] The specific contents of IOU threshold setting and preliminary judgment are as follows: set the IOU threshold according to factors such as the type of road damage, image resolution, and actual detection requirements. For example, a suitable value can be selected between 0.5 and 0.7. Calculate the IOU value of the detection frame A and frame B to be compared. If the IOU value is greater than the set threshold, proceed to the next step of comprehensive judgment; if it is less than the threshold, it can be preliminarily judged that frame A and frame B do not represent the same target, and continue to compare other detection frames.

[0092] The specific content of the detection frame category judgment is: determine the categories of detection frame A and frame B, and judge whether they are the same. Only when the categories of the two are the same, indicating that they are the same type of road disease objects, can they be further judged as the same target. For example, if frame A is a crack disease category, frame B must also be a crack disease category to continue the subsequent judgment, otherwise they can be judged as different disease objects and other detection frames can be processed.

[0093] The specific content of the confidence comparison judgment is: when the detection box categories are the same and the IOU value meets the conditions, obtain the confidence of box A and box B. Calculate the confidence difference, for example, to determine whether it is greater than 0.1 (can be adjusted according to actual conditions). If the confidence of one box is higher than the other box to a set degree, the box is considered more reliable and retained, and the other box is considered redundant; if the confidence difference does not reach the set degree, proceed to the next step of other rule judgment.

[0094] S5: Uploading of road damage information, which includes information addition and integration, human-computer interactive display and mobile module uploading to the terminal server;

[0095] The specific contents of information attachment and integration are as follows: after completing the disease identification, the system automatically obtains the GPS coordinate information of the current location and matches it with the geographic information database to determine the road name. The identified road diseases are classified in detail, such as cracks are divided into longitudinal, transverse, and cracking, and the information such as the depth range of the subsidence is marked. The disease information is associated with the image and attached, so that each disease image contains key information such as GPS coordinates, road name, and disease type.

[0096] The specific contents of the human-computer interaction display are as follows: The human-computer interaction module uses a high-resolution display screen to display disease information with a clear image interface. The image display area presents high-definition disease images, and uses different colors and logos to highlight the location of the disease, such as red lines to outline cracks and yellow areas to mark subsidence. The information display area lists text information such as road names, GPS coordinates, and disease types, and provides operation functions. It has an information retrieval function, and can query historical disease records by keywords.

[0097] The specific content of the mobile module uploading to the terminal server is as follows: the mobile module is connected to the terminal server through the 4G / 5G network. The disease information is compressed and encrypted before uploading. After receiving, the terminal server decrypts and decompresses it, and stores the disease information in the database in a preset manner.

[0098] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A road disease inspection method by fusing visual images and radar images, characterized in that: The following steps are involved: S1: Cameras and LiDAR collect road data, which includes the installation of collection equipment and synchronization of collected data; S2: LiDAR data preprocessing operations, which include scanning measurement and information generation, point cloud downsampling, and ground data filtering; S3: Fusion of image information and radar information, which includes feature input, splicing of input features, feature conversion, feature fusion generation, image feature post-processing and fusion result judgment; S4: Redundant result removal, which includes IOU value calculation, IOU threshold setting and preliminary judgment, detection box category judgment and confidence comparison judgment; S5: Upload of road damage information, which includes information addition and integration, human-computer interactive display and mobile module uploading to the terminal server.

2. The road damage inspection method of claim 1, characterized in that: The specific contents of the acquisition equipment construction in S1 are: to identify road defects such as longitudinal cracks, transverse cracks, cracks and collapses, use an area array camera and a 64-line laser radar, use the camera position as the origin of the local coordinate system, and calibrate the camera and laser radar through a chessboard and a calibration plate; The specific content of data acquisition synchronization is: use an external synchronization signal generator to simultaneously trigger the acquisition of the camera and lidar, set the acquisition frequency of the two to be consistent, and add a unique identifier and timestamp to each data file.

3. The road damage inspection method of claim 1, characterized in that: The specific contents of the scanning measurement and information generation in S2 are: using a laser scanner to perform high-speed scanning measurement of road defects, recording information such as the three-dimensional coordinates of the measured target, and combining it with the data of the Global Integrated Navigation Satellite System (GNSS) and the Inertial Measurement System (IMU) to generate a spatial point with position information and depth information; The specific content of point cloud downsampling is as follows: using the downsampling method based on the voxel grid method, establishing an axial bounding box, dividing the bounding box into n equal parts along the coordinate axis, and then for each voxel (i.e., the small space unit after division), calculating the centroid of all points in it as the sampling value; The specific content of filtering out ground data is: using a slope-based filtering algorithm, calculating the elevation difference between the current point and its neighboring points, setting an elevation difference threshold to determine whether the point is a ground point, traversing the point cloud data, retaining non-ground points, and discarding ground points.

4. The road damage inspection method of claim 1, characterized in that: The specific content of the feature input in S3 is as follows: the feature input is an RGB image with a size of 648×648×3 and a dense depth map with a size of 648×648×2 generated from the point cloud and preprocessed, and then the RGB image and the point cloud depth map are input in square slices using a sliding window method; The specific content of splicing the input features is as follows: the feature map input in the previous stage is used to fuse the two feature maps through the feature fusion module (CRC), and then a 1×1 convolution kernel is used to keep the size of the feature map unchanged, increase the nonlinear characteristics of the feature map, and promote the feature map to achieve cross-channel information interaction, thereby fully fusing the data features of the two feature maps; The specific content of feature conversion is: using the linear transformation method, the transformation parameters x1, x2, y1, y2 are generated through the component T. In the component T, the merged features are received as input through the transformation module (TFN), and these transformation parameters are output, and finally these parameters are used to generate the fused features; The specific content of feature fusion generation is as follows: Fusion feature generation (G) combines the original features with the transformed features and outputs the fused feature map. The output feature maps are RGB fusion map F rgb ' and PL fusion graph F pl '; The specific contents of image feature post-processing are as follows: in the post-processing part, the output feature features of different stage levels are sent to the pyramid pooling module (PPM) so that the features at different scales will not be lost; The specific content of the fusion result judgment is as follows: According to the different situations of laser radar and visual detection, the output of the detection result can be divided into the following situations: Ⅰ: When the matching degree between the depth information of the laser radar and the detection result in the image exceeds the set threshold, the detection algorithm will add the category information of the image detection to the target and output the category and depth information of the target; II: If the 3D depth information of the LiDAR is projected on the image, but does not overlap with the 2D detection box in the image or the overlap is lower than the set threshold, the algorithm will not perform further fusion processing and only output the 2D detection box and classification information of the target; III: When only the image recognition algorithm detects the target and the lidar does not detect the corresponding depth information, the algorithm will retain the two-dimensional detection box in the image.

5. The road damage inspection method of claim 1, characterized in that: The specific content of the IOU value calculation in S4 is: for a pair of boxes A and B to be compared, the intersection area (Intersection) and the union area (Union) are calculated: the IOU value is calculated by the formula IOU=Intersection / Union; The specific contents of IOU threshold setting and preliminary judgment are as follows: the IOU threshold is set according to factors such as the type of road damage, image resolution, and actual detection requirements, and the IOU value of the detection frame A and frame B to be compared is calculated. If the IOU value is greater than the set threshold, the next step of comprehensive judgment is entered; If it is less than the threshold, it can be preliminarily determined that box A and box B do not represent the same target, and the comparison with other detection boxes continues; The specific content of the detection frame category judgment is: determine the categories of the detection frame A and frame B, and judge whether they are the same. Only when the categories of the two are the same, indicating that they are the same type of road disease objects, can they be further judged as the same target; The specific content of the confidence comparison judgment is: when the detection box category is the same and the IOU value meets the conditions, obtain the confidence of box A and box B, and calculate the confidence difference. If the confidence of one box is higher than the other box to a set degree, the box is considered more reliable and retained, and the other box is considered redundant. If the confidence difference does not reach the set degree, proceed to the next step of other rule judgment.

6. The road damage inspection method of claim 1 by fusing visual images and radar images, characterized in that: The specific contents of information addition and integration in S5 are as follows: after completing the disease identification, the system automatically obtains the GPS coordinate information of the current location, and matches it with the geographic information database to determine the road name; the specific contents of human-computer interaction display are as follows: the human-computer interaction module uses a high-resolution display screen to display the disease information with a clear image interface; the specific contents of the mobile module uploading to the terminal server are as follows: the mobile module is connected to the terminal server through the 4G / 5G network.

Images