A highway intelligent inspection method and equipment based on multi-dimensional visual fusion

Abstract

The present invention discloses a method and equipment for intelligent highway inspection based on multi-dimensional visual fusion, which relates to the technical field of highway inspection. The method comprises the following steps: using a visual detection module to obtain foreground photos and laser point clouds, obtaining navigation data through a combined positioning module, and transmitting the data to an industrial control module; using an image algorithm to perform uniform light enhancement on the foreground photos to obtain intermediate recognition result data, and using the laser point cloud after online motion compensation to process the intermediate recognition result data in real time to obtain road surface disease inspection results and important highway facility target inspection results; based on the important highway facility target inspection results, combined with the intermediate recognition result data and the corresponding laser point cloud, the missing and deformed damage of the facility target is judged. The present invention integrates the multi-dimensional information of 2D images and 3D laser point clouds, which can not only solve the intelligent inspection of road surface diseases, but also automatically judge the missing and deformed damage status of important highway facility targets.

Description

Technical Field

[0001] The present invention relates to the technical field of highway inspection, and in particular to a method and equipment for intelligent highway inspection based on multi-dimensional vision fusion. Background Art

[0002] Highway inspection is an important part of road maintenance work, and its main purpose is to quickly discover and record road diseases so that road repairs can be carried out in a timely manner. Traditional highway inspection methods mainly rely on manual inspections, and workers need to observe and record the location and type of road diseases with the naked eye; however, with the increase in traffic volume and the expansion of road networks, the limitations of manual inspections are becoming increasingly apparent. Not only does it require a large amount of manpower and material resources, but there are also problems of missed inspections and false inspections, and the detection accuracy and efficiency are low; in addition, activities such as taking photos and measuring during manual inspections may also affect the normal operation of traffic.

[0003] With the development of machine vision and artificial intelligence technology, highway inspection technology is gradually moving towards automation. At present, mainstream automatic inspection equipment mostly uses a single front-view camera installed on the roof, mainly using 2D vision technology to identify road defects. This automated inspection method has improved the efficiency and accuracy of inspections to a certain extent, reduced reliance on manual inspections, and can quickly discover and record road defects.

[0004] However, the existing highway inspection technology still has some shortcomings. First, the visual method of a single forward-looking camera is difficult to fully identify the defects of infrastructure along the highway, especially in complex road conditions, such as bridges and tunnels, and its detection effect is limited; second, the existing inspection technology lacks the ability to reliably locate the defects and cannot accurately reflect the dynamic changes in road conditions throughout the life cycle of the highway; in addition, with the diversified development of highway inspection service scenarios, it is not only necessary for the inspection equipment to be able to efficiently identify road defects, but also to be able to provide information on the status of highway facilities and traffic risk status in a timely manner, so as to provide higher-quality data support for daily maintenance decisions, and the existing highway inspection technology is still insufficient in responding to the diversified inspection needs.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the deficiencies in the prior art, the present invention provides an intelligent highway inspection method and equipment based on multi-dimensional visual fusion. The method and equipment have the advantages of not only being able to solve the intelligent inspection of road surface defects but also being able to automatically identify the missing and deformed and damaged status of important highway facility targets by fusing 2D images and 3D laser point cloud multi-dimensional information. The method further solves the problems that the existing highway inspection technology is difficult to comprehensively identify the defects of infrastructure along the highway, lacks the ability to reliably locate defects, cannot accurately reflect the dynamic changes of road conditions throughout the life cycle of the highway, and is difficult to provide information on the status of highway facilities and traffic risk status in a timely manner, and is still insufficient in responding to diversified inspection needs.

[0008] (II) Technical solution

[0009] In order to achieve the advantages of not only solving the intelligent inspection of road surface diseases but also automatically identifying the missing and deformed damaged states of important highway facilities by fusing 2D images and 3D laser point cloud multi-dimensional information, the specific technical solutions adopted by the present invention are as follows:

[0010] According to one aspect of the present invention, a highway intelligent inspection method based on multi-dimensional visual fusion is provided, and the highway intelligent inspection method based on multi-dimensional visual fusion comprises the following steps:

[0011] S1. According to the newly built inspection project, configure the inspection route, direction and stake number starting point, and synchronize the satellite timing of each module; based on the preset frequency, use the visual detection module to obtain the foreground photo and laser point cloud, obtain the navigation data through the combined positioning module, and transmit the foreground photo, laser point cloud and navigation data to the industrial control module;

[0012] S2. Based on the industrial control module, the image algorithm is used to enhance the uniform light of the foreground photo to obtain the intermediate recognition result data, and the laser point cloud after online motion compensation is used to process the intermediate recognition result data in real time to obtain the road surface disease inspection results and the important highway facility target inspection results;

[0013] S3. Use the industrial control module to store the pavement disease inspection results and upload them to the cloud, and output the pavement disease inspection result report and the corresponding pavement disease photos;

[0014] S4. Based on the inspection results of important highway facility targets, combined with the intermediate recognition result data uploaded to the cloud by the industrial control module and the corresponding laser point cloud, the missing, deformed and damaged facility targets are identified, and the inspection result report of important highway facility targets and the corresponding photos of abnormal facilities are exported.

[0015] Furthermore, the intermediate recognition result data includes the corner pixel coordinates and the center pixel coordinates of the road surface disease recognition frame, the corner pixel coordinates and the center pixel coordinates of the important highway facility target recognition frame, the uniformly illuminated foreground photo containing the recognition frame, the facility target type, and the lane where the road surface disease is located;

[0016] Facility target types include pavement damage types and important highway facility target types; pavement damage types include spills, potholes, transverse cracks, longitudinal cracks, block cracks and strip repairs; important highway facility target types include lane lines, mileage posts, anti-glare panels, contour markers and signboards.

[0017] Furthermore, based on the industrial control module, the image algorithm is used to enhance the uniform light of the foreground photo to obtain the intermediate recognition result data, and the laser point cloud after online motion compensation is used to process the intermediate recognition result data in real time to obtain the road surface disease inspection results and the important highway facility target inspection results, including:

[0018] S21, based on the foreground photo, using the image brightening curve model to process the pixels in the foreground photo to obtain a brightened foreground photo, and based on the trained image recognition model, inputting the brightened foreground photo to output intermediate recognition result data;

[0019] S22. Based on the laser point cloud, the camera distortion projection model is used to perform camera-lidar calibration, and the laser radar coordinate system-local independent coordinate system conversion is performed using the laser radar-inertial navigation system calibration model to obtain camera-lidar calibration parameters and laser radar-integrated navigation system calibration parameters;

[0020] S23, calculating the physical size of the identification frame according to the pixel coordinates of the corner points of the road surface disease identification frame, the pixel coordinates of the corner points of the important highway facility target identification frame, and the camera-lidar calibration parameters, and obtaining the length attribute and area attribute of the road surface disease;

[0021] S24. Based on the central pixel coordinates of the pavement disease identification frame, the central pixel coordinates of the important highway facility target identification frame, the camera-lidar calibration parameters and the lidar-combined navigation system calibration parameters, the longitude and latitude coordinates of the central pixel point of the identification frame are calculated to obtain the geographical locations of the pavement disease and the important highway facility target.

[0022] Furthermore, based on the foreground photo, the image brightening curve model is used to process the pixels in the foreground photo to obtain a brightened foreground photo, and based on the trained image recognition model, the brightened foreground photo is input, and the output intermediate recognition result data includes:

[0023] S211, establishing a bright and dark sample data set of images according to sample images under different sensors, different weather conditions and different lighting conditions;

[0024] S212, based on the image brightening curve model, using the brightening function to process each pixel in the foreground photo to obtain a brightened foreground photo;

[0025] S213, building a model network based on depth-separable convolution and dilated convolution, and constructing a loss function based on spatial consistency error and color constant error to obtain an image processing model, and using an image brightness and darkness sample data set to train an image processing recognition model;

[0026] The expression of spatial consistency error is:

[0027]

[0028] In the formula, is the spatial consistency error, K is the number of pixels, i To traverse the pixels, For the i The 4-neighborhood of pixels, Y i and Y j Respectively i The enhanced image and j An enhanced image, I i and I j Respectively i The input image and j input images;

[0029] The expression of color constancy error is:

[0030]

[0031] In the formula, is the color constant error, For all the two combinations of the three color channels traversed, is the average brightness of color channel p; For color channels q The average brightness of

[0032] S214, based on the trained image processing recognition model, input the brightened foreground photo, output the intermediate recognition result data, and transmit it to the industrial control module through the ROS2 node.

[0033] Furthermore, based on the laser point cloud, the camera distortion projection model is used to perform camera-lidar calibration, and the lidar coordinate system-local independent coordinate system conversion is performed using the lidar-inertial navigation system calibration model. The camera-lidar calibration parameters and lidar-integrated navigation system calibration parameters include:

[0034] S221. Based on radial distortion and tangential distortion, establish a camera distortion projection model, and determine camera-lidar calibration parameters;

[0035] S222, based on the laser radar-inertial navigation system calibration model, converting the laser radar point from the laser radar coordinate system to the local independent coordinate system to obtain the point coordinates in the local independent coordinate system;

[0036] S223, using a rotation matrix, by aligning the directions of the local coordinate system and the earth-centered earth-fixed space rectangular coordinate system, transforming the local independent coordinate system into the earth-centered earth-fixed space rectangular coordinate system;

[0037] S224. Translate the middle aligned coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system to obtain the laser radar-integrated navigation system calibration parameters.

[0038] Furthermore, the expression of radial distortion is:

[0039]

[0040] In the formula, for x Axial distortion, for y Axial distortion, are the image coordinates without distortion correction, is the first radial distortion parameter, is the second radial distortion parameter, is the third radial distortion parameter, r is the distance from the current point to the center of the image;

[0041] The expression of tangential distortion is:

[0042]

[0043] In the formula, for x Axis tangential distortion, for y Axis tangential distortion, is the first tangential distortion parameter, is the second tangential distortion parameter;

[0044] The expression of the camera distortion projection model is:

[0045]

[0046]

[0047] In the formula, is the pixel coordinate collected by the camera, is the camera internal parameter, and They are the rotation matrix and translation matrix from the laser radar to the camera respectively; forx Total axis distortion, for y Total axis distortion, for x Axial distortion, for y Axial distortion, for x Axis tangential distortion, for y Axis tangential distortion, For LiDAR z Axis point cloud coordinates;

[0048] The expression of the laser radar-inertial navigation system calibration model is:

[0049]

[0050] In the formula, is the coordinate of the laser point in the Earth-centered Earth-fixed space rectangular coordinate system, is the coordinate value of the Earth-centered Earth-fixed space rectangular coordinate system of the current inertial navigation receiver, is the rotation matrix from the inertial navigation coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system, is the rotation matrix from the laser radar coordinate system to the inertial navigation coordinate system, is the translation between the laser radar and the inertial navigation receiver, is the spatial rectangular coordinate of the laser point in the laser radar coordinate system.

[0051] According to another aspect of the present invention, there is also provided a highway intelligent inspection equipment based on multi-dimensional vision fusion, the highway intelligent inspection equipment based on multi-dimensional vision fusion comprising:

[0052] Automobile carrier, used to install visual inspection module, combined positioning module, synchronous control module, industrial control module and power supply module;

[0053] Visual inspection module, used to obtain 2D foreground images and 3D laser point cloud data within the highway inspection field of view, and identify inspection targets;

[0054] Combined positioning module, used to provide location information of vehicle carrier and inspection target;

[0055] Synchronous control module, used to complete the clock synchronization of the visual inspection module and the industrial control module;

[0056] Industrial control module, used to control the opening, closing and configuration of the visual inspection module, combined positioning module and synchronous control module, and automatically analyze the inspection data;

[0057] The power supply module is used to provide continuous power supply to the visual inspection module, combined positioning module, synchronous control module and industrial control module.

[0058] Furthermore, the vehicle carrier includes an inspection vehicle body; a roof clamp is provided on one side of the top of the inspection vehicle body, a bottom plate is provided on the top of the roof clamp, an outer shell is provided on the top of the bottom plate, GPS antennas are penetrated at both ends of the inner side of the outer shell, a partition is penetrated through the middle part of the inner side of the outer shell, the partition is fixedly connected to the outer shell through a snap-in groove, and a laser radar is provided on one side of the top of the partition, an industrial camera lens assembly is provided on the other side of the top of the partition, an inertial navigation is provided on one side of the bottom end of the partition, and a time synchronization box is provided on the other side of the bottom end of the partition.

[0059] Furthermore, the inspection vehicle body includes a driver's seat and a trunk, a display is provided on one side of the interior of the driver's seat, a switch is provided on one side of the interior of the trunk, a power adapter is provided on the other side of the interior of the trunk, an industrial computer is provided at one end of the interior of the trunk, and a battery is provided at the other end of the interior of the trunk; the visual inspection module includes a laser radar and an industrial camera lens assembly; the combined positioning module includes a GPS antenna and an inertial navigation; the synchronous control module includes a time synchronization box; the industrial control module includes a switch and an industrial computer; the power supply module includes a power adapter and a battery.

[0060] Furthermore, the industrial camera lens assembly includes a camera arranged in the middle of the inner side of the shell, a lens is arranged at one end of the camera, a first fixing plate is arranged at the bottom end of the camera, connecting arms are symmetrically arranged on both sides of the bottom end of the first fixing plate, a supporting arm is arranged at one end of the connecting arm, and a second fixing plate is arranged at the bottom end of the supporting arm; adjustment slots are opened on both sides of the second fixing plate, and an adjustment connecting rod is arranged at the other end of the connecting arm through the adjustment slot, and a plurality of nuts matching the second fixing plate are sleeved on the outer side of the adjusting connecting rod.

[0061] (III) Beneficial effects

[0062] Compared with the prior art, the present invention provides a highway intelligent inspection method and equipment based on multi-dimensional visual fusion, which has the following beneficial effects:

[0063] (1) The present invention adopts a 2D+3D multi-dimensional vision fusion mode. Compared with the mainstream mode of highway inspection that only uses 2D vision in the current market, it has a more comprehensive inspection capability. By combining 2D images and 3D laser point cloud multi-dimensional information, it can not only solve the intelligent inspection of road surface diseases, but also has the ability to automatically determine the missing, deformed and damaged status of important highway facilities.

[0064] (2) The present invention adopts a tightly coupled combined positioning method of GNSS positioning + IMU inertial measurement + vehicle wheel speed meter compensation, combined with the 3D ranging information provided by the lidar, which can achieve centimeter-level accuracy in geographical positioning of road surface defects and important highway facilities, and has the ability to compare and trace road surface defects and facility status over multiple periods.

[0065] (3) The present invention has the ability to identify pavement defects with millimeter-level accuracy and can achieve objective measurement of the physical dimensions of pavement defects based on 3D lidar point cloud data.

[0066] (4) Each module of the present invention adopts an integrated structure and is directly installed on the roof through a luggage rack. It is lightweight and easy to install. The installation height and angle of the visual inspection module can be adjusted inside the structure, and it can be adapted to different models of vehicle carriers.

[0067] (5) The present invention has strong scalability and can be used in a variety of application scenarios. The point cloud data collected by the three-dimensional laser radar can also be used for digital modeling of highway infrastructure, asset inventory and high-precision map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 is a flow chart of a highway intelligent inspection method based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0070] Figure 2 It is a principle block diagram of a highway intelligent inspection equipment based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0071] Figure 3 is a structural schematic diagram of a highway intelligent inspection equipment based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0072] Figure 4 is a structural schematic diagram of another angle of the intelligent highway inspection equipment based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0073] Figure 5 is a structural schematic diagram of an industrial camera lens assembly in a highway intelligent inspection equipment based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0074] Figure 6is a structural schematic diagram of another angle of the industrial camera lens assembly in the highway intelligent inspection equipment based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0075] Figure 7 is a camera-lidar coordinate system transformation relationship diagram in a highway intelligent inspection method based on multi-dimensional visual fusion according to an embodiment of the present invention;

[0076] Figure 8 It is a specific implementation diagram of the highway intelligent inspection method based on multi-dimensional visual fusion according to an embodiment of the present invention.

[0077] In the figure:

[0078] 1. Inspection vehicle body; 2. Roof clamp; 3. Bottom plate; 4. Casing; 5. Switch; 6. Power adapter; 7. Battery; 8. Industrial computer; 9. Display; 10. GPS antenna; 11. LiDAR; 12. Industrial camera lens assembly; 1201. Camera; 1202. Lens; 1203. First fixing plate; 1204. Second fixing plate; 1205. Adjusting connecting rod; 1206. Nut; 1207. Connecting arm; 1208. Support arm; 13. Time synchronization box; 14. Inertial navigation. DETAILED DESCRIPTION

[0079] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0080] According to an embodiment of the present invention, a method and equipment for intelligent highway inspection based on multi-dimensional visual fusion are provided.

[0081] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 and Figure 8 As shown, according to one embodiment of the present invention, a method for intelligent highway inspection based on multi-dimensional visual fusion is provided, comprising the following steps:

[0082] S1. According to the newly built inspection project, configure the inspection route, direction and stake number starting point, and synchronize the satellite timing of each module; based on the preset frequency, use the visual detection module to obtain the foreground photo and laser point cloud, obtain the navigation data through the combined positioning module, and transmit the foreground photo, laser point cloud and navigation data to the industrial control module;

[0083] S2. Based on the industrial control module, the image algorithm is used to enhance the uniform light of the foreground photo to obtain the intermediate recognition result data, and the laser point cloud after online motion compensation is used to process the intermediate recognition result data in real time to obtain the road surface disease inspection results and the important highway facility target inspection results;

[0084] S3. Use the industrial control module to store the pavement disease inspection results and upload them to the cloud, and output the pavement disease inspection result report and the corresponding pavement disease photos;

[0085] S4. Based on the inspection results of important highway facility targets, combined with the intermediate recognition result data uploaded to the cloud by the industrial control module and the corresponding laser point cloud, the missing, deformed and damaged facility targets are identified, and the inspection result report of important highway facility targets and the corresponding photos of abnormal facilities are exported.

[0086] In one embodiment, the intermediate recognition result data includes the corner pixel coordinates and the center pixel coordinates of the road surface disease recognition frame, the corner pixel coordinates and the center pixel coordinates of the important highway facility target recognition frame, the uniformly illuminated foreground photo containing the recognition frame, the facility target type, and the lane where the road surface disease is located;

[0087] Facility target types include pavement damage types and important highway facility target types; pavement damage types include spills, potholes, transverse cracks, longitudinal cracks, block cracks and strip repairs; important highway facility target types include lane lines, mileage posts, anti-glare panels, contour markers and signboards.

[0088] In one embodiment, based on the industrial control module, the image algorithm is used to perform uniform light enhancement on the foreground photo to obtain intermediate recognition result data, and the laser point cloud after online motion compensation is used to process the intermediate recognition result data in real time to obtain the road surface disease inspection results and the important highway facility target inspection results including:

[0089] S21, based on the foreground photo, using the image brightening curve model to process the pixels in the foreground photo to obtain a brightened foreground photo, and based on the trained image recognition model, inputting the brightened foreground photo to output intermediate recognition result data;

[0090] S22. Based on the laser point cloud, the camera distortion projection model is used to perform camera-lidar calibration, and the laser radar coordinate system-local independent coordinate system conversion is performed using the laser radar-inertial navigation system calibration model to obtain camera-lidar calibration parameters and laser radar-integrated navigation system calibration parameters;

[0091] S23, calculating the physical size of the identification frame according to the pixel coordinates of the corner points of the road surface disease identification frame, the pixel coordinates of the corner points of the important highway facility target identification frame, and the camera-lidar calibration parameters, and obtaining the length attribute and area attribute of the road surface disease;

[0092] S24. Based on the central pixel coordinates of the pavement disease identification frame, the central pixel coordinates of the important highway facility target identification frame, the camera-lidar calibration parameters and the lidar-combined navigation system calibration parameters, the longitude and latitude coordinates of the central pixel point of the identification frame are calculated to obtain the geographical locations of the pavement disease and the important highway facility target.

[0093] In one embodiment, based on the foreground photo, the pixel points in the foreground photo are processed using the image brightening curve model to obtain a brightened foreground photo, and based on the trained image recognition model, the brightened foreground photo is input, and the output intermediate recognition result data includes:

[0094] S211, establishing a bright and dark sample data set of images according to sample images under different sensors, different weather conditions and different lighting conditions;

[0095] S212, based on the image brightening curve model, using the brightening function to process each pixel in the foreground photo to obtain a brightened foreground photo;

[0096] The expression of the brightening function is:

[0097]

[0098] In the formula, x is the pixel coordinate; is a learnable parameter, and ; For input I ( x ) (three color channels are processed separately); I ( x ) is the intensity function of the image, which is equal to x The intensity value at the pixel coordinate; For the n After iterations, the image x The intensity value at the pixel coordinate. Each iteration further adjusts the brightness of the image based on the previous result. For the n The learnable parameters at the iteration are used to control n The intensity of image enhancement in the iteration can be gradually adjusted in each round of iteration;

[0099] S213, building a model network based on depth-separable convolution and dilated convolution, and constructing a loss function based on spatial consistency error and color constant error to obtain an image processing model, and using an image brightness and darkness sample data set to train an image processing recognition model;

[0100] The expression of spatial consistency error is:

[0101]

[0102] In the formula, is the spatial consistency error, K is the number of pixels, i To traverse the pixels, For the i The four neighborhoods of pixels, Y i and Y j Respectively i The enhanced image and j An enhanced image, I i and I j Respectively i The input image and j input images;

[0103] The expression of color constancy error is:

[0104]

[0105] In the formula, is the color constant error, For all the two combinations of the three color channels traversed, is the average brightness of color channel p; For color channels q The average brightness of

[0106] S214, based on the trained image processing recognition model, input the brightened foreground photo, output the intermediate recognition result data, and transmit it to the industrial control module through the ROS2 node.

[0107] Specifically, ① the robot operating system (ROS2 is used in this embodiment) is installed and run in the industrial control module. It should be noted that ROS2 is an open source robot operating system framework that provides a series of tools and libraries for developers to build robot applications. In addition to supporting robot applications, ROS2 is also suitable for various control and data processing scenarios that require real-time performance, as well as integration with industrial equipment. The basic building blocks of ROS2 are composed of many ROS2 nodes, and the nodes can exchange information through forwarding and subscription mechanisms. In the above embodiment, the foreground photo is forwarded by the industrial control module through the ROS2 node, and is subscribed by the image algorithm on the industrial control module through the ROS2 node.

[0108] Specifically, the image algorithm ② performs uniform light enhancement processing on the foreground photo, which mainly includes the following five steps:

[0109] 1) Establish a dataset of image brightness and darkness samples. In order to train and evaluate image feature enhancement algorithms that take into account multi-scale sampling features of images, first build a dataset containing image brightness and darkness samples. It is necessary to include images from different sensors, different weather conditions, or different lighting conditions. Ensure that the data is diverse to improve the model's generalization ability to various scenarios.

[0110] 2) Establish an image brightening curve model (Light-Enhancement Curve, LE-curve). This curve can apply the same brightening function to each pixel in the image. If the brightness of the pixel is high, the pixel will not be processed.

[0111] 3) Introduce depthwise separable convolution and dilated convolution to improve the network structure. When building the network, it is planned to introduce depthwise separable convolution and dilated convolution to replace the traditional standard convolution operation to improve the algorithm's computational efficiency and receptive field.

[0112] 4) Model training. Use the established image dataset for training, and select Spatial Consistency Loss and Color Constancy Loss as loss functions. Ensure the stability of model training by adjusting hyperparameters such as learning rate and batch size.

[0113] 5) Accuracy assessment: Use PSNR (Peak signal-to-noise ratio) and SSIM (Structural Similarity Index) as evaluation indicators to assess the accuracy of the trained model. When the performance indicators meet the requirements, save the optimal weights of the model.

[0114] Specifically, deep neural networks are used to intelligently identify road defects (spillage, potholes, transverse cracks, longitudinal cracks, block cracks, strip repairs, etc.) and important highway facility targets (lane lines, mileage posts, anti-glare panels, contour markers, signboards), and the intermediate recognition result data is transmitted to the industrial control module through the ROS2 node for the next step of processing. The recognition result data includes the pixel coordinates of the four corner points and the center pixel coordinates of the road defect recognition frame and the pixel coordinates of the four corner points and the center pixel coordinates of the important highway facility target recognition frame, the foreground photo after uniform illumination containing the recognition frame, the types of road defects and important highway facility targets, and the lane where the road defect is located.

[0115] In one embodiment, based on the laser point cloud, the camera-lidar calibration is performed using the camera distortion projection model, and the lidar coordinate system-local independent coordinate system conversion is performed using the lidar-inertial navigation system calibration model, and the camera-lidar calibration parameters and the lidar-integrated navigation system calibration parameters are obtained, including:

[0116] S221. Based on radial distortion and tangential distortion, establish a camera distortion projection model, and determine camera-lidar calibration parameters;

[0117] S222, based on the laser radar-inertial navigation system calibration model, converting the laser radar point from the laser radar coordinate system to the local independent coordinate system to obtain the point coordinates in the local independent coordinate system;

[0118] S223, using a rotation matrix, by aligning the directions of the local coordinate system and the earth-centered earth-fixed space rectangular coordinate system, transforming the local independent coordinate system into the earth-centered earth-fixed space rectangular coordinate system;

[0119] S224. Translate the middle aligned coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system to obtain the laser radar-integrated navigation system calibration parameters.

[0120] Specifically, the industrial control module ③ integrates the 3D laser point cloud data after online motion compensation to process the intermediate recognition result data of the image algorithm in real time, including a) the pixel coordinates of the four corner points of the image algorithm recognition frame, the camera-lidar calibration parameters As input, calculate the physical size of the identification box, that is, obtain the length and area attributes of the road disease; b) use the image algorithm to identify the center pixel coordinates of the box and the camera-lidar calibration parameters and LiDAR-INS calibration parameters As input, the longitude and latitude coordinates of the central pixel of the identification frame are calculated, that is, the geographical location of the road surface disease and important highway facilities is obtained;

[0121] Specifically, ④ For the intelligent inspection results of pavement defects, the industrial control module directly exports and stores them locally and uploads them to the cloud. After each inspection project, the pavement defect inspection result report and the corresponding pavement defect photos are output. The result report includes the inspection route, direction, defect type, corresponding photo ID, lane, pile number position, longitude and latitude coordinates, and size attributes;

[0122] Specifically, ⑤ For the intelligent inspection of important highway facility targets, the missing of small facility targets (such as mileage piles, anti-glare plates and contour marks) and the deformation and damage of large facility targets (such as large signboards) need to be distinguished in combination with the intermediate results obtained in ②-③ (facility target type, geographic location coordinates, 2D image recognition frame and its corresponding 3D point cloud data) for further judgment. The intermediate results obtained in each inspection are uploaded to the cloud by the industrial control module. The cloud server comprehensively compares the intermediate result data of multiple periods to automatically judge the missing, deformed and damaged status of the facility target, and then exports the inspection result report of important highway facility targets and the corresponding abnormal facility photos. The result report content includes the inspection route, direction, facility abnormality category, corresponding photo id, pile number location, longitude and latitude coordinates.

[0123] In one embodiment, the expression of radial distortion is:

[0124]

[0125]

[0126] In the formula, for x Axial distortion, for y Axial distortion, is the image coordinate without distortion correction (ideal image coordinate), is the first radial distortion parameter, is the second radial distortion parameter, is the third radial distortion parameter, r is the distance from the current point to the center of the image, and ;

[0127] The expression of tangential distortion is:

[0128]

[0129] In the formula, for x Axis tangential distortion, for y Axis tangential distortion, is the first tangential distortion parameter, is the second tangential distortion parameter;

[0130] Specifically, the distortion parameters can be obtained after the camera is calibrated. In the above embodiment, the distortion parameter result obtained by camera calibration is:

[0131]

[0132] in, .

[0133] The expression of the camera distortion projection model is:

[0134]

[0135]

[0136] In the formula, is the pixel coordinate collected by the camera, is the camera internal parameter, and They are the rotation matrix and translation matrix from the laser radar to the camera respectively; forx Total axis distortion, for y Total axis distortion, for x Axial distortion, for y Axial distortion, for x Axis tangential distortion, for y Axis tangential distortion, For LiDAR z Axis point cloud coordinates;

[0137] The expression of the laser radar-inertial navigation system calibration model is:

[0138]

[0139] In the formula, is the coordinate of the laser point in the Earth-centered Earth-fixed space rectangular coordinate system, is the coordinate value of the Earth-centered Earth-fixed space rectangular coordinate system of the current inertial navigation receiver, is the rotation matrix from the inertial navigation coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system, is the rotation matrix from the laser radar coordinate system to the inertial navigation coordinate system, is the translation between the laser radar and the inertial navigation receiver, is the spatial rectangular coordinate of the laser point in the laser radar coordinate system.

[0140] Specifically, the conversion parameters of the laser radar coordinate system coordinate point and the camera coordinate system coordinate point are the external parameters of the laser radar 11 and the camera 1201. The calculation of their external parameters requires finding their common features, that is, features with the same name. is the pixel coordinate collected by camera 1201, is the point cloud coordinate of the laser radar 11, and the camera internal parameter is , is the rotation and translation matrix from the laser radar 11 to the camera 1201, then the principle of external parameter calibration between the laser radar 11 and the camera 1201 is:

[0141]

[0142] In the formula, is the pixel coordinate collected by the camera, is the point cloud coordinate of the laser radar, is the camera internal parameter, The rotation and translation matrices from the lidar to the camera.

[0143] Specifically, calibration is the process of unifying the parameters of each device into the same coordinate system and finding the conversion relationship between the coordinate systems.

[0144] Specifically, the rotation and translation matrices are obtained through a calibration process, which is described in yellow later.

[0145] The camera lens used is a short focal length lens (8mm). In order to solve the problem of serious lens distortion in practical applications, the present invention establishes a camera distortion projection model, including radial distortion and tangential distortion.

[0146] Specifically, first convert the coordinates into the camera coordinate system:

[0147]

[0148] Specifically, the radial distortion is:

[0149]

[0150] Specifically, the tangential distortion is:

[0151]

[0152] Specifically, the total distortion is:

[0153]

[0154] Specifically, the complete camera distortion projection model including distortion is:

[0155]

[0156] in is the radial distortion parameter, is the tangential distortion parameter. The above parameters can be obtained through camera calibration. .

[0157] Specifically, all parameters can be obtained through the calibration process, and the calibration process is as follows:

[0158] a) Camera-LiDAR calibration is performed using a checkerboard target. Camera 1201 and LiDAR 11 simultaneously shoot and scan the checkerboard target at different positions and postures to obtain 2D checkerboard images and 3D checkerboard point clouds, respectively. Then, the pixel coordinates and point cloud coordinates of the checkerboard corners are extracted and brought into the above calibration formula to obtain the camera 1201 internal parameters. , the rotation and translation matrices of the laser radar 11 and the camera 1201 .

[0159] b) After the camera-lidar calibration is completed, the laser radar coordinate system is converted to the earth-centered earth-fixed rectangular coordinate system. Assuming that the laser radar 11 scans a ground point at a certain moment, the position and attitude information of the inspection vehicle body 1 at the corresponding moment are obtained by interpolating the position and attitude data collected by the GPS antenna 10 and the inertial navigation 14. The data collected at the same moment mainly includes and .in, Indicates the current time of the GPS antenna 10 coordinate; Indicates the roll angle, describing the left-right tilt of the inspection vehicle body 1 relative to the horizontal plane; represents the pitch angle, describing the degree of front-to-back tilt of the inspection vehicle body 1 relative to the horizontal plane; It represents the azimuth, and describes the azimuth of the forward direction of the inspection vehicle body 1.

[0160] Specifically, in actual operation, the point cloud in the laser radar 11 coordinate system can be first translated to the GPS antenna 10, and then the coordinate system attitude can be adjusted according to the attitude information collected by the inertial navigation system 14 (inertial navigation system) so that the origin of the new coordinate system is located at the center of the GPS antenna 10. The axis points due east. The axis points due north. The axis points to the zenith direction to form a right-handed coordinate system, which is defined as a local independent coordinate system; then, according to the local longitude and latitude, the relative attitude relationship between the current coordinate system and the earth-centered earth-fixed rectangular coordinate system (WGS84 coordinate system in this embodiment) is obtained and the attitude is adjusted so that the coordinate system is consistent with the WGS84 coordinate system; finally, according to the position information collected by the GPS antenna 10, the intermediate coordinate system is translated to the WGS84 coordinate system, and the conversion of the laser radar coordinate system to the earth-centered earth-fixed rectangular coordinate system can be completed. In this way, the calibration of the equipment is completed, and the camera-lidar calibration parameters are obtained. and LiDAR-integrated navigation system calibration parameters .

[0161] Specifically, the coordinate conversion in the calibration process of the lidar-inertial navigation system of the present invention is divided into several steps. First, the lidar point is converted from the lidar coordinate system to the local independent coordinate system, the point in the lidar coordinate system is translated to the position of the GPS antenna 10, and rotated according to the attitude information of the inertial navigation 14 (inertial navigation system, INS) to obtain the point coordinates in the local independent coordinate system.

[0162]

[0163] Specifically, the local independent coordinate system is then converted to the Earth-centered Earth-fixed rectangular coordinate system (WGS84 coordinate system in this embodiment). This step requires the use of a rotation matrix To align the direction of the local coordinate system and the Earth-centered, Earth-fixed space rectangular coordinate system.

[0164]

[0165] Specifically, finally, the middle aligned coordinate system is translated to the position of the GPS antenna 10 in the geocentric earth-fixed space rectangular coordinate system, thereby obtaining the final position of the point in the geocentric earth-fixed space rectangular coordinate system.

[0166]

[0167] Specifically, the laser radar-inertial navigation system calibration model is obtained as follows:

[0168]

[0169]

[0170] In the formula, is the coordinate of the laser point in the Earth-centered Earth-fixed space rectangular coordinate system, is the coordinate value of the Earth-centered Earth-fixed space rectangular coordinate system of the current inertial navigation receiver, is the rotation matrix from the inertial navigation coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system, where They are the yaw angle, pitch angle and roll angle of the laser radar relative to the inertial navigation. is the rotation matrix from the laser radar coordinate system to the inertial navigation coordinate system, where are the yaw angle, pitch angle and roll angle of the inertial navigation relative to the Earth-centered Earth-fixed space rectangular coordinate system, is the translation between the laser radar and the inertial navigation receiver, is the spatial rectangular coordinate of the laser point in the laser radar coordinate system.

[0171] like Figure 2-Figure 6 As shown, according to another embodiment of the present invention, there is also provided a highway intelligent inspection equipment based on multi-dimensional vision fusion, the highway intelligent inspection equipment based on multi-dimensional vision fusion comprises:

[0172] Automobile carrier, used to install visual inspection module, combined positioning module, synchronous control module, industrial control module and power supply module;

[0173] Visual inspection module, used to obtain 2D foreground images and 3D laser point cloud data within the highway inspection field of view, and identify inspection targets;

[0174] Combined positioning module, used to provide location information of vehicle carrier and inspection target;

[0175] Synchronous control module, used to complete the clock synchronization of the visual inspection module and the industrial control module;

[0176] Industrial control module, used to control the opening, closing and configuration of the visual inspection module, combined positioning module and synchronous control module, and automatically analyze the inspection data;

[0177] The power supply module is used to provide continuous power supply to the visual inspection module, combined positioning module, synchronous control module and industrial control module.

[0178] Specifically, after the calibration is completed, the inspection vehicle body 1 can work. First, turn on the power supply module, which includes a battery 7 and a power adapter 6, and can stably output 220V AC and 12V DC to power the remaining modules. At the same time, start the visual inspection module, combined positioning module, synchronous control module and industrial control module, create a new inspection project, and configure the inspection route, direction and stake number starting point.

[0179] Specifically, the combined positioning module includes an inertial navigation system 14 and a GPS antenna 10, wherein the GPS antenna 10 can output a real-time satellite positioning trajectory. Combined with the data of the inertial navigation system 14, it can output an accurate trajectory and posture change. At the same time, the GPS antenna 10 also outputs high-precision satellite timing. Using this satellite timing as the time reference, the clocks of the remaining modules are synchronized using the PTP protocol through the synchronization control module, i.e., the time synchronization box 13, thereby completing the time synchronization of the entire system.

[0180] Specifically, after the camera 1201 and the laser radar 11 receive the satellite timing transmitted by the time synchronization box 13 using the PTP protocol, they start collecting road foreground photos and laser point clouds at a frequency of 5 Hz, respectively, and the inertial navigation 14 and the GPS antenna 10 start collecting navigation data at a frequency of 200 Hz;

[0181] Specifically, the industrial control module includes an industrial computer 8 and a switch 5, wherein the switch 5 can solve the problem of insufficient interfaces of the industrial computer 8. The road foreground photo data from the camera 1201 is transmitted to the industrial computer 8 via the GIGE interface via the switch 5. The industrial computer 8 displays and stores the transmitted road foreground photo data in real time, and the storage format is a *.jpg file. At the same time, the road foreground photo data is forwarded by the industrial computer 8 through the ROS2 node, and is subscribed by the image algorithm on the industrial computer 8 through the ROS2 node. The image algorithm performs uniform light enhancement processing on the foreground photo, and intelligently identifies road surface diseases (including spilled objects, potholes, transverse cracks, longitudinal cracks, block cracks, strip repairs) and important highway facility targets (including lane lines, mileage posts, anti-glare plates, contour marks, and signboards) through a deep neural network, and transmits the recognition result data to the industrial computer 8 through the ROS2 node for the next step of processing. The recognition result data includes the pixel coordinates of the four corner points and the center pixel coordinates of the road surface disease and important highway facility target identification frame, the uniform light foreground photo containing the identification frame, the type of road surface disease and important highway facility target, and the lane where the road surface disease is located.

[0182] Specifically, the navigation data from the inertial navigation 14 and the GPS antenna 10 is transmitted to the industrial computer 8 via the RS422 interface via the switch 5. The industrial computer 8 displays and stores the transmitted navigation data in real time in the format of *.txt files. The laser point cloud data from the laser radar 11 is transmitted to the industrial computer 8 via the GIGE interface via the switch 5. The industrial computer 8 displays and stores the transmitted laser data in real time in the format of *.pcap files. The industrial computer 8 integrates the 3D laser point cloud data after online motion compensation to process the pixel coordinates of the four corner points of the target identification box of road surface diseases and important highway facilities in real time, and uses the camera-lidar calibration parameters As input, the physical size of the identification box is calculated, that is, the length and area attributes of the road surface disease are obtained; at the same time, the center pixel coordinates of the road surface disease and important highway facility target identification box, the camera-lidar calibration parameters and LiDAR-integrated navigation system calibration parameters As input, the longitude and latitude coordinates of the central pixel of the identification frame are calculated to obtain the geographical location of the road surface defects and important highway facilities.

[0183] Specifically, the results of the intelligent inspection of road surface diseases are directly exported by the industrial computer 8 and stored locally and uploaded to the cloud. After each inspection project, a road surface disease inspection result report and corresponding road surface disease photos are output. The result report content includes the inspection route, direction, disease type, corresponding photo ID, lane, pile number location, longitude and latitude coordinates, and size attributes.

[0184] Specifically, for the intelligent inspection of important highway facility targets, the missing of small facility targets (including mileage piles, anti-glare plates and contour marks) and the deformation and damage of large facility targets (including large signboards) need to be distinguished in combination with the intermediate results obtained by previous real-time processing (including facility target type, geographic location coordinates, 2D image recognition box and its corresponding 3D point cloud data) for further judgment. The intermediate results obtained from each inspection are uploaded to the cloud by the industrial control module. The cloud server comprehensively compares the intermediate result data of multiple periods to automatically determine the missing, deformed and damaged status of the facility target, and then exports the inspection result report of important highway facility targets and the corresponding abnormal facility photos. The result report content includes the inspection route, direction, facility abnormality category, corresponding photo id, pile number location, longitude and latitude coordinates.

[0185] In one embodiment, the vehicle carrier includes an inspection vehicle body 1, a roof clamp 2 is provided on one side of the top of the inspection vehicle body 1, a bottom plate 3 is provided on the top of the roof clamp 2, a shell 4 is provided on the top of the bottom plate 3, GPS antennas 10 are provided at both ends of the inner side of the shell 4, a partition is provided at the middle part of the inner side of the shell 4 (in addition, in specific applications, a plurality of equidistantly arranged snap-in grooves are provided in the middle part of the inner side of the shell 4), the partition is fixedly connected to the shell 4 through the snap-in groove, a laser radar 11 is provided on one side of the top of the partition, an industrial camera lens assembly 12 is provided on the other side of the top of the partition, an inertial navigation 14 is provided on one side of the bottom end of the partition, and a time synchronization box 13 is provided on the other side of the bottom end of the partition.

[0186] Specifically, Figure 3-Figure 4 As shown, various hardware devices are installed on the inspection vehicle, wherein the GPS antenna 10, the laser radar 11, the industrial camera lens assembly 12, the time synchronization box 13 and the inertial navigation 14 are first installed in the housing 4, thereby integrating visual detection, combined positioning and time synchronization into one component, wherein the laser radar 11 and the industrial camera lens assembly 12 are installed on a partition, and the partition can be installed on the card slot on the housing 4. There are multiple card slots on the housing 4, so the installation height of the laser radar 11 and the industrial camera lens assembly 12 can be adjusted. The industrial camera lens assembly 12 is as shown in FIG. Figure 1 The method shown can adjust the pitch angle of the camera 1201. The entire assembly is mounted on a base plate 3 fixed on a roof fixture 2, thereby fixing the entire assembly on the roof. The remaining equipment, including a switch 5, a power adapter 6, a battery 7, an industrial computer 8, and a display 9, are all installed in the inspection vehicle body 1, wherein the display 9 is installed in front of the co-pilot seat for easy viewing by the staff, and the switch 5, the power adapter 6, the battery 7, and the industrial computer 8 are all installed in the trunk of the vehicle.

[0187] In one embodiment, the inspection vehicle body 1 includes a driver's seat and a trunk, a display 9 is provided on one side of the driver's seat, a switch 5 is provided on one side of the trunk, a power adapter 6 is provided on the other side of the trunk, an industrial computer 8 is provided on one end of the trunk, and a battery 7 is provided on the other end of the trunk;

[0188] The visual detection module includes a laser radar 11 and an industrial camera lens assembly 12; the combined positioning module includes a GPS antenna 10 and an inertial navigation system 14; the synchronous control module includes a time synchronization box 13; the industrial control module includes a switch 5 and an industrial computer 8; and the power supply module includes a power adapter 6 and a battery 7.

[0189] Specifically, after installing each device on the inspection vehicle body 1, the camera-lidar and lidar-combined navigation can be calibrated, and the GPS antenna 10, camera 1201, lidar 11, and inertial navigation 14 data are unified to the WGS84 coordinate system, mainly including: a) camera-lidar coordinate system transformation; b) lidar coordinate system-WGS84 coordinate system conversion. Among them, the camera-lidar coordinate system transformation relationship is as follows Figure 5 As shown in , the calibration system composed of the two includes four coordinate systems, namely the lidar coordinate system, the camera coordinate system, the pixel coordinate system and the world coordinate system, as shown in Figure 7 shown.

[0190] Specifically, the visual inspection module includes an industrial camera lens assembly 12 and a laser radar 11; the camera 1201 has a resolution of 12 million pixels, an imaging size of 3000*4096 pixels, uses a GIGE interface for image data transmission, and supports a PTP time synchronization protocol; the wide-angle fixed-focus lens has a focal length of 8mm, which can achieve a physical resolution of 3mm and a field of view width of 12m at a viewing distance of 7m; the laser radar 11 is a three-dimensional laser radar, which uses a one-dimensional rotating mirror for scanning, has 128 channels, a field of view of 120° (horizontal)*25.4° (vertical), an angular resolution of 0.1° (horizontal)*0.2° (vertical), uses a GIGE interface for point cloud data transmission, and supports a PTP time synchronization protocol.

[0191] In one embodiment, the industrial camera lens assembly 12 includes a camera 1201 disposed in the middle of the inner side of the housing 4, a lens 1202 is disposed at one end of the camera 1201, a first fixing plate 1203 is disposed at the bottom end of the camera 1201, connecting arms 1207 are symmetrically disposed on both sides of the bottom end of the first fixing plate 1203, a supporting arm 1208 is disposed at one end of the connecting arm 1207 (in addition, in a specific application, one end of the connecting arm 1207 is connected to the supporting arm 1208 through a pin shaft), and a second fixing plate 1204 is disposed at the bottom end of the supporting arm 1208;

[0192] Adjustment slots are provided on both sides of the second fixing plate 1204 , and an adjustment connecting rod 1205 is provided at the other end of the connecting arm 1207 passing through the adjustment slot. A plurality of nuts 1206 matching with the second fixing plate 1204 are sleeved on the outer side of the adjustment connecting rod 1205 .

[0193] Specifically, the lens 1202 (a wide-angle fixed-focus lens in the present embodiment) is connected to the camera 1201 (in addition, in specific applications, the camera 1201 adopts an industrial area array camera) via a C-Mount interface, and the camera 1201 and the laser radar 11 are mounted on a partition in the middle of the inner side of the outer shell 4 via mechanical screws; the partition can adjust the overall installation height of the visual detection module by inserting different card slots to uniformly adapt to automobile carriers with different roof heights.

[0194] Specifically, Figure 5-Figure 6 As shown, the camera 1201 is installed on a camera fixing part, and the camera fixing part is divided into two upper and lower fixing plates. The camera 1201 is fixed on the first fixing plate 1203, and the two fixing plates are connected by a slider connecting rod mechanism. Two cross beams protrude from the bottom of the first fixing plate 1203 to serve as connecting arms 1207 of the two connecting rod slider mechanisms. One end of the connecting arm 1207 is fixed to a hinge on the second fixing plate 1204 and can rotate around the hinge. Another adjusting connecting rod 1205 is threaded on the rod body, and two notches are opened at corresponding positions of the second fixing plate 1204 and the partition. The adjusting connecting rod 1205 with a thread passes through the second fixing plate 1204 and the partition notches and acts as a slider to move in the notches, and is fixed to the second fixing plate 1204 using four nuts 1206. Therefore, the first fixing plate 1203 can be controlled to rotate around the hinge of the second fixing plate 1204 by adjusting the nut 1206 at the position of the adjusting connecting rod 1205, thereby controlling the change of the pitch angle of the camera 1201.

[0195] In order to facilitate understanding of the above technical solution of the present invention, the following is a specific description of the intelligent inspection of a highway by the inspection equipment of the present invention as an example:

[0196] like Figure 8 As shown in the figure, the operation process of the inspection equipment is as follows:

[0197] 1) Turn on the power supply module, start the visual inspection module, combined positioning module, synchronous control module and industrial control module at the same time, create a new inspection project, and configure the inspection route, direction and stake number starting point;

[0198] 2) The synchronization control module receives satellite timing from the combined positioning module and synchronizes the clocks of the array industrial camera and lidar through the PTP protocol;

[0199] 3) The time-synchronized area array industrial camera and laser radar start collecting road foreground photos and laser point clouds at a frequency of 5 Hz respectively, and the combined positioning module starts collecting navigation data at a frequency of 200 Hz;

[0200] 4) The navigation data is transmitted to the industrial control module through the RS422 interface, and the road foreground photos and laser point cloud data are transmitted to the industrial control module through the GIGE interface. The industrial control module displays and stores the transmitted navigation data, foreground photos, and laser point cloud in real time. The storage formats are *.txt, *.jpg, and *.pcap respectively, and the industrial control module performs real-time processing at the same time;

[0201] 5) The industrial control module performs intelligent fusion processing on the transmitted multi-source data, and realizes the precise positioning and attribute measurement of road surface defects by fusing the 3D laser point cloud data after online motion compensation and the image algorithm recognition results, and automatically generates inspection result reports; for important highway facilities, the system can also identify the missing of small facilities and the deformation and damage of large facilities, and output detailed facility inspection reports through multi-period data comparison on the cloud server.

[0202] In summary, with the help of the above technical solution of the present invention, by adopting the mode of 2D+3D multi-dimensional vision fusion, compared with the mainstream mode of highway inspection using only 2D vision in the current market, it has more comprehensive inspection capabilities. By combining 2D images and 3D laser point cloud multi-dimensional information, it can not only solve the intelligent inspection of road diseases, but also has the ability to automatically identify the missing, deformed and damaged state of important highway facilities. The present invention adopts a tightly coupled combined positioning method of GNSS positioning + IMU inertial measurement + vehicle wheel speed meter compensation, combined with the 3D ranging information provided by the laser radar, it can achieve centimeter-level accuracy of geographical positioning of road diseases and important highway facilities, and has the ability to compare and trace road diseases and facility status in multiple periods. The present invention has the ability to identify road diseases with millimeter-level accuracy, and can realize objective measurement of the physical size of road diseases based on 3D laser radar point cloud data. Each module of the present invention adopts an integrated structure, which is directly installed on the roof through the luggage rack, and has the characteristics of lightweight and convenient installation. The installation height and angle of the visual detection module can be adjusted inside the structure, and it can adapt to different models of vehicle carriers.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A highway intelligent inspection method based on multi-dimensional visual fusion, characterized in that: The method includes: S1. According to the newly built inspection project, configure the inspection route, direction and stake number starting point, and synchronize satellite timing; obtain foreground photos, laser point cloud and navigation data; S2. Perform uniform light enhancement on the foreground photo to obtain the intermediate recognition result data, and use the laser point cloud after online motion compensation for real-time processing to obtain the road surface disease inspection results and important highway facility target inspection results; specifically including: S21, using the image brightening curve model to process the pixels in the foreground photo, and outputting intermediate recognition result data based on the trained image recognition model; S22, performing camera-lidar calibration using a camera distortion projection model, and performing coordinate system conversion using a lidar-inertial navigation system calibration model to obtain calibration parameters; S23, calculating the physical size of the identification frame according to the pixel coordinates of the corner points of the road surface disease identification frame, the pixel coordinates of the corner points of the important highway facility target identification frame and the calibration parameters, and obtaining the length attribute and the area attribute of the road surface disease; S24, based on the central pixel coordinates of the pavement disease identification frame, the central pixel coordinates of the important highway facility target identification frame, the camera-lidar calibration parameters and the lidar-integrated navigation system calibration parameters, calculate the longitude and latitude coordinates of the central pixel point of the identification frame to obtain the geographical locations of the pavement disease and the important highway facility target; S3, storing and uploading the pavement disease inspection results, and outputting a pavement disease inspection result report and corresponding pavement disease photos; S4. Based on the inspection results of important highway facilities, combined with the intermediate recognition result data and the corresponding laser point cloud, the missing, deformed and damaged facility targets are identified, and the inspection result report of important highway facilities and the corresponding abnormal facility photos are exported; The intermediate recognition result data includes the corner pixel coordinates and center pixel coordinates of the road surface disease recognition frame, the corner pixel coordinates and center pixel coordinates of the important highway facility target recognition frame, the uniformly illuminated foreground photo containing the recognition frame, the facility target type, and the lane where the road surface disease is located.

2. The method for intelligent highway inspection based on multi-dimensional visual fusion according to claim 1 is characterized in that: The facility target types include pavement damage types and important highway facility target types; the pavement damage types include spills, potholes, transverse cracks, longitudinal cracks, block cracks and strip repairs; the important highway facility target types include lane lines, mileage posts, anti-glare panels, contour markers and signboards.

3. The method for intelligent highway inspection based on multi-dimensional visual fusion according to claim 1 is characterized in that: The method of processing the pixels in the foreground photo by using the image brightening curve model and outputting the intermediate recognition result data based on the trained image recognition model includes: S211, establishing a bright and dark sample data set of images according to sample images under different sensors, different weather conditions and different lighting conditions; S212, based on the image brightening curve model, using the brightening function to process each pixel in the foreground photo to obtain a brightened foreground photo; S213, building a model network based on depth-separable convolution and dilated convolution, and constructing a loss function based on spatial consistency error and color constant error to obtain an image processing model, and using an image brightness and darkness sample data set to train an image processing recognition model; The expression of the spatial consistency error is: Where, L spa is the spatial consistency error, K is the number of pixels, i is the traversal of pixels, Ω(i) is the 4-neighborhood of the i-th pixel, Y i With Y j are the i-th enhanced image and the j-th enhanced image, respectively. i with I j are the i-th input image and the j-th input image respectively; The expression of the color constancy error is: Where, L col is the color constant error, ε is all the two-to-two combinations of the three color channels traversed, j p is the average brightness of color channel p; j q is the average brightness of color channel q; S214, based on the trained image processing recognition model, input the brightened foreground photo, output the intermediate recognition result data, and transmit it to the industrial control module through the ROS2 node.

4. The method for intelligent highway inspection based on multi-dimensional visual fusion according to claim 1 is characterized in that: The camera-lidar calibration is performed using the camera distortion projection model, and the coordinate system is converted using the lidar-inertial navigation system calibration model to obtain calibration parameters including: S221. Based on radial distortion and tangential distortion, establish a camera distortion projection model, and determine camera-lidar calibration parameters; S222, based on the laser radar-inertial navigation system calibration model, convert the laser radar point from the laser radar coordinate system to the local independent coordinate system to obtain the point coordinates in the local independent coordinate system; S223, using a rotation matrix, by aligning the directions of the local coordinate system and the earth-centered earth-fixed space rectangular coordinate system, transforming the local independent coordinate system into the earth-centered earth-fixed space rectangular coordinate system; S224. Translate the middle aligned coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system to obtain the laser radar-integrated navigation system calibration parameters.

5. The method for intelligent highway inspection based on multi-dimensional visual fusion according to claim 4 is characterized in that: The expression of the radial distortion is: x′=x c (1+k1r 2 +k2r 4 +k3r 6 ) and′=y c (1+k1r 2 +k2r 4 +k3r 6 ) Where x′ is the radial distortion of the x-axis, y′ is the radial distortion of the y-axis, (x c ,y c ) is the image coordinate without distortion correction, k1 is the first radial distortion parameter, k2 is the second radial distortion parameter, k3 is the third radial distortion parameter, and r is the distance from the current point to the center of the image; The expression of the tangential distortion is: In the formula, x ″ is the tangential distortion of the x-axis, ″ is the y-axis tangential distortion, p1 is the first tangential distortion parameter, and p2 is the second tangential distortion parameter; The expression of the camera distortion projection model is: x ″′ =x ′ +x ″ y″′=y′+ty″ In the formula, (u, v) is the pixel coordinates collected by the camera, K is the camera internal parameter, and R l With t l are the rotation matrix and translation matrix from the laser radar to the camera respectively; x″′ is the total distortion of the x-axis, y″′ is the total distortion of the y-axis, x′ is the radial distortion of the x-axis, y′ is the radial distortion of the y-axis, x″ is the tangential distortion of the x-axis, y″ is the tangential distortion of the y-axis, z l is the z-axis point cloud coordinate of the laser radar; The expression of the laser radar-inertial navigation system calibration model is: In the formula, is the coordinate of the laser point in the Earth-centered Earth-fixed space rectangular coordinate system, is the coordinate value of the Earth-centered Earth-fixed rectangular coordinate system of the current inertial navigation receiver, R W is the rotation matrix from the inertial navigation coordinate system to the Earth-centered Earth-fixed space rectangular coordinate system, R IMU is the rotation matrix from the laser radar coordinate system to the inertial navigation coordinate system, is the translation between the laser radar and the inertial navigation receiver, is the spatial rectangular coordinate of the laser point in the laser radar coordinate system.

6. A highway intelligent inspection equipment based on multi-dimensional vision fusion, used to implement the highway intelligent inspection method based on multi-dimensional vision fusion as described in any one of claims 1 to 5, characterized in that: The intelligent highway inspection equipment based on multi-dimensional visual fusion includes: Automobile carrier, used to install visual inspection module, combined positioning module, synchronous control module, industrial control module and power supply module; Visual inspection module, used to obtain 2D foreground images and 3D laser point cloud data within the highway inspection field of view, and identify inspection targets; A combined positioning module, used to provide position information of the vehicle carrier and the inspection target; A synchronization control module, used to complete the clock synchronization of the visual inspection module and the industrial control module; An industrial control module, used to control the opening, closing and configuration of the visual inspection module, the combined positioning module and the synchronous control module, and automatically analyze the inspection data; A power supply module is used to provide continuous power supply to the visual detection module, the combined positioning module, the synchronous control module and the industrial control module.

7. The intelligent highway inspection equipment based on multi-dimensional visual fusion according to claim 6 is characterized in that: The vehicle carrier includes an inspection vehicle body; a roof clamp is provided on one side of the top of the inspection vehicle body, a bottom plate is provided on the top of the roof clamp, an outer shell is provided on the top of the bottom plate, GPS antennas are penetrated at both ends of the inner side of the outer shell, a partition is penetrated through the middle of the inner side of the outer shell, a laser radar is provided on one side of the top of the partition, an industrial camera lens assembly is provided on the other side of the top of the partition, an inertial navigation is provided on one side of the bottom end of the partition, and a time synchronization box is provided on the other side of the bottom end of the partition.

8. The intelligent highway inspection equipment based on multi-dimensional visual fusion according to claim 7 is characterized in that: The inspection vehicle body includes a driver's seat and a trunk, a display is provided on one side of the driver's seat, a switch is provided on one side of the trunk, a power adapter is provided on the other side of the trunk, an industrial computer is provided on one end of the trunk, and a battery is provided on the other end of the trunk; The visual detection module includes a laser radar and an industrial camera lens assembly; the combined positioning module includes a GPS antenna and an inertial navigation system; the synchronization control module includes a time synchronization box; the industrial control module includes a switch and an industrial computer; and the power supply module includes a power adapter and a battery.

9. The intelligent highway inspection equipment based on multi-dimensional visual fusion according to claim 8 is characterized in that: The industrial camera lens assembly includes a camera arranged in the middle of the inner side of the shell, a lens is arranged at one end of the camera, a first fixing plate is arranged at the bottom end of the camera, connecting arms are symmetrically arranged on both sides of the bottom end of the first fixing plate, a supporting arm is arranged at one end of the connecting arm, and a second fixing plate is arranged at the bottom end of the supporting arm; adjustment slots are opened on both sides of the second fixing plate, and an adjustment connecting rod is arranged at the other end of the connecting arm passing through the adjustment slot, and a plurality of nuts matching the second fixing plate are sleeved on the outer side of the adjustment connecting rod.

Images