Pavement defect detection method, device and equipment and storage medium

Abstract

The invention discloses a road surface defect detection method, device and equipment and a storage medium, relates to the technical field of road engineering detection, and is applied to an unmanned vehicle, and a camera, a laser radar and an air cannon are mounted on a chassis of the unmanned vehicle. Collecting current road surface data of a road surface area of the to-be-measured road surface after air is emitted by the current air cannon through the camera and the laser radar; performing crack detection by using a pavement elevation model constructed based on the current pavement data, and extracting geometric parameter information of a crack when the pavement elevation model is detected and determined to have the crack; and processing the geometric parameter information through a pre-training language model in combination with road environment information corresponding to the to-be-detected road surface to generate a road surface defect detection result. Therefore, interference of protruding objects on pavement defect detection can be avoided by means of the air cannon; and the detection result is generated in combination with the multi-modal pavement data and the road environment information, so that the pavement defect detection effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road engineering detection, and in particular relates to a road surface defect detection method, device, equipment and storage medium. BACKGROUND

[0002] In the current commonly used road surface defect detection technology, manual inspection is low in efficiency and high in missed detection rate; traditional vehicle-mounted detection equipment and some unmanned detection vehicles still have the following problems: protruding objects are generally interfering: fallen leaves, stones, paper scraps and other protruding objects are easy to be misjudged as road surface pits or protruding defects, resulting in a high invalid detection rate; and the road environment is complex, and the same kind of defects on the road surface under different environmental conditions may need to be dealt with in different ways, that is, the current commonly used road surface defect detection method has low effectiveness and applicability of defect detection results.

[0003] It can be seen that how to avoid the interference of protruding objects in the process of road surface defect detection and improve the accuracy of defect detection and the effectiveness of the detection results is a problem to be solved in the field. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a road surface defect detection method, device, equipment and storage medium, which can avoid the interference of protruding objects on road surface defect detection with the help of an air cannon; and generate detection results by combining multi-modal road surface data and road environment information, thereby improving the road surface defect detection effect. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a road surface defect detection method applied to an unmanned vehicle, wherein a chassis of the unmanned vehicle is installed with a camera, a laser radar and an air cannon, and the method comprises:

[0006] When the unmanned vehicle drives on a to-be-detected road surface, current road surface data of a road surface area after air is emitted by the current air cannon is collected by the camera and the laser radar;

[0007] A road surface elevation model constructed based on the current road surface data is used for crack detection, and when it is detected that the road surface elevation model has cracks, geometric parameter information of the cracks is extracted;

[0008] By combining road environment information corresponding to the to-be-detected road surface, the geometric parameter information is processed by a pre-trained language model to generate road surface defect detection results corresponding to the road surface area.

[0009] Optionally, the current road surface data of the road surface area after air is emitted by the current air cannon is collected by the camera and the laser radar, comprising:

[0010] Collecting, by the camera and the laser radar, original road surface data of a road surface region matching a position of the unmanned vehicle;

[0011] Three-dimensionally reconstructing the road surface region by using the original road surface data, and determining an object height of an object in the reconstructed three-dimensional model of the road surface region relative to a road surface reference plane corresponding to the road surface region;

[0012] When the object height is not less than a preset height threshold, emitting air to the road surface region by the air cannon, and re-collecting, by the camera and the laser radar, road surface data of the road surface region to obtain current road surface data.

[0013] Optionally, before the re-collecting, by the camera and the laser radar, road surface data of the road surface region to obtain current road surface data, the method further includes:

[0014] Determining a current illumination intensity of the road surface region by a pre-set light sensor;

[0015] When the current illumination intensity is less than a preset illumination intensity threshold, performing light compensation on the road surface region by a pre-set light compensation lamp, so that the re-collecting, by the camera and the laser radar, road surface data of the road surface region to obtain current road surface data is performed when an illumination intensity corresponding to the road surface region is not less than the preset illumination intensity threshold.

[0016] Optionally, the crack detection by using a road surface elevation model constructed based on the current road surface data includes:

[0017] Processing point cloud data in the current road surface data by point cloud voxelization to obtain a corresponding point cloud feature pyramid, and performing feature extraction on image data in the current road surface data by a pre-set convolutional neural network to obtain a corresponding image feature;

[0018] Performing feature fusion operation on the point cloud feature pyramid and the image feature based on an alignment weight corresponding to the point cloud data and the image data, and constructing a road surface elevation model corresponding to the current road surface data by using a corresponding fused feature in combination with a CRF optimization framework;

[0019] Performing crack detection by using the road surface elevation model.

[0020] Optionally, the processing of the geometric parameter information by the pre-trained language model to generate a road surface defect detection result corresponding to the road surface region in combination with road environment information corresponding to the road surface to be detected includes:

[0021] The road environment information corresponding to the to-be-tested road surface and the geometric parameter information are input into a pre-trained language model through a prompt learning technology to generate a defect parameter threshold related to a road surface defect and a defect level corresponding to the defect parameter threshold;

[0022] A road surface defect detection result corresponding to the road surface area is generated according to a similarity between a semantic vector of the geometric parameter information and the defect level.

[0023] Optionally, the method further comprises:

[0024] When the unmanned vehicle is driving on the to-be-tested road surface, a surrounding environment map corresponding to the unmanned vehicle is constructed through a simultaneous localization and mapping technology in combination with data collected by the laser radar and inertial navigation data of the unmanned vehicle;

[0025] In combination with Beidou positioning information of the unmanned vehicle and road condition data collected by the surround-view camera, a driving path is generated in real time so that the unmanned vehicle drives on the to-be-tested road surface based on the driving path.

[0026] Optionally, the combination of the Beidou positioning information of the unmanned vehicle and the data collected by the surround-view camera to generate the driving path in real time comprises:

[0027] If the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection position of the to-be-tested road surface, and the current road surface defect detection result indicates that the road surface area has a crack defect, a corresponding driving path is generated in real time in combination with road condition data collected by the surround-view camera according to a preset driving strategy; the preset driving strategy is to continue driving in the order of straight line, right turn and left turn.

[0028] If the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection position of the to-be-tested road surface, and the current road surface defect detection result indicates that the road surface area does not have a crack defect, a driving path is generated in real time in combination with road condition data corresponding to the intersection position collected by the surround-view camera.

[0029] In a second aspect, the present application provides a road surface defect detection device applied to an unmanned vehicle, wherein a chassis of the unmanned vehicle is provided with a camera, a laser radar and an air cannon, and the device comprises:

[0030] A road surface data acquisition module is configured to acquire current road surface data of a road surface area of the to-be-tested road surface after air is emitted by the current air cannon through the camera and the laser radar when the unmanned vehicle drives on the to-be-tested road surface.

[0031] a crack detection module configured to perform crack detection using a road elevation model constructed based on the current road surface data, and extract geometric parameter information of the cracks when it is determined that the road elevation model has cracks;

[0032] a result generation module configured to process the geometric parameter information by a pre-trained language model to generate a road surface defect detection result corresponding to the road surface region in combination with road environment information corresponding to the road surface to be detected.

[0033] In a third aspect, the present application provides an electronic device, comprising:

[0034] a memory configured to save a computer program;

[0035] a processor configured to execute the computer program to implement the road surface defect detection method as described above.

[0036] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, which is executed by a processor to implement the road surface defect detection method as described above.

[0037] As can be seen, in the present application, the chassis of the unmanned vehicle is provided with a camera, a laser radar and an air cannon. When the unmanned vehicle drives on the road surface to be detected, the camera and the laser radar collect current road surface data of the road surface region after the air cannon emits air. Then, crack detection is performed using a road elevation model constructed based on the current road surface data, and geometric parameter information of the cracks is extracted when it is determined that the road elevation model has cracks. Then, in combination with road environment information corresponding to the road surface to be detected, the geometric parameter information is processed by a pre-trained language model to generate a road surface defect detection result corresponding to the road surface region. In this way, the protruding objects on the road surface can be removed by the air cannon, so that the interference of the protruding objects on the road surface defect detection can be avoided. In addition, the road elevation model of the road surface can be constructed in combination with the multi-modal data of the laser radar and the camera, and the geometric parameter information of the cracks extracted from the road elevation model is analyzed and processed in combination with the road environment information, so that the generated defect detection result conforms to the actual road environment information, and the effectiveness and scene applicability of the road surface defect detection result are improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0039] Figure 1 A flow chart of a road defect detection method disclosed in the present application;

[0040] Figure 2 A chassis structure schematic diagram of an unmanned vehicle disclosed in the present application;

[0041] Figure 3 A flow chart of a specific road defect detection method disclosed in the present application;

[0042] Figure 4 A road defect detection device structure schematic diagram disclosed in the present application;

[0043] Figure 5 An electronic device structure diagram disclosed in the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0045] Referring to Figure 1 The embodiments of the present application disclose a road defect detection method, applied to an unmanned vehicle, wherein a chassis of the unmanned vehicle is installed with a camera, a laser radar and an air cannon, and specifically comprises:

[0046] In step S11, when the unmanned vehicle is driving on a to-be-tested road, current road data of a road area after air emitted by the air cannon on the to-be-tested road is collected by the camera and the laser radar.

[0047] In the embodiments, the unmanned vehicle can detect defects of the road in real time when driving on the to-be-tested road. Specifically, road data of the to-be-tested road needs to be collected, and the camera and the laser radar can collect comprehensive multi-modal data of the road. When the unmanned vehicle is driving on the to-be-tested road, the road area to be collected dynamically changes with the driving condition of the unmanned vehicle, for example, road data of a road area under the chassis of the unmanned vehicle needs to be collected. In combination with the air cannon, obstacles on the road can be removed, and more clear and accurate current road data of the corresponding road area can be collected.

[0048] In a specific embodiment, the collecting, by the camera and the lidar, of the current road surface data of the road surface region of the road surface to be tested after the air is fired by the current air cannon can include: collecting, by the camera and the lidar, original road surface data of a road surface region of the road surface to be tested that matches the position of the unmanned vehicle; performing three-dimensional reconstruction on the road surface region using the original road surface data and determining an object height of an object in the road surface region of the reconstructed three-dimensional model relative to a road surface reference surface corresponding to the road surface region; when the object height is not less than a preset height threshold, firing air on the road surface region by the air cannon and re-collecting road surface data of the road surface region by the camera and the lidar to obtain current road surface data. Specifically, in the process of collecting road surface data, the original road surface data of the road surface region can be obtained by the camera and the lidar in cooperation, which can include road surface images, depth data, and point cloud data. Then, based on the collected original road surface data, three-dimensional reconstruction can be performed, and the height of the protruding object in the road surface region relative to the object height of the road surface reference surface of the road surface region can be calculated based on the reconstructed three-dimensional model. If the object height exceeds a certain threshold, it can affect the chassis of the unmanned vehicle or interfere with the identification of road defects, such as collision hazards with cameras and other devices installed on the chassis, and road surface shielding affecting road surface data collection. At this time, air can be sent to the road surface region by the air cannon to remove the object. Then, after the object is removed by the air cannon, the data of the road surface region can be collected again by the camera and the lidar to obtain current road surface data. As can be seen, the obstacles and other objects on the road surface can be removed by the air cannon, making the re-collected road surface data more accurate and avoiding the interference of obstacles and other objects on road defect detection.

[0049] In another specific embodiment, before the current road surface data of the road surface area is reacquired by the camera and the laser radar, it can further include: determining the current illumination intensity of the road surface area by a pre-set light sensor; when the current illumination intensity is less than a preset illumination intensity threshold, using a pre-set light compensation lamp to compensate light for the road surface area, so that when the corresponding illumination intensity of the road surface area is not less than the preset illumination intensity threshold, the current road surface data of the road surface area is reacquired by the camera and the laser radar. Specifically, before the current road surface data of the road surface area is acquired, the road surface area can be compensated for light, which needs to be combined with the relationship between the current illumination intensity of the road surface area and the preset illumination intensity threshold, and the light compensation lamp pre-set in the chassis is used to compensate light for the road surface area, and when the illumination intensity of the road surface area is not less than the preset illumination intensity threshold, the current road surface data is reacquired by the camera and the laser radar. Wherein, when the light compensation, the light data of the road surface area can be fed back based on the light sensor, and the brightness and color temperature are automatically adjusted by a light compensation adjustment algorithm, so that the illumination intensity of the road surface area meets the corresponding illumination condition.

[0050] Step S12, crack detection is performed using the road surface elevation model constructed based on the current road surface data, and when it is determined that the road surface elevation model has cracks, the geometric parameter information of the cracks is extracted.

[0051] In this embodiment, the current road surface data of the road surface area of the road to be measured can be acquired by the above steps, and then the road surface elevation model constructed using the current road surface data can be used for crack detection. If the road surface elevation model is determined to have no cracks after crack detection, the next road surface area can be detected for defects. Correspondingly, if the road surface elevation model is determined to have cracks after crack detection, the geometric parameter information of the related cracks can be extracted. The geometric parameter information specifically includes the width and depth extracted by calculating the cross-sectional profile of the cracks.

[0052] In a specific embodiment, the crack detection using the road elevation model constructed based on the current road surface data can include: processing the point cloud data in the current road surface data in a point cloud voxelization manner to obtain a corresponding point cloud feature pyramid, and extracting features of the image data in the current road surface data by a preset convolutional neural network to obtain corresponding image features; performing feature fusion operation on the point cloud feature pyramid and the image features based on the alignment weight corresponding to the point cloud data and the image data, and combining a CRF (Conditional Random Field) optimization framework to construct the road elevation model corresponding to the current road surface data using the corresponding fused features; and performing crack detection using the road elevation model. Specifically, in the process of crack detection using the road elevation model constructed based on the current road surface data, feature extraction operation is involved for the point cloud data and the image data in the current road surface data; the point cloud data can be processed in a point cloud voxelization manner to extract a corresponding point cloud feature pyramid; the image data can be processed by a convolutional neural network to extract corresponding image features; further, after the multi-modal features are extracted, cross-modal feature alignment can be performed, the alignment weight of the point cloud and the image features is calculated based on the attention mechanism of the Transformer (attention-based sequence model), and then the image features and the point cloud feature pyramid are fused to obtain fused features; and combining the CRF optimization framework, the road elevation model corresponding to the road surface area can be constructed using the corresponding fused features. Further, the road elevation model can be used for crack detection to determine whether there is a crack cross-section contour in the elevation model and to collect geometric parameter information of the related cracks.

[0053] In step S13, the geometric parameter information is processed by a pre-trained language model in combination with the road environment information of the to-be-tested road surface to generate a road surface defect detection result corresponding to the road surface area.

[0054] In this embodiment, the geometric parameter information of the related cracks can be extracted from the road elevation model corresponding to the current road surface data of the road surface area through the above steps; then, in combination with the road environment information (road type, traffic flow, and surrounding buildings, public facilities, and other environmental information corresponding to the road section) of the to-be-tested road surface, the geometric parameter information is processed by a pre-trained language model to generate a road surface area detection result corresponding to the road surface area.

[0055] In a specific embodiment, in combination with the road environment information corresponding to the to-be-tested pavement, the geometric parameter information is processed by a pre-trained language model to generate the pavement defect detection result corresponding to the pavement area, which can include: through prompt learning technology, the road environment information corresponding to the to-be-tested pavement and the geometric parameter information are input into the pre-trained language model to generate a defect parameter threshold related to the pavement defect and a defect level corresponding to the defect parameter threshold; according to the similarity between the semantic vector of the geometric parameter information and the defect level, the pavement defect detection result corresponding to the pavement area is generated. Specifically, through prompt engineering technology, the geometric parameter information of the crack and the road environment information can be input into the pre-trained language model to generate the description of the corresponding defect parameter threshold and defect level; further, by comparing the similarity between the semantic vector of the geometric parameter information of the crack and the vector corresponding to the description information of these defect levels, the defect level corresponding to the current pavement area is determined, and the corresponding pavement defect detection result is generated.

[0056] In another specific embodiment, it can also include: when the unmanned vehicle is driving on the to-be-tested pavement, through a simultaneous localization and mapping technology, in combination with the data collected by the laser radar and the inertial navigation data of the unmanned vehicle, the surrounding environment map corresponding to the unmanned vehicle is constructed; in combination with the Beidou positioning information of the unmanned vehicle and the road condition data collected by the surround-view camera, a driving path is generated in real time, so that the unmanned vehicle drives on the to-be-tested pavement based on the driving path. Specifically, when the unmanned vehicle drives on the to-be-tested pavement, the environment map is constructed based on the SLAM (Simultaneous Localization and Mapping) technology to fuse the laser radar and inertial navigation data, the driving path is generated in combination with the real-time road condition collected by the Beidou positioning and surround-view camera, and the vehicle power and steering system is controlled to realize automatic driving.

[0057] In another specific embodiment, the Beidou positioning information of the unmanned vehicle is combined with the road condition data collected by the surround-view camera to generate a driving path in real time, which can include: if the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection position of the road to be detected, and the current road defect detection result indicates that there are crack defects in the road area, then according to a preset driving strategy, the corresponding driving path is generated in real time in combination with the road condition data collected by the surround-view camera; the preset driving strategy is to continue driving in the order of straight line, right turn and left turn; if the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection position of the road to be detected, and the current road defect detection result indicates that there are no crack defects in the road area, then the driving path is generated in real time in combination with the road condition data collected by the surround-view camera corresponding to the intersection position. Specifically, in the process of generating the driving path, the position of the unmanned vehicle can be considered, for example, when driving on a straight road, the unmanned vehicle can directly drive along the road; when at an intersection position, the current road area detection result indicates that there are defects, indicating that the current road needs to be continuously detected for defects, at this time, a suitable driving path can be generated based on the preset driving strategy in combination with real-time road condition information; in order to reduce the road safety risk of road defect detection, the straight driving can be selected first, followed by the right turn, and then the left turn in sequence, and finally the real-time road condition data needs to be considered to calculate the final path; in this way, the road defect detection efficiency can be ensured while the road safety risk caused by intersection decision is minimized. Further, if the unmanned vehicle is at an intersection position and the current road defect detection result indicates that there are no defects, then other roads can be selected for defect detection again, without considering the continuity of defects and the like, only considering the real-time road condition information for path planning.

[0058] As can be seen, the application can use the air cannon to remove protruding objects on the road, which can avoid the interference of protruding objects on road defect detection; and the multi-modal data of the laser radar and the camera can be combined to construct a road elevation model, and the geometric parameter information of the cracks extracted from the road elevation model can be analyzed and processed in combination with the road environment information, so that the generated defect detection result conforms to the actual road environment information; further, the unmanned vehicle can automatically construct a surrounding environment map, generate a driving path in combination with real-time road conditions, and automatically detect road defects, thereby improving the effectiveness and scene applicability of the road defect detection result.

[0059] As Figure 2 Fig. 1 shows a chassis structure diagram of an unmanned vehicle disclosed by the application, which includes a laser radar, a camera and an air cannon; the following embodiments will be described in detail in combination with a road defect detection method flowchart shown in Figure 3 Fig. 2, which specifically includes:

[0060] It can be seen that in the embodiment, the unmanned vehicle chassis installs cameras and lidar, which can obtain road color images, depth data and point cloud data in cooperation. The shooting area is completely covered by the chassis shadow (the chassis width is greater than or equal to 1.2 m, which ensures that the detection area is free of direct sunlight or shadow); and the chassis can also integrate adjustable light compensation lamps. The light intensity is collected in real time by the light sensor, and the brightness is dynamically adjusted by the light compensation adjustment algorithm, so that the illumination intensity deviation of the day and night shooting area is less than or equal to 5%; the air cannon is installed on the side of the chassis and has an angle adjustment function (such as pitch -30° to +30°, rotation 0° to 180°), which can release high-pressure air instantly. Further, through the chassis structure design (the chassis is 0.4-0.6m away from the ground, and the detection component covers a width of 1.5m laterally), the shooting area can be completely under the chassis shadow, avoiding the formation of car / tree shadow on the road surface by sunlight, street light and the like; during road surface data collection, the light sensor data can be used to automatically adjust the brightness and color temperature of the light compensation system, and the formula involved is: L=k×L0+b; wherein, L is the light compensation brightness, L0 is the ambient light intensity, k is the adjustment coefficient (-0.3~0.2), b is the reference brightness, which ensures that the illumination intensity of the day and night shooting area is stable within a certain range.

[0061] Further, the air cannon can remove the interference of objects on the road surface on defect detection; based on the three-dimensional reconstruction data corresponding to the chassis lidar and the camera, the height h of the object protrusion on the road surface (relative to the road surface reference surface) can be calculated, and when h>threshold value (default 5mm, adjustable), it is determined that the protruding object needs to be removed; the air cannon angle can be adjusted by the control algorithm, and high-pressure air is emitted. If the object displacement is greater than or equal to 1cm or it is out of the shooting area, it is determined that the disease is removed (not a road surface flaw or defect), that is, the interference on the road surface defect detection is excluded and is not included in the flaw detection; otherwise, it is retained as a suspected flaw.

[0062] Further, based on the chassis lidar point cloud and camera data, combined with the CRF optimization framework, a road elevation model can be constructed; based on the elevation model, crack detection can be performed, involving parameter extraction of cracks (width W, depth D), pits (area Area, volume Volume), etc., combined with image enhancement algorithms in shadow environment (suppressing glare noise), which can improve the feature recognition accuracy. Specifically, for the current road data collected, feature extraction can be performed; first, multi-scale feature extraction is performed, and a 4-layer feature pyramid is constructed by voxelizing the point cloud, and multi-scale semantic features are extracted from the image through a deep convolutional neural network; then, cross-modal feature alignment is performed, which can be based on the attention mechanism of Transformer to calculate the alignment weight of the point cloud and image features; then, feature pyramid fusion is performed, and the multi-scale fusion features are generated by layer-by-layer fusion through skip connection. Further, combined with the multi-scale fusion features, a road elevation model is constructed using the CRF optimization framework. And the extraction of crack parameters specifically includes: calculating the crack section profile in the model, extracting the width (W) and depth (D): W = max(dist(pi, pj)), D = max(href-hi); wherein pi and pj are crack boundary points obtained by edge detection and contour extraction on the model; dist(pi, pj) represents the Euclidean distance between boundary points pi and pj; href is the reference plane height, defined as the average elevation of non-crack points within a 5cm range around the crack; hi is the elevation value of the crack point pi, which is obtained by interpolating the points in the crack region.

[0063] Then, based on the extracted crack parameters, road defect detection can be performed; first, feature semantic conversion is performed, which converts crack geometric features into semantic vectors: S = fembed([W, D, Area, Volume]) wherein [W, D, Area, Volume] is the geometric feature vector of the crack, fembed is a feature embedding function, which is implemented by a multilayer perceptron to map the geometric features to a high-dimensional semantic space. Defect detection involves defect threshold reasoning, which can use a pre-trained language model to build a reasoning template: "When the road crack width ≥ {Wthres} mm and the depth ≥ {Dthres} mm, it is determined as the dangerous level {Level}"; wherein Wthres and Dthres are defect threshold parameters, which are dynamically generated by the pre-trained language model according to the road type, traffic flow and other context information; Level is the dangerous level, which can be divided into mild, moderate, severe and extremely severe; through the prompt learning technology, the crack geometric features and road environment information can be input into the pre-trained language model to generate the corresponding dangerous threshold and level description. Then the final defect situation can be determined by the cosine similarity between vectors.

[0064] Therefore, the protruding object on the road surface can be removed by the air cannon, so that the interference of the protruding object on the road defect detection can be avoided, and the light supplementing operation can reduce the defect detection deviation under different light intensities, and improve the accuracy of the road defect detection. Further, the multi-modal data of the laser radar and the camera can be combined to construct a road surface elevation model, and the geometric parameter information of the cracks extracted from the road surface elevation model is analyzed and processed in combination with the road environment information, so that the generated defect detection result conforms to the actual road environment information, and the effectiveness and scene applicability of the road defect detection result are improved.

[0065] As shown in Figure 4 The embodiment of the application discloses a road defect detection device applied to an unmanned vehicle, wherein a chassis of the unmanned vehicle is installed with a camera, a laser radar and an air cannon, and the device comprises:

[0066] A road data acquisition module 11 is configured to acquire current road data of a road surface region of the to-be-tested road surface after the air cannon emits air when the unmanned vehicle drives on the to-be-tested road surface through the camera and the laser radar;

[0067] A crack detection module 12 is configured to perform crack detection by using a road surface elevation model constructed based on the current road data, and extract geometric parameter information of the cracks when it is determined that the road surface elevation model has cracks;

[0068] A result generation module 13 is configured to process the geometric parameter information by a pre-trained language model in combination with road environment information corresponding to the to-be-tested road surface, so as to generate a road defect detection result corresponding to the road surface region.

[0069] Therefore, the protruding object on the road surface can be removed by the air cannon, so that the interference of the protruding object on the road defect detection can be avoided, and the light supplementing operation can reduce the defect detection deviation under different light intensities, and improve the accuracy of the road defect detection. Further, the multi-modal data of the laser radar and the camera can be combined to construct a road surface elevation model, and the geometric parameter information of the cracks extracted from the road surface elevation model is analyzed and processed in combination with the road environment information, so that the generated defect detection result conforms to the actual road environment information, and the effectiveness and scene applicability of the road defect detection result are improved.

[0070] In a specific embodiment, the road data acquisition module 11 can comprise:

[0071] A first data acquisition unit is configured to acquire original road data of a road surface region of the to-be-tested road surface matched with the position of the unmanned vehicle through the camera and the laser radar;

[0072] An object height determination unit is configured to reconstruct the road surface region in three dimensions using the original road surface data, and determine an object height of an object in the reconstructed three-dimensional model of the road surface region relative to a road surface reference plane corresponding to the road surface region;

[0073] A second data acquisition unit is configured to, when the object height is not less than a preset height threshold, fire air at the road surface region by the air cannon, and reacquire road surface data of the road surface region by the camera and the lidar to obtain current road surface data.

[0074] In another specific embodiment, the device can further include:

[0075] An illumination intensity determination module is configured to determine a current illumination intensity of the road surface region by a pre-set light sensor;

[0076] A light supplementing module is configured to, when the current illumination intensity is less than a preset illumination intensity threshold, supplement light to the road surface region by a pre-set light supplementing lamp, so that the road surface data of the road surface region is reacquired by the camera and the lidar to obtain current road surface data when the corresponding illumination intensity of the road surface region is not less than the preset illumination intensity threshold.

[0077] In a specific embodiment, the crack detection module 12 can include:

[0078] A feature extraction unit is configured to process point cloud data in the current road surface data by point cloud voxelization to obtain a corresponding point cloud feature pyramid, and extract features of image data in the current road surface data by a preset convolutional neural network to obtain corresponding image features;

[0079] A model construction unit is configured to perform feature fusion operations on the point cloud feature pyramid and the image features based on alignment weights corresponding to the point cloud data and the image data, and construct a road elevation model corresponding to the current road surface data by using corresponding fused features in combination with a CRF optimization framework;

[0080] A crack detection unit is configured to perform crack detection by using the road elevation model.

[0081] In a specific embodiment, the result generation module 13 can include:

[0082] A defect level generation unit is configured to input road environment information and the geometric parameter information corresponding to the to-be-tested road surface into a pre-trained language model by a prompt learning technique to generate a defect parameter threshold related to a road surface defect and a defect level corresponding to the defect parameter threshold;

[0083] A result generation unit is configured to generate a road surface defect detection result corresponding to the road surface region according to a similarity between the semantic vector of the geometric parameter information and the defect grade.

[0084] In a specific embodiment, the device can further include:

[0085] A map construction module is configured to construct a surrounding environment map corresponding to the unmanned vehicle by using a simultaneous localization and mapping technology in combination with data collected by the laser radar and inertial navigation data of the unmanned vehicle when the unmanned vehicle is driving on the road surface to be detected.

[0086] A path generation module is configured to generate a driving path in real time in combination with Beidou positioning information of the unmanned vehicle and road condition data collected by the surround-view camera, so that the unmanned vehicle drives on the road surface to be detected based on the driving path.

[0087] In another specific embodiment, the path generation module can include:

[0088] A first path generation unit is configured to generate a corresponding driving path in real time according to a preset driving strategy in combination with road condition data collected by the surround-view camera when the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection position of the road surface to be detected and the current road surface defect detection result indicates that there is a crack defect in the road surface region; the preset driving strategy is to continue driving in the order of straight line, right turn and left turn.

[0089] A second path generation unit is configured to generate a driving path in real time in combination with road condition data corresponding to the intersection position collected by the surround-view camera when the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection position of the road surface to be detected and the current road surface defect detection result indicates that there is no crack defect in the road surface region.

[0090] Further, the embodiments of the present application also disclose an electronic device, Figure 5 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0091] Figure 5A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the road defect detection method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiments of the present application can be specifically an electronic computer.

[0092] In the embodiments of the present application, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0093] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0094] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the road defect detection method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0095] Further, the present application further discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the road defect detection method disclosed above. For the specific steps of the method, reference can be made to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0096] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, reference can be made to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and reference can be made to the method part.

[0097] Those skilled in the art will further appreciate that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functionality, which has been described generally and symbolically in flow charts. Having thus described the functionality of the examples, a person of ordinary skill in the art will be able to implement such functionality in hardware and / or software, and will recognize that the bounds of the examples are not limited by one approach or the other. The various examples can be realized in a centralized fashion in one computer system or network, or in a distributed fashion where different elements are spread across several computer systems or sub-networks. Any kind of computer system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, carries out the methods described herein.

[0098] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hard disk can be used as a storage medium.

[0099] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply singular or plural. Moreover, the terms "include", "have", or any other variant thereof are intended to encompass non-exclusive inclusions, such that processes, methods, articles, or apparatuses that comprise a set of elements not expressly listed are also within the scope of the present application. In addition, the articles "a" and "an" are used herein to refer to one or to more than one (i.e., to one or at least one) of the grammatical object of the article. By way of example, "an element" means one element or one or more elements.

[0100] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application have been described in the text. The above description of the examples is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the description should not be understood as limiting the present application.

Claims

1. A road surface defect detection method, characterized in that: Applied to an unmanned vehicle, the chassis of which is equipped with a camera, a laser radar, and an air cannon, the method includes: When the unmanned vehicle is traveling on the road to be tested, current road surface data of the road area of ​​the road to be tested after the air cannon currently emits air is collected by the camera and the laser radar; Performing crack detection using a road surface elevation model constructed based on the current road surface data, and extracting geometric parameter information of the crack when cracks are detected in the road surface elevation model; Combined with the road environment information corresponding to the road surface to be tested, the geometric parameter information is processed through a pre-trained language model to generate a road surface defect detection result corresponding to the road surface area.

2. The road surface defect detection method according to claim 1, characterized in that: The collecting of current road surface data of the road surface area after the air cannon currently emits air on the road surface to be tested by the camera and the laser radar includes: Collecting original road surface data of the road area to be measured that matches the position of the unmanned vehicle by the camera and the laser radar; Performing three-dimensional reconstruction of the road surface area using the original road surface data, and determining the height of an object in the road surface area of ​​the reconstructed three-dimensional model relative to a road surface reference plane corresponding to the road surface area; When the height of the object is not less than a preset height threshold, air is emitted to the road surface area through the air cannon, and road surface data of the road surface area is recollected through the camera and the lidar to obtain current road surface data.

3. The road surface defect detection method according to claim 2, characterized in that: Before re-collecting the road surface data of the road area by the camera and the laser radar to obtain the current road surface data, the method further includes: determining the current light intensity of the road area by a pre-set light sensor; When the current light intensity is less than a preset light intensity threshold, the road surface area is supplemented with light using a preset fill light, so that when the light intensity corresponding to the road surface area is not less than the preset light intensity threshold, the road surface data of the road surface area is recollected by the camera and the lidar to obtain the current road surface data.

4. The road surface defect detection method according to claim 1, characterized in that: The method of detecting cracks using a road surface elevation model constructed based on the current road surface data includes: Processing the point cloud data in the current road surface data by voxelization to obtain a corresponding point cloud feature pyramid, and performing feature extraction on the image data in the current road surface data by a preset convolutional neural network to obtain corresponding image features; Based on the alignment weights corresponding to the point cloud data and the image data, a feature fusion operation is performed on the point cloud feature pyramid and the image features, and in combination with a CRF optimization framework, a road surface elevation model corresponding to the current road surface data is constructed using the corresponding fused features; The pavement elevation model is used to perform crack detection.

5. The road surface defect detection method according to claim 1, characterized in that: The processing of the geometric parameter information by a pre-trained language model in combination with the road environment information corresponding to the road surface to be tested to generate a road surface defect detection result corresponding to the road surface area includes: Inputting the road environment information and the geometric parameter information corresponding to the road surface to be tested into a pre-trained language model through a prompt learning technology to generate a defect parameter threshold value related to the road surface defect and a defect level corresponding to the defect parameter threshold value; A road surface defect detection result corresponding to the road surface area is generated according to the similarity between the semantic vector of the geometric parameter information and the defect level.

6. The road surface defect detection method according to any one of claims 1 to 5, characterized in that: Also includes: When the unmanned vehicle is driving on the road to be tested, a map of the surrounding environment corresponding to the unmanned vehicle is constructed by combining the data collected by the laser radar and the inertial navigation data of the unmanned vehicle through synchronous positioning and mapping technology; Combining the Beidou positioning information of the unmanned vehicle and the road condition data collected by the surround-view camera, a driving path is generated in real time so that the unmanned vehicle can travel on the road to be tested based on the driving path.

7. The road surface defect detection method according to claim 6, characterized in that: The method of combining the Beidou positioning information of the unmanned vehicle and the road condition data collected by the surround-view camera to generate a driving path in real time includes: If the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection of the road surface to be tested, and the current road surface defect detection result indicates that crack defects exist in the road surface area, then a corresponding driving path is generated in real time according to a preset driving strategy combined with road condition data collected by a surround-view camera; the preset driving strategy is to continue driving in a straight line, turn right, and turn left in this order; If the Beidou positioning information of the unmanned vehicle indicates that the unmanned vehicle is located at an intersection of the road surface to be tested, and the current road surface defect detection result indicates that there are no crack defects in the road surface area, then the driving path is generated in real time in combination with the road condition data corresponding to the intersection position collected by the surround-view camera.

8. A road surface defect detection device, characterized in that: Applied to unmanned vehicles, the chassis of which is equipped with a camera, a laser radar, and an air cannon. The device includes: a road surface data acquisition module, configured to collect, by means of the camera and the laser radar, current road surface data of a road surface area of ​​the road surface to be tested after the air cannon currently emits air, when the unmanned vehicle is traveling on the road surface to be tested; a crack detection module, configured to detect cracks using a road elevation model constructed based on the current road surface data, and extract geometric parameter information of the cracks when cracks are detected in the road elevation model; The result generation module is used to combine the road environment information corresponding to the road surface to be tested and process the geometric parameter information through a pre-trained language model to generate a road surface defect detection result corresponding to the road surface area.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the road surface defect detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the road surface defect detection method according to any one of claims 1 to 7.

Images