A method and system for detecting road surface unevenness

Abstract

The present invention relates to the technical field of data processing, and relates to a method and system for detecting road bumps. The method includes: acquiring a binocular road image and converting it into a disparity map, and optimizing the disparity map to obtain an optimized disparity map; inputting the binocular road image into a bump target detection model to obtain a bump target detection frame in the binocular road image, where the bump target detection frame is used to frame bump targets; mapping the bump target detection frame to the optimized disparity map according to the image coordinates of the points on the bump target detection frame; obtaining feature key points on the optimized disparity map according to the bump target detection frame, and using the feature key points for elevation measurement. The bump target detection method of the present invention can improve the detection accuracy of bump targets.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing. More specifically, the present invention relates to a method and system for detecting bumps and depressions on a road surface. Background Art

[0002] At present, there have been some studies on detecting bumps and depressions on the road surface in complex scenarios. Generally, they can be roughly divided into mechanical device detection, inertial sensor detection, radar detection, camera acquisition detection, etc. Among them, mechanical devices generally rely on measuring heads for measurement, which greatly limits their application scenarios and cannot be applied to some systems that require real-time response; there are problems such as large volume and slow detection process. Inertial sensors usually have low accuracy and need to be combined with other methods to achieve higher detection accuracy. Radar detection usually generates a 3D point cloud map of the road surface with the help of radar equipment for bump and depression detection. Although it has high accuracy and fast response, it has a high cost and cannot detect some bump and depression targets with small elevations; camera acquisition detection extracts information from the road surface images collected by the camera, using a lower cost while having a fast response speed and high detection accuracy; however, for the accuracy of camera acquisition detection, in addition to being affected by irresistible factors such as weather conditions, it more depends on the image acquisition quality of the camera and the image processing process. Although many current bump and depression target recognition models perform well in recognizing a single type of bump and depression target such as speed bumps, in complex road surface scenarios, their effects are not satisfactory. The complex road surface increases the interference of the image, making the detection of the target blurred and inaccurate. For autonomous vehicles, since they need to perform real-time monitoring in a changing environment, the working difficulty is further exacerbated. Although some image processing technologies can improve the video effect and are beneficial to subsequent bump and depression target recognition tasks, their accompanying high complexity and long inference time become bottlenecks. Integrating these technologies into the bump and depression target detection process, although improving the detection accuracy, is not conducive to achieving fast and real-time recognition and positioning. Summary of the Invention

[0003] To solve the technical problem of poor detection accuracy of bumps and depressions on the road surface in the prior art, the present invention provides solutions in the following aspects.

[0004] In a first aspect, the present invention provides a method for detecting bumps and depressions on a road surface, including:

[0005] Obtaining a binocular image of the road surface and converting it into a disparity map, and optimizing the disparity map to obtain an optimized disparity map;

[0006] Input the binocular road surface image into the convex and concave target detection model to obtain the convex and concave target detection frame in the binocular road surface image, where the convex and concave target detection frame is used to frame the convex and concave targets; the convex and concave target detection model includes a feature extraction module, the output of the feature extraction module is connected to the input of the feature fusion module, and the output of the feature fusion module is connected to the input of the dynamic detection head module. The feature extraction module includes a DSconv module and a dual-path architecture. The dual-path architecture includes a local information extraction branch and a global information extraction branch. The output of the DSconv module is respectively connected to the inputs of the local information extraction branch and the global information extraction branch; the dynamic detection head module includes a multi-layer stacked dynamic detection head for detecting the convex and concave road surface targets based on the fused image generated by the feature fusion module.

[0007] Map the convex and concave target detection frame to the optimized disparity map according to the image coordinates of the points on the convex and concave target detection frame.

[0008] Obtain the feature key points on the optimized disparity map according to the convex and concave target detection frame, and use the feature key points for elevation measurement to obtain the elevation measurement result.

[0009] The beneficial effects are as follows: In the present invention, by setting a global information extraction branch and a local information extraction branch in the feature extraction module of the convex and concave target detection model to extract the global information and local information of the binocular road surface image, the bias or incomplete cognition brought by a single perspective can be eliminated, and the feature extraction module has high performance in capturing feature diversity. By setting the dynamic detection head module, the extracted high-margin features can be effectively utilized to achieve accurate recognition and positioning of the convex and concave road surface targets; in addition, by optimizing the disparity map, the error information in the disparity map can be filtered out, thereby improving the quality of the disparity map and further improving the accuracy of convex and concave target recognition. Therefore, the convex and concave target detection method of the present invention can improve the convex and concave target detection accuracy and help the autonomous driving vehicle make a more accurate judgment on the road conditions in complex road surface scenarios.

[0010] Preferably, the global information extraction branch includes a plurality of cross-connected MBConv modules and EfficientVit modules, and an SPPF module located at the end of the global information extraction branch; the local information extraction branch includes a plurality of cross-connected residual modules, Dwconv modules and Concat modules; several EfficientVit modules of the global information extraction branch are connected to the Concat module of the local information extraction branch.

[0011] Its beneficial effects are as follows: By setting the EfficientVit module in the global information extraction branch, multi-scale linear attention can be used to obtain the global receptive field and learn multi-scale features, achieving efficient feature extraction; By setting the MBConv module in the global information extraction branch to perform the operation of first increasing the dimension and then decreasing the dimension on the feature map output by the DSconv module, the channel dimension features can be obtained while accelerating the calculation speed; By setting the SPPF module, the spatial pyramid pooling operation can be performed on the feature map output by the third EfficientVit module to extract features from feature maps of different scales and obtain the features of feature maps of different scales.

[0012] By setting the residual module in the local information extraction branch, higher-level feature information can be obtained, and the computational complexity can be reduced while maintaining a high representation ability; By setting the Dwconv module, the channel features and spatial features of the image can be extracted; By setting the first Concat module, the feature map with a large amount of semantic information at low resolution and the feature map with detailed information at high resolution can be spliced to obtain a larger feature map.

[0013] Preferably, the first residual module includes a first Conv module and a second Conv module. The output of the second Conv module is connected to the input of the BottleNeck module. The output of the first Conv module and the output of the BottleNeck module are connected to the input of the fifth Concat module. The output of the fifth Concat module is connected to the input of the third Conv module. The output of the third Conv module is the output of the first residual module. The inputs of the first Conv module and the second Conv module are the inputs of the first residual module.

[0014] Preferably, the feature fusion module includes a fourth Dwconv module, a first UpSample module, a third Concat module, a fifth residual module, a fifth Dwconv module, a second UpSample module, a fourth Concat module, a sixth residual module, a sixth Dwconv module, a seventh residual module, a sixth Dwconv module, and an eighth residual module, which are connected end to end in sequence. Among them, the output of the sixth residual module is the first fused image, the output of the seventh residual module is the second fused image, and the output of the eighth residual module is the third fused image.

[0015] The beneficial effects are as follows: By setting the Dwconv module in the feature fusion module, the channel features and spatial features of feature maps with different scales can be extracted; through Upsample for upsampling, a mapping operation from a feature map with a small resolution to a large resolution is performed on the feature map output by the fourth Dwconv module, facilitating subsequent feature extraction; by setting the fourth Concat module, the feature map containing a large amount of semantic information output by the Upsample module and the feature map with high resolution and containing detailed information output by the fourth residual module are spliced to obtain a larger feature map.

[0016] Preferably, the shape of the convex-concave target detection frame is rectangular. Obtaining feature key points on the optimized disparity map based on the convex-concave target detection frame, and performing elevation measurement on the feature key points includes:

[0017] Taking the pixel points on the middle horizontal line of the convex-concave target detection frame as the first pixel points, and selecting second pixel points vertically corresponding to each of the first pixel points from the lower side of the convex-concave target detection frame; the vertically corresponding second pixel points refer to the intersection points of the perpendiculars drawn from the points on the middle horizontal line to the lower side and the lower side.

[0018] Obtaining the world coordinate system coordinates corresponding to each of the first pixel points and the second pixel points according to the parameter information of the binocular camera; then calculating the Euclidean distance between each of the first pixel points and the corresponding second pixel points, taking the average value thereof, and using this average value as the height of the convex-concave target.

[0019] Preferably, optimizing the disparity map includes:

[0020] Performing bilinear interpolation processing on the disparity map to obtain an enlarged disparity map;

[0021] Performing dilation processing on the enlarged disparity map;

[0022] Performing erosion processing on the dilated disparity map.

[0023] Preferably, the dynamic detection head module includes 4 layers of stacked dynamic detection heads.

[0024] Preferably, the road surface binocular image is obtained by a binocular camera. Before obtaining the road surface binocular image, it further includes: calibrating the binocular camera to obtain the parameter information of the binocular camera; the parameter information includes: the internal parameter matrix, external parameter matrix, radial distortion, and tangential distortion of the binocular camera.

[0025] Preferably, the Zhang Zhengyou calibration method is used when calibrating the binocular camera.

[0026] In a second aspect, the present invention provides a road surface unevenness detection system, including a binocular camera for collecting road surface images, a calibration system for calibrating the binocular camera, and a measurement system connected to the binocular camera and the calibration system, where the measurement system is used to execute the road surface unevenness detection method of the present invention.

[0027] The beneficial effects of the present invention are as follows: By adopting the uneven target detection method of the present invention, the detection accuracy of uneven targets can be improved, which helps autonomous driving vehicles make more accurate judgments on road conditions in complex road surface scenarios. Description of the Drawings

[0028] By referring to the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become readily understandable. In the drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, where:

[0029] Figure 1 is a schematic flowchart showing the road surface unevenness detection method according to an embodiment of the present invention;

[0030] Figure 2 is a schematic structural diagram showing a dynamic detection head according to an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram showing the installation position of the binocular camera according to an embodiment of the present invention;

[0032] Figure 4 is a schematic diagram showing the image detection error rate of Group 1 according to an embodiment of the present invention;

[0033] Figure 5 is a schematic diagram showing the image detection error rate of Group 2 according to an embodiment of the present invention;

[0034] Figure 6 is a schematic diagram showing the image detection error rate of Group 3 according to an embodiment of the present invention;

[0035] Figure 7 is a schematic diagram showing the image detection error rate of Group 4 according to an embodiment of the present invention;

[0036] Figure 8 is a schematic diagram showing the image detection error rate of Group 5 according to an embodiment of the present invention;

[0037] Figure 9 is a schematic diagram showing the image detection error rate of Group 6 according to an embodiment of the present invention;

[0038] Figure 10It schematically shows a road surface image with speed bumps after labeling tags according to an embodiment of the present invention;

[0039] Figure 11 It schematically shows a road surface image with manhole covers after labeling tags according to an embodiment of the present invention;

[0040] Figure 12 It schematically shows a schematic diagram of the speed bump recognition result of the baseline model according to an embodiment of the present invention;

[0041] Figure 13 It schematically shows a schematic diagram of the speed bump recognition result of the convex - concave target detection model according to an embodiment of the present invention;

[0042] Figure 14 It schematically shows a schematic diagram of the manhole cover recognition result of the baseline model according to an embodiment of the present invention;

[0043] Figure 15 It schematically shows a schematic diagram of the manhole cover recognition result of the convex - concave target detection model according to an embodiment of the present invention;

[0044] Figure 16 It schematically shows a schematic diagram of the structure of the feature extraction module according to an embodiment of the present invention;

[0045] Figure 17 It schematically shows a schematic diagram of the structure of the feature fusion module according to an embodiment of the present invention;

[0046] Figure 18 It schematically shows a schematic diagram of the structure of the measurement system according to an embodiment of the present invention. Detailed implementation manners

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

[0048] Next, the detailed implementation manners of the present invention will be described in detail with reference to the accompanying drawings.

[0049] Embodiment of the road surface convex - concave detection method:

[0050] As Figure 1 shown, the road surface convex - concave detection method of the present invention includes:

[0051] S101. Obtain an optimized disparity map. Specifically: obtain a binocular image of the road surface and convert it into a disparity map, and optimize the disparity map to obtain an optimized disparity map;

[0052] By optimizing the disparity map, the error information in the disparity map can be filtered out, thereby improving the quality of the disparity map.

[0053] S102. Generate convex and concave object detection frames, specifically: input the binocular road surface image into the convex and concave object detection model to obtain the convex and concave object detection frames in the binocular road surface image, and the convex and concave object detection frames are used to frame convex and concave objects; the convex and concave object detection model includes a feature extraction module, the output of the feature extraction module is connected to the input of the feature fusion module, the output of the feature fusion module is connected to the input of the dynamic detection head module, the feature extraction module includes a DSconv module and a dual-path architecture, the dual-path architecture includes a local information extraction branch and a global information extraction branch, and the output of the DSconv module is respectively connected to the inputs of the local information extraction branch and the global information extraction branch; the dynamic detection head module includes multiple stacked dynamic detection heads for detecting convex and concave road surface objects according to the fused image generated by the feature fusion module.

[0054] In this embodiment, the dynamic detection head module includes 4 stacked dynamic detection heads. In other embodiments, the number of stacked dynamic detection heads can also be other appropriate values.

[0055] By repeatedly stacking the dynamic detection heads multiple times, the detection performance of the convex and concave detection model is further improved. This multiple enhancement structure in terms of hierarchy, space, and task enables the model to more comprehensively perceive the features in complex scenes, thereby significantly improving the accuracy and robustness of model detection.

[0056] The structure of the Dynamic Head is as Figure 2 shown, and the module includes scale-aware attention π L , spatial-aware attention π S , and task-aware attention π C . As Figure 3 shown, the binocular camera 1 can be installed on the vehicle. During the vehicle's driving process, the binocular camera continuously collects road surface images within a certain distance in front of the vehicle and outputs left and right binocular images, providing a data source for subsequent image processing. The shooting interval of the camera should be determined according to the vehicle's speed, the road surface range that the camera can capture, etc. Ideally, the camera should be able to capture all the road surface images during the vehicle's driving process to achieve a comprehensive detection effect; in the figure, 2 is a convex and concave object.

[0057] By setting up a feature extraction module to extract the global and local information of the road surface binocular images, the bias or incomplete cognition caused by a single perspective can be eliminated, enabling the feature extraction module to have high performance in capturing feature diversity. By setting up a dynamic detection head module, the extracted high-margin features can be effectively utilized to achieve accurate recognition and positioning of the convex and concave road surface targets.

[0058] S103. Map the convex and concave target detection frame to the optimized disparity map according to the image coordinates of the points on the convex and concave target detection frame;

[0059] S104. Obtain the feature key points on the optimized disparity map according to the convex and concave target detection frame, and use the feature key points for elevation measurement to obtain the elevation measurement result.

[0060] To verify the effect of the road surface convex and concave detection method of this embodiment, the detection result of the road surface convex and concave detection method of this embodiment is tested. The test method is as follows:

[0061] Select 6 groups of speed bumps from the collected road scene data, arrange them in the order from far to near the vehicle, and each group of data contains 10 images; for the images in each group of data, execute the steps of the road surface convex and concave detection method of this embodiment to obtain the test results.

[0062] The 6 groups of speed bumps are Group 1, Group 2, Group 3, Group 4, Group 5, and Group 6 in the order from far to near the binocular camera. The detection error rate of the images in Group 1 is as Figure 4 shown, the detection error rate of the images in Group 2 is as Figure 5 shown, the detection error rate of the images in Group 3 is as Figure 6 shown, the detection error rate of the images in Group 4 is as Figure 7 shown, the detection error rate of the images in Group 5 is as Figure 8 shown, and the detection error rate of the images in Group 6 is as Figure 9 shown. According to the test results, when the distance between the vehicle and the speed bump is far, the displacement information of the speed bump target in the disparity map is relatively small, resulting in a relatively large elevation estimation error. As the speed bump gradually approaches the vehicle, the displacement information in the disparity map becomes more obvious and accurate, and the height prediction error rate of the speed bump decreases. In this experiment, the actual height of the speed bump is 4.5 cm. Through the analysis and calculation of 60 images, the average error rate obtained is 14.20%. Under different distance conditions, due to the change of the target information in the disparity map, the accuracy and stability of the height estimation will change, and the elevation estimation error of the speed bump shows a downward trend from far to near.

[0063] To verify the performance of the convex-concave target detection model of this embodiment, a road surface image with a manhole cover and a road surface image with a speed bump are selected from the road surface convex-concave dataset. The road surface images of the manhole cover and the speed bump are labeled with tags, and then the baseline model and the convex-concave target detection model of this embodiment are respectively used to detect the convex-concave targets in these two road surface images.

[0064] The road surface image with a speed bump after labeling the tags is as Figure 10 shown, and the road surface image with a manhole cover after labeling the tags is as Figure 11 shown. The result of using the baseline model to detect the convex-concave target in the road surface image with a speed bump is as Figure 12 shown, and the result of using the convex-concave target detection model of this embodiment to detect the convex-concave target in the road surface image with a speed bump is as Figure 13 shown. The result of using the baseline model to detect the convex-concave target in the road surface image with a manhole cover is as Figure 14 shown, and the result of using the convex-concave target detection model of this embodiment to detect the convex-concave target in the road surface image with a manhole cover is as Figure 15 shown. By comparing the detection effect diagrams, it can be clearly observed that when facing smaller targets, such as the misdetection and missed detection of manhole covers, PB-Det can detect the targets more accurately compared to the baseline model. At the same time, for targets such as speed bumps with a length significantly greater than the width, PB-Det can also detect the targets more precisely, reducing the occurrence of misdetection and missed detection. Through the intuitive comparison of the detection effect diagrams, it further verifies the significant advantages of the model PB-Det in the detection of road surface convex-concave targets.

[0065] In this embodiment, by setting a global information extraction branch and a local information extraction branch in the feature extraction module of the convex-concave target detection model to extract the global information and local information of the road surface binocular image, the bias or incomplete cognition brought by a single perspective can be eliminated, making the feature extraction module have high performance in capturing feature diversity. By setting the dynamic detection head module, the extracted high-margin features can be effectively utilized to achieve the accurate recognition and positioning of road surface convex-concave targets; in addition, by optimizing the disparity map, the error information in the disparity map can be filtered out, thereby improving the quality of the disparity map and further improving the accuracy of convex-concave target recognition. Therefore, adopting the convex-concave target detection method of the present invention can improve the accuracy of convex-concave target detection, which helps the autonomous driving vehicle to make more accurate judgments on the road conditions in complex road surface scenarios.

[0066] In one embodiment, the training process of the convex-concave target detection model is as follows:

[0067] S201. Use a stereo camera to collect binocular road surface images on campus roads and social roads in the automotive industrial park; and obtain images of corresponding types of convex and concave road surface targets through the network;

[0068] Since the roads in the campus and experimental park are generally flat, it is very likely that data of the two types of potholes and ruts will not be encountered during the actual collection process. Therefore, it is necessary to obtain images of corresponding types of convex and concave road surface targets through the network to make up for this lack, and additional data of corresponding types of convex and concave road surface targets is obtained to ensure the comprehensiveness of algorithm training.

[0069] S202. Perform data screening and cleaning on the collected binocular road surface images and the obtained images of convex and concave road surface targets to obtain a convex and concave road surface dataset; and divide the convex and concave road surface dataset into a training set, a validation set, and a test set;

[0070] Data screening and cleaning can improve the quality of the collected binocular road surface images and the obtained images of convex and concave road surface targets.

[0071] S203. Use the training set, validation set, and test set to train the convex and concave target detection model to obtain the best training weights.

[0072] The convex and concave target detection model can be trained through a loss function.

[0073] In one embodiment, the global information extraction branch includes a plurality of cross-connected MBConv modules and EfficientVit modules, and an SPPF module located at the end of the global information extraction branch; the local information extraction branch includes a plurality of cross-connected residual modules, Dwconv modules, and Concat modules; and several EfficientVit modules of the global information extraction branch are connected to the Concat module of the local information extraction branch.

[0074] Such as Figure 16As shown in the figure, the global information extraction branch includes a first MBConv module. The output of the first MBConv module is connected to the input of the second MBConv module. The output of the second MBConv module is connected to the input of the first EfficientVit module. The output of the first EfficientVit module is respectively connected to the input of the third MBConv module and the input of the first Concat module. The output of the third MBConv module is connected to the input of the second EfficientVit module. The output of the second EfficientVit module is respectively connected to the input of the fourth MBConv module and the input of the second Concat module. The output of the fourth MBConv module is connected to the input of the third EfficientVit module. The output of the third EfficientVit module is connected to the input of the SPPF module.

[0075] The local information extraction branch includes a first residual module. The output of the first residual module is connected to the input of the first Dwconv module. The output of the first Dwconv module is connected to the input of the second residual module. The output of the second residual module is connected to the input of the first Concat module. The output of the first Concat module is connected to the input of the second Dwconv module. The output of the second Dwconv module is connected to the input of the third residual module. The output of the third residual module is connected to the input of the second Concat module. The output of the second Concat module is connected to the input of the third Dwconv module. The output of the third Dwconv module is connected to the input of the fourth residual module.

[0076] Among them, the output of the SPPF module is the first feature map to be fused, the output of the fourth residual module is the second feature map to be fused, and the output of the second Dwconv module is the third feature map to be fused. The output of the DSconv module is respectively connected to the input of the first MBConv module and the input of the first residual module.

[0077] As the depth of the model deepens, the feature width gradually increases, and the final output feature size of the feature extraction module is 64×40×40.

[0078] The DSconv module refers to the depthwise separable convolution, which is an algorithm obtained by improving the standard convolution calculation in the convolutional neural network. By splitting the correlation of the spatial dimension and the channel (depth) dimension, it reduces the number of parameters required for the convolution calculation and has been proven to improve the utilization efficiency of the convolutional kernel parameters in some studies. The MBConv module refers to the Mobile Bottleneck Convolution module.

[0079] In the global information extraction branch, the first MBConv module and the second MBConv module are used to perform an operation of first increasing the dimension and then decreasing the dimension on the feature map output by the DSConv module, reducing the parameters used in the extraction process while extracting features. The first EfficientVit module is used to obtain the global receptive field of the feature map output by the second MBConv module through multi-scale linear attention, learn multi-scale features, achieve efficient feature extraction, and output a feature map containing a large amount of semantic information. The third MBConv module is used to perform an operation of first increasing the dimension and then decreasing the dimension on the feature map output by the first EfficientVit module, thereby reducing the parameters used in the extraction process while extracting features. The second EfficientVit module is used to obtain the global receptive field of the feature map output by the third MBConv module and learn multi-scale features to achieve efficient feature extraction; the fourth MBConv module is used to perform an operation of first increasing the dimension and then decreasing the dimension on the feature map output by the second EfficientVit module, reducing the parameters used in the extraction process while extracting features. The third EfficientVit module is used to obtain the global receptive field of the feature map output by the fourth MBConv module and learn multi-scale features to achieve efficient feature extraction; the SPPF module is used to perform spatial pyramid pooling on the feature map output by the third EfficientVit module, extract features from feature maps of different scales, and obtain the features of feature maps of different scales.

[0080] By setting the EfficientVit module in the global information extraction branch, the global receptive field can be obtained through multi-scale linear attention, multi-scale features can be learned, and efficient feature extraction can be achieved. By setting the MBConv module in the global information extraction branch to perform an operation of first increasing the dimension and then decreasing the dimension on the feature map output by the DSconv module, the calculation speed can be accelerated while obtaining the features in the channel dimension; by setting the SPPF module, spatial pyramid pooling can be performed on the feature map output by the third EfficientVit module, features can be extracted from feature maps of different scales, and the features of feature maps of different scales can be obtained.

[0081] In the local information extraction branch, the first residual module is used to obtain higher-level feature information from the feature map output by the DSConv module, reducing the computational complexity while maintaining a high representation ability; the first Dwconv module is used to perform depthwise separable convolution operations on the feature map output by the first residual module to extract channel features and spatial features; the second residual module is used to obtain higher-level feature information from the feature map output by the first Dwconv module, reducing the computational complexity while maintaining a high representation ability; the first Concat module is used to concatenate the high-resolution feature map containing detailed information output by the second residual module and the feature map containing a large amount of semantic information output by the first EfficientVit module to obtain a larger feature map; the second Dwconv module is used to perform depthwise separable convolution operations on the concatenated feature map to obtain features containing detailed and semantic information; the third residual module is used to obtain higher-level feature information from the feature map output by the second Dwconv module, reducing the computational complexity while maintaining a high representation ability; the second Concat module is used to concatenate the high-resolution feature map containing detailed information output by the second Dwconv module and the feature map containing a large amount of semantic information output by the second EfficientVit module to obtain a larger feature map; the third Dwconv module is used to perform depthwise separable convolution operations on the feature map output by the second Concat module to obtain features containing detailed and semantic information; the fourth residual module is used to obtain higher-level feature information from the feature map output by the third Dwconv module, thereby reducing the computational complexity while maintaining a high representation ability.

[0082] By setting residual modules in the local information extraction branch, higher-level feature information can be obtained, reducing the computational complexity while maintaining a high representation ability; by setting Dwconv modules, the channel features and spatial features of the image can be extracted; by setting the first Concat module, the feature map with a large amount of semantic information at low resolution can be concatenated with the feature map with detailed information at high resolution to obtain a larger feature map.

[0083] In one embodiment, the first residual module includes a first Conv module and a second Conv module. The output of the second Conv module is connected to the input of the BottleNeck module. The output of the first Conv module and the output of the BottleNeck module are connected to the input of the fifth Concat module. The output of the fifth Concat module is connected to the input of the third Conv module. The output of the third Conv module is the output of the first residual module. The inputs of the first Conv module and the second Conv module are the inputs of the first residual module.

[0084] In one embodiment, asFigure 17 As shown, the feature fusion module includes a fourth Dwconv module, a first UpSample module, a third Concat module, a fifth residual module, a fifth Dwconv module, a second UpSample module, a fourth Concat module, a sixth residual module, a sixth Dwconv module, a seventh residual module, a sixth Dwconv module, and an eighth residual module that are connected end to end in sequence. Among them, the input of the third Concat module is also connected to the output of the fourth residual module, the input of the fourth Concat module is also connected to the output of the second Dwconv module, the output of the sixth residual module is the first fused image, the output of the seventh residual module is the second fused image, and the output of the eighth residual module is the third fused image.

[0085] The working process of the feature fusion module in this embodiment is as follows: The fourth Dwconv module performs depthwise separable convolution (Dwconv) operations on the first feature map to be fused output by the SPPF module to extract the channel features and spatial features of feature maps with different densities; the first UpSample module performs a mapping operation from a small resolution to a large resolution on the feature map output by the fourth Dwconv module to facilitate subsequent feature extraction; the third Concat module concatenates the feature map output by the first UpSample module and the second feature map to be fused output by the fourth residual module to obtain a larger feature map; the fifth residual module obtains a larger feature map based on the feature map output by the third Concat module, and the fifth Dwconv module performs depthwise separable convolution (Dwconv) operations on the feature map output by the fifth residual module to extract the channel features and spatial features of feature maps with different densities; the second UpSample module performs upsampling on the feature map output by the fifth Dwconv module and performs a mapping operation from a small resolution to a large resolution on the input feature map to facilitate subsequent feature extraction; the fourth Concat module concatenates the feature map output by the second UpSample module and the third feature map to be fused output by the second Dwconv module to obtain a larger feature map; the sixth residual module obtains higher-level feature information based on the feature map output by the fourth Concat module, reduces the computational complexity while maintaining a high representation ability, and outputs the features of this layer to the dynamic detection head; the sixth Dwconv module performs depthwise separable convolution (Dwconv) operations on the feature map output by the sixth residual module to extract the channel features and spatial features of feature maps with different densities; the seventh residual module obtains higher-level feature information based on the feature map output by the sixth Dwconv module, reduces the computational complexity while maintaining a high representation ability, and outputs the features of this layer to the detection head; the seventh Dwconv module performs depthwise separable convolution (Dwconv) operations on the feature map output by the seventh residual module to extract the channel features and spatial features of feature maps with different densities; the eighth residual module obtains higher-level feature information based on the feature map output by the seventh Dwconv module, reduces the computational complexity while maintaining a high representation ability, and outputs the feature map output by the seventh Dwconv module to the detection head.

[0086] By setting the Dwconv module in the feature fusion module, the channel features and spatial features of feature maps with different scales can be extracted; by performing upsampling through Upsample, a mapping operation from a small resolution to a large resolution is performed on the feature map output by the fourth Dwconv module to facilitate subsequent feature extraction; by setting the fourth Concat module, the feature map with a large amount of semantic information output by the Upsample module and the high-resolution feature map with detailed information output by the fourth residual module are concatenated to obtain a larger feature map.

[0087] In one embodiment, the shape of the convex-concave target detection frame is rectangular. Obtaining feature key points on the optimized disparity map according to the convex-concave target detection frame, and performing elevation measurement on the feature key points includes:

[0088] S301: Taking the pixel points on the middle horizontal line of the convex-concave target detection frame as the first pixel points, and selecting second pixel points corresponding vertically to each of the first pixel points from the lower side of the convex-concave target detection frame; the second pixel points corresponding vertically refer to the intersection points of the perpendiculars drawn from the points on the middle horizontal line to the lower side and the lower side.

[0089] S302: Obtaining the world coordinate system coordinates corresponding to each of the first pixel points and the second pixel points according to the parameter information of the binocular camera; then calculating the Euclidean distance between each of the first pixel points and the corresponding second pixel points, and taking the average value thereof, and using this average value as the height of the convex-concave target; wherein, the Euclidean distance between a certain first pixel point and the corresponding second pixel point is calculated by the following expression:

[0090] ;

[0091] In the formula, represents the X-axis coordinate of this first pixel point in the world coordinate system, represents the X-axis coordinate of the corresponding second pixel point in the world coordinate system, represents the Y-axis coordinate of this first pixel point in the world coordinate system, represents the Y-axis coordinate of the corresponding second pixel point in the world coordinate system, represents the Z-axis coordinate of this first pixel point in the world coordinate system, represents the Z-axis coordinate of the corresponding second pixel point in the world coordinate system.

[0092] In one embodiment, optimizing the disparity map includes:

[0093] S401: Performing bilinear interpolation processing on the disparity map to obtain an enlarged disparity map;

[0094] S402: Performing dilation processing on the enlarged disparity map;

[0095] S403: Performing erosion processing on the dilated disparity map.

[0096] In one embodiment, the road surface binocular image is obtained by a binocular camera. Before obtaining the road surface binocular image, it further includes: calibrating the binocular camera to obtain the parameter information of the binocular camera; the parameter information includes: the internal parameter matrix, the external parameter matrix, the radial distortion, and the tangential distortion of the binocular camera.

[0097] In this embodiment, the Zhang-Zhengyou calibration method can be used to calibrate the binocular camera. In other embodiments, other suitable methods can also be used. Calibration can be performed when the binocular camera is installed or debugged to obtain the parameter information of the binocular camera.

[0098] The calibration process is as follows: First, fix the binocular camera at the specified installation position with the baseline of the binocular camera parallel to the ground, and hold the calibration board to capture images of the calibration board from different distances and angles. After obtaining the captured calibration board images, use the Matlab binocular camera calibration toolbox to solve the camera internal parameters, external parameters, radial distortion, and tangential distortion parameters to complete the calibration of the binocular camera. The calibration process corrects the lens distortion of the binocular camera and the epipolar lines of the binocular camera, which can greatly reduce the computational amount of corresponding point matching of the binocular camera and improve the matching accuracy. For a well-calibrated binocular camera, the output binocular images can effectively represent depth information, and this step is the basis for subsequent elevation measurement.

[0099] Embodiment of the road surface unevenness detection system:

[0100] The present invention also provides a road surface unevenness detection system, including a binocular camera for collecting road surface images, a calibration system for calibrating the binocular camera, and a measurement system connected to the binocular camera and the calibration system. The measurement system is used to execute the road surface unevenness detection method described in the above embodiments.

[0101] As Figure 18 shown, the measurement system includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the road surface unevenness detection method described in the above embodiments is implemented.

[0102] The measurement system also includes other components well-known to those skilled in the art such as a communication bus and a communication interface, and their settings and functions are known in the art, so they will not be elaborated here.

[0103] In the present invention, the aforementioned memory may be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, the computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that can be stored or otherwise held by such a computer-readable medium.

[0104] In the description of this specification, the meanings of "a plurality of" and "several" are at least two, for example, two, three, or more, etc., unless otherwise specifically defined.

[0105] Although this specification has shown and described multiple embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will think of many changes, alterations, and alternative ways without departing from the spirit and scope of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A method for detecting road surface unevenness, characterized in that, Including: Obtain a binocular image of the road surface and convert it into a disparity map, and optimize the disparity map to obtain an optimized disparity map; Input the binocular image of the road surface into a convex and concave target detection model to obtain a convex and concave target detection frame in the binocular image of the road surface. The convex and concave target detection frame is used to frame the convex and concave targets. The convex and concave target detection model includes a feature extraction module. The output of the feature extraction module is connected to the input of the feature fusion module. The output of the feature fusion module is connected to the input of the dynamic detection head module. The feature extraction module includes a DSconv (depthwise separable convolution) module and a dual-path architecture. The dual-path architecture includes a local information extraction branch and a global information extraction branch. The output of the DSconv module is respectively connected to the inputs of the local information extraction branch and the global information extraction branch. The dynamic detection head module includes a multi-layer stacked dynamic detection head for detecting convex and concave road surface targets based on the fused image generated by the feature fusion module; According to the image coordinates of the points on the convex and concave target detection frame, map the convex and concave target detection frame to the optimized disparity map; Obtain feature key points on the optimized disparity map according to the convex and concave target detection frame, and use the feature key points for elevation measurement to obtain an elevation measurement result; The global information extraction branch includes a plurality of cross-connected MBConv modules and EfficientVit modules, and an SPPF module located at the end of the global information extraction branch. The local information extraction branch includes a plurality of cross-connected residual modules, Dwconv modules and Concat modules; Wherein, several EfficientVit modules of the global information extraction branch are connected to the Concat module of the local information extraction branch.

2. The road surface unevenness detection method according to claim 1, characterized in that, The first residual module includes a first Conv module and a second Conv module. The output of the second Conv module is connected to the input of the BottleNeck module. The output of the first Conv module and the output of the BottleNeck module are connected to the input of the fifth Concat module. The output of the fifth Concat module is connected to the input of the third Conv module. The output of the third Conv module is the output of the first residual module. The inputs of the first Conv module and the second Conv module are the inputs of the first residual module.

3. The road surface unevenness detection method according to claim 1, characterized in that, The feature fusion module includes a fourth Dwconv module, a first UpSample module, a third Concat module, a fifth residual module, a fifth Dwconv module, a second UpSample module, a fourth Concat module, a sixth residual module, a sixth Dwconv module, a seventh residual module, a sixth Dwconv module and an eighth residual module connected end to end in sequence. Among them, the output of the sixth residual module is the first fused image, the output of the seventh residual module is the second fused image, and the output of the eighth residual module is the third fused image.

4. The road surface unevenness detection method according to claim 1, wherein, The shape of the convex and concave target detection frame is rectangular. Obtaining feature key points on the optimized disparity map according to the convex and concave target detection frame, and performing elevation measurement on the feature key points includes: Take the pixel points on the middle horizontal line of the convex-concave target detection frame as the first pixel points, and select the second pixel points corresponding vertically to each of the first pixel points from the lower side of the convex-concave target detection frame; the second pixel points corresponding vertically refer to the intersection points of the perpendiculars drawn from the points on the middle horizontal line to the lower side and the lower side. Obtain the world coordinate system coordinates corresponding to each of the first pixel points and the second pixel points according to the parameter information of the binocular camera; then calculate the Euclidean distance between each of the first pixel points and the corresponding second pixel points, and take the average value thereof, and use this average value as the height of the convex-concave target.

5. The road surface unevenness detection method according to claim 1, characterized in that, Optimizing the disparity map includes: Performing bilinear interpolation processing on the disparity map to obtain an enlarged disparity map; Performing dilation processing on the enlarged disparity map; Performing erosion processing on the dilated disparity map.

6. The road surface unevenness detection method according to claim 1, characterized in that, The dynamic detection head module includes 4 stacked dynamic detection heads.

7. The road surface unevenness detection method according to claim 1, wherein, The road surface binocular image is obtained by using a binocular camera. Before obtaining the road surface binocular image, it further includes: calibrating the binocular camera to obtain the parameter information of the binocular camera; the parameter information includes: the internal parameter matrix, the external parameter matrix, the radial distortion, and the tangential distortion of the binocular camera.

8. The road surface unevenness detection method according to claim 7, characterized in that, The Zhang Zhengyou calibration method is used when calibrating the binocular camera.

9. A road surface unevenness detection system, comprising a binocular camera for collecting road surface images, a calibration system for calibrating the binocular camera, and a measurement system connected to the binocular camera and the calibration system, characterized in that, The measurement system is used to execute the road surface convex-concave detection method according to any one of claims 1 to 8.

Images