Road surface convex-concave detection method and system

Abstract

The invention relates to the technical field of data processing, in particular to a road surface convex-concave detection method and system. The method comprises the following steps: acquiring a pavement binocular image, converting the pavement binocular image into a disparity map, and optimizing the disparity map to obtain an optimized disparity map; inputting the pavement binocular image into a convex-concave target detection model to obtain a convex-concave target detection frame in the pavement binocular image, the convex-concave target detection frame being used for frame selection of convex-concave targets; mapping the convex-concave target detection frame to the optimized disparity map according to the image coordinates of the points on the convex-concave target detection frame; and obtaining feature key points on the optimized disparity map according to the convex-concave target detection frame, and performing elevation measurement by using the feature key points. By adopting the convex-concave target detection method, the convex-concave target detection precision can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a method and system for detecting road surface unevenness. Background Art

[0002] At present, there have been some studies on road bump detection in complex scenes. According to the different detection methods, it can be roughly divided into mechanical device detection, inertial sensor detection, radar detection and camera acquisition detection. Among them, mechanical devices generally rely on measuring heads for measurement, which greatly limits their application scenarios and cannot be applied to some systems that require real-time response; there are problems such as large size and slow detection process. Inertial sensors are usually not accurate, and other methods must be combined to achieve higher detection accuracy. Radar detection usually uses radar equipment to generate 3D point cloud maps of the road surface for bump detection. Although it has high accuracy and fast response, it is expensive and cannot detect some bump targets with small elevations; camera acquisition detection, by extracting information from the road surface image collected by the camera, has a faster response speed and higher detection accuracy while using a lower cost; however, for the accuracy of camera acquisition detection, in addition to being affected by uncontrollable factors such as weather conditions, it depends more on the image acquisition quality of the camera and the image processing process. Although many bump target recognition models currently perform well in identifying a single type of bump target, such as speed bumps, their effects are not satisfactory in complex road scenes. Complex road surfaces increase image interference, making target detection blurry and inaccurate. For autonomous vehicles, the need for real-time monitoring in a changing environment further exacerbates the difficulty of the work. Although some image processing technologies can improve video effects and benefit subsequent convex and concave target recognition tasks, their high complexity and long inference time become bottlenecks. Integrating these technologies into the convex and concave target detection process improves the accuracy of detection, but is not conducive to achieving fast and real-time recognition and positioning. Summary of the invention

[0003] In order to solve the technical problem of poor road surface unevenness detection accuracy in the prior art, the present invention provides solutions in the following aspects.

[0004] In a first aspect, the present invention provides a method for detecting road surface unevenness, comprising: Acquire a binocular image of the road surface and convert it into a disparity map, and optimize the disparity map to obtain an optimized disparity map; The road binocular image is input into a convex-concave target detection model to obtain a convex-concave target detection frame in the road binocular image, and the convex-concave target detection frame is used to select convex-concave targets; the convex-concave target detection model includes a feature extraction module, the output of the feature extraction module is connected to the input of a feature fusion module, the output of the feature fusion module is connected to the input of a dynamic detection head module, the feature extraction module includes a DSconv module and a dual-path architecture, the dual-path architecture includes a local information extraction branch and a global information extraction branch, the output of the DSconv module is connected to the input of the local information extraction branch and the global information extraction branch respectively; the dynamic detection head module includes a multi-layer stacked dynamic detection head, which is used to detect convex-concave targets on the road surface according to the fused image generated by the feature fusion module; Mapping the convex and concave object detection frame onto the optimized disparity map according to the image coordinates of the points on the convex and concave object detection frame; The characteristic key points on the optimized disparity map are obtained according to the convex and concave target detection frame, and the elevation measurement is performed using the characteristic key points to obtain the elevation measurement result.

[0005] The beneficial effects are as follows: the present invention extracts global information and local information of the binocular image of the road surface by setting a global information extraction branch and a local information extraction branch in the feature extraction module of the convex-concave target detection model, which can eliminate the bias or incomplete cognition caused by a single perspective, so that the feature extraction module has higher performance in capturing feature diversity. By setting a dynamic detection head module, the extracted high-margin features can be effectively utilized to achieve accurate recognition and positioning of convex-concave targets on the road surface; in addition, by optimizing the disparity map, the erroneous information in the disparity map can be filtered out, thereby improving the quality of the disparity map, and then improving the accuracy of convex-concave target recognition. Therefore, the convex-concave target detection method of the present invention can improve the accuracy of convex-concave target detection, which helps autonomous driving vehicles to make more accurate judgments on road conditions in complex road scenes.

[0006] Preferably, the global information extraction branch includes a plurality of cross-connected MBConv modules and EfficientVit modules, and an SPPF module located at the end of the global information extraction branch; the local information extraction branch includes a plurality of cross-connected residual modules, Dwconv modules and Concat modules; wherein several EfficientVit modules of the global information extraction branch are connected to the Concat module of the local information extraction branch.

[0007] Its beneficial effects are as follows: by setting the EfficientVit module in the global information extraction branch, the global receptive field can be obtained through multi-scale linear attention, and multi-scale features can be learned to achieve efficient feature extraction; by setting the MBConv module in the global information extraction branch to perform dimensionality increase and then dimensionality reduction operations on the feature map output by the DSconv module, the calculation speed can be accelerated while obtaining the features of the channel dimension; by setting the SPPF module, the feature map output by the third EfficientVit module can be subjected to spatial pyramid pooling operations, feature extraction can be performed on feature maps of different scales, and features of feature maps of different scales can be obtained.

[0008] By setting a residual module in the local information extraction branch, higher-level feature information can be obtained, and the computational complexity can be reduced while maintaining a high representation capability; by setting the Dwconv module, the channel features and spatial features of the image can be extracted; by setting the first Concat module, the low-resolution feature map containing a large amount of semantic information and the high-resolution feature map containing detail information can be spliced ​​to obtain a larger feature map.

[0009] Preferably, the first residual module includes a first Conv module and a second Conv module, the output of the second Conv module is connected to the input of the BottleNeck module, the output of the first Conv module and the output of the BottleNeck module are connected to the input of the fifth Concat module, the output of the fifth Concat module is connected to the input of the third Conv module, the output of the third Conv module is the output of the first residual module, and the input of the first Conv module and the second Conv module are the input of the first residual module.

[0010] Preferably, the feature fusion module includes a fourth Dwconv module, a first UpSample module, a third Concat module, a fifth residual module, a fifth Dwconv module, a second UpSample module, a fourth Concat module, a sixth residual module, a sixth Dwconv module, a seventh residual module, a sixth Dwconv module and an eighth residual module which are connected end to end in sequence, wherein the output of the sixth residual module is a first fused image, the output of the seventh residual module is a second fused image, and the output of the eighth residual module is a third fused image.

[0011] Its beneficial effects are as follows: by setting the Dwconv module in the feature fusion module, the channel features and spatial features of feature maps of different scales can be extracted; by upsampling through Upsample, the feature map output by the fourth Dwconv module is mapped from small resolution to large resolution, which is convenient for subsequent feature extraction; by setting the fourth Concat module, the feature map output by the Upsample module containing a large amount of semantic information and the feature map output by the fourth residual module containing high resolution detail information are spliced ​​to obtain a larger feature map.

[0012] Preferably, the shape of the convex-concave target detection frame is a rectangle, and obtaining the feature key points on the optimized disparity map according to the convex-concave target detection frame, and performing elevation measurement on the feature key points includes: The pixel points on the middle horizontal line of the convex-concave object detection frame are taken as first pixel points, and second pixel points vertically corresponding to each of the first pixel points are selected from the lower side of the convex-concave object detection frame; the vertically corresponding second pixel points refer to the intersection points of the vertical line drawn from the point on the middle horizontal line to the lower side and the lower side; The world coordinate system coordinates corresponding to each first pixel point and the second pixel point are obtained according to the parameter information of the binocular camera; then the Euclidean distance between each first pixel point and the corresponding second pixel point is calculated, and the average is calculated, and the average is used as the height of the convex and concave target Preferably, optimizing the disparity map includes: Performing bilinear interpolation processing on the disparity map to obtain an enlarged disparity map; Performing dilation processing on the enlarged disparity map; The expanded disparity map is corroded.

[0013] Preferably, the dynamic detection head module comprises four layers of stacked dynamic detection heads.

[0014] Preferably, the road surface binocular image is acquired using a binocular camera, and before acquiring the road surface binocular image, the process also includes: calibrating the binocular camera to acquire parameter information of the binocular camera; the parameter information includes: an intrinsic parameter matrix, an extrinsic parameter matrix, radial distortion, and tangential distortion of the binocular camera.

[0015] Preferably, the Zhang Zhengyou calibration method is used when calibrating the binocular camera.

[0016] In a second aspect, the present invention provides a road surface bump detection system, comprising a binocular camera for acquiring road surface images, a calibration system for calibrating the binocular camera, and a measurement system connected to the binocular camera and the calibration system, wherein the measurement system is used to execute the road surface bump detection method of the present invention.

[0017] The beneficial effect of the present invention is that the convex and concave target detection method of the present invention can improve the convex and concave target detection accuracy, which helps the autonomous driving vehicle to make more accurate judgments on road conditions in complex road scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 is a flow chart schematically showing a method for detecting road surface unevenness according to an embodiment of the present invention; Figure 2 is a schematic diagram schematically showing the structure of a dynamic detection head according to an embodiment of the present invention; Figure 3 is a schematic diagram schematically showing the installation position of a binocular camera according to an embodiment of the present invention; Figure 4 Schematically illustrates the image detection error rate of group 1 according to an embodiment of the present invention; Figure 5 Schematically illustrates the image detection error rate of group 2 according to an embodiment of the present invention; Figure 6 Schematically illustrates the image detection error rate of group 3 according to an embodiment of the present invention; Figure 7 is a schematic diagram showing the image detection error rate of group 4 according to an embodiment of the present invention; Figure 8 is a schematic diagram showing the image detection error rate of group 5 according to an embodiment of the present invention; Fig. 9 is a schematic diagram showing the image detection error rate of group 6 according to an embodiment of the present invention; Fig.10 Schematically shows a road image with speed bumps after being labeled according to an embodiment of the present invention; Fig.11 is a schematic diagram showing a road surface image with a manhole cover after being labeled according to an embodiment of the present invention; Fig.12 is a schematic diagram schematically showing a speed bump recognition result of a baseline model according to an embodiment of the present invention; Fig.13 is a schematic diagram schematically showing a speed bump recognition result of a convex-concave target detection model according to an embodiment of the present invention; Fig.14is a schematic diagram schematically showing a result of manhole cover recognition of a baseline model according to an embodiment of the present invention; Fig.15 is a schematic diagram schematically showing a manhole cover recognition result of a convex-concave target detection model according to an embodiment of the present invention; Fig.16 is a schematic diagram schematically showing the structure of a feature extraction module according to an embodiment of the present invention; Fig.17 is a schematic diagram schematically showing the structure of a feature fusion module according to an embodiment of the present invention; Fig.18 FIG. 1 is a schematic diagram schematically showing the structure of a measurement system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] Embodiment of the method for detecting unevenness of road surface: like Figure 1 As shown, the road surface convexity and concavity detection method of the present invention comprises: S101, obtaining an optimized disparity map, specifically: obtaining a binocular image of the road surface and converting it into a disparity map, and optimizing the disparity map to obtain an optimized disparity map; By optimizing the disparity map, erroneous information in the disparity map can be filtered out, thereby improving the quality of the disparity map.

[0022] S102, generating a convex-concave target detection frame, specifically: inputting the road binocular image into a convex-concave target detection model, thereby obtaining a convex-concave target detection frame in the road binocular image, wherein the convex-concave target detection frame is used to select convex-concave targets; the convex-concave target detection model includes a feature extraction module, the output of the feature extraction module is connected to the input of a feature fusion module, the output of the feature fusion module is connected to the input of a dynamic detection head module, the feature extraction module includes a DSconv module and a dual-path architecture, the dual-path architecture includes a local information extraction branch and a global information extraction branch, the output of the DSconv module is connected to the input of the local information extraction branch and the global information extraction branch respectively; the dynamic detection head module includes a multi-layer stacked dynamic detection head, which is used to detect convex-concave targets on the road surface according to the fused image generated by the feature fusion module; In this embodiment, the dynamic detection head module includes 4 layers of stacked dynamic detection heads. In other embodiments, the number of stacked dynamic detection heads may also be other appropriate values.

[0023] The detection performance of the convex-concave detection model is further improved by repeatedly stacking the dynamic detection heads. This multi-enhancement structure in terms of hierarchy, space, and tasks enables the model to more comprehensively perceive the features in complex scenes, thus significantly improving the accuracy and robustness of the model detection.

[0024] The structure of the dynamic detection head (Dynamic Head) is as follows Figure 2 As shown, the module includes scale-aware attention π L , spatial perception attention π S , and task-aware attention π C .like Figure 3 As shown, the binocular camera 1 can be installed on a car. When the vehicle is driving, the binocular camera continuously collects road images within a certain distance in front of the vehicle and outputs left and right binocular images to provide a data source for subsequent image processing. The shooting interval of the camera should be determined according to the speed of the vehicle, the road range that the camera can capture, etc. Ideally, the camera should be able to collect all road images during the vehicle's driving process to achieve a comprehensive detection effect; 2 in the figure is a convex or concave target.

[0025] By setting up a feature extraction module to extract global and local information from the road binocular image, the bias or incomplete cognition caused by a single perspective can be eliminated, making the feature extraction module have higher performance in capturing feature diversity. By setting up a dynamic detection head module, the extracted high-margin features can be effectively utilized to achieve accurate recognition and positioning of convex and concave targets on the road surface.

[0026] S103, mapping the convex and concave object detection frame to the optimized disparity map according to the image coordinates of the points on the convex and concave object detection frame; S104, acquiring feature key points on the optimized disparity map according to the convex and concave target detection frame, and performing elevation measurement using the feature key points to obtain an elevation measurement result.

[0027] In order to verify the effect of the road surface convexity and concavity detection method of this embodiment, the detection result of the road surface convexity and concavity detection method of this embodiment is tested, and the test method is: Six groups of speed bumps are selected from the collected road scene data and arranged in order from far to near distance from the vehicle. Each group of data contains 10 images. For the images in each group of data, the steps of the road surface unevenness detection method of this embodiment are executed to obtain the test results.

[0028] The 6 groups of speed bumps are group 1, group 2, group 3, group 4, group 5 and group 6 in the order of distance from the binocular camera. The detection error rate of the image of group 1 is as follows: Figure 4 As shown, the detection error rate of group 2 images is Figure 5 As shown, the detection error rate of group 3 images is Figure 6 As shown, the detection error rate of group 4 images is Figure 7 As shown, the detection error rate of group 5 images is Figure 8 As shown in Figure 2, the detection error rate of group 6 images is as follows: Fig. 9 As shown in the figure. According to the test results, when the distance between the vehicle and the speed bump is far, the speed bump target displacement information in the disparity map is relatively small, resulting in a relatively large elevation estimation error. As the speed bump gradually approaches the vehicle, the displacement information in the disparity map becomes more obvious and accurate, and the speed bump height prediction error rate decreases. In this experiment, the height of the actual speed bump is 4.5 cm. Through the analysis and calculation of 60 images, the average error rate is 14.20%. Under different distance conditions, due to the change of target information in the disparity map, the accuracy and stability of height estimation will change, and the speed bump elevation estimation error shows a downward trend from far to near.

[0029] In order to verify the performance of the convex-concave target detection model of this embodiment, a road surface image with a manhole cover and a road surface image with a speed bump are selected from the road surface convex-concave dataset, the road surface image of the manhole cover and the road surface image of the speed bump are labeled, and then the baseline model and the convex-concave target detection model of this embodiment are used to detect the convex-concave targets in these two road surface images respectively.

[0030] The road image with speed bumps after labeling is as follows Fig.10 As shown in the figure, the road image with manhole covers after labeling is as follows Fig.11 As shown in Figure 2, the results of using the baseline model to detect convex and concave targets on road images with speed bumps are shown in Figure 2. Fig.12 As shown in FIG. 1 , the result of using the convex-concave target detection model of this embodiment to detect convex-concave targets on a road surface image with speed bumps is shown in FIG. Fig.13 As shown in Figure 2, the results of using the baseline model to detect convex and concave objects on road images with manhole covers are shown in Figure 2. Fig.14 As shown in FIG. 1 , the result of using the convex-concave target detection model of this embodiment to detect convex-concave targets on a road surface image with a manhole cover is as follows: Fig.15As shown. By comparing the detection effect diagram, it can be clearly observed that when facing smaller targets, such as manhole covers, PB-Det can detect targets more accurately than the baseline model. At the same time, for targets such as speed bumps whose length is significantly greater than their width, PB-Det can also detect targets more accurately, reducing the occurrence of false detections and missed detections. By intuitively comparing the detection effect diagrams, the significant advantages of the model PB-Det in detecting bumpy targets on the road surface are further confirmed.

[0031] This embodiment extracts global information and local information of the binocular image of the road surface by setting a global information extraction branch and a local information extraction branch in the feature extraction module of the convex-concave target detection model, which can eliminate the bias or incomplete cognition caused by a single perspective, so that the feature extraction module has higher performance in capturing feature diversity. By setting a dynamic detection head module, the extracted high-margin features can be effectively utilized to achieve accurate recognition and positioning of convex-concave targets on the road surface; in addition, by optimizing the disparity map, the erroneous information in the disparity map can be filtered out, thereby improving the quality of the disparity map, and then improving the accuracy of convex-concave target recognition. Therefore, the convex-concave target detection method of the present invention can improve the accuracy of convex-concave target detection, which helps autonomous driving vehicles to make more accurate judgments on road conditions in complex road scenes.

[0032] In one embodiment, the training process of the convex-concave object detection model is: S201, using a double-sided camera to collect road binocular images on campus roads and social roads in the automobile industrial park; and obtaining images of corresponding types of convex and concave targets on the road surface through the network; Since the roads in the campus and experimental park are generally flat, it is very likely that the two types of data, potholes and potholes, will not be encountered during the actual collection process. Therefore, it is necessary to obtain images of the corresponding types of road surface convex and concave targets through the network to make up for this lack. In addition, the corresponding types of road surface convex and concave target data are obtained to ensure the comprehensiveness of the algorithm training.

[0033] S202, screening and cleaning the collected road surface binocular images and the acquired road surface convex and concave target images to obtain a road surface convex and concave data set; and dividing the road surface convex and concave data set into a training set, a verification set, and a test set; The quality of the acquired road surface binocular images and images of convex and concave targets on the road surface can be improved through data screening and cleaning.

[0034] S203, training the convex and concave object detection model using the training set, the validation set and the test set to obtain the best training weights.

[0035] The convex and concave object detection model can be trained through the loss function.

[0036] In one embodiment, the global information extraction branch includes a plurality of cross-connected MBConv modules and EfficientVit modules, and an SPPF module located at the end of the global information extraction branch; the local information extraction branch includes a plurality of cross-connected residual modules, Dwconv modules and Concat modules; wherein several EfficientVit modules of the global information extraction branch are connected to the Concat module of the local information extraction branch.

[0037] like Fig.16 As shown, the global information extraction branch includes a first MBConv module, the output of the first MBConv module is connected to the input of the second MBConv module, the output of the second MBConv module is connected to the input of the first EfficientVit module, the output of the first EfficientVit module is respectively connected to the input of the third MBConv module and the input of the first Concat module, the output of the third MBConv module is connected to the input of the second EfficientVit module, the output of the second EfficientVit module is respectively connected to the input of the fourth MBConv module and the input of the second Concat module, the output of the fourth MBConv module is connected to the input of the third EfficientVit module, and the output of the third EfficientVit module is connected to the input of the SPPF module.

[0038] The local information extraction branch includes a first residual module, the output of the first residual module is connected to the input of the first Dwconv module, the output of the first Dwconv module is connected to the input of the second residual module, the output of the second residual module is connected to the input of the first Concat module, the output of the first Concat module is connected to the input of the second Dwconv module, the output of the second Dwconv module is connected to the input of the third residual module, the output of the third residual module is connected to the input of the second Concat module, the output of the second Concat module is connected to the input of the third Dwconv module, and the output of the third Dwconv module is connected to the input of the fourth residual module.

[0039] The output of the SPPF module is the first feature map to be fused, the output of the fourth residual module is the second feature map to be fused, and the output of the second Dwconv module is the third feature map to be fused. The output of the DSconv module is connected to the input of the first MBConv module and the input of the first residual module respectively.

[0040] As the model depth increases, the feature width gradually increases, and the feature extraction module finally outputs a feature size of 64×40×40. DSconv module refers to depthwise separable convolution, which is an algorithm obtained by improving the standard convolution calculation in convolutional neural networks. It reduces the number of parameters required for convolution calculation by splitting the correlation between spatial dimensions and channel (depth) dimensions, and has been proven in some studies to improve the efficiency of convolution kernel parameters. MBConv module refers to Mobile Bottleneck Convolution module.

[0041] In the global information extraction branch, the first MBConv module and the second MBConv module are used to perform dimension-up and dimension-down operations on the feature map output by the DSConv module, thereby reducing the parameters used in the extraction process while extracting features. The first EfficientVit module is used to obtain the global receptive field of the feature map output by the second MBConv module through multi-scale linear attention, learn multi-scale features, achieve efficient feature extraction, and output a feature map containing a large amount of semantic information. The third MBConv module is used to perform dimension-up and dimension-down operations on the feature map output by the first EfficientVit module, thereby reducing the parameters used in the extraction process while extracting features. The second EfficientVit module is used to obtain the global receptive field of the feature map output by the third MBConv module, learn multi-scale features, and achieve efficient feature extraction; the fourth MBConv module is used to perform dimension-up and dimension-down operations on the feature map output by the second EfficientVit module, thereby reducing the parameters used in the extraction process while extracting features. The third EfficientVit module is used to obtain the global receptive field of the feature map output by the fourth MBConv module, and learn multi-scale features to achieve efficient feature extraction; the SPPF module is used to perform spatial pyramid pooling operations on the feature map output by the third EfficientVit module, perform feature extraction on feature maps of different scales, and obtain features of feature maps of different scales.

[0042] By setting the EfficientVit module in the global information extraction branch, the global receptive field can be obtained through multi-scale linear attention, and multi-scale features can be learned to achieve efficient feature extraction. By setting the MBConv module in the global information extraction branch to first increase the dimension and then reduce the dimension of the feature map output by the DSconv module, the calculation speed can be accelerated while obtaining the features of the channel dimension; by setting the SPPF module, the feature map output by the third EfficientVit module can be subjected to spatial pyramid pooling, feature extraction can be performed on feature maps of different scales, and features of feature maps of different scales can be obtained.

[0043] In the local information extraction branch, the first residual module is used to obtain higher-level feature information using the feature map output by the DSConv module, thereby reducing the computational complexity while maintaining a high representation capability; the first Dwconv module is used to perform a depth-separable convolution operation on the feature map output by the first residual module to extract channel features and spatial features; the second residual module is used to obtain higher-level feature information based on the feature map output by the first Dwconv module, thereby reducing the computational complexity while maintaining a high representation capability; the first Concat module is used to splice the high-resolution feature map containing detail information output by the second residual module and the feature map containing a large amount of semantic information output by the first EfficientVit module, thereby obtaining a larger feature map; the second Dwconv module is used to perform a depth-separable convolution operation on the spliced ​​feature map The operation of separating convolution is used to obtain features containing details and semantic information; the third residual module is used to obtain higher-level feature information based on the feature map output by the second Dwconv module, thereby reducing the computational complexity while maintaining a high representation capability; the second Concat module is used to splice the high-resolution feature map containing detail information output by the second Dwconv module and the feature map containing a large amount of semantic information output by the second EfficientVit module to obtain a larger feature map; the third Dwconv module is used to perform a deep separable convolution operation on the feature map output by the second Concat module to obtain features containing details and semantic information; the fourth residual module is used to obtain higher-level feature information based on the feature map output by the third Dwconv module, thereby reducing the computational complexity while maintaining a high representation capability.

[0044] By setting a residual module in the local information extraction branch, higher-level feature information can be obtained, and the computational complexity can be reduced while maintaining a high representation capability; by setting the Dwconv module, the channel features and spatial features of the image can be extracted; by setting the first Concat module, the low-resolution feature map containing a large amount of semantic information and the high-resolution feature map containing detail information can be spliced ​​to obtain a larger feature map.

[0045] In one embodiment, the first residual module includes a first Conv module and a second Conv module, the output of the second Conv module is connected to the input of the BottleNeck module, the output of the first Conv module and the output of the BottleNeck module are connected to the input of the fifth Concat module, the output of the fifth Concat module is connected to the input of the third Conv module, the output of the third Conv module is the output of the first residual module, and the input of the first Conv module and the second Conv module are the input of the first residual module.

[0046] In one embodiment, Fig.17 As shown, the feature fusion module includes a fourth Dwconv module, a first UpSample module, a third Concat module, a fifth residual module, a fifth Dwconv module, a second UpSample module, a fourth Concat module, a sixth residual module, a sixth Dwconv module, a seventh residual module, a sixth Dwconv module and an eighth residual module which are connected end to end in sequence, wherein the input of the third Concat module is also connected to the output of the fourth residual module, the input of the fourth Concat module is also connected to the output of the second Dwconv module, the output of the sixth residual module is the first fused image, the output of the seventh residual module is the second fused image, and the output of the eighth residual module is the third fused image.

[0047] The working process of the feature fusion module of this embodiment is as follows: the fourth Dwconv module performs a deep separable convolution (Dwconv) operation on the first feature map to be fused output by the SPPF module to extract the channel features and spatial features of feature maps of different degrees of multiplicity; the first UpSample module performs a mapping operation from a small resolution to a large resolution on the feature map output by the fourth Dwconv module to facilitate subsequent feature extraction; the third Concat module performs a splicing operation on the feature map output by the first UpSample module and the second feature map to be fused output by the fourth residual module to obtain a larger feature map; the fifth residual module obtains a larger feature map based on the feature map output by the third Concat module, and the fifth Dwconv module performs a deep separable convolution (Dwconv) operation on the feature map output by the fifth residual module to extract the channel features and spatial features of feature maps of different degrees of multiplicity; the second UpSample module upsamples the feature map output by the fifth Dwconv module, and performs a mapping operation from a small resolution to a large resolution on the input feature map to facilitate subsequent feature extraction; the fourth Concat module performs a splicing operation on the second UpSample module and the second Dwconv module. The feature map output by the mple module is concatenated with the third feature map to be fused output by the second Dwconv module to obtain a larger feature map; the sixth residual module obtains higher-level feature information based on the feature map output by the fourth Concat module, reduces the computational complexity while maintaining a high representation capability, and outputs the features of this layer to the dynamic detection head; the sixth Dwconv module performs a deep separable convolution (Dwconv) operation on the feature map output by the sixth residual module to extract channel features and spatial features of feature maps of different degrees of multiplicity; the seventh residual module obtains higher-level feature information based on the feature map output by the sixth Dwconv module, reduces the computational complexity while maintaining a high representation capability, and outputs the features of this layer to the detection head; the seventh Dwconv module performs a deep separable convolution (Dwconv) operation on the feature map output by the seventh residual module to extract channel features and spatial features of feature maps of different degrees of multiplicity; the eighth residual module obtains higher-level feature information based on the feature map output by the seventh Dwconv module, reduces the computational complexity while maintaining a high representation capability, and outputs the feature map output by the seventh Dwconv module to the detection head.

[0048] By setting the Dwconv module in the feature fusion module, the channel features and spatial features of feature maps of different scales can be extracted; by upsampling through Upsample, the feature map output by the fourth Dwconv module is mapped from small resolution to large resolution to facilitate subsequent feature extraction; by setting the fourth Concat module, the feature map output by the Upsample module containing a large amount of semantic information and the high-resolution feature map output by the fourth residual module containing detail information are spliced ​​to obtain a larger feature map.

[0049] In one embodiment, the shape of the convex-concave target detection frame is a rectangle, and obtaining feature key points on the optimized disparity map according to the convex-concave target detection frame and performing elevation measurement on the feature key points includes: S301, taking pixel points on the middle horizontal line of the convex-concave object detection frame as first pixel points, and selecting second pixel points vertically corresponding to each of the first pixel points from the lower side of the convex-concave object detection frame; the vertically corresponding second pixel points refer to intersection points of a perpendicular line drawn from a point on the middle horizontal line to the lower side and the lower side; S302, obtaining the world coordinate system coordinates corresponding to each first pixel point and the second pixel point according to the parameter information of the binocular camera; then calculating the Euclidean distance between each first pixel point and the corresponding second pixel point, and averaging them, and using the average as the height of the convex-concave target; wherein the Euclidean distance between a first pixel point and the corresponding second pixel point The calculation expression is: ; In the formula, Indicates the X-axis coordinate of the first pixel in the world coordinate system, Indicates the X-axis coordinate of the corresponding second pixel point in the world coordinate system, Indicates the Y-axis coordinate of the first pixel in the world coordinate system, Indicates the Y-axis coordinate of the corresponding second pixel in the world coordinate system, Indicates the Z-axis coordinate of the first pixel in the world coordinate system, Indicates the Z-axis coordinate of the corresponding second pixel point in the world coordinate system.

[0050] In one embodiment, optimizing the disparity map includes: S401, performing bilinear interpolation processing on the disparity map to obtain an enlarged disparity map; S402, dilating the enlarged disparity map; S403: performing corrosion processing on the expanded disparity map.

[0051] In one embodiment, the road surface binocular image is acquired using a binocular camera. Before acquiring the road surface binocular image, the process also includes: calibrating the binocular camera to acquire parameter information of the binocular camera; the parameter information includes: an intrinsic parameter matrix, an extrinsic parameter matrix, radial distortion, and tangential distortion of the binocular camera.

[0052] In this embodiment, the binocular camera can be calibrated by Zhang Zhengyou calibration method, and in other embodiments, other suitable methods can also be used. The binocular camera can be calibrated when it is installed or debugged to obtain parameter information of the binocular camera.

[0053] The calibration process is as follows: first fix the binocular camera at the specified installation position, and the baseline of the binocular camera is parallel to the ground, and hold the calibration plate to capture images of the calibration plate from different distances and angles. After obtaining the captured calibration plate image, use the Matlab binocular camera calibration toolbox to solve the camera intrinsic parameters, extrinsic parameters, radial distortion and tangential distortion parameters to complete the calibration of the binocular camera. The calibration process corrects the lens distortion of the binocular camera and the epipolar lines of the binocular camera, which can greatly reduce the calculation amount of the binocular camera corresponding point matching and improve the matching accuracy. The binocular image output by a well-calibrated binocular camera can effectively represent the depth information. This step is the basis for subsequent elevation measurement.

[0054] Road surface bump detection system implementation example: The present invention also provides a road surface unevenness detection system, comprising a binocular camera for acquiring road surface images, a calibration system for calibrating the binocular camera, and a measuring system connected to the binocular camera and the calibration system, wherein the measuring system is used to execute the road surface unevenness detection method described in the above embodiments.

[0055] like Fig.18 As shown, the measurement system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the road surface unevenness detection method described in the above embodiment is implemented.

[0056] The measurement system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface, whose configuration and functions are known in the art and thus will not be described in detail here.

[0057] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0058] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0059] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A method for detecting road surface unevenness, characterized in that: include: Acquire a binocular image of the road surface and convert it into a disparity map, and optimize the disparity map to obtain an optimized disparity map; The road binocular image is input into a convex-concave target detection model to obtain a convex-concave target detection frame in the road binocular image, and the convex-concave target detection frame is used to select convex-concave targets; the convex-concave target detection model includes a feature extraction module, the output of the feature extraction module is connected to the input of a feature fusion module, the output of the feature fusion module is connected to the input of a dynamic detection head module, the feature extraction module includes a DSconv module and a dual-path architecture, the dual-path architecture includes a local information extraction branch and a global information extraction branch, the output of the DSconv module is connected to the input of the local information extraction branch and the global information extraction branch respectively; the dynamic detection head module includes a multi-layer stacked dynamic detection head, which is used to detect convex-concave targets on the road surface according to the fused image generated by the feature fusion module; Mapping the convex and concave object detection frame onto the optimized disparity map according to the image coordinates of the points on the convex and concave object detection frame; The characteristic key points on the optimized disparity map are obtained according to the convex and concave target detection frame, and the elevation measurement is performed using the characteristic key points to obtain the elevation measurement result.

2. The method for detecting road surface unevenness according to claim 1, characterized in that: The global information extraction branch includes a plurality of cross-connected MBConv modules and EfficientVit modules, and an SPPF module located at the end of the global information extraction branch; the local information extraction branch includes a plurality of cross-connected residual modules, Dwconv modules and Concat modules; Among them, several EfficientVit modules of the global information extraction branch are connected to the Concat module of the local information extraction branch.

3. The method for detecting road surface unevenness as claimed in claim 2, characterized in that: The first residual module includes a first Conv module and a second Conv module, the output of the second Conv module is connected to the input of the BottleNeck module, the output of the first Conv module and the output of the BottleNeck module are connected to the input of the fifth Concat module, the output of the fifth Concat module is connected to the input of the third Conv module, the output of the third Conv module is the output of the first residual module, and the input of the first Conv module and the second Conv module is the input of the first residual module.

4. The method for detecting road surface unevenness as claimed in claim 2, characterized in that: The feature fusion module includes a fourth Dwconv module, a first UpSample module, a third Concat module, a fifth residual module, a fifth Dwconv module, a second UpSample module, a fourth Concat module, a sixth residual module, a sixth Dwconv module, a seventh residual module, a sixth Dwconv module and an eighth residual module, which are connected end to end in sequence, wherein the output of the sixth residual module is a first fused image, the output of the seventh residual module is a second fused image, and the output of the eighth residual module is a third fused image.

5. The method for detecting road surface unevenness according to claim 1, characterized in that: The shape of the convex-concave target detection frame is a rectangle, and the feature key points on the optimized disparity map are obtained according to the convex-concave target detection frame, and the elevation measurement of the feature key points includes: The pixel points on the middle horizontal line of the convex-concave object detection frame are taken as first pixel points, and second pixel points vertically corresponding to each of the first pixel points are selected from the lower side of the convex-concave object detection frame; the vertically corresponding second pixel points refer to the intersection points of the vertical line drawn from the point on the middle horizontal line to the lower side and the lower side; The world coordinate system coordinates corresponding to each first pixel point and the second pixel point are obtained according to the parameter information of the binocular camera; then the Euclidean distance between each first pixel point and the corresponding second pixel point is calculated, and the average is calculated, and the average is used as the height of the convex-concave target.

6. The method for detecting road surface unevenness according to claim 1, characterized in that: Optimizing the disparity map includes: Performing bilinear interpolation processing on the disparity map to obtain an enlarged disparity map; Performing dilation processing on the enlarged disparity map; The expanded disparity map is corroded.

7. The method for detecting road surface unevenness according to claim 1, characterized in that: The dynamic detection head module includes four layers of stacked dynamic detection heads.

8. The method for detecting road surface unevenness according to any one of claims 1 to 7, characterized in that: The road binocular image is obtained by using a binocular camera. Before obtaining the road binocular image, the process also includes: calibrating the binocular camera to obtain parameter information of the binocular camera; the parameter information includes: an intrinsic parameter matrix, an extrinsic parameter matrix, radial distortion and tangential distortion of the binocular camera.

9. The method for detecting road surface unevenness as claimed in claim 8, characterized in that: The Zhang Zhengyou calibration method is used to calibrate the binocular camera.

10. A road surface bump detection system, comprising a binocular camera for collecting road surface images, a calibration system for calibrating the binocular camera, and a measurement system connected to the binocular camera and the calibration system, characterized in that: The measuring system is used to execute the road surface unevenness detection method described in any one of claims 1 to 9.

Images