Vehicle control method, road concave-convex feature detection method, system, device and readable storage medium based on binocular stereo vision

Through the binocular stereoscopic visual detection method, combined with deep learning and point cloud processing, the problem of insufficient time and accuracy of road unevenness detection is solved, and the precise road condition information is provided for intelligent driving is achieved, which improves driving comfort.

CN117808868BActive Publication Date: 2024-11-05SHANGHAI BAOLONG AUTOMOTIVE CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311849996.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-28
Publication Date
2024-11-05
Estimated Expiration
2043-12-28

AI Technical Summary

Technical Problem

The existing road surface unevenness detection methods take a long time to calculate when processing point cloud data, and the pothole feature detection is not comprehensive enough, and the image detection is insufficient for depth and positioning accuracy, making it difficult to provide accurate road conditions information for intelligent driving.

Method used

Using a road concave and convex feature detection method based on binocular stereo vision, images are acquired through binocular cameras, 2D prediction frames are detected using deep learning algorithms, parallax point clouds are generated, concave and convex information are projected and calculated, and combined with image recognition and point cloud processing, the calculation is reduced. Resource consumption and improve detection accuracy.

Benefits of technology

Effectively identify road convex information, improve the comfort and computing efficiency of intelligent driving, and provide more accurate road conditions information to the autonomous driving controller.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117808868B_ABST
    Figure CN117808868B_ABST
Patent Text Reader

Abstract

The present invention relates to a vehicle control method, a road concave-convex feature detection method based on binocular stereo vision, a system, a device and a readable storage medium. The road concave-convex feature detection method based on binocular stereo vision comprises the following steps: S1, road surface concave-convex feature detection; S2, generating binocular parallax point cloud; S3, feature area point cloud projection; S4, area feature calculation, calculating the concave-convex information of the corresponding area point cloud. A vehicle control method comprises the following steps: T1, predicting the vehicle trajectory according to the vehicle motion state; T2, calculating the correlation with the corresponding area point cloud based on the predicted vehicle trajectory; T3, sending the concave-convexity, height position and acquired correlation of the corresponding area point cloud as the summary information of each road concave-convex feature to an ADAS controller. The present invention can effectively identify road concave-convex information, reduce computing resource consumption, and thus improve the comfort of intelligent driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a vehicle control method, a road concave-convex feature detection method based on binocular stereo vision, a system, a device and a readable storage medium. Background Art

[0002] Among the existing methods for detecting road roughness, there are many neural network detection methods based on point cloud data or image data, which can achieve relatively good detection results. Among them, the roughness detection based on point cloud data usually rasterizes the point cloud. Because the amount of point cloud data is huge and the processing is time-consuming, rasterization is performed to speed up the calculation speed and improve performance, but there is also a disadvantage that features such as potholes will be rasterized to different areas, making it impossible to extract more comprehensive and complete pothole road features, and even the detection of the feature may be lost. Image detection of road conditions can accurately identify different road features in the image, but the depth and positioning accuracy of pothole information will be reduced. Summary of the invention

[0003] In response to the above-mentioned problems in the prior art, the present invention proposes a vehicle control method, a road bump feature detection method based on binocular stereo vision, a system, a device and a readable storage medium, which can effectively identify road bump information, reduce computing resource consumption, and thereby improve the comfort of intelligent driving.

[0004] Specifically, the present invention proposes a method for detecting road concave-convex features based on binocular stereo vision, comprising the steps of:

[0005] S1, road surface uneven feature detection, obtaining a binocular image of the vehicle's driving direction through a binocular camera, wherein the binocular image includes a left image and a right image obtained by a left camera and a right camera, and using the left image or the right image as the main image, detecting the road surface uneven feature by using a deep learning algorithm to obtain a 2D prediction frame;

[0006] S2, generating a binocular disparity point cloud, calculating a disparity map D between a left-eye image and a right-eye image of the same scene based on a deep learning binocular stereo matching algorithm SGBM, and obtaining a binocular disparity point cloud in a binocular camera coordinate system based on the disparity map D;

[0007] S3, feature area point cloud projection, projecting the 2D prediction box to the binocular disparity point cloud to obtain the corresponding area point cloud;

[0008] S4, regional feature calculation, calculates the concave and convex information of the corresponding regional point cloud.

[0009] According to one embodiment of the present invention, in step S1, the road surface unevenness feature includes a road pothole or a speed bump, and the 2D prediction frame corresponds to a road pothole area or a speed bump area;

[0010] 2D prediction box ((Xi, Yj), (Xm, Yn)), where (Xi, Yj) represents the coordinates of the upper left corner of the 2D prediction box, and (Xm, Yn) represents the coordinates of the lower right corner of the 2D prediction box.

[0011] According to one embodiment of the present invention, the disparity refers to the difference in pixel position of a spatial point P between the left-eye image and the right-eye image. The disparity d of the spatial point P is d=x l -x r , where x l is the horizontal coordinate of the left camera, x r is the horizontal coordinate of the right camera; step S2 comprises the steps:

[0012] S21, setting a disparity consistency test, calculating disparity values ​​for pixel points of the left eye image and the right eye image that pass the consistency test, and obtaining a disparity map;

[0013] S22, extracting connected areas from the disparity map, and performing linear interpolation on the disparity between adjacent pixels to obtain a dense disparity map D;

[0014] S23, obtaining depth information of the spatial point P;

[0015] S24, based on the depth information of the spatial point P, obtain the position coordinates of the spatial point P in the camera coordinate system.

[0016] According to one embodiment of the present invention, in step S23, the depth Z of the spatial point P is calculated as follows:

[0017]

[0018] Wherein, T is the length of the optical center line of the binocular camera, and f is the focal length of the camera.

[0019] According to one embodiment of the present invention, in step S24, the calculation formulas for the X coordinate and the Y coordinate of the space point P in the camera coordinate system are:

[0020]

[0021]

[0022] Among them, (X image ,Y image ) is the image coordinate of the spatial point p in the disparity map, (u x ,v y ) is the pixel coordinate of the spatial point p in the disparity map, (u0 ,v 0 ) are the pixel coordinates of the camera principal point in the master image.

[0023] According to an embodiment of the present invention, in step S3, the 2D prediction frame is projected onto the binocular disparity point cloud, and the point cloud coordinates corresponding to the road concave-convex features are obtained as follows:

[0024]

[0025] According to one embodiment of the present invention, step S4 comprises the steps of:

[0026] S41, calculating the concavity and convexity of the point cloud of the corresponding area;

[0027] Based on the four corner points of the 2D prediction box, a spatial plane S is constructed, and a spatial point P1 is selected. As the observation point, the minimum spanning tree algorithm in the open3d library is used to estimate the normal vector of the spatial point P1, which is recorded as v1;

[0028] Use the spatial point P1 to project the plane S. The spatial point P1 and the projection point of the plane S form another vector V2.

[0029] Calculate the angle between v1 and v2, and determine whether the point cloud in the corresponding area is concave or convex based on the angle;

[0030] S42, calculating the height position of the point cloud of the corresponding area;

[0031] Taking the spatial point P1 as the starting point, the distance from the spatial point p1 to the plane S is calculated, and the gradient ascent method is used to find the maximum distance from a point in the concave-convex feature area to the plane S and the position coordinates of the point.

[0032] The present invention also provides a vehicle control method, which is applicable to the above-mentioned road concave-convex feature detection method based on binocular stereo vision, and comprises the steps of:

[0033] T1, predict the trajectory of the vehicle according to the vehicle's motion state;

[0034] T2, based on the predicted trajectory of the vehicle, calculate the correlation with the point cloud of the corresponding area;

[0035] T3, the concavity and convexity, height position, and obtained correlation of the corresponding area point cloud are sent to the ADAS controller as summary information of each road concavity and convexity feature.

[0036] The present invention also provides a road concave-convex feature detection system based on binocular stereo vision, which is applicable to the above-mentioned road concave-convex feature detection method based on binocular stereo vision. The road concave-convex feature detection system comprises:

[0037] A feature detection unit is used to obtain a binocular image of the vehicle's driving direction through a binocular camera, wherein the binocular image includes a left image and a right image obtained by a left camera and a right camera, and the left image or the right image is used as a main image to detect the concave and convex features of the road surface by using a deep learning algorithm to obtain a 2D prediction frame;

[0038] A point cloud generation unit is used to calculate a disparity map D between a left-eye image and a right-eye image of the same scene based on a deep learning binocular stereo matching algorithm SGBM, and obtain a binocular disparity point cloud in a binocular camera coordinate system based on the disparity map D;

[0039] A projection unit, used for projecting the 2D prediction frame onto the binocular disparity point cloud to obtain a corresponding area point cloud;

[0040] The calculation unit calculates the concave and convex information of the point cloud of the corresponding area.

[0041] The present invention also provides a road bump feature detection device based on binocular stereo vision, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the aforementioned road bump feature detection methods based on binocular stereo vision are implemented.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the aforementioned methods for detecting road bump features based on binocular stereo vision are implemented.

[0043] The present invention provides a vehicle control method, a road bump feature detection method based on binocular stereo vision, a system, a device and a readable storage medium. Based on deep learning, a single camera in a binocular camera is used to perform preliminary detection and image positioning of bump feature areas in the road surface, and a dense point cloud in the camera coordinate system is generated by combining binocular stereo vision. The bump features of the road detected by the image are projected and mapped to the point cloud, so as to calculate and analyze the point cloud in the projection area to obtain bump information. This technical solution combines the advantages of image recognition and point cloud processing, reduces the consumption of computing resources, and provides more accurate and richer road condition information for the automatic driving controller module, thereby improving the comfort of intelligent driving under some special road conditions.

[0044] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are included to provide further explanation of the present invention, and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principle of the present invention.

[0046] In the attached figure:

[0047] Figure 1 A flowchart of a method for detecting road concave-convex features based on binocular stereo vision according to an embodiment of the present invention is shown.

[0048] Figure 2 A schematic diagram showing the pixel position relationship of a spatial point P in left and right cameras according to an embodiment of the present invention.

[0049] Figure 3 A schematic diagram of a main image coordinate system of a binocular camera according to an embodiment of the present invention is shown.

[0050] Figure 4 A flow chart of a vehicle control method according to an embodiment of the present invention is shown.

[0051] Figure 5 A schematic structural diagram of a road concave-convex feature detection system based on binocular stereo vision according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0056] In the description of the present application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present application; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0057] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0058] Figure 1 A flowchart of a method for detecting road concave-convex features based on binocular stereo vision according to an embodiment of the present invention is shown. As shown in the figure, a method for detecting road concave-convex features based on binocular stereo vision comprises the following steps:

[0059] S1, road surface bump feature detection, obtain a binocular image of the vehicle's driving direction through a binocular camera, the binocular image includes a left image and a right image obtained by a left camera and a right camera, take the left image or the right image as the main image, use a deep learning algorithm to detect the road surface bump features, and obtain a 2D prediction frame. Usually, the left image can be used as the main image, and the deep learning algorithm can be used to detect the bump features of the left image. It should be noted that the binocular camera is composed of two horizontally placed left and right cameras, so as to obtain a pair of two-dimensional images of the same scene.

[0060] S2, generate binocular disparity point cloud, calculate the disparity map D of the left and right images of the same scene based on the deep learning binocular stereo matching algorithm SGBM, and obtain the binocular disparity point cloud in the binocular camera coordinate system based on the disparity map D;

[0061] S3, feature area point cloud projection, projecting the 2D prediction box to the binocular disparity point cloud to obtain the corresponding area point cloud;

[0062] S4, regional feature calculation, calculates the concave and convex information of the corresponding regional point cloud.

[0063] Preferably, in step S1, the road surface unevenness features include potholes or speed bumps, and the 2D prediction box corresponds to the road surface pothole area or speed bump area. 2D prediction box ((Xi, Yj), (Xm, Yn)), where (Xi, Yj) represents the upper left corner coordinates of the 2D prediction box, and (Xm, Yn) represents the lower right corner coordinates of the 2D prediction box.

[0064] Preferably, disparity refers to the difference in pixel position of a spatial point P between the left-eye image and the right-eye image. Figure 2 The schematic diagram shows the pixel position relationship of the spatial point P in the left and right cameras according to an embodiment of the present invention. As shown in the figure, the parallax d of the spatial point P is d=x l -x r , where x l is the horizontal coordinate of the left camera, x r is the horizontal coordinate of the right camera, f is the focal length of the camera, and T is the binocular camera baseline (the length of the line connecting the optical centers of the left camera and the right camera). Step S2 includes the following steps:

[0065] S21, setting disparity consistency detection, calculating disparity values ​​for the pixels of the left and right images that pass the consistency detection, and obtaining a disparity map. For example, if |x l -x r |>m, assuming m is 1, if the absolute value of the disparity d is greater than 1, it fails the consistency check.

[0066] S22, in order to remove noise points, the disparity map is subjected to connected area extraction, that is, if the disparity value of two adjacent pixels is less than a preset threshold, then the two pixels can be considered to belong to the same area, and the disparity between adjacent pixels is linearly interpolated to obtain a dense disparity map D;

[0067] S23, obtaining the depth information of the spatial point P, that is, calculating the depth Z;

[0068] S24, based on the depth information of the spatial point P, obtain the position coordinates of the spatial point P in the camera coordinate system.

[0069] Preferably, in step S23, the depth Z of the spatial point P is calculated by:

[0070]

[0071] Wherein, T is the length of the optical center line of the binocular camera, and f is the focal length of the camera.

[0072] refer to Figure 1 According to the principle of similar triangles, the following relationship holds true:

[0073] Where L is the horizontal size of the image. The relationship can be further simplified to obtain the above calculation formula for the spatial point P: It is easy to understand that for the same binocular camera, the closer the object is, the smaller the depth Z is and the larger the parallax is; the farther the object is, the larger the depth Z is and the smaller the parallax is.

[0074] Figure 3 The schematic diagram of the main image coordinate system of the binocular camera of an embodiment of the present invention is shown. As shown in the figure, (x, y) is the binocular main camera image coordinate system, o is the coordinate origin (the origin of the image formed by the main camera), (Xc, Yc, Zc) is the camera coordinate system, and Oc is the coordinate origin (camera center). Point P is the imaging point of the spatial point p in the image, and the spatial point p, the imaging point p and the camera center Oc are connected into a straight line. According to the triangle similarity principle, we get: ΔABO C ~ΔoCO C ; ΔPBO C ~ΔpCO C , so we have: f is the focal length of the camera, and x, y represent the distances of the spatial point p from the x and y axes in the image coordinates, respectively.

[0075] Preferably, in step S24, the calculation formulas for the X coordinate and the Y coordinate of the spatial point P in the camera coordinate system are:

[0076]

[0077]

[0078] Among them, (X image ,Y image ) is the image coordinate of the spatial point p in the disparity map, (u x ,v y ) is the pixel coordinate of the spatial point p in the disparity map, (u 0 ,v 0 ) is the pixel coordinate of the camera principal point in the main image. After obtaining the 3D point cloud coordinates, the dense 3D point cloud in the camera coordinate system can be obtained based on the disparity map. X represents the horizontal distance of the 3D point from the camera, Y represents the vertical distance of the 3D point from the camera, and Z represents the depth distance of the 3D point from the camera.

[0079] Preferably, in step S3, the 2D prediction frame is projected onto the binocular disparity point cloud, and the point cloud coordinates corresponding to the road concave-convex features can be obtained according to the calculation of the above point cloud coordinates:

[0080]

[0081] Preferably, step S4 is to calculate the concave-convex information based on the regional point cloud corresponding to each concave-convex feature of the road surface. Usually, the amount of point cloud data provided by binocular stereo vision can reach millions of levels. The feature calculation of such a large amount of point cloud is very computationally intensive. Therefore, after the potholes and speed bumps detected by visual features are projected onto the point cloud, only the point cloud data of the specified features need to be extracted, and a small batch of point cloud processing and calculation is performed, which saves computational resources and combines image recognition with point cloud processing, thereby improving computational efficiency. Specifically, step S4 includes the steps of:

[0082] S41, calculating the concavity and convexity of the point cloud of the corresponding area;

[0083] Based on the four corner points of the 2D prediction box, four 3D points in the point cloud are obtained, and any three of them are selected to construct a spatial plane. In this implementation, three points in the point cloud corresponding to the three points (i, j), (m, j), and (m, n) in the 2D prediction box are selected to construct the plane S:

[0084]

[0085] Select a point in space As the observation point, the minimum spanning tree algorithm in the open3d library is used to estimate the normal vector of the spatial point P1, which is recorded as v1;

[0086] Use the spatial point P1 to project the plane S. The spatial point P1 and the projection point of the plane S form another vector V2.

[0087] Calculate the angle between v1 and v2, and determine whether the point cloud in the corresponding area is concave or convex based on the angle. Specifically, if the angle is less than 90°, it is concave, otherwise it is convex.

[0088] S42, calculating the height position of the point cloud of the corresponding area, that is, calculating the maximum depth of the pothole on the road surface or the maximum height of the speed bump;

[0089] Taking the spatial point P1 as the starting point, calculate the distance from the spatial point p1 to the plane S, and use the gradient ascent method to find the maximum distance from a point in the concave-convex feature area to the plane S and the position coordinates of the point. The maximum value is the maximum depth or maximum height of the concave-convex surface.

[0090] The present invention also proposes a vehicle control method, which is applicable to the above-mentioned road concave-convex feature detection method. Figure 4 A flow chart of a vehicle control method according to an embodiment of the present invention is shown. As shown in the figure, the vehicle control method comprises the following steps:

[0091] T1, predict the trajectory of the vehicle according to the vehicle's motion state;

[0092] T2, based on the predicted trajectory of the vehicle, calculate the correlation with the point cloud of the corresponding area;

[0093] T3, the concavity and convexity, height position, and obtained correlation of the corresponding area point cloud are sent to the ADAS controller as summary information of each road concavity and convexity feature.

[0094] Preferably, in step T1, the parameters related to the vehicle's motion state include:

[0095] Vehicle wheel deflection angle: wheel_angle, unit: rad (radian); vehicle yaw angular velocity: yaw_rate, unit: rad / s (radian / second); vehicle speed: vehicle_speed, unit: m / s.

[0096] First, define the predicted output trajectory equation of the vehicle as a cubic curve, and the expression is defined as follows:

[0097]

[0098] where a 0 That is, the lateral offset of the trajectory, a 1 Indicates the speed direction and the curve direction. Since it is the prediction of the vehicle trajectory, a 1 The default value is 0, a 2 represents the curvature of the curve, a 3 Represents the rate of change of the curve curvature. Next, according to the received vehicle state parameter signal, the coefficients of the vehicle trajectory curve equation are predicted.

[0099] Predicted curvature a2 for:

[0100] When the vehicle speed is less than 1m / s: Among them, k and j are calibration quantities, k = 57.3, j = 41.2, both of which are empirical values;

[0101] When the vehicle speed is greater than 1m / s:

[0102] Predicted curvature change rate a 3 :

[0103] Prediction bias a 0 for:

[0104] Select the offset value of the left and right wheels of the vehicle relative to the center axis of the vehicle as a 0 , the left tire is -d, and the right tire is d.

[0105] Preferably, in step T2, based on the predicted trajectory (curve) of the vehicle and the point cloud area where the potholes and speed bumps are located, the correlation between the point cloud area and the driving trajectory of the vehicle is calculated.

[0106] The Z coordinates of the corner points of the point cloud feature areas of the road potholes and speed bumps are respectively (i,j) , Z (m,j) , Z (i,n) , Z (m,n) Substitute the y value into the left wheel trajectory equation obtained by the above formula (1) and calculate the x value, which is recorded as Then substitute the y value of the right wheel trajectory equation and find the x value, recorded as Calculate the X coordinates of the corner points of the feature area respectively The lateral position relationship between the x value obtained from the left and right wheel trajectories is used to determine the correlation between the road surface concave and convex feature area and the vehicle's driving trajectory.

[0107] Calculate the maximum offset d of the road surface concave and convex feature area into the left and right wheel tracks max , if d max >d k ,d k is the calibration quantity, then the road surface concave-convex characteristic area is within the vehicle's driving trajectory; otherwise, the road surface concave-convex characteristic area is not within the vehicle's driving trajectory.

[0108] Preferably, in step T3, the concavity and convexity of the point cloud of the corresponding area, the height position (maximum depth of potholes and maximum height of speed bumps), and the obtained correlation are sent to the ADAS controller as summary information of each road concavity and convexity feature. It is used by the vehicle backend to perform vehicle control to improve the comfort of intelligent driving.

[0109] Figure 5 The schematic diagram of the structure of a road concave-convex feature detection system based on binocular stereo vision according to an embodiment of the present invention is shown. As shown in the figure, a road concave-convex feature detection system 500 based on binocular stereo vision is applicable to the above-mentioned road concave-convex feature detection method based on binocular stereo vision. The road concave-convex feature detection system 500 includes:

[0110] The feature detection unit 501 is used to obtain a binocular image of the vehicle's driving direction through a binocular camera, wherein the binocular image includes a left image and a right image obtained by a left camera and a right camera, and the left image or the right image is used as a main image, and a road surface concave-convex feature detection is performed on the left image or the right image using a deep learning algorithm to obtain a 2D prediction frame;

[0111] The point cloud generating unit 502 is used to calculate the disparity map D between the left eye image and the right eye image of the same scene based on the deep learning binocular stereo matching algorithm SGBM, and obtain the binocular disparity point cloud in the binocular camera coordinate system based on the disparity map D;

[0112] A projection unit 503 is used to project the 2D prediction frame to the binocular disparity point cloud to obtain a corresponding area point cloud;

[0113] The calculation unit 504 calculates the concave-convex information of the point cloud of the corresponding area.

[0114] The present invention also provides a road bump feature detection device based on binocular stereo vision, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements any of the steps of the aforementioned road bump feature detection method based on binocular stereo vision.

[0115] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for detecting road bump features based on binocular stereo vision are implemented.

[0116] Among them, the specific implementation methods and technical effects of the road bump feature detection device based on binocular stereo vision and the computer-readable storage medium can be referred to the embodiments of the road bump feature detection method based on binocular stereo vision provided by the above-mentioned present invention, which will not be repeated here.

[0117] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0118] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0119] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0120] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0121] It will be apparent to those skilled in the art that various modifications and variations may be made to the above exemplary embodiments of the present invention without departing from the spirit and scope of the present invention. Therefore, it is intended that the present invention covers modifications and variations of the present invention that fall within the scope of the appended claims and their equivalent technical solutions.

Claims

1. A method for detecting road concave-convex features based on binocular stereo vision, comprising the steps of: S1, road surface unevenness feature detection, obtain a binocular image of the vehicle's driving direction through a binocular camera, the binocular image includes a left image and a right image obtained by a left camera and a right camera, take the left image or the right image as the main image, use a deep learning algorithm to detect the road surface unevenness feature, and obtain a 2D prediction frame; the road surface unevenness feature includes potholes or speed bumps, and the 2D prediction frame corresponds to the pothole area or the speed bump area; 2D prediction frame ((Xi, Yj), (Xm, Yn)), where (Xi, Yj) represents the upper left corner coordinate of the 2D prediction frame, and (Xm, Yn) represents the lower right corner coordinate of the 2D prediction frame; S2, generating a binocular disparity point cloud, calculating a disparity map D between a left-eye image and a right-eye image of the same scene based on a deep learning binocular stereo matching algorithm SGBM, and obtaining a binocular disparity point cloud in a binocular camera coordinate system based on the disparity map D; S3, feature area point cloud projection, projecting the 2D prediction box to the binocular disparity point cloud to obtain the corresponding area point cloud; S4, regional feature calculation, calculating the concave-convex information of the corresponding regional point cloud, including the following steps: S41, calculating the concavity and convexity of the point cloud of the corresponding area; Construct a spatial plane S based on the four corner points of the 2D prediction box, and select a spatial point P1 as the point cloud coordinate of the center point of the 2D prediction box projected to the binocular disparity point cloud As the observation point, the minimum spanning tree algorithm in the open3d library is used to estimate the normal vector of the spatial point P1, which is recorded as v1; Use the spatial point P1 to project the plane S. The spatial point P1 and the projection point of the plane S form another vector V2. Calculate the angle between v1 and v2, and determine whether the point cloud in the corresponding area is concave or convex based on the angle; S42, calculating the height position of the point cloud of the corresponding area; Taking the spatial point P1 as the starting point, calculate the distance from the spatial point P1 to the plane S, and use the gradient ascent method to find the maximum distance from a point in the concave-convex feature area to the plane S and the position coordinates of the point; in, (u0,v0) is the pixel coordinate of the camera principal point in the main image, f is the camera focal length, is the depth of spatial point P1.

2. The method for detecting road concave-convex features based on binocular stereo vision as claimed in claim 1, characterized in that: Disparity refers to the difference in pixel position between the left and right images of a spatial point P. The disparity of a spatial point P is d = x l -x r , where x l is the horizontal coordinate of the left camera, x r is the horizontal coordinate of the right camera; step S2 comprises the steps: S21, setting a disparity consistency test, calculating disparity values ​​for pixel points of the left eye image and the right eye image that pass the consistency test, and obtaining a disparity map; S22, extracting connected areas from the disparity map, and performing linear interpolation on the disparity between adjacent pixels to obtain a dense disparity map D; S23, obtaining depth information of the spatial point P; S24, based on the depth information of the spatial point P, obtain the position coordinates of the spatial point P in the camera coordinate system.

3. The method for detecting road concave-convex features based on binocular stereo vision as claimed in claim 2, characterized in that: In step S23, the depth Z of the spatial point P is calculated as: Where T is the length of the optical center line of the binocular camera.

4. The method for detecting road concave-convex features based on binocular stereo vision as claimed in claim 3, characterized in that: In step S24, the calculation formulas for the X coordinate and Y coordinate of the spatial point P in the camera coordinate system are: Among them, (X image ,Y image ) is the image coordinate of the spatial point p in the disparity map, (u x ,v y ) is the pixel coordinate of the spatial point P in the disparity map.

5. The method for detecting road concave-convex features based on binocular stereo vision as claimed in claim 4, characterized in that: In step S3, the 2D prediction frame is projected onto the binocular disparity point cloud, and the point cloud coordinates corresponding to the road concave-convex features are obtained as follows:

6. A vehicle control method, applicable to the road concave-convex feature detection method based on binocular stereo vision as claimed in claim 5, characterized in that: Includes steps: T1, predict the trajectory of the vehicle according to the vehicle's motion state; T2, based on the predicted trajectory of the vehicle, calculate the correlation with the point cloud of the corresponding area; T3, the concavity and convexity, height position, and obtained correlation of the corresponding area point cloud are sent to the ADAS controller as summary information of each road concavity and convexity feature.

7. A road concave-convex feature detection system based on binocular stereo vision, applicable to the road concave-convex feature detection method based on binocular stereo vision according to claim 1, the road concave-convex feature detection system comprising: A feature detection unit is used to obtain a binocular image of the vehicle's driving direction through a binocular camera, wherein the binocular image includes a left image and a right image obtained by a left camera and a right camera, and the left image or the right image is used as a main image to detect the concave and convex features of the road surface by using a deep learning algorithm to obtain a 2D prediction frame; A point cloud generation unit is used to calculate a disparity map D between a left-eye image and a right-eye image of the same scene based on a deep learning binocular stereo matching algorithm SGBM, and obtain a binocular disparity point cloud in a binocular camera coordinate system based on the disparity map D; A projection unit, used for projecting the 2D prediction frame onto the binocular disparity point cloud to obtain a corresponding area point cloud; The calculation unit calculates the concave and convex information of the point cloud of the corresponding area.

8. A road concave-convex feature detection device based on binocular stereo vision, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for detecting road concave-convex features based on binocular stereo vision as described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting road concave-convex features based on binocular stereo vision as described in any one of claims 1 to 5 are implemented.