Vehicle control method, binocular stereoscopic vision-based road unevenness feature detection method, system and device, and computer readable storage medium

Abstract

The present invention relates to a vehicle control method, a binocular stereoscopic vision-based road unevenness feature detection method, system and device, and a computer readable storage medium. The binocular stereoscopic vision-based road unevenness feature detection method comprises the steps: S1, road surface unevenness feature detection; S2, generating a binocular disparity point cloud; S3, feature region point cloud projection; and S4, region feature calculation: calculating unevenness information of a corresponding region point cloud. The vehicle control method comprises the steps: T1, performing ego-vehicle trajectory prediction on the basis of the motion state of the vehicle; T2, on the basis of the predicted ego-vehicle trajectory, calculating a degree of correlation to a corresponding region point cloud; and T3, sending to an ADAS controller the concavity and convexity and height position of the corresponding region point cloud and the acquired degree of correlation as summarized information of each road unevenness feature. The present invention can effectively identify road unevenness information, reduce calculation resource consumption, and thus improve the comfort of intelligent driving.

Description

Vehicle control method, road concave-convex feature detection method based on binocular stereo vision, detection system, detection equipment and computer-readable storage medium Technical Field

[0001] The present invention relates to the field of assisted driving technology, and in particular to a vehicle control method, a road concave-convex feature detection method based on binocular stereo vision, a detection system, a detection device, and a computer-readable storage medium. Background Art

[0002] Among the existing methods for detecting road roughness, there are many neural network detection methods based on point cloud data or image data, which can achieve relatively good detection results. Among them, roughness detection based on point cloud data usually involves rasterizing the point cloud. Because the amount of point cloud data is huge and processing is time-consuming, rasterization is performed to speed up calculations and improve performance. However, there is a disadvantage that features such as potholes will be rasterized into different areas, making it impossible to extract more comprehensive and complete pothole road surface features, and may even lose the detection of such features. Image-based detection of road conditions can more accurately identify different road features in the image, but the depth and positioning accuracy of pothole information will be reduced.

[0003] Summary of the Invention

[0004] In response to the above-mentioned problems in the prior art, the present invention proposes a vehicle control method, a road bump feature detection method based on binocular stereo vision, a detection system, a detection device and a computer-readable storage medium, which can effectively identify road bump information, reduce computing resource consumption, and thus improve the comfort of intelligent driving.

[0005] Specifically, the present invention proposes a method for detecting road concave-convex features based on binocular stereo vision, comprising the following steps:

[0006] S1, road surface concave-convex feature detection, using a binocular camera to obtain a binocular image in the vehicle's travel direction, the binocular image including a left image and a right image obtained by a left camera and a right camera, using the left image or the right image as the main image, and detecting road surface concave-convex features using a deep learning algorithm to obtain a 2D prediction frame;

[0007] S2, generating a binocular disparity point cloud, calculating the disparity map D between the left and right eye images of the same scene based on the deep learning binocular stereo matching algorithm SGBM, and obtaining the binocular disparity point cloud in the binocular camera coordinate system based on the disparity map D;

[0008] S3, feature area point cloud projection, projecting the 2D prediction box onto the binocular disparity point cloud to obtain the corresponding area point cloud;

[0009] S4, regional feature calculation, calculates the concave and convex information of the corresponding regional point cloud.

[0010] According to one embodiment of the present invention, in step S1, the road surface unevenness feature includes a road pothole or a speed bump, and the 2D prediction frame corresponds to the road pothole area or the speed bump area;

[0011] 2D prediction box ((Xi, Yj), (Xm, Yn)), where (Xi, Yj) represents the coordinates of the upper left corner of the 2D two-dimensional box, and (Xm, Yn) represents the coordinates of the lower right corner of the 2D two-dimensional box.

[0012] According to one embodiment of the present invention, the disparity refers to the difference in pixel position of a spatial point P between the left image and the right image. The disparity of the spatial point P is d=x l -x r , where x l is the horizontal coordinate of the left camera, x r is the horizontal coordinate of the right camera; step S2 includes the steps:

[0013] S21, setting a disparity consistency test, calculating disparity values ​​for the pixels of the left and right images that pass the consistency test, and obtaining a disparity map;

[0014] S22, extracting connected areas from the disparity map, and performing linear interpolation on the disparity between adjacent pixels to obtain a dense disparity map D;

[0015] S23, obtaining depth information of the spatial point P;

[0016] S24, based on the depth information of the spatial point P, obtain the position coordinates of the spatial point P in the camera coordinate system. According to one embodiment of the present invention, in step S23, the calculation formula of the depth Z of the spatial point P is:

[0017] Where T is the length of the optical center line of the binocular camera, and f is the focal length of the camera.

[0018] According to one embodiment of the present invention, in step S24, the calculation formulas for the X coordinate and the Y coordinate of the spatial point P in the camera coordinate system are:

[0019] Among them, (X image , Y image ) is the image coordinate of the spatial point p in the disparity map, (u x , v y ) are the pixel coordinates of the spatial point p in the disparity map, and (u0, v0) are the pixel coordinates of the camera principal point in the main image.

[0020] According to one embodiment of the present invention, in step S3, the 2D prediction frame is projected onto the binocular disparity point cloud, and the point cloud coordinates corresponding to the concave-convex features of the road are obtained as follows:

[0021] According to one embodiment of the present invention, step S4 includes the steps of:

[0022] S41, calculating the concavity and convexity of the point cloud of the corresponding area;

[0023] Based on the four corner points of the 2D prediction box, a spatial plane S is constructed and the spatial point As the observation point, the minimum spanning tree algorithm in the open3d library is used to estimate the normal vector of the spatial point P1, which is recorded as v1;

[0024] Use the spatial point P1 to project the plane S. The spatial point P1 and the projection point of the plane S form another vector V2.

[0025] Calculate the angle between v1 and v2, and determine whether the point cloud in the corresponding area is concave or convex based on the angle;

[0026] S42, calculating the height position of the point cloud corresponding to the area;

[0027] Taking the spatial point P1 as the starting point, the distance from the spatial point p1 to the plane S is calculated, and the gradient ascent method is used to find the maximum distance from a point in the concave-convex feature area to the plane S and the position coordinates of the point.

[0028] The present invention also provides a vehicle control method, which is applicable to the aforementioned road concave-convex feature detection method based on binocular stereo vision, comprising the steps of:

[0029] T1, predict the vehicle trajectory based on the vehicle's motion state;

[0030] T2, based on the predicted vehicle trajectory, calculate the correlation with the corresponding area point cloud;

[0031] T3, the concavity and convexity, height position, and obtained correlation of the corresponding area point cloud are sent to the ADAS controller as the summary information of each road concavity and convexity feature.

[0032] The present invention also provides a road concave-convex feature detection system based on binocular stereo vision, which is applicable to the above-mentioned road concave-convex feature detection method based on binocular stereo vision. The road concave-convex feature detection system includes:

[0033] A feature detection unit is configured to acquire a binocular image of the vehicle's travel direction using a binocular camera, the binocular image including a left image and a right image acquired by a left camera and a right camera, and to detect road surface concave and convex features using a deep learning algorithm on the left image or the right image as a primary image to obtain a 2D prediction frame;

[0034] A point cloud generation unit is configured to calculate a disparity map D between a left-eye image and a right-eye image of the same scene based on a deep learning binocular stereo matching algorithm SGBM, and obtain a binocular disparity point cloud in a binocular camera coordinate system based on the disparity map D;

[0035] A projection unit, configured to project the 2D prediction frame onto the binocular disparity point cloud to obtain a corresponding area point cloud;

[0036] The calculation unit calculates the concave and convex information of the point cloud of the corresponding area.

[0037] The present invention also provides a road concave-convex feature detection device based on binocular stereo vision, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of any of the aforementioned methods for detecting road concave-convex features based on binocular stereo vision are implemented.

[0038] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods for detecting road concave-convex features based on binocular stereo vision.

[0039] The present invention provides a vehicle control method, a binocular stereo vision-based road bump detection method, a detection system, a detection device, and a computer-readable storage medium. Based on deep learning, this method uses a single camera in a binocular camera to perform preliminary detection and image localization of bumpy road surface areas. Combining binocular stereo vision, the method generates a dense point cloud in the camera coordinate system. The detected bumpy road features are projected onto the point cloud, and the point cloud within the projected area is calculated and analyzed to obtain bump information. This technical solution combines the advantages of image recognition and point cloud processing, reducing computing resource consumption and providing the autonomous driving controller module with more accurate and comprehensive road condition information, thereby improving the comfort of intelligent driving under certain special road conditions.

[0040] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are included to provide further explanation of the present invention, are incorporated into and constitute a part of this application, illustrate embodiments of the present invention, and together with this specification serve to explain the principles of the present invention. In the drawings:

[0042] FIG1 shows a flowchart of a method for detecting road concave-convex features based on binocular stereo vision according to an embodiment of the present invention.

[0043] FIG2 is a schematic diagram showing the pixel position relationship of a spatial point P in left and right cameras according to an embodiment of the present invention.

[0044] FIG3 is a schematic diagram showing a main image coordinate system of a binocular camera according to an embodiment of the present invention.

[0045] FIG4 shows a flow chart of a vehicle control method according to an embodiment of the present invention.

[0046] FIG5 shows a schematic structural diagram of a road concave-convex feature detection system based on binocular stereo vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] Unless otherwise specified, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. Technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0051] In the description of this application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of this application; the directional words "inside and outside" refer to the inside and outside relative to the outline of each component itself.

[0052] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is solely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. Furthermore, while the terms used in this application are selected from commonly known and commonly used terms, some terms mentioned in this specification may have been selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant sections of this description. Furthermore, this application should be understood not only by the actual terms used, but also by the meaning implied by each term.

[0053] Figure 1 shows a flowchart of a method for detecting road concave-convex features based on binocular stereo vision according to an embodiment of the present invention. As shown in the figure, a method for detecting road concave-convex features based on binocular stereo vision includes the following steps:

[0054] S1, road surface bump feature detection: A binocular camera acquires a binocular image in the vehicle's travel direction. The binocular image includes a left image and a right image, respectively, acquired by the left and right cameras. Using the left or right image as the primary image, a deep learning algorithm is used to detect road surface bump features and obtain a 2D prediction frame. Typically, the left image is used as the primary image, and a deep learning algorithm is used to detect bump features on the left image. It should be noted that a binocular camera consists of two horizontally positioned left and right cameras, capturing a pair of 2D images of the same scene.

[0055] S2, generate binocular disparity point cloud, calculate the disparity map D between the left and right images of the same scene based on the deep learning binocular stereo matching algorithm SGBM, and obtain the binocular disparity point cloud in the binocular camera coordinate system based on the disparity map D;

[0056] S3, feature area point cloud projection, projecting the 2D prediction box to the binocular disparity point cloud to obtain the corresponding area point cloud;

[0057] S4, regional feature calculation, calculates the concave and convex information of the corresponding regional point cloud.

[0058] Preferably, in step S1, the road surface unevenness feature includes a pothole or a speed bump, and the 2D prediction box corresponds to the pothole area or the speed bump area. The 2D prediction box is ((Xi, Yj), (Xm, Yn)), where (Xi, Yj) represents the coordinates of the upper left corner of the 2D box, and (Xm, Yn) represents the coordinates of the lower right corner of the 2D box.

[0059] Preferably, disparity refers to the difference in pixel position of a spatial point P between the left and right images. FIG2 shows a schematic diagram of the pixel position relationship of a spatial point P in the left and right cameras according to an embodiment of the present invention. As shown in the figure, the disparity d of the spatial point P is x l -x r , where x l is the horizontal coordinate of the left camera, x r is the horizontal coordinate of the right camera, f is the focal length of the camera, and T is the binocular camera baseline (the length of the line connecting the optical centers of the left and right cameras). Step S2 includes the following steps:

[0060] S21, set up disparity consistency detection, calculate the disparity value for the pixels of the left image and the right image that pass the consistency detection, and obtain a disparity map. For example, if |x l -x r |>m, assuming m is 1. If the absolute value of the disparity d is greater than 1, the consistency check fails.

[0061] S22: To remove noise points, the disparity map is subjected to connected region extraction. That is, if the disparity value of two adjacent pixels is less than a preset threshold, the two pixels are considered to belong to the same region. The disparity between adjacent pixels is linearly interpolated to obtain a dense disparity map D.

[0062] S23, obtaining the depth information of the spatial point P, that is, calculating the depth Z;

[0063] S24, based on the depth information of the spatial point P, obtain the position coordinates of the spatial point P in the camera coordinate system.

[0064] Preferably, in step S23, the depth Z of the spatial point P is calculated as follows:

[0065] Where T is the length of the optical center line of the binocular camera, and f is the focal length of the camera.

[0066] Referring to Figure 1, according to the principle of similar triangles, the following relationship holds true:

[0067] Where L is the horizontal size of the image. The relationship can be further simplified to obtain the above calculation formula for the spatial point P: It is easy to understand that for the same binocular camera, the closer the object is, the smaller the depth Z is and the larger the parallax is; the farther the object is, the larger the depth Z is and the smaller the parallax is.

[0068] Figure 3 shows a schematic diagram of the main image coordinate system of a binocular camera according to an embodiment of the present invention. As shown in the figure, (x, y) is the binocular main camera image coordinate system, o is the coordinate origin (the origin of the image formed by the main camera), (Xc, Yc, Zc) is the camera coordinate system, and Oc is the coordinate origin (camera center). Point P is the imaging point of spatial point p in the image, and spatial point p, imaging point p, and camera center Oc are connected into a straight line. According to the triangle similarity principle, we get: ΔABO C ~ΔoCO C ;ΔPBO C ~ΔpCO C , so we have: f is the focal length of the camera, x and y represent the distance of the spatial point p from the x and y axes in the image coordinates, respectively.

[0069] Preferably, in step S24, the calculation formulas for the X coordinate and Y coordinate of the spatial point P in the camera coordinate system are:

[0070] Among them, (X image , Y image ) is the image coordinate of the spatial point p in the disparity map, (u x , v y) are the pixel coordinates of spatial point p in the disparity map, and (u0, v0) are the pixel coordinates of the camera's principal point in the main image. Once the 3D point cloud coordinates are obtained, a dense 3D point cloud in the camera coordinate system can be obtained based on the disparity map. X represents the lateral distance of the 3D point from the camera, Y represents the longitudinal distance of the 3D point from the camera, and Z represents the depth distance of the 3D point from the camera.

[0071] Preferably, in step S3, the 2D prediction frame is projected onto the binocular disparity point cloud. According to the calculation of the above-mentioned point cloud coordinates, the point cloud coordinates corresponding to the concave-convex features of the road can be obtained as follows:

[0072] Preferably, step S4 is to calculate the concave-convex information based on the regional point cloud corresponding to each concave-convex feature of the road surface. Generally, the amount of point cloud data provided by binocular stereo vision can reach millions of levels. The feature calculation of such a large amount of point cloud is very computationally intensive. Therefore, after projecting the potholes and speed bumps detected by visual features onto the point cloud, only the point cloud data of the specified features need to be extracted and a small batch of point cloud processing and calculation is performed, which saves computational resources and combines image recognition with point cloud processing, thereby improving computational efficiency. Specifically, step S4 includes the following steps:

[0073] S41, calculating the concavity and convexity of the point cloud of the corresponding area;

[0074] Based on the four corner points of the 2D prediction box, we get four 3D points in the point cloud and select any three of them to construct a spatial plane. In this implementation, we select three points in the point cloud corresponding to the three points (i, j), (m, j), and (m, n) in the 2D prediction box to construct plane S:

[0075] Select a spatial point As the observation point, the minimum spanning tree algorithm in the open3d library is used to estimate the normal vector of the spatial point P1, which is recorded as v1;

[0076] Use the spatial point P1 to project the plane S. The spatial point P1 and the projection point of the plane S form another vector V2.

[0077] Calculate the angle between v1 and v2, and based on the angle, determine whether the corresponding area of ​​the point cloud is concave or convex. Specifically, if the angle is less than 90°, it is concave, otherwise it is convex.

[0078] S42, calculating the height position of the point cloud of the corresponding area, that is, calculating the maximum depth of the pothole or the maximum height of the speed bump;

[0079] Starting from spatial point P1, calculate the distance from spatial point P1 to plane S. Using the gradient ascent method, find the maximum distance from a point within the concave-convex feature area to plane S and the coordinates of that point. The maximum value is the maximum depth or height of the concave-convex surface.

[0080] The present invention also proposes a vehicle control method, which is applicable to the above-mentioned road concave-convex feature detection method. FIG4 shows a flow chart of a vehicle control method according to an embodiment of the present invention. As shown in the figure, the vehicle control method includes the following steps:

[0081] T1, predict the vehicle trajectory based on the vehicle's motion state;

[0082] T2, based on the predicted vehicle trajectory, calculate the correlation with the corresponding area point cloud;

[0083] T3, the concavity and convexity, height position, and obtained correlation of the corresponding area point cloud are sent to the ADAS controller as the summary information of each road concavity and convexity feature.

[0084] Preferably, in step T1, the parameters related to the vehicle's motion state include:

[0085] Vehicle wheel deflection angle: wheel_angle, unit: rad (radian); vehicle yaw angular velocity: yaw_rate, unit: rad / s (radian / second); vehicle speed: vehicle_speed, unit: m / s.

[0086] First, define the predicted output trajectory equation of the vehicle as a cubic curve. The expression is defined as follows:

[0087] Where a0 is the lateral offset of the trajectory, a1 represents the velocity and curve direction. Since this is a prediction of the ego vehicle trajectory, a1 defaults to 0. a2 represents the curve curvature, and a3 represents the rate of change of the curve curvature. Next, based on the incoming ego vehicle state parameter signal, the coefficients of the ego vehicle trajectory equation are predicted.

[0088] The predicted curvature a2 is:

[0089] When the vehicle speed is less than 1m / s: Among them, k and j are calibration quantities, k = 57.3, j = 41.2, both of which are empirical values;

[0090] When the vehicle speed is greater than 1m / s:

[0091] Predicted curvature change rate a3:

[0092] The prediction bias a0 is:

[0093] The offset value of the left and right wheels of the vehicle relative to the vehicle center axis is selected as a0, the left tire is -d, and the right tire is d.

[0094] Preferably, in step T2, the correlation between the predicted trajectory (curve) of the vehicle and the point cloud areas where potholes and speed bumps are located and the driving trajectory of the vehicle is calculated.

[0095] The Z coordinates of the corner points of the point cloud feature areas of the potholes and speed bumps are {Z (i,j) , Z (m，j) , Z (i,n) , Z (m，n) Substitute the y value into the left wheel trajectory equation obtained by the above formula (1) and calculate the x value respectively, which is recorded as Then substitute the y value of the right wheel trajectory equation and calculate the x value respectively, which is recorded as Calculate the X coordinates of the corner points of the feature area respectively The lateral position relationship between the x-values ​​obtained from the left and right wheel trajectories is used to determine the correlation between the road surface concave and convex feature areas and the vehicle's driving trajectory.

[0096] Calculate the maximum offset d of the road surface concave and convex feature area into the left and right wheel tracks max , if d max >d k , d k is the calibration quantity, then the road surface concave-convex characteristic area is within the vehicle's driving trajectory; otherwise, the road surface concave-convex characteristic area is not within the vehicle's driving trajectory.

[0097] Preferably, in step T3, the concavity and convexity of the corresponding area point cloud, the height position (maximum depth of potholes and maximum height of speed bumps), and the obtained correlation are sent to the ADAS controller as summary information of each road concavity and convexity feature. This is used by the vehicle backend to perform vehicle control and improve the comfort of intelligent driving.

[0098] Figure 5 shows a schematic diagram of the structure of a binocular stereo vision-based road concavity and convexity detection system according to one embodiment of the present invention. As shown in the figure, a binocular stereo vision-based road concavity and convexity detection system 500 is applicable to the aforementioned binocular stereo vision-based road concavity and convexity detection method. The road concavity and convexity detection system 500 includes:

[0099] The feature detection unit 501 is configured to acquire a binocular image of the vehicle's travel direction using a binocular camera. The binocular image includes a left image and a right image acquired by a left camera and a right camera, respectively. The left image or the right image is used as the primary image, and a deep learning algorithm is used to detect road surface concave and convex features to obtain a 2D prediction frame.

[0100] The point cloud generation unit 502 is configured to calculate a disparity map D between a left-eye image and a right-eye image of the same scene based on a deep learning binocular stereo matching algorithm SGBM, and obtain a binocular disparity point cloud in a binocular camera coordinate system based on the disparity map D.

[0101] The projection unit 503 is used to project the 2D prediction frame onto the binocular disparity point cloud to obtain the corresponding area point cloud;

[0102] The calculation unit 504 calculates the concave-convex information of the point cloud corresponding to the area.

[0103] The present invention also provides a road concave-convex feature detection device based on binocular stereo vision, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of any one of the aforementioned road concave-convex feature detection methods based on binocular stereo vision are implemented.

[0104] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of any of the aforementioned methods for detecting road concave-convex features based on binocular stereo vision.

[0105] Among them, the specific implementation methods and technical effects of the road bump feature detection device based on binocular stereo vision and the computer-readable storage medium can be referred to the embodiments of the road bump feature detection method based on binocular stereo vision provided by the above-mentioned present invention, and will not be repeated here.

[0106] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0107] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0108] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside in a user terminal as discrete components.

[0109] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0110] It will be apparent to those skilled in the art that various modifications and variations may be made to the above exemplary embodiments of the present invention without departing from the spirit and scope of the present invention. Therefore, it is intended that the present invention cover modifications and variations of the present invention that fall within the scope of the appended claims and their equivalent technical solutions.

Claims

1. A method for detecting road bump features based on binocular stereo vision, comprising the steps: S1, Road bump feature detection: Obtain binocular images in the vehicle driving direction through a binocular camera. The binocular images include left-eye images and right-eye images obtained by the left camera and the right camera. Using the left-eye image or the right-eye image as the main image, use a deep learning algorithm to detect road bump features therein to obtain a 2D prediction box; S2, Generate binocular disparity point cloud: Based on the deep learning binocular stereo matching algorithm SGBM, calculate the disparity map D of the left-eye image and the right-eye image of the same scene, and obtain the binocular disparity point cloud in the binocular camera coordinate system based on the disparity map D; S3, Feature region point cloud projection: Project the 2D prediction box onto the binocular disparity point cloud to obtain the corresponding region point cloud; S4, Region feature calculation: Calculate the bump information of the corresponding region point cloud.

2. The method for detecting road bump features based on binocular stereo vision according to claim 1, wherein, In step S1, the road bump features include road potholes or speed bumps, and the 2D prediction box corresponds to the road pothole area or the speed bump area; The 2D prediction box ((Xi, Yj), (Xm, Yn)), where (Xi, Yj) represents the upper left corner coordinates of the 2D two-dimensional box, and (Xm, Yn) represents the lower right corner coordinates of the 2D two-dimensional box.

3. The method for detecting road unevenness features based on binocular stereo vision according to claim 2, wherein Parallax refers to the pixel position difference of the spatial point P between the left-eye image and the right-eye image. The parallax d of the spatial point P = x l - x r , where x l is the horizontal coordinate of the left camera, and x r is the horizontal coordinate of the right camera; Step S2 includes the steps: S21, Set disparity consistency detection: Calculate the disparity value for the pixel points of the left-eye image and the right-eye image that pass the consistency detection to obtain a disparity map; S22, Perform connected region extraction on the disparity map, perform linear interpolation on the disparity between adjacent pixels to obtain a dense disparity map D; S23, Obtain the depth information of the spatial point P; S24, Based on the depth information of the spatial point P, obtain the position coordinates of the spatial point P in the camera coordinate system.

4. The method for detecting road unevenness features based on binocular stereo vision according to claim 3, wherein In step S23, the calculation formula for the depth Z of the spatial point P is: Wherein, T is the length of the line connecting the optical centers of the binocular cameras, and f is the camera focal length.

5. The method for detecting road unevenness features based on binocular stereo vision according to claim 4, wherein In step S24, the calculation formulas for the X coordinate and Y coordinate of the spatial point P in the camera coordinate system are as follows: Among them, (X image , Y image ) is the image coordinate of the spatial point p in the disparity map, (u x , v y ) is the pixel coordinate of the spatial point p in the disparity map, and (u0, v0) is the pixel coordinate of the camera principal point in the main image.

6. The method for detecting road bump features based on binocular stereo vision according to claim 5, wherein In step S3, project the 2D prediction box onto the binocular disparity point cloud, and the point cloud coordinates corresponding to the road unevenness feature are obtained as follows:

7. The method for detecting road unevenness features based on binocular stereo vision according to claim 5, wherein, Step S4 includes the steps: S41, Calculate the concavity and convexity of the corresponding region point cloud; Construct a spatial plane S based on the four corner points of the 2D prediction box, and select a spatial point As an observation point, use the minimum spanning tree algorithm in the open3d library to estimate the normal vector of the spatial point P1, denoted as v1; Project the spatial point P1 onto the plane S, and the spatial point P1 and the projection point of the plane S form another vector V2; Calculate the angle between v1 and v2, and judge whether the corresponding region point cloud is a concave surface or a convex surface based on the angle; S42, Calculate the height position of the corresponding region point cloud; Starting from the spatial point P1, calculate the distance from the spatial point p1 to the plane S, and use the gradient ascent method to find the maximum distance from a point in the bump feature region to the plane S and the position coordinates of this point.

8. A vehicle control method, applicable to the road unevenness feature detection method based on binocular stereo vision as described in claim 7, characterized in that, Including the steps: T1, Predict the ego-vehicle trajectory according to the vehicle motion state; T2, Based on the predicted ego-vehicle trajectory, calculate the correlation degree with the corresponding region point cloud; T3, Send the concavity and convexity, height position of the corresponding region point cloud, and the obtained correlation degree as the summary information of each road bump feature to the ADAS controller.

9. A road bump feature detection system based on binocular stereo vision, applicable to the road bump feature detection method based on binocular stereo vision described in claim 1, the road bump feature detection system includes: A feature detection unit, configured to obtain binocular images of the vehicle driving direction through a binocular camera, where the binocular images include left-eye images and right-eye images obtained by a left camera and a right camera. Using the left-eye image or the right-eye image as the main image, a deep learning algorithm is used to detect road unevenness features therein, and a 2D prediction box is obtained; A point cloud generation unit, configured to calculate a disparity map D of the left-eye image and the right-eye image of the same scene based on the deep learning binocular stereo matching algorithm SGBM, and obtain a binocular disparity point cloud in the binocular camera coordinate system based on the disparity map D; A projection unit, configured to project the 2D prediction box onto the binocular disparity point cloud to obtain a corresponding region point cloud; A calculation unit, which calculates the unevenness information of the corresponding region point cloud.

10. A road unevenness feature detection device based on binocular stereo vision, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the binocular stereo vision-based road unevenness feature detection method according to any one of claims 1-7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the binocular stereo vision-based road unevenness feature detection method according to any one of claims 1-7.

Images