Vehicle positioning and vehicle 3D detection method based on binocular vision SLAM
By fixing the binocular camera on the front of the autonomous vehicle and using the binocular vision SLAM algorithm for vehicle positioning and 3D detection, the problem of poor positioning effect and high cost of lidar sensors in bad weather is solved, achieving high accuracy and low cost Vehicle positioning and 3D detection.
Patent Information
- Application Number
- CN202011361776.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-11-27
AI Technical Summary
In the field of autonomous driving, lidar sensors have poor positioning effects and high cost in weather such as rain, snow and dust, resulting in limited popularity of autonomous driving vehicles.
Using vehicle positioning and vehicle 3D detection methods based on binocular vision SLAM, the binocular image is obtained by fixing the binocular camera in the front of the vehicle, and using preset object detection, feature matching and fitting algorithms to realize vehicle 3D detection and positioning , reduce hardware requirements and reduce costs.
It realizes high-precision positioning and 3D detection of vehicles in severe weather conditions, reduces costs, improves intelligence, and avoids the high cost and positioning accuracy of lidar sensors.
Smart Images

Figure CN112489080B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a vehicle positioning and vehicle 3D detection method based on binocular vision SLAM, and belongs to the field of automatic driving. Background Art
[0002] In the field of autonomous driving, accurate positioning is a very important part. The current mainstream vehicle positioning solution uses a LiDAR sensor and a laser SLAM algorithm for positioning, which has the advantages of being simple and mature. However, due to the inherent shortcomings of LiDAR sensors, the positioning effect is poor in rainy, snowy and dusty weather, and LiDAR sensors are often expensive, which greatly increases the cost, which has brought great disadvantages to the popularization of autonomous vehicles. Summary of the invention
[0003] The technical problem to be solved by the present invention is to provide a vehicle positioning and vehicle 3D detection method based on binocular vision SLAM to solve the high cost problem of vehicle positioning and vehicle detection in the field of autonomous driving vehicles.
[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions: the present invention designs a vehicle positioning and vehicle 3D detection method based on binocular vision SLAM, based on a binocular camera fixedly placed at a designated position in front of the main vehicle, and the image capture direction of the binocular camera pointing to the front of the vehicle, to achieve the positioning of the main vehicle during driving and the 3D detection of each vehicle around the main vehicle. Specifically, the 3D detection of each vehicle around the main vehicle is achieved by steps A1 to A3, and the positioning of the main vehicle during driving is achieved by step B;
[0005] Step A1. Obtain the intrinsic parameter matrix, distortion coefficient, binocular baseline, and transformation matrix from the left eye of the binocular camera to the center of the main vehicle body through a preset camera calibration method;
[0006] Step A2. During the driving process of the vehicle, the binocular camera is used to continuously acquire 2D binocular images, and each vehicle around the main vehicle is taken as a detection target. The detection targets in the 2D binocular image are detected respectively using a preset target detection algorithm, so as to obtain 2D detection frames of each vehicle around the main vehicle in the 2D binocular image;
[0007] Step A3. According to the 2D detection frames of each vehicle around the main vehicle obtained in step A2, the 3D detection frames of each vehicle around the main vehicle are obtained by following steps A3-1 to A3-4;
[0008] Step A3-1. Use a preset feature detection algorithm to perform feature detection on the image area in each 2D detection frame in the 2D binocular image, obtain the image feature points corresponding to each 2D binocular image, and then proceed to step A3-2;
[0009] Step A3-2. For the image feature points in each 2D binocular image obtained in step A3-1, a preset feature matching algorithm is used to match the feature points, thereby obtaining pairs of matching image feature points that can be matched in the 2D binocular image, and the 2D image coordinates of each matching image feature point in each 2D binocular image;
[0010] Step A3-3. For each pair of matching image feature points, the 3D coordinates of the space point corresponding to each pair of matching image feature points relative to the main vehicle in the world coordinate system are obtained by applying a preset ranging algorithm based on the intrinsic parameter matrix, distortion coefficient, binocular baseline and transformation matrix from the left eye of the binocular camera to the center of the main vehicle body obtained in step A1;
[0011] Step A3-4. Use a preset fitting algorithm to fit the spatial points corresponding to each pair of matching image feature points to obtain a 3D detection frame of each vehicle around the main vehicle;
[0012] Step B. During the driving process of the vehicle, the 2D binocular images continuously acquired by the binocular camera are used as the input images of the binocular vision SLAM algorithm, and then the SLAM algorithm is used to complete the positioning of the vehicle.
[0013] As a preferred technical solution of the present invention, the camera calibration method preset in step A1 is the Zhang Zhengyou calibration method.
[0014] As a preferred technical solution of the present invention, the target detection algorithm preset in step A2 is an SSD detection algorithm.
[0015] As a preferred technical solution of the present invention, the feature detection algorithm preset in step A3-1 is the ORB feature detection sub-detection algorithm.
[0016] As a preferred technical solution of the present invention, the feature matching algorithm preset in step A3-2 is the ORB feature descriptor matching algorithm.
[0017] As a preferred technical solution of the present invention, the distance measurement algorithm preset in step A3-3 is a triangulation algorithm.
[0018] As a preferred technical solution of the present invention, the fitting algorithm preset in step A3-4 is a minimum circumscribed cube fitting algorithm.
[0019] As a preferred technical solution of the present invention, in step B, the image acquired by the binocular camera is first processed according to a preset method, and then the motion trajectory is determined using the SLAM algorithm.
[0020] As a preferred technical solution of the present invention, the preset method is to remove the image area of the surrounding moving vehicles relative to the main vehicle in the 2D binocular image acquired by the binocular camera.
[0021] Compared with the existing laser SLAM technology solutions based on laser radar sensors for autonomous vehicles, the solution proposed by the present invention has low hardware requirements and only requires a binocular camera and corresponding sensors to be fixed in front of the vehicle, which greatly reduces the cost. In addition, the visual sensor has a better advantage in similarity matching than the laser radar sensor, and can also complete the function of 3D detection of vehicles in front of the road, with a more complete level of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is a flow chart of a vehicle positioning and vehicle 3D detection method based on binocular vision SLAM of the present invention. DETAILED DESCRIPTION
[0023] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.
[0024] The present invention designs a vehicle positioning and vehicle 3D detection method based on binocular vision SLAM, such as Figure 1 As shown, a binocular camera is fixedly placed at a designated position in front of the main vehicle, and the image capture direction of the binocular camera points to the front of the vehicle, so as to realize the positioning of the main vehicle during driving and the 3D detection of each vehicle around the main vehicle. Specifically, the 3D detection of each vehicle around the main vehicle is realized by steps A1 to A3, and the positioning of the main vehicle during driving is realized by step B.
[0025] Step A1. Obtain the intrinsic parameter matrix, distortion coefficient, binocular baseline, and transformation matrix from the left eye of the binocular camera to the center of the vehicle body through the Zhang Zhengyou camera calibration method; place a calibration plate in front of the camera, obtain the left and right eye images of the binocular camera at the same time, and calibrate using the Zhang Zhengyou calibration method to obtain the camera intrinsic parameters, distortion coefficient, and binocular baseline. Then calibrate the transformation matrix from the left eye of the binocular camera to the center of the vehicle body. After fixing the binocular camera in front of the vehicle, use the scale to measure the offset distances of the x, y, and z axes respectively.
[0026] Step A2. During the driving process of the vehicle, the binocular camera is used to continuously obtain two 2D binocular images corresponding to the left and right targets at the same time, and each vehicle around the main vehicle is taken as a detection target. The SSD detection algorithm is used to detect the detection targets in the 2D binocular image respectively, so as to obtain the 2D detection frames of each vehicle around the main vehicle in the 2D binocular image.
[0027] Step A3. Based on the 2D detection frames of each vehicle around the main vehicle obtained in step A2, the 3D detection frames of each vehicle around the main vehicle are obtained by following steps A3-1 to A3-4.
[0028] Step A3-1. Use the ORB feature detection sub-detection algorithm to perform feature detection on the image areas in each 2D detection frame in the 2D binocular image, obtain the image feature points corresponding to each 2D binocular image, and then enter step A3-2.
[0029] Step A3-2. For the image feature points in each 2D binocular image obtained in step A3-1, feature point matching is performed using the ORB feature descriptor matching algorithm to obtain pairs of matching image feature points that can be matched in the 2D binocular image, and the 2D image coordinates of each matching image feature point in each 2D binocular image.
[0030] Step A3-3. For each pair of matching image feature points, based on the intrinsic parameter matrix, distortion coefficient, binocular baseline and transformation matrix from the left eye of the binocular camera to the center of the body of the main vehicle obtained in step A1, a triangulation algorithm is applied to obtain the 3D coordinates of the spatial points corresponding to each pair of matching image feature points relative to the main vehicle in the world coordinate system.
[0031] Step A3-4. Use the minimum circumscribed cube fitting algorithm to fit the spatial points corresponding to each pair of matching image feature points to obtain the 3D detection frames of each vehicle around the main vehicle.
[0032] Step B. During the driving process of the vehicle, for the 2D binocular images continuously acquired by the binocular camera, first remove the image area of the surrounding moving vehicles relative to the main vehicle in the 2D binocular images acquired by the binocular camera. This is because the surrounding moving vehicles are not suitable for use as fixed environmental feature points in the SLAM algorithm, so they need to be excluded. Then, the processed image is used as the input image of the binocular vision SLAM algorithm, and then the SLAM algorithm is used to complete the tracking and positioning of the motion trajectory of the main vehicle. Specifically, 1) the binocular camera continuously acquires binocular images during the driving process of the vehicle; 2) removes the moving moving vehicles in the binocular images; 3) uses the ORB feature detection sub-detection algorithm to extract the image feature points of the binocular images from the image part after the moving vehicles are removed; 4) uses the ORB feature descriptor matching algorithm to perform feature matching on the image feature points of the left and right images to obtain a stable and unchanging pairing relationship of the environment; 5) uses the triangular geometry algorithm to calculate the position and distance information of the environmental feature points; 6) Repeat the above process to calculate the motion trajectory of the camera (the main vehicle) relative to the surrounding environment.
[0033] The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM designed by the above technical solution has low hardware requirements and only requires a binocular camera and corresponding sensors to be fixed in front of the vehicle, which greatly reduces the cost. In addition, the visual sensor has a better advantage in similarity matching than the laser radar sensor, and can also complete the function of 3D detection of vehicles in front of the road, with a more complete degree of intelligence.
[0034] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. Vehicle positioning and vehicle 3D detection method based on binocular vision SLAM, characterized in that: Based on a binocular camera fixedly placed at a designated position in front of the main vehicle, and the image capture direction of the binocular camera pointing to the front of the vehicle, the positioning of the main vehicle during driving and the 3D detection of each vehicle around the main vehicle are realized. Specifically, the 3D detection of each vehicle around the main vehicle is realized by steps A1 to A3, and the positioning of the main vehicle during driving is realized by step B; Step A1. Obtain the intrinsic parameter matrix, distortion coefficient, binocular baseline, and transformation matrix from the left eye of the binocular camera to the center of the main vehicle body through a preset camera calibration method; Step A2. During the driving process of the vehicle, the binocular camera is used to continuously acquire 2D binocular images, and each vehicle around the main vehicle is taken as a detection target. The detection targets in the 2D binocular image are detected using a preset target detection algorithm, thereby obtaining 2D detection frames of each vehicle around the main vehicle in the 2D binocular image; Step A3. According to the 2D detection frames of each vehicle around the main vehicle obtained in step A2, the 3D detection frames of each vehicle around the main vehicle are obtained by following steps A3-1 to A3-4; Step A3-1. Use a preset feature detection algorithm to perform feature detection on the image area in each 2D detection frame in the 2D binocular image, obtain the image feature points corresponding to each 2D binocular image, and then proceed to step A3-2; Step A3-2. For the image feature points in each 2D binocular image obtained in step A3-1, a preset feature matching algorithm is used to match the feature points, thereby obtaining pairs of matching image feature points that can be matched in the 2D binocular image, and the 2D image coordinates of each matching image feature point in each 2D binocular image; Step A3-3. For each pair of matching image feature points, the 3D coordinates of the space point corresponding to each pair of matching image feature points relative to the main vehicle in the world coordinate system are obtained by applying a preset ranging algorithm based on the intrinsic parameter matrix, distortion coefficient, binocular baseline and transformation matrix from the left eye of the binocular camera to the center of the main vehicle body obtained in step A1; Step A3-4. Use a preset fitting algorithm to fit the spatial points corresponding to each pair of matching image feature points to obtain a 3D detection frame of each vehicle around the main vehicle; Step B. During the driving process of the vehicle, the 2D binocular image continuously acquired by the binocular camera is used as the input image of the binocular vision SLAM algorithm, and then the SLAM algorithm is used to complete the positioning of the main vehicle; The feature detection algorithm preset in step A3-1 is the ORB feature detection sub-detection algorithm; The distance measurement algorithm preset in step A3-3 is a triangulation algorithm.
2. The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM according to claim 1 is characterized in that: The camera calibration method preset in step A1 is the Zhang Zhengyou calibration method.
3. The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM according to claim 1 is characterized in that: The target detection algorithm preset in step A2 is the SSD detection algorithm.
4. The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM according to claim 1 is characterized in that: The feature matching algorithm preset in step A3-2 is the ORB feature descriptor matching algorithm.
5. The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM according to claim 1 is characterized in that: The preset fitting algorithm in step A3-4 is the minimum circumscribed cube fitting algorithm.
6. The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM according to claim 1 is characterized in that: In step B, the image acquired by the binocular camera is first processed according to a preset method, and then the motion trajectory is determined using the SLAM algorithm.
7. The vehicle positioning and vehicle 3D detection method based on binocular vision SLAM according to claim 6 is characterized in that: The preset method is to remove the image area of the surrounding moving vehicles relative to the main vehicle in the 2D binocular image acquired by the binocular camera.
Citation Information
Patent Citations
Vehicle mileage calculation method based on double-eye vision
CN105300403A
Landmark map vehicle instant positioning method based on binocular vision system
CN108534782A
Indoor multi-balance cars autonomous navigation method based on sparse map and driver identification
CN108827317A
ORB-SLAM based high-precision vehicle positioning method
CN109631855A
Intelligent vehicle track measurement method based on binocular stereo vision system
CN110285793A
