A binocular camera guidance based sensor fusion method

Abstract

The application discloses a kind of sensor fusion methods based on binocular camera guidance, belong to intelligent vehicle environment perception technical field, the fusion method of binocular vision guidance is used in the application, can use visual target information to establish its motion trajectory, and then predict its position in three-dimensional space;The fusion method of data layer is used in the application, the prediction result of vision is used to assist laser radar to demarcate detection range, greatly reduce the data amount of operation, the requirement of lower to computing platform performance;The target detection result obtained based on binocular vision in the application can accurately obtain target depth, expand target detection information and enhance the robustness of target detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle environmental perception technology, and in particular to a sensor fusion method based on binocular camera guidance. Background Technology

[0002] Currently, specific research and engineering applications utilize single-type sensors such as vision or lidar to achieve target detection. Vision sensors are widely used in target detection due to their low cost, ease of target identification, and rich data acquisition. LiDAR has the ability to actively detect targets and offers high ranging accuracy; in recent years, with the optimization and upgrading of radar working principles, mass production has become possible. However, with increasingly complex traffic conditions, single-sensor-based detection methods are constrained by complex algorithms and insufficient detection information, making it difficult to meet current requirements for robustness and speed in target detection.

[0003] To compensate for the shortcomings of single-sensor detection, multi-sensor fusion detection methods have gradually developed. This detection method can also improve the robustness of the system in complex environments, so multi-sensor information fusion detection methods are increasingly used in target detection research. Currently, multi-sensor data fusion is divided into three levels: decision-level fusion, feature-level fusion, and data-level fusion. In his paper "Outdoor Positioning Based on Fusion of Binocular Vision and 2D LiDAR," Liu Zhengxuan used binocular vision to detect the relative pose of a vehicle and fused it with LiDAR data from multiple moments within a time window to obtain a local sub-image. After noise reduction and matching processing, high-precision vehicle positioning was achieved. In his paper "Research on SLAM Method Based on LiDAR and Vision Fusion," Gong Chaoguang designed a classification fusion framework for LiDAR and binocular cameras. By classifying and matching extracted visual features with LiDAR point clouds, and then merging feature line segments based on the matching results, the fused point cloud information was obtained. Both papers perform data fusion at the data-level of binocular cameras and LiDAR, achieving high accuracy but being less suitable for engineering implementation.

[0004] Patent "CN114463303A" proposes a road target detection method based on the fusion of binocular cameras and LiDAR. This method combines LiDAR technology with visual sensors and integrates the YOLOv4 detection model to fuse the radar and binocular vision results, thus providing a more intelligent and accurate technical means for target detection. Patent "CN111340797A" proposes a detection method and system based on the fusion of LiDAR and binocular camera data. This method fuses the detection results from the binocular camera with those from the LiDAR and determines whether the fusion overlap rate is greater than a certain threshold. If it is greater than a set value, the fusion result is output; otherwise, target detection needs to be performed again. This method improves measurement accuracy. These two patents fuse binocular cameras and LiDAR at the decision layer, which has good engineering effects but is not conducive to performance improvement. Most existing research uses traditional methods to fuse camera and LiDAR data layers or decision layers for target detection. These methods are difficult to balance between engineering applicability and detection performance.

[0005] Based on the current research status, this invention proposes a sensor fusion method guided by a binocular camera, which indirectly realizes data fusion between the binocular camera and LiDAR at the data layer. It can combine the advantages of binocular camera and LiDAR sensors to effectively improve the accuracy and robustness of target detection and meet the needs of engineering applications. Summary of the Invention

[0006] This invention provides a sensor fusion method based on binocular camera guidance to solve the problems existing in the background technology.

[0007] A sensor fusion method guided by a binocular camera includes the following steps:

[0008] Step 1: Based on the visual image information acquired by the binocular camera and the 3D point cloud information acquired by the lidar, perform camera calibration on the left and right cameras that make up the binocular camera, and perform joint calibration on the lidar and the left camera.

[0009] Step 2: Collect forward environmental information of the electronically guided rubber-wheeled vehicle using a binocular camera and lidar;

[0010] Step 3: Calculate the depth information of the forward environment of the electronically guided rubber-wheeled vehicle using a stereo matching algorithm;

[0011] Step 4: Using a YOLOv7-based target detection neural network, obtain visual information of the target in the image captured by the camera, including the target category and two-dimensional planar position information;

[0012] Step 5: Combine the target visual information, environmental depth information, and target time information output by the neural network to obtain the target detection results based on stereo vision;

[0013] Step Six: Based on the target time information obtained in Step Five, establish the target's motion trajectory and predict the target's three-dimensional position information at the time of lidar acquisition;

[0014] Step 7: Based on the target's stereo position information predicted in Step 6, segment and cluster the point cloud collected by the lidar, and output the target detection results;

[0015] Step 8: Track the target detected by the lidar based on the Kalman filter algorithm and the nearest neighbor algorithm, and correct the output of the fused target based on the operational scenario.

[0016] The specific steps for camera calibration in step one are as follows:

[0017] Using visual image information acquired by the binocular cameras, the Zhang Zhengyou calibration method is used to obtain the intrinsic parameter M of the left camera. L Distortion coefficient D L Right camera internal parameters M R Distortion coefficient D R And the external parameter M between the left and right cameras LR .

[0018] The specific steps for joint calibration in step one are as follows:

[0019] By jointly calibrating the 3D point cloud information acquired by the LiDAR and the 2D position information in the visual image information acquired by the left camera using a calibration board, the external parameter M between the LiDAR and the left camera is obtained. LC .

[0020] The calculation method in step three is as follows:

[0021] By matching the corresponding pixels in the left and right views under a unified scene, a disparity map is obtained. The projection matrix Q from the two-dimensional position of the image to the three-dimensional position in space is obtained based on the Bouguet algorithm, and the target depth information is calculated.

[0022] Step four includes: collecting and creating a dataset of target images in the operating environment, using the existing target-annotated dataset as the training set of the neural network, obtaining the weight coefficients of the neural network through training the model, and then predicting the images collected by the left camera to obtain target detection results containing target category and two-dimensional bounding box information describing the target's planar position.

[0023] Step five includes: using the calculation method in step three to project the target plane information obtained in step four into three-dimensional space, obtaining the target detection result described by the three-dimensional bounding box, and then combining the image timestamp to obtain the target detection result based on stereo vision.

[0024] Step seven includes:

[0025] The RANSAC algorithm is used to segment the ground and extract non-ground target point clouds. Based on the stereo position information obtained in step six, point cloud clustering is performed to obtain target point cloud clusters. The point cloud target detection results described by stereo information in three-dimensional bounding boxes are output. The target information obtained by the binocular camera guides the lidar to accurately measure the distance to the target.

[0026] The steps for point cloud clustering are as follows:

[0027] (1) LiDAR acquires target category and historical trajectory information detected by binocular camera, models and predicts the spatial position of the target, and assists in delineating the detection range;

[0028] (2) Select any point M in the point cloud within the detection range, and adaptively set the clustering threshold according to the detection range. Search for the k nearest neighbors of M based on the KDTree data structure, filter the points whose distance from point M is less than the clustering threshold and store them in set Q. If the number of points in Q increases, select any point other than point M in Q to update point M. Continue searching until the number of elements in set Q no longer increases.

[0029] (3) Repeat steps (1) and (2) for all detected objects in the images acquired by the binocular camera, and then use the clustering detection algorithm to complete the detection of all targets, outputting the target category, point cloud, 3D bounding box and timestamp.

[0030] Before clustering the point cloud data collected by lidar, motion distortion compensation is used to eliminate motion distortion within the point cloud, as follows:

[0031] An IMU is used for motion distortion compensation of the lidar. While receiving lidar scanning data, the data is stored in the same circular queue as the vehicle attitude angle data collected by the IMU. The processor calculates the sampling time of each laser point based on the lidar sampling time interval and data timestamp. It searches for two adjacent frames of data in the IMU data queue and uses spherical linear interpolation to pair the laser points at the same measurement time with the vehicle's motion attitude. This allows the coordinates of each laser point in a lidar scanning data frame to be transformed into the corresponding position coordinate system of the electronically guided rubber-tired vehicle at the same time. This unifies all point clouds collected by the lidar over a period of time to a single moment, thus achieving motion distortion compensation for the lidar.

[0032] The method for tracking the target based on the Kalman filter algorithm and the nearest neighbor algorithm in step eight is as follows:

[0033] Based on the target motion model, a Kalman filter bank is constructed and initialized, and relevant parameters are configured to calculate the centroid position of the target point cloud. Using the Kalman filter bank, the centroid position of the target at the current timestamp is used to predict the centroid position of the target in the next frame of the point cloud. In the next frame of the point cloud, a nearest neighbor algorithm is used to search for matching combinations of real and predicted targets, achieving inter-frame target association, i.e., obtaining the spatial position of the same target in two adjacent frames. The target state of the Kalman filter bank is updated using the target's spatial position and timestamp information in the next frame of the point cloud, and then the target state is predicted again. Repeating this process allows for target tracking.

[0034] Compared with the prior art, the beneficial effects of the present invention are: the present invention adopts a binocular vision-guided fusion method, which can use visual target information to establish its motion trajectory and then predict its position in three-dimensional space; the present invention adopts a data layer fusion method, which uses the prediction results of vision to assist the lidar in delineating the detection range, greatly reducing the amount of computational data and lowering the performance requirements of the computing platform.

[0035] Target detection results obtained based on binocular vision can accurately obtain target depth, expand target detection information, and enhance the robustness of target detection. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the sensor fusion method of the present invention;

[0037] Figure 2 This is a flowchart of the sensor fusion method of the present invention;

[0038] Figure 3 A flowchart for clustering point clouds from a lidar system;

[0039] Figure 4 This is a diagram showing the result of point cloud clustering.

[0040] Figure 5 This is a schematic diagram of target tracking. Detailed Implementation

[0041] The following detailed description of a specific embodiment of the present invention is provided in conjunction with the accompanying drawings. However, it should be understood that the scope of protection of the present invention is not limited to the specific embodiment.

[0042] like Figures 1 to 5 As shown in the figure, an embodiment of the present invention provides a sensor fusion method based on a binocular camera, comprising the following steps:

[0043] Step 1: Based on the visual image information acquired by the binocular camera and the 3D point cloud information acquired by the lidar, perform camera calibration on the left and right cameras that make up the binocular camera, and perform joint calibration on the lidar and the left camera.

[0044] The fusion method proposed in this invention first performs target detection and network tracking based on a binocular camera. A binocular camera is generally composed of two horizontally placed cameras, one on the left and one on the right. The two cameras image the target respectively. Since there is a certain length of baseline between the cameras, the imaging position of the target will be different. The positional relationship between the target and the camera can be calculated by using the difference in imaging position and the baseline length.

[0045] The specific steps for camera calibration in Step 1 are as follows:

[0046] Using visual image information acquired by the binocular cameras, the Zhang Zhengyou calibration method is used to obtain the intrinsic parameter M of the left camera. L Distortion coefficient D L Right camera internal parameters M R Distortion coefficient D R And the external parameter M between the left and right cameras LR ;

[0047] The specific steps for joint calibration in Step 1 are as follows:

[0048] By jointly calibrating the 3D point cloud information acquired by the LiDAR and the 2D position information in the visual image information acquired by the left camera using a calibration board, the external parameter M between the LiDAR and the left camera is obtained. LC This allows the measured values ​​from their respective coordinate systems to be transformed into the train coordinate system.

[0049] Binocular correction corrects distortion and aligns the views captured by the two cameras based on the relative positional relationship obtained during camera calibration and the internal parameter data of the left and right cameras, so that the origin coordinates of the left and right views are consistent, the optical axes of the two cameras are parallel, the epipolar lines are aligned and the imaging planes of the left and right cameras are coplanar.

[0050] Step 2: Collect forward environmental information of the electronically guided rubber-wheeled vehicle using a binocular camera and lidar;

[0051] Step two is performed while keeping the positions of the binocular camera and the lidar unchanged;

[0052] Step 3: Calculate the depth information of the forward environment of the electronically guided rubber-wheeled vehicle using a stereo matching algorithm;

[0053] The calculation method in step three is as follows: Match the corresponding pixels of the left and right views under the same scene to obtain the disparity map, obtain the projection matrix Q from the two-dimensional position of the image to the three-dimensional position in space based on the Bouguet algorithm, and calculate the target depth information;

[0054] Step 4: Using a YOLOv7-based target detection neural network, obtain visual information of the target in the image captured by the camera, including the target category and two-dimensional planar position information;

[0055] The above step four specifically involves: collecting and creating a dataset of target images in the operating environment; using the existing target-annotated dataset as the training set for the YOLOv7 neural network; obtaining the weight coefficients of the neural network through training the model; then predicting the images collected by the left camera; and obtaining target detection results containing target category and two-dimensional bounding box information describing the target's planar position. This invention utilizes a deep learning-based target detection algorithm to perform target detection on a single image, which can identify and classify traffic participants such as pedestrians and vehicles, and obtain target detection results based on stereo vision.

[0056] Furthermore, the YOLOv7 used in this invention improves detection speed and accuracy by reforming multiple internal architectures based on YOLOv5. YOLOv7 mainly uses the MP structure and ELAN, which can perform deep learning more efficiently by controlling the shortest gradient path and a deeper network structure. The loss function of YOLOv7 is divided into three main parts: target confidence loss, coordinate loss, and classification loss. It mainly performs data candidate selection during the matching strategy process, retaining the data with the minimum loss to achieve high-precision target detection.

[0057] Step 5: Combine the target visual information, environmental depth information, and target time information output by the neural network to obtain the target detection results based on stereo vision;

[0058] Step five above uses the calculation method in step three to project the target plane information obtained in step four into three-dimensional space, obtaining the target detection result described by the three-dimensional bounding box. Then, combined with the image timestamp, the target detection result based on stereo vision is obtained.

[0059] Step Six: Based on the target time information obtained in Step Five, establish the target's motion trajectory and predict the target's three-dimensional position information at the time of lidar acquisition;

[0060] Step 7: Based on the target's stereo position information predicted in Step 6, segment and cluster the point cloud collected by the lidar, and output the target detection results;

[0061] Step seven above specifically involves: combining the operating environment characteristics of the electronically guided rubber-wheeled vehicle, using the RANSAC algorithm to segment the ground, extracting non-ground target point clouds, clustering the point clouds based on the stereo position information obtained in step six, obtaining target point cloud clusters, and outputting the point cloud target detection results described by three-dimensional bounding boxes. The target information acquired by the binocular camera guides the LiDAR to accurately measure the distance to the target. Generally, when using a binocular camera to acquire the category and location of a target, the sampling delay of the image and the processing time of the target detection algorithm need to be considered. Currently, LiDAR point cloud clustering mainly uses the Euclidean clustering method. This clustering method assumes that in all point cloud data collected by the LiDAR, the distance between any two points in the point cloud of the same object will be less than a certain value. Therefore, points with a distance less than a certain value can be grouped into one class, ultimately completing the target clustering. This clustering method has high accuracy, but its clustering speed is not advantageous. Although this method uses KD-Tree to speed up data retrieval, it is still insufficient in rapidly changing scenarios. This invention innovatively adopts a binocular camera-guided LiDAR for target clustering, which not only improves the clustering speed but also provides better accuracy and robustness of the final identification results. The point cloud clustering steps are as follows:

[0062] (1) LiDAR acquires target category and historical trajectory information detected by binocular camera, models and predicts the spatial position of the target, and assists in delineating the detection range;

[0063] (2) Select any point M in the point cloud within the detection range, and adaptively set the clustering threshold according to the detection range. Search for the k nearest neighbors of M based on the KDTree data structure, filter the points whose distance from point M is less than the clustering threshold and store them in set Q. If the number of points in Q increases, select any point other than point M in Q to update point M. Continue searching until the number of elements in set Q no longer increases.

[0064] (3) Repeat steps (1) and (2) for all detected objects in the images acquired by the binocular camera, and then use the clustering detection algorithm to complete the detection of all targets, outputting the target category, point cloud, 3D bounding box and timestamp.

[0065] Furthermore, during the process of the lidar following the movement of the rubber-wheeled vehicle, the reference pose of the electronically guided vehicle varies at different times due to its high-speed movement. This causes motion distortion in the laser point cloud collected by the lidar. Additionally, if the lidar scanning frequency is low, the laser point data is not acquired instantaneously, which also leads to motion distortion in the laser point cloud. Therefore, before clustering the point cloud collected by the lidar, motion distortion compensation is used to eliminate the motion distortion within the point cloud, as detailed below:

[0066] An IMU is used for motion distortion compensation of the lidar. While receiving lidar scanning data, the data is stored in the same circular queue as the vehicle attitude angle data collected by the IMU. The processor calculates the sampling time of each laser point based on the lidar sampling time interval and data timestamp. It searches for two adjacent frames of data in the IMU data queue and uses spherical linear interpolation to pair the laser points at the same measurement time with the vehicle's motion attitude. This allows the coordinates of each laser point in a lidar scanning data frame to be transformed into the corresponding position coordinate system of the electronically guided rubber-tired vehicle at the same time. This unifies all point clouds collected by the lidar over a period of time to a single moment, thus achieving motion distortion compensation for the lidar.

[0067] Step 8: Track the target detected by the lidar based on the Kalman filter algorithm and the nearest neighbor algorithm, and correct the output of the fused target based on the operational scenario;

[0068] The method for tracking the target based on the Kalman filter algorithm and the nearest neighbor algorithm in step eight above is as follows:

[0069] Based on the target motion model, a Kalman filter bank is constructed and initialized, and relevant parameters are configured. The centroid position of the target point cloud is calculated. Based on the centroid position of the target at the current timestamp, the centroid position of the target in the next frame of the point cloud is predicted using the Kalman filter bank. In the next frame of the point cloud, a nearest neighbor algorithm is used to search for matching combinations of real and predicted targets, realizing inter-frame target association, i.e., obtaining the spatial position of the same target in two adjacent frames. The target state of the Kalman filter bank is updated using the spatial position and timestamp information of the target in the next frame of the point cloud, and the target state is predicted again. By repeating the above process, target tracking can be achieved. This invention proposes a sensor fusion framework guided by a binocular camera, providing a fusion method for the sensor data layer guided by a binocular camera. It takes into account both target detection performance and engineering applications, realizes the complementary advantages between sensors with different functions, and improves the intelligence level of the target detection system. Compared with 3D target detection systems, deep learning-based visual target detection systems are more mature and have lower performance requirements for computing platforms.

[0070] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit and essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0071] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A sensor fusion method based on binocular camera guidance, characterized in that, Includes the following steps: Step 1: Based on the visual image information acquired by the binocular camera and the 3D point cloud information acquired by the lidar, perform camera calibration on the left and right cameras that make up the binocular camera, and perform joint calibration on the lidar and the left camera. Step 2: Collect forward environmental information of the electronically guided rubber-wheeled vehicle using a binocular camera and lidar; Step 3: Calculate the depth information of the forward environment of the electronically guided rubber-wheeled vehicle using a stereo matching algorithm; Step 4: Using a YOLOv7-based target detection neural network, obtain visual information of the target in the image captured by the camera, including the target category and two-dimensional planar position information; Step 5: Combine the target visual information, environmental depth information, and target time information output by the neural network to obtain the target detection results based on stereo vision; Step Six: Based on the target time information obtained in Step Five, establish the target's motion trajectory and predict the target's three-dimensional position information at the time of lidar acquisition; Step 7: Based on the target's stereo position information predicted in Step 6, segment and cluster the point cloud collected by the lidar, and output the target detection results; Step 8: Track the target detected by the lidar based on the Kalman filter algorithm and the nearest neighbor algorithm, and correct the output of the fused target based on the operational scenario; Step seven includes: The RANSAC algorithm is used to segment the ground and extract non-ground target point clouds. Based on the stereo position information obtained in step six, point cloud clustering is performed to obtain target point cloud clusters. The point cloud target detection results described by stereo information in three-dimensional bounding boxes are output. The target information obtained by the binocular camera guides the lidar to accurately measure the distance to the target. The steps for point cloud clustering are as follows: (1) LiDAR models and predicts the spatial location of the target by acquiring the target category and historical trajectory information detected by the binocular camera, and assists in delineating the detection range; (2) Select any point M in the point cloud within the detection range, and adaptively set the clustering threshold according to the detection range. Search for the k nearest neighbors of M based on the KDTree data structure, filter the points whose distance from point M is less than the clustering threshold and store them in set Q. If the number of points in Q increases, select any point other than point M in Q to update point M. Continue searching until the number of elements in set Q no longer increases. (3) Repeat steps (1) and (2) for all detected objects in the images acquired by the binocular camera, and then use the clustering detection algorithm to complete the detection of all targets, and output the target category, point cloud, 3D bounding box and timestamp; Before clustering the point cloud data collected by lidar, motion distortion compensation is used to eliminate motion distortion within the point cloud, as follows: An IMU is used for motion distortion compensation of the lidar. While receiving lidar scanning data, the data is stored in the same circular queue as the vehicle attitude angle data collected by the IMU. The processor calculates the sampling time of each laser point based on the lidar sampling time interval and data timestamp. It searches for two adjacent frames of data in the IMU data queue and uses spherical linear interpolation to pair the laser points at the same measurement time with the vehicle's motion attitude. This allows the coordinates of each laser point in a lidar scanning data frame to be transformed into the corresponding position coordinate system of the electronically guided rubber-tired vehicle at the same time. This unifies all point clouds collected by the lidar over a period of time to a single moment, thus achieving motion distortion compensation for the lidar.

2. The sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that, The specific steps for camera calibration in step one are as follows: Using visual image information acquired by the binocular cameras, the Zhang Zhengyou calibration method is employed to obtain the internal parameters of the left camera. Distortion coefficient Right camera internal parameters Distortion coefficient and external parameters between the left and right cameras .

3. A sensor fusion method based on binocular camera guidance as described in claim 1 or 2, characterized in that, The specific steps for joint calibration in step one are as follows: By jointly calibrating the 3D point cloud information acquired by the LiDAR and the 2D position information in the visual image information acquired by the left camera using a calibration board, the external parameters between the LiDAR and the left camera are obtained. .

4. The sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that, The calculation method in step three is as follows: By matching the corresponding pixels in the left and right views under a unified scene, a disparity map is obtained. The projection matrix Q from the two-dimensional position of the image to the three-dimensional position in space is obtained based on the Bouguet algorithm, and the target depth information is calculated.

5. The sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that, Step four includes: collecting and creating a dataset of target images in the operating environment, using the existing target-annotated dataset as the training set of the neural network, obtaining the weight coefficients of the neural network through training the model, and then predicting the images collected by the left camera to obtain target detection results containing target category and two-dimensional bounding box information describing the target's planar position.

6. The sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that, Step five includes: using the calculation method in step three to project the target plane information obtained in step four into three-dimensional space, obtaining the target detection result described by the three-dimensional bounding box, and then combining the image timestamp to obtain the target detection result based on stereo vision.

7. The sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that, The method for tracking the target based on the Kalman filter algorithm and the nearest neighbor algorithm in step eight is as follows: Based on the target motion model, a Kalman filter bank is constructed and initialized, and relevant parameters are configured. The centroid position of the target point cloud is calculated. Based on the centroid position of the target at the current timestamp, the centroid position of the target at the next frame of the point cloud is predicted. In the next frame of the point cloud, the nearest neighbor algorithm is used to search for matching combinations of real targets and predicted targets to achieve inter-frame target association, that is, to obtain the spatial position of the same target in two adjacent frames. The target state of the Kalman filter bank is updated using the spatial position and timestamp information of the target in the next frame of the point cloud, and the target state is predicted again. By repeating this process, target tracking can be achieved.

Images