Vehicle Detection Method and Detection Device for Target Data Fusion

Abstract

The present invention relates to a vehicle detection method and a detection device for target data fusion. The vehicle detection method for target data fusion comprises the following steps: firstly, performing spatiotemporal synchronization on a laser radar and a camera sensor; then, performing preliminary processing on the point cloud data through voxel sampling, pass-through filtering, and a RANSAC plane fitting algorithm, compressing the point cloud data while retaining the vehicle point cloud information to improve the operation speed, and then using an adaptive Euclidean clustering algorithm to obtain a three-dimensional vehicle detection frame based on the point cloud data; then, using an improved Yolov4 algorithm to perform target detection on an RGB image to obtain a two-dimensional vehicle detection frame based on the RGB image. The present invention retains the original detection accuracy of the Yolov4 algorithm while greatly improving the operation speed of the model, and at the same time, while improving the algorithm operation speed, it also improves the clustering accuracy of the point cloud, and can meet the complex perception requirements of automatic driving during the actual driving process of intelligent vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and particularly to a vehicle detection method and detection device based on the fusion of lidar and camera target data. Background Art

[0002] An intelligent vehicle system generally includes three parts: environmental perception, decision-making and planning, and execution control. Among them, the perception system, as the first step of information interaction between the intelligent vehicle and the external environment, is the core component of the intelligent vehicle. It helps the intelligent vehicle to more comprehensively understand the surrounding environment through the input of perception information, and provides high-quality perception results as support for subsequent vehicle trajectory prediction and path planning steps. At present, the sensors used in environmental perception generally include lidar, cameras, millimeter-wave radars, ultrasonic radars, etc. The multi-sensor system obtains richer and more reliable target information than a single sensor, and will surely become the mainstream trend of future intelligent driving development. In the multi-sensor system, the multi-source sensor information fusion technology is very crucial and has always attracted the attention of researchers. Therefore, how to make up for the defects of single sensors working independently through sensor information fusion technology and improve the safety and reliability of the entire intelligent driving system has important theoretical and practical significance. Summary of the Invention

[0003] To solve the technical problems that the existing intelligent vehicle system cannot meet the environmental perception requirements in complex traffic environments and the existing multi-sensor fusion algorithms are difficult to be deployed on lightweight platforms, the present invention provides a vehicle detection method and detection device based on the fusion of lidar and camera target data.

[0004] The present invention is implemented by the following technical solutions: A vehicle detection method for target data fusion, including the following steps:

[0005] Step S1, the camera sensor and the lidar sensor are synchronized in time and space;

[0006] Step S2, obtain a visual image through the camera sensor, and perform visual detection through the improved Yolov4 algorithm based on the lightweight network MobileNet to obtain a two-dimensional vehicle detection frame based on the visual image;

[0007] Step S3, obtain point cloud data through the lidar sensor, and preprocess the point cloud data;

[0008] Step S4, perform clustering analysis on the preprocessed point cloud data through an adaptive Euclidean clustering algorithm to obtain a three-dimensional vehicle detection frame based on the point cloud data;

[0009] Step S5: Project the 3D vehicle detection frame based on the point cloud data according to the projection matrix to obtain a 2D vehicle detection frame based on the point cloud data. Then, preliminarily fuse the 2D vehicle detection frame based on the point cloud data and the 2D vehicle detection frame based on the visual image, and calculate the area overlap degree of the two in the RGB image.

[0010] Step S6: Determine whether the area overlap degree is higher than the set threshold through the intersection-over-union operation. When the area overlap degree is greater than the set threshold, perform preliminary matching on it.

[0011] Step S7: For the targets with successful preliminary matching, record their trajectory information under the lidar sensor and the trajectory information under the camera sensor respectively, and calculate the local trajectory similarity of their trajectory information through the improved longest common subsequence algorithm based on the fuzzy membership function.

[0012] When the local trajectory similarity is greater than the set threshold, it is determined that the two trajectories are similar, the lidar sensor target and the camera sensor target are successfully associated, and the fusion information is output.

[0013] For the targets that fail to be successfully associated, the processing results of the lidar sensor and the camera sensor are output respectively.

[0014] As a preferred example, in step S1, the method for spatial synchronization between the camera sensor and the lidar sensor is as follows: Define that there is a point [x i y i z i T and [x c y c z c T on the lidar coordinate system and the camera coordinate system respectively. First, unify the directions of the lidar coordinate system and the camera coordinate system through the rotation matrix R, and then make the two coordinate systems completely coincide through the translation matrix T. Transform the point [x i y i z i T into the in the camera coordinate system. Finally, use the conversion formula between the camera coordinate system and the pixel coordinate system to project the vehicle detection frame based on the point cloud data onto the visual image, and take the maximum values of the horizontal and vertical coordinates of the projected pixel points as the 2D projection result of the 3D stereo detection frame.

[0015] Among them, the rotation matrix R is:

[0016]

[0017] In the formula, α is the rotation angle of the X-axis, β is the rotation angle of the Y-axis, and θ is the rotation angle of the Z-axis.

[0018] The translation matrix T is as follows:

[0019]

[0020] Where x d , y d , z d are the coordinates of the origin of the lidar coordinate system in the camera coordinate system;

[0021] The conversion relationship from the lidar coordinate system to the camera coordinate system is:

[0022]

[0023] The conversion formula between the camera coordinate system and the pixel coordinate system is:

[0024]

[0025] Where U and V are the coordinate points of a single point in the pixel coordinate system, f is the focal length; dx and dy respectively represent the sizes of the actual physical values represented by a unit pixel; u0 and v0 respectively represent the horizontal and vertical pixel numbers difference between the image center pixel coordinate and the image origin pixel coordinate; H is the camera intrinsic parameter matrix.

[0026] As a preferred example, in step S3, preprocessing the point cloud data includes the following steps:

[0027] S31. Voxelize and sample the original point cloud data;

[0028] S32. Perform pass-through filtering on the voxelized and sampled point cloud data;

[0029] S33. Obtain a plane model by fitting the ground through the RANSAC plane fitting algorithm, and delete the points within the plane model from the point cloud data.

[0030] As a preferred example, in step S4, an adaptive Euclidean clustering algorithm is used to cluster the point cloud data, and the size information of each point cloud after successful clustering is extracted, and then a rectangular box capable of containing the point cloud data is generated according to its size information, and this rectangular box is the three-dimensional vehicle detection box based on the point cloud data.

[0031] As a preferred example, the adaptive Euclidean clustering algorithm stores and sorts the input point cloud data in the way of KD-Tree, and creates a clustering set obstacle{o1, o2,..., o n}, and a temporary storage queue "quene". Then, calculate the clustering distance threshold for each point cloud according to the clustering distance threshold calculation formula, and use KD-Tree for nearest neighbor search. Identify the point cloud data within the same threshold range as a whole to achieve the clustering function;

[0032] Among them, the clustering distance threshold d j The calculation formula is:

[0033]

[0034]

[0035] d j = R j ×sin(λΔα) + σ

[0036] In the formula, x j , y j , z j represent the coordinate information of any point cloud, λ is the angular resolution coefficient, σ is the measurement error of the lidar point cloud, Δα is the angular resolution of the lidar, R j is the ranging result of the lidar point cloud, and R' is the detection distance of the grid corresponding to the lidar point cloud.

[0037] As a preferred example, in step S6, after obtaining the lidar target projection frame and the camera target detection frame, construct the judgment condition for preliminary fusion through the intersection over union (IOU) between the two. If IOU > 0.5, it is determined that the two targets are preliminarily associated and matched;

[0038] Among them, the intersection over union calculation formula is:

[0039]

[0040] In the formula, R lidar is the two-dimensional vehicle detection frame based on the point cloud data, and R camera is the two-dimensional vehicle detection frame based on the visual image.

[0041] As a preferred example, in step S7, define the lidar target trajectory A m = ((a x,1 , a y,1 ),...,(a x,m , a y,m )) and the camera target trajectory B n = ((b x,1 , b y,1 ),...,(b x,n , b y,n)) Calculate the trajectory similarity value using the improved F_LCSS algorithm based on the fuzzy membership function, calculate the trajectory similarity through the trajectory similarity calculation formula, and use it as the judgment criterion. When the trajectory similarity is greater than the set similarity threshold of 50%, it is determined that the two targets are successfully associated fusion targets, and the relevant information is output;

[0042] Among them, the core formula of F_LCSS is:

[0043]

[0044] In the formula, dis(a i , b j ) is the Euclidean distance between points a i and b j , and m is the fuzzy membership function.

[0045] The trajectory similarity calculation formula is:

[0046]

[0047] In the formula, F LCSSε (i, j) is the length of the common subsequence, and min(n, m) is the minimum length value in the lidar target trajectory and the camera target trajectory.

[0048] As a preferred example, the fuzzy membership function is used to calculate the distance and matching degree between each pair of trajectory points;

[0049] Among them, the calculation formula of the fuzzy membership function is:

[0050]

[0051] In the formula, c and d are artificially set distance parameters, and dis(a i , b j ) is the Euclidean distance between points a i and b j .

[0052] As a preferred example, in step S7, if there are multiple two-dimensional vehicle detection frames based on visual images and two-dimensional vehicle detection frames based on point cloud data that meet the intersection-over-union preliminary matching requirements, it is determined that the two targets with the largest local trajectory similarity value are the finally successfully associated sensor targets, and the fusion information is output.

[0053] The present invention also provides a vehicle detection device for target data fusion, which applies the vehicle detection method for target data fusion as described above, and includes:

[0054] A first target detection module based on vision, which is used to obtain two-dimensional target detection frames in visual (RGB) images;

[0055] The second target detection module based on point cloud data is used to obtain the 3D target detection box under point cloud data;

[0056] The projection module is used to project the 3D detection box generated based on point cloud data onto the camera coordinate system;

[0057] The intersection over union calculation module is used for the preliminary fusion of the detection targets of the first target detection module and the second target detection module, and calculates the area overlap ratio between the 2D detection box generated based on point cloud data and the 2D target detection box based on vision in the image coordinate system;

[0058] The trajectory tracking module is used for the final fusion of the detection targets of the first target detection module and the second target detection module.

[0059] The beneficial effects of the vehicle detection method and detection device for target data fusion provided by this application are as follows:

[0060] 1. In the vehicle detection technology, the present invention proposes to replace the YOLOv4 feature extraction network CSPDarkNet-53 with the MobileNet lightweight network, which significantly reduces the model size while ensuring that the target detection accuracy is not significantly reduced, and improves the detection speed, facilitating the deployment of this algorithm on low-computing-power platforms and terminal products.

[0061] 2. In the vehicle detection technology, the present invention proposes to use the adaptive Euclidean clustering algorithm to process point cloud data, which improves the running speed of the clustering algorithm and avoids the under-segmentation or truncation phenomena that are prone to occur in the process of processing distant point cloud data by the traditional Euclidean clustering algorithm, improving the clustering accuracy of point clouds.

[0062] 3. In the vehicle detection technology, the present invention proposes to use the improved longest common subsequence (F_LCSS) algorithm based on the fuzzy membership function to finally fuse the point cloud detection target and the camera detection target. The trapezoidal membership function is used to reduce the probability of fusion failure caused by sensor detection errors, and the trajectory similarity is calculated using the longest common subsequence algorithm, thereby further ensuring the correctness of target fusion at the time level and avoiding phenomena such as false detection and missed detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flowchart of the vehicle detection method for target data fusion based on lidar and camera of the present invention.

[0064] Figure 2 It is a schematic diagram of setting the camera coordinate system and the lidar coordinate system in Embodiment 1 of the present invention.

[0065] Figure 3This is the flowchart for processing lidar point cloud data in Embodiment 1 of the present invention. Detailed implementation manners

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0067] It should be noted that when a component is referred to as being "installed on" another component, it can be directly on the other component or there may also be an intermediate component. When a component is considered to be "disposed on" another component, it can be directly disposed on the other component or there may be an intermediate component at the same time. When a component is considered to be "fixed to" another component, it can be directly fixed to the other component or there may be an intermediate component at the same time.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used herein in the description of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more of the related listed items.

[0069] Embodiment 1

[0070] Please refer to Figure 1 、 Figure 2 and Figure 3, this embodiment provides a vehicle detection method based on the fusion of lidar and camera target data. This method first synchronizes the lidar and camera sensors in time and space; then performs preliminary processing on the point cloud data through voxelization sampling, pass-through filtering, and RANSAC plane fitting algorithms, compressing the point cloud data while retaining the vehicle point cloud information to improve the operation speed, and then uses an adaptive Euclidean clustering algorithm to obtain a three-dimensional vehicle detection box based on the point cloud data; then uses the improved Yolov4 algorithm to perform target detection on the RGB image to obtain a two-dimensional vehicle detection box based on the RGB image; then projects the three-dimensional vehicle detection box based on the point cloud data into the image coordinate system according to the image imaging principle, and performs preliminary fusion on the lidar detection target and the camera detection target through intersection over union calculation; finally, uses the longest common subsequence algorithm improved based on the fuzzy membership function to calculate the trajectory similarity between the lidar detection target trajectory and the camera trajectory within the same time period, realizing the final fusion of the lidar and the camera, generating more accurate multi-dimensional environmental perception data, and normally outputting the working process of installing a single sensor for the detection targets that fail to be successfully fused. Specifically, the vehicle detection method based on the fusion of lidar and camera target data mainly includes steps S1-S9, and the following will elaborate on each step in detail.

[0071] Step S1: Synchronize the camera sensor and the lidar sensor in time and space;

[0072] In this step, time synchronization is performed through multi-threading technology, and spatial synchronization is achieved by deriving the projection matrix from the lidar coordinate system to the image coordinate system based on the camera imaging principle.

[0073] Specifically, the camera sensor and the lidar sensor are installed at the corresponding positions shown in Figure 2 , and it is defined that there is a point [x i y i z i T and [x c y c z c T in the lidar sensor coordinate system and the camera coordinate system respectively. First, the direction of the lidar sensor coordinate system and the camera coordinate system is unified through the rotation matrix R, and then the complete coincidence of the two coordinate systems is achieved through the translation matrix T. The point [x i y i z i T is transformed into in the camera coordinate system, and the camera internal parameter H is obtained through camera calibration technology. The spatial synchronization between the lidar sensor and the camera sensor can be achieved through the rotation matrix R, the translation matrix T, and the camera internal parameter matrix H.

[0074] Among them, the rotation matrix R is as follows:

[0075]

[0076] In the formula, α is the rotation angle of the X-axis, β is the rotation angle of the Y-axis, and θ is the rotation angle of the Z-axis.

[0077] The translation matrix T is as follows:

[0078]

[0079] In the formula, x d , y d , z d are the coordinates of the origin of the laser sensor coordinate system in the camera coordinate system.

[0080] The conversion relationship from the laser sensor coordinate system to the camera coordinate system is:

[0081]

[0082] The conversion formula between the camera coordinate system and the pixel coordinate system is:

[0083]

[0084] In the formula, U and V are the coordinate points of a single point in the pixel coordinate system, f is the focal length; dx and dy respectively represent the sizes of the actual physical values represented by a single pixel; u0 represents the number of horizontal pixels between the image center pixel coordinate and the image origin pixel coordinate, and v0 represents the number of vertical pixels between the image center pixel coordinate and the image origin pixel coordinate; H is the camera internal parameter matrix.

[0085] Step S2: Obtain a visual image through the camera sensor, and perform visual detection through the Yolov4 algorithm improved based on the lightweight network MobileNet to obtain a two-dimensional vehicle detection frame based on the visual image;

[0086] In this step, the MobileNet lightweight network is used to replace the YOLOv4 feature extraction network CSPDarkNet-53, and the depthwise separable convolution technology is used to further reduce the parameter data in the convolutional neural network, so as to realize the lightweight deployment of the Yolov4 object detection algorithm, further improve the running speed of the visual detection algorithm while reducing the model size, so as to facilitate the deployment of this fusion algorithm on platforms with insufficient computing power, and obtain a two-dimensional vehicle detection frame based on the RGB image faster and more conveniently in this application.

[0087] Step S3: Obtain point cloud data through the lidar sensor and preprocess the point cloud data;

[0088] Step S4: Perform clustering analysis on the preprocessed point cloud data through an adaptive Euclidean clustering algorithm to obtain a three-dimensional vehicle detection frame based on the point cloud data.

[0089] In the above steps, the preprocessing process includes voxelization sampling, pass-through filtering, and the RANSAC plane fitting algorithm. While retaining the vehicle point cloud information, the point cloud data is compressed to improve the operation speed. Then, an adaptive Euclidean clustering algorithm is used to obtain a three-dimensional vehicle detection frame based on the point cloud data. The specific acquisition process includes the following steps:

[0090] S31: Perform voxelization sampling on the original point cloud data, with the voxel grid size being 0.1 meters, to ensure maximum compression of the point cloud data while preserving the original features of the point cloud, thus ensuring the real-time performance of the algorithm;

[0091] S32: Perform pass-through filtering on the point cloud data after voxelization sampling. Create a right-handed coordinate system with the centroid of the vehicle as the origin, and only retain the point cloud data in the horizontal direction [-15m, 15m] and the vertical direction [0m, 80m] to further improve the operation speed;

[0092] S33: Obtain a plane model through the RANSAC plane fitting algorithm for ground fitting, and delete the points within the plane model from the point cloud data, so that the preprocessed point cloud data only retains the vehicle point cloud data within the visible range as much as possible, facilitating subsequent point cloud processing;

[0093] S34: Use the adaptive Euclidean clustering algorithm to perform clustering processing on the preprocessed point cloud data, extract the size information of each point cloud after successful clustering, and then generate a rectangular box that can contain the point cloud data according to its size information. This rectangular box is the three-dimensional vehicle detection frame based on the point cloud data.

[0094] Specifically, the adaptive Euclidean clustering algorithm stores and sorts the input point cloud data in the form of a KD-Tree, and creates a clustering set obstacle{o1, o2,..., o n} and a temporary storage queue quene. Then, calculate the clustering distance threshold for each point cloud according to the clustering distance threshold calculation formula, and perform a nearest neighbor search using the KD-Tree. The point cloud data within the same threshold range is recognized as a whole to achieve the clustering function. The process of the adaptive Euclidean clustering algorithm is as follows:

[0095] Step1: For each input frame of point cloud P, create a Kd-Tree object;

[0096] Step2: Assume there are n detection targets, create a clustering set obstacle{o1, o2,..., o n} for storing the point cloud index numbers, and initialize it to be empty;

[0097] Step 3: Create a temporary storage queue quene for the point cloud in the clustering process;

[0098] Step 4: Determine whether the unclassified point cloud in P is in the queue quene. If not, place it in quene; if so, execute Step 5;

[0099] Step 5: According to the formula d j = R j ×sin(λΔα)+σ, calculate the adaptive clustering threshold d j for any point cloud p j in the queue quene;

[0100] Step 6: Use KD-Tree for nearest neighbor search to search for all point clouds within the threshold d j of p j to form a set p J {p1, p2,..., p m}; and determine whether the points in the set p J {p1, p2,..., p m} are in the queue quene. If not, add them to quene;

[0101] Step 7: When the amount of point cloud data in the queue quene no longer changes, place the point cloud in the queue quene into the subset of obstacle {o1, o2,..., o n} to form a clustering set. Empty the queue quene and repeat Step 4 - Step 7 until all point cloud data is processed.

[0102] Among them, the calculation formula for the clustering distance threshold d j is:

[0103]

[0104]

[0105] d j = R j ×sin(λΔα)+σ

[0106] In the formula, x j , y j , z j represent the coordinate information of any point cloud, λ is the angular resolution coefficient, σ is the measurement error of the radar point cloud, Δα is the angular resolution of the lidar, R j is the ranging result of the radar point cloud, and R' is the detection distance of the grid corresponding to the radar point cloud.

[0107] Step S5: Project the 3D vehicle detection box based on the point cloud data according to the above projection matrix. The projected 2D vehicle detection box based on the point cloud data is obtained on the visual image. The 2D vehicle detection box based on the point cloud data and the 2D vehicle detection box based on the visual image are preliminarily fused, and the area overlap degree between the two in the RGB image is calculated. In this step, using the obtained projection matrix, project each vertex of the 3D vehicle detection box based on the point cloud data onto the pixel coordinate system, and take the largest rectangular box formed by these points in the pixel coordinate system as the 2D vehicle detection box based on the point cloud data.

[0108] Step S6: Determine whether the above area overlap degree is higher than the set threshold through the intersection over union (IoU) operation. When the area overlap degree is greater than the set threshold, perform preliminary matching on it.

[0109] In this step, the IoU calculation formula used is:

[0110]

[0111] where R lidar is the 2D vehicle detection box based on the point cloud data, and R camera is the 2D vehicle detection box based on the visual image. Set the fusion judgment threshold to 0.5. When it is greater than this threshold, the lidar sensor and the camera sensor are preliminarily fused.

[0112] Step S7: For the targets with successful preliminary matching, define the lidar target trajectory A m = ((a x,1 , a y,1 ),...,(a x,m , a y,m )) and the camera target trajectory B n = ((b x,1 , b y,1 ),...,(b x,n , b y,n )) Use the longest common subsequence algorithm to calculate the trajectory similarity value, and calculate the trajectory similarity through the trajectory similarity calculation formula, and use it as the judgment criterion. When the trajectory similarity is greater than the set similarity threshold of 50%, the two targets are identified as the finally successfully associated sensor targets, and the fusion information is output. For the targets that fail to be successfully associated, determine the two targets with the largest similarity as the finally successfully associated sensor targets.

[0113] It should be noted that if multiple two-dimensional vehicle detection frames based on visual images and two-dimensional vehicle detection frames based on point cloud data meet the intersection-over-union preliminary matching requirements, the two targets with the largest local trajectory similarity value are determined as the finally successfully associated fusion targets. For the successfully associated fusion targets, fusion information is output, including lidar detection position information, camera detection frame, detection category, and confidence information. For the targets that fail to be successfully associated, the detection position information of the lidar sensor, the visual image of the camera sensor, and the camera detection frame information are respectively output and stored in the log.

[0114] In this embodiment, the longest common subsequence method is used for calculation. However, since the calculation accuracy of the traditional longest common subsequence algorithm depends on the artificially set distance threshold, its accuracy in calculating the trajectory similarity value is relatively low in the face of complex trajectories, and it often leads to the loss of effective information as a criterion for target association. Therefore, this application also proposes an improved method: introducing a fuzzy membership function to replace the fixed distance threshold for calculation. The improved longest common subsequence ((F_LCSS)) algorithm can, on the one hand, reduce the influence of the fixed threshold on the trajectory similarity value, and on the other hand, effectively reduce the influence of the sensor ranging error on the point matching degree and the overall similarity of the two trajectories.

[0115] Among them, the calculation formula of the fuzzy membership function is:

[0116]

[0117] In the formula, c and d are artificially set distance parameters, and dis(a i ,b j ) is the Euclidean distance between point a i and point b j .

[0118] The core formula of F_LCSS is:

[0119]

[0120] In the formula, dis(a i ,b J ) is the Euclidean distance between point a I and point b J , and m is the fuzzy membership function.

[0121] The trajectory similarity calculation formula is:

[0122]

[0123] In the formula, is the length of the common subsequence, and min(n, m) is the minimum length value in the lidar target trajectory and the camera target trajectory.

[0124] Example 2

[0125] This embodiment provides a vehicle detection device for target data fusion. This detection device applies the vehicle detection method based on lidar and camera target data fusion in Embodiment 1, and specifically includes a first target detection module based on vision, a second target detection module based on point cloud data, a projection module, an intersection over union calculation module, and a trajectory tracking module.

[0126] The first target detection module based on vision obtains a two-dimensional target detection box in the RGB image through an improved target detection algorithm;

[0127] The second target detection module based on point cloud data obtains a three-dimensional target detection box based on the point cloud data through point cloud data compression processing and an adaptive Euclidean clustering algorithm;

[0128] In this embodiment, the first target detection module selects a camera, and the second target detection module selects a lidar. For the convenience of understanding, the camera and lidar are directly referred to in the following description.

[0129] The projection module calculates the transformation matrix from the lidar coordinate system to the pixel coordinate system through the camera internal parameter matrix obtained by the camera calibration algorithm and the relative positions of the lidar and the camera installation, so as to realize the transformation from the three-dimensional target detection box based on the point cloud data to the two-dimensional target detection box based on the point cloud data;

[0130] The intersection over union calculation module realizes the preliminary fusion of the lidar and the camera by calculating the area overlap ratio between the two-dimensional detection box generated based on the point cloud data and the two-dimensional target detection box based on vision in the pixel coordinate system;

[0131] The trajectory tracking module is used for the final fusion of the lidar detection target and the camera detection. It improves the longest common subsequence algorithm by replacing the fixed threshold with a fuzzy membership function, calculates the trajectory similarity according to the improved longest common subsequence algorithm, and realizes the final fusion based on its calculation results.

[0132] Combined with Embodiment 1 and Embodiment 2, compared with the multi-source sensor fusion vehicle detection scheme in the prior art, the present invention proposes that in the vehicle detection technology, in order to facilitate the deployment of this algorithm on a low-computing-power platform and terminal products, the original feature extraction network of the Yolov4 algorithm is replaced with the lightweight network MobileNet proposed by Google, and depthwise separable convolution is used to replace the original convolution algorithm. While retaining the original detection accuracy of the Yolov4 algorithm, the operation speed of the model is greatly improved.

[0133] Meanwhile, in the vehicle detection technology, to avoid the under-segmentation or truncation phenomena that are prone to occur in the traditional Euclidean clustering algorithm during the processing of distant point cloud data, an adaptive Euclidean clustering algorithm is used to process the point cloud data, which improves the clustering accuracy of the point cloud while increasing the running speed of the algorithm.

[0134] It is also proposed that in the vehicle detection technology, to avoid the disadvantages such as low robustness and sensitivity to sensor ranging errors caused by the traditional longest common subsequence algorithm using a fixed distance threshold, a fuzzy membership function is introduced to replace the fixed distance threshold, and a longest common subsequence (F_LCSS) algorithm improved based on the fuzzy membership function is proposed for the final fusion of point cloud detection targets and camera detection targets. This significantly reduces the probability of fusion failure caused by sensor detection errors and improves the robustness and accuracy of the fusion algorithm, meeting the complex perception requirements in the actual driving process of intelligent vehicles for autonomous driving.

[0135] Embodiment 3

[0136] The present invention also provides a computer terminal, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the vehicle detection method based on the fusion of lidar and camera target data in Embodiment 1 are implemented.

[0137] When the vehicle detection method based on the fusion of lidar and camera target data is applied, it can be applied in the form of software, such as designed as an independently running program and installed on a computer terminal, which can be a computer, a smart phone, a control system, and other Internet of Things devices, etc. The vehicle detection method based on the fusion of lidar and camera target data can also be designed as an embedded running program and installed on a computer terminal, such as installed on a single-chip microcomputer.

[0138] Embodiment 4

[0139] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the vehicle detection method based on the fusion of lidar and camera target data in Embodiment 1 are implemented. When the vehicle detection method based on the fusion of lidar and camera target data is applied, it can be applied in the form of software, such as designed as an independently running program on the computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, or a storage medium in the form of a USB key, or a program designed to start the whole method by external triggering through a USB flash drive.

[0140] The above-described embodiments merely represent several implementation manners of the present invention. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A vehicle detection method for target data fusion, characterized in that, It includes the following steps: Step S1: Synchronize the camera sensor and the lidar sensor in space and time; Step S2: Obtain a visual image through the camera sensor, and perform visual detection through the Yolov4 algorithm improved based on the lightweight network MobileNet to obtain a two-dimensional vehicle detection frame based on the visual image; Step S3: Obtain point cloud data through the lidar sensor and preprocess the point cloud data; Step S4: Perform clustering analysis on the preprocessed point cloud data through the adaptive Euclidean clustering algorithm to obtain a three-dimensional vehicle detection frame based on the point cloud data; Step S5: Project the three-dimensional vehicle detection frame based on the point cloud data according to the projection matrix to obtain a two-dimensional vehicle detection frame based on the point cloud data, preliminarily fuse the two-dimensional vehicle detection frame based on the point cloud data and the two-dimensional vehicle detection frame based on the visual image, and calculate the area overlap degree of the two in the visual image; Step S6: Determine whether the area overlap degree is higher than the set threshold through the intersection over union operation; when the area overlap degree is greater than the set threshold, perform preliminary matching on it; Step S7: For the targets that are preliminarily matched successfully, record their trajectory information under the lidar sensor and the trajectory information under the camera sensor respectively, and define the lidar target trajectory and the camera target trajectory , use the improved F_LCSS algorithm based on the fuzzy membership function to calculate the trajectory similarity value, calculate the trajectory similarity through the trajectory similarity calculation formula, and use it as the judgment criterion. When the trajectory similarity is greater than the set similarity threshold of 50%, it is determined that the two targets are successfully associated fusion targets, and the relevant information is output; for the targets that fail to be successfully associated, the processing results of the lidar sensor and the camera sensor are output respectively; The core formula of F_LCSS is: Wherein, is and the Euclidean distance between two points, is the fuzzy membership function; The trajectory similarity calculation formula is: In the formula, is the length of the common subsequence, is the minimum length value in the lidar target trajectory and the camera target trajectory.

2. The vehicle detection method for target data fusion according to claim 1, wherein In step S1, the method for spatial synchronization between the camera sensor and the lidar sensor is as follows: Define a point on each of the lidar coordinate system and the camera coordinate system , . First, the direction of the lidar coordinate system and the camera coordinate system is unified through the rotation matrix , and then the complete coincidence of the two coordinate systems is achieved through the translation matrix . The point is transformed into in the camera coordinate system. Finally, the conversion formula between the camera coordinate system and the pixel coordinate system can be used to project the vehicle detection frame based on the point cloud data onto the visual image, and the maximum values of the horizontal and vertical coordinates of the projected pixel points are taken as the two-dimensional projection result of the three-dimensional stereo detection frame; Among them, the rotation matrix is as follows: wherein is the rotation angle about the X-axis, is the rotation angle about the Y-axis, is the rotation angle about the Z-axis; Translation matrix is as follows: where , , are the coordinates of the origin of the lidar coordinate system in the camera coordinate system; The conversion relationship from the lidar coordinate system to the camera coordinate system is: The conversion formula between the camera coordinate system and the pixel coordinate system is: In the formula, , are the coordinate points of a single point in the pixel coordinate system, is the focal length; , respectively represent the size of the actual physical value represented by a unit pixel; , respectively represent the number of horizontal and vertical pixels between the pixel coordinates of the image center and the pixel coordinates of the image origin; is the camera internal parameter matrix.

3. The vehicle detection method for target data fusion according to claim 1, wherein In step S3, the preprocessing of the point cloud data includes the following steps: S31: Perform voxelization sampling on the original point cloud data; S32: Perform direct filtering on the point cloud data after voxelization sampling; S33: Obtain a plane model through the RANSAC plane fitting algorithm for ground fitting, and delete the points within the plane model from the point cloud data.

4. The vehicle detection method for target data fusion according to claim 1, characterized in that, In step S4, use the adaptive Euclidean clustering algorithm to perform clustering processing on the point cloud data, extract the size information of each point cloud after successful clustering, and then generate a rectangular frame that can contain the point cloud data according to its size information. This rectangular frame is the three-dimensional vehicle detection frame based on the point cloud data.

5. The vehicle detection method for target data fusion according to claim 4, characterized in that, The adaptive Euclidean clustering algorithm stores and sorts the input point cloud data in the form of a KD-Tree and creates a clustering set and a temporary storage queue , then calculates the clustering distance threshold of each point cloud according to the clustering distance threshold calculation formula, and uses the KD-Tree for nearest neighbor search. The point cloud data within the same threshold range is identified as a whole to achieve the clustering function; Among them, the clustering distance threshold The calculation formula is as follows: where , , represent the coordinate information of any point cloud, is the angular resolution coefficient, is the measurement error of the radar point cloud, is the angular resolution of the lidar, is the ranging result of the radar point cloud, is the detection distance of the grid corresponding to the radar point cloud.

6. The vehicle detection method for target data fusion according to claim 1, wherein In step S6, after obtaining the lidar target projection box and the camera target detection box, the intersection over union between the two is used to construct the judgment condition for preliminary fusion. If then it is determined that the two targets are preliminarily associated and matched; Among them, the intersection over union calculation formula is: where , is a two-dimensional vehicle detection box based on visual images.

7. The vehicle detection method for target data fusion according to claim 1, wherein Use the fuzzy membership function to calculate the distance and matching degree between each pair of trajectory points; Among them, the fuzzy membership function calculation formula is: where c and d are artificially set distance parameters, is and the Euclidean distance between two points.

8. The vehicle detection method for target data fusion according to claim 1, characterized in that, In step S7, if there are multiple two-dimensional vehicle detection frames based on the visual image and two-dimensional vehicle detection frames based on the point cloud data that meet the preliminary matching requirements of the intersection over union, determine that the two targets with the largest local trajectory similarity value are the finally successfully associated sensor targets, and output the fusion information.

9. A vehicle detection device for target data fusion, which applies the vehicle detection method for target data fusion according to any one of claims 1 to 8, characterized in that, It includes: The first target detection module based on vision, which is used to obtain the two-dimensional target detection frame in the visual image; The second target detection module based on point cloud data, which is used to obtain the three-dimensional target detection frame under the point cloud data; The projection module, which is used to project the three-dimensional detection frame generated based on the point cloud data into the camera coordinate system; The intersection over union calculation module, which is used for the preliminary fusion of the detection targets of the first target detection module and the detection targets of the second target detection module, and calculates the area overlap ratio between the two-dimensional detection frame generated based on the point cloud data and the two-dimensional target detection frame based on vision in the image coordinate system; A trajectory tracking module for the final fusion of the detection targets of the first target detection module and the second target detection module.

Images