Vehicle identification method based on laser radar and visual information fusion

Abstract

The invention discloses a vehicle identification method based on laser radar and visual information fusion, and the method comprises the steps: carrying out the preprocessing of target vehicle driving data obtained based on sensing equipment, and carrying out the feature combination of image features and point cloud features; extracting a motion feature description factor by using the target vehicle driving data after feature combination according to trajectory sampling point speed and driving direction information, and constructing a target feature vector; and according to the constructed target feature vector, based on a supervised class learning method and in combination with a random forest model, identifying a target vehicle type, and outputting an identification result. According to the method, target vehicle driving data acquired by sensing equipment is preprocessed, image and point cloud features are combined, motion feature description factors are extracted to construct feature vectors, and then the vehicle type is identified based on supervised learning and a random forest model. The accuracy and robustness of target vehicle type identification are improved, and reliable target identification support is provided for an unmanned driving system.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle recognition, and particularly relates to a vehicle recognition method based on the fusion of lidar and visual information. Background Art

[0002] Currently, existing vehicle type recognition algorithms mainly rely on cameras or lidar to extract vehicle features and construct corresponding classifiers to achieve accurate recognition of vehicle types. Although this method reduces the computational cost to a certain extent and makes it more efficient in practical applications, it also exposes some obvious problems. Since these algorithms rely solely on cameras or lidar, they are easily interfered by various external factors such as light changes, obstacles, and weather conditions, resulting in unstable recognition results. In addition, in the complex and changeable open-pit mine environment, this single recognition method often fails to ensure a high recognition accuracy, posing challenges to the safety and reliability of the unmanned driving system.

[0003] The deficiencies of the existing technologies are that they fail to fully overcome the influence of external environmental interference on the recognition accuracy. Although cameras and lidar each have certain advantages, their performance often drops significantly when facing extreme weather, complex terrain, or insufficient lighting conditions. This not only limits the wide application of the unmanned driving system in complex environments such as open-pit mines but also may lead to system misjudgment, thus triggering potential safety hazards. Therefore, in order to improve the accuracy of the unmanned driving system in open-pit mines for recognizing the vehicle front environment, it is urgent to develop more advanced and stable vehicle type recognition technologies. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the existing technologies. To achieve the above purpose, a vehicle recognition method based on the fusion of lidar and visual information is adopted to solve the problems raised in the above background art.

[0005] A vehicle recognition method based on the fusion of lidar and visual information includes the following steps:

[0006] S1. After preprocessing the target vehicle driving data obtained based on the sensing device, perform feature combination of image features and point cloud features;

[0007] S2. Using the target vehicle driving data after feature combination, extract motion feature description factors according to the speed and driving direction information of the trajectory sampling points, and construct a target feature vector;

[0008] S3. Based on the constructed target feature vector, perform recognition of the target vehicle type based on the supervised learning method and in combination with the random forest model, and output the recognition result.

[0009] As a further solution of the present invention: The specific steps in S1 include:

[0010] Step S11: Obtain the coordinate system space conversion relationship according to the internal and external parameters of the camera and the position relationship between the radar and the camera;

[0011] Step S12: Filter the background by the exponential weighting method;

[0012] Step S13: Perform preprocessing operations on the signals received by the radar to filter out the noise of the target vehicle therein; and

[0013] Step S14: Process the window image using the Canny edge detection operator to extract its edge information, generate a binary edge image, and then perform symmetry analysis on the obtained edge image;

[0014] Step S15: After feature extraction, obtain an image feature and a point cloud feature at each moment through the histogram of gradients; concatenate the two types of features to generate a joint feature with feature complementarity.

[0015] As a further solution of the present invention: The specific steps in S12 include:

[0016] Assign weights to the historical background templates at the current moment and the previous moment respectively by the exponential weighting method. The formula is:

[0017] y t =αy t-1 +(1 - α)r t , 0 ≤ α < 1;

[0018] In the formula, r t is the data at the current moment, and α is the exponential weighting factor, which determines the weight ratio of the historical background and the current moment; as time increases, the background farther from the current moment has less influence.

[0019] As a further solution of the present invention: The specific steps in S2 include:

[0020] Step S21: Calculate the basic information of the driving direction and speed of each sampling point by correlating the spatio-temporal information of adjacent sampling points in the associated trajectory and applying the uniform motion model;

[0021] Step S22: Based on the speed and driving direction of the trajectory sampling points, use the feature extraction algorithm to extract the high-order feature factors that uniformly describe the motion characteristics of the target vehicle;

[0022] Step S23: Standardize and combine the extracted high-order feature factors to construct a target feature vector with a unified dimension.

[0023] As a further solution of the present invention: The specific steps of the high-order feature factor extraction and processing include:

[0024] Define the target trajectory as Z = [z 1 , z 2 , …, z N , where z i (i = 1, 2, …, N) represents the i-th sampling point on the trajectory, and N is the trajectory length. The speed and driving direction corresponding to the trajectory Z are respectively described in vector form as V = [v 1 , v 2 , …, v N and H = [h 1 , h 2 , …, h N ;

[0025] Define the average speed as the average of all speeds in the speed vector V, and the formula is:

[0026]

[0027] Use the speed standard deviation to describe the speed change characteristics of the target, and the formula is:

[0028]

[0029] Use the deflection angle deviation of the driving direction to describe the uncertainty of the target's spatial position, and the formula is:

[0030]

[0031] Use the maneuverability factor to quantitatively describe the maneuverability of the target, and the formula is:

[0032]

[0033] By calculating the relative difference value of the driving direction between each adjacent sampling point in the trajectory, obtain a quantitative description of the trajectory fluctuation characteristics by the fluctuation factor, and the formula is:

[0034]

[0035] Use the trajectory motion feature description factor to construct the target motion feature vector, and the formula is:

[0036] S = [v mean , v std , h std , h mean , σ, ξ].

[0037] As a further solution of the present invention: The specific steps in S3 include:

[0038] Step S31: Use the trajectory motion feature description factor to construct a feature vector that reflects the motion characteristics of the target.

[0039] Step S32: Perform normalized numerical processing on all trajectory features to ensure a unified description standard for maneuverability among different trajectories.

[0040] Step S33: Use the constructed feature vector, combined with the supervised learning method and the random forest model, to identify the type of the target vehicle and output the identification result.

[0041] Compared with the prior art, the present invention has the following technical effects:

[0042] With the above technical solution, by preprocessing the driving data of the target vehicle obtained by the sensing device, then combining the image features and the point cloud features, extracting the motion feature description factors in the speed and driving direction information of the trajectory sampling points, and constructing the target feature vector. Finally, based on the supervised learning method, combined with the random forest model, the accurate identification of the target vehicle type is realized and the identification result is output. The feature combination enhances the comprehensiveness and accuracy of the data, and improves the accuracy of vehicle type identification. Combining the supervised learning and the random forest model realizes efficient and stable identification, providing reliable support for the unmanned driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The following will describe in detail the specific embodiments of the present invention with reference to the accompanying drawings:

[0044] Figure 1 It is a schematic diagram of the steps of the vehicle identification method according to the disclosed embodiment of the present application;

[0045] Figure 2 It is a flowchart of the vehicle identification method according to the disclosed embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0047] Please refer to Figure 1 and Figure 2 In the embodiments of the present invention, a vehicle identification method based on the fusion of lidar and visual information includes the following steps:

[0048] S1: After preprocessing the driving data of the target vehicle obtained based on the sensing device, perform feature combination of image features and point cloud features. The specific steps include:

[0049] Step S11: Obtain the coordinate system space conversion relationship according to the internal and external parameters of the camera and the positional relationship between the radar and the camera;

[0050] In the specific implementation steps, after the millimeter-wave radar and the monocular camera are installed, parameter calibration is carried out. The camera calibration uses the standard checkerboard corner point extraction method.

[0051] Step S12: Filter the background by the exponential weighting method. The specific steps include:

[0052] Generally, the vehicle moves with a large amplitude while the background changes little. The background can be filtered by the exponential weighting method.

[0053] Weights are assigned to the historical background template at the current moment and the previous moment respectively by the exponential weighting method. The formula is:

[0054] y t = αy t-1 +(1 - α)r t , 0 ≤ α < 1;

[0055] In the formula, r t is the data at the current moment, and α is the exponential weighting factor, which determines the weight ratio of the historical background and the current moment; as time increases, the influence of the background farther from the current moment becomes smaller.

[0056] Step S13: Perform preprocessing operations on the signals received by the radar to filter out the noise of the target vehicle therein;

[0057] Specifically, the original target signal of the lidar contains various detected obstacles. Therefore, it is necessary to preprocess the signal that activates the radar first to eliminate the target noise.

[0058] Step S14: Use a fixed-size window as the sampling template to perform horizontal sampling on the region of interest in the front to obtain different image window sequences. Due to different illuminations and changes in the camera angle, the Canny edge detection operator is used to process the window images to extract their edge information, generate a binary edge image, and then perform symmetry analysis on the obtained edge image;

[0059] Step S15: After feature extraction, obtain an image feature and a point cloud feature at each moment through the histogram of gradients; concatenate the two types of features to generate a joint feature with complementary features.

[0060] Specifically, after feature extraction, an image feature A S and a point cloud feature A EAlthough the two features come from different types of sensors, both the image feature and the point cloud feature are obtained through the histogram of gradients and have the same physical meaning in the corresponding dimensions.

[0061] Concatenate the two types of features to generate a combined feature A with complementary features, A = A S ∪A E 。

[0062] S2. Using the driving data of the target vehicle after feature combination, according to the speed and driving direction information of the trajectory sampling points, extract the motion feature description factors and construct the target feature vector. The specific steps include:

[0063] Step S21. By correlating the spatio-temporal information of adjacent sampling points in the trajectory and applying the uniform motion model, calculate the basic information of the driving direction and speed of each sampling point;

[0064] Step S22. Based on the speed and driving direction of the trajectory sampling points, use the feature extraction algorithm to extract the high-order feature factors that uniformly describe the motion characteristics of the target vehicle;

[0065] Step S23. Standardize and combine the extracted high-order feature factors to construct a target feature vector with a unified dimension.

[0066] In this embodiment, the existing unmanned driving perception subsystem all adopts the steps of "detection - tracking - recognition". In the perception data, the target object trajectory is composed of trajectory sampling point vectors, and the sampling points contain time and space information.

[0067] Considering that the rotation rate of the lidar is very high relative to the driving speed of the ground vehicle, it can be approximately considered that the target vehicle moves uniformly between adjacent sampling points, and the driving direction is the vector of adjacent sampling points.

[0068] Therefore, the driving direction and speed information of each trajectory sampling point can be calculated by correlating the spatio-temporal information of adjacent points in the trajectory and combining the target motion model.

[0069] However, there are the following limitations in directly using the speed and driving direction information of the trajectory sampling points to describe the motion characteristics of the target vehicle:

[0070] The inconsistent length of the target trajectory leads to non-uniform dimensions of the feature vector;

[0071] The speed and driving direction respectively describe the first-order motion characteristics of the target in the spatio-temporal dimensions, resulting in weak correlation.

[0072] Therefore, it is necessary to further explore the target motion characteristics contained in the trajectory and construct a feature space with a unified dimension in order to enable the subsequent module to accurately identify the target vehicle.

[0073] The motion feature extraction algorithm based on the target trajectory extracts the motion feature description factors according to the speed of the trajectory sampling points and the driving direction information, and constructs the target feature vector.

[0074] Specifically, the specific steps for extracting and processing the high-order feature factors include:

[0075] Define the target trajectory as Z = [z 1 , z 2 , …, z N , where z i (i = 1, 2, …, N) represents the i-th sampling point on the trajectory, and N is the trajectory length. The speed and driving direction corresponding to the trajectory Z are respectively described in vector form as V = [v 1 , v 2 , …, v N and H = [h 1 , h 2 , …, h N ;

[0076] Define the average speed as the average of all speeds in the speed vector V, and the formula is:

[0077]

[0078] The average speed is suitable for distinguishing targets with significantly different motion characteristics, such as driverless vehicles and auxiliary vehicles, but it is not suitable for distinguishing manned mining vehicles and driverless mining vehicles. It can usually be used as a basic description and combined with other description factors to describe the motion characteristics of the target. For example: For refueling vehicles, except for the refueling mode, the average driving speed of the vehicle is relatively close in most cases, while the average speed of the trajectory of a driverless mining vehicle is related to the uphill and downhill and the load size, and usually the driving speed is slower than that of refueling vehicles and sprinkler trucks.

[0079] Use the standard deviation of speed to describe the speed change characteristics of the target, and the formula is:

[0080]

[0081] Use the deviation of the driving direction deflection angle to describe the uncertainty of the target's spatial position, and the formula is:

[0082]

[0083] There is a large range of variation in the standard deviation of the target vehicle's speed, which has the potential to distinguish target vehicles. For example: The speed of a sprinkler truck usually changes little during normal driving; the speed of a driverless mining vehicle is relatively uniform during flat driving, however, in different loading, unloading, and operation modes, the uncertainty characteristics of its driving speed are relatively complex, and there is a certain change in its driving speed when going uphill and downhill.

[0084] The quantitative description of the target maneuverability using the maneuver factor is given by the formula:

[0085]

[0086] In this embodiment, the maneuver factor:

[0087] When a high-speed vehicle exhibits a large direction deflection, it indicates a strong maneuverability. This maneuverability is a characterization of the motion characteristics of the target vehicle in the spatio-temporal joint dimension. The maneuver factor is used to achieve a quantitative description of the target maneuverability.

[0088] By calculating the relative difference value of the driving direction between each adjacent sampling point in the trajectory, a quantitative description of the trajectory fluctuation characteristics is obtained, and the fluctuation coefficient = overall standard deviation / overall average. The formula is:

[0089]

[0090] In this embodiment, the fluctuation factor:

[0091] By calculating the relative difference value of the driving direction between each adjacent sampling point in the trajectory, a quantitative description of the trajectory fluctuation characteristics is achieved.

[0092] Using the trajectory motion characteristic description factor, a target motion characteristic vector is constructed, and the formula is:

[0093] S = [v mean , v std , h std , h mean , σ, ξ].

[0094] S3. Based on the constructed target feature vector, using the supervised learning method and combined with the random forest model, identify the type of the target vehicle and output the identification result. The specific steps include:

[0095] Step S31. Using the trajectory motion characteristic description factor, construct a feature vector that reflects the target motion characteristics;

[0096] Step S32. Perform normalized numerical processing on all trajectory features to ensure a unified description standard for maneuverability among different trajectories;

[0097] Step S33. Using the constructed feature vector, combined with the supervised learning method and the random forest model, identify the type of the target vehicle and output the identification result.

[0098] The constructed target motion characteristic vector:

[0099] S = [v mean , v std , h std, h mean , σ, ξ]

[0100] Among them: for the unified description of maneuverability, normalized numerical processing is adopted, that is, the normalized value is calculated by defining the numerical range.

[0101] The vector S contains the motion characteristics of the target vehicle in different dimensions, which can ensure that the feature vectors of different trajectory lengths have the same dimension.

[0102] Beneficial effects:

[0103] (1) Although lidar can provide a target trajectory with high accuracy, environmental interference cannot guarantee 100% trajectory accuracy. Therefore, the sample set inevitably has uncertainties. The advantage of random forest is that it can perform repeated random sampling on some features. By constructing different decision tree classifiers, the model has strong anti-noise ability and a lower overfitting rate.

[0104] (2) The computational complexity of random forest is usually much smaller than that of other classification models. Given the complexity of vehicle motion patterns, vehicle type recognition usually requires adjusting the target trajectory training set. In practical scenarios, the random forest model has advantages in terms of efficiency and sample flexibility.

[0105] (3) The vehicle recognition rate of the experimental results is 92.1%, showing good stability in various weather conditions and road environments. The average total processing time of single-frame fusion data is 35 mS, which can effectively improve the real-time processing speed on the premise of ensuring the recognition accuracy. It also reduces the hardware requirements for in-vehicle computing platforms, meeting the high requirements of driverless technology for accuracy and stability.

[0106] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the protection scope of the present invention.

Claims

1. A vehicle recognition method based on laser radar and visual information fusion, characterized in that: The following steps are involved: S1. After preprocessing the target vehicle driving data obtained by the sensor device, the image features and the point cloud features are combined; S2, using the target vehicle driving data after feature combination, extracting motion feature description factors according to the speed and driving direction information of the trajectory sampling points, and constructing the target feature vector; S3. According to the constructed target feature vector, based on the supervised learning method and combined with the random forest model, the target vehicle type is identified and the identification result is output.

2. The vehicle identification method based on laser radar and visual information fusion according to claim 1 is characterized in that: The specific steps in S1 include: Step S11, obtaining a coordinate system space conversion relationship according to the internal and external parameters of the camera and the positional relationship between the radar and the camera; Step S12, filtering the background by exponential weighting method; Step S13, performing a preprocessing operation on the signal received by the radar to filter out the noise of the target vehicle; and Step S14, using the Canny edge detection operator to process the window image to extract its edge information, generate a binary edge image, and then perform symmetry analysis on the obtained edge image; Step S15: After feature extraction, an image feature and a point cloud feature are obtained at each moment through the gradient histogram; the two types of features are connected in series to generate a joint feature with feature complementarity.

3. The vehicle identification method based on laser radar and visual information fusion according to claim 2 is characterized in that: The specific steps in S12 include: The exponential weighting method is used to assign weights to the historical background templates at the current moment and the previous moment respectively. The formula is: y t =αy t-1 +(1-a)r t ,0≤α<1; In the formula, r t is the current moment data, α is the exponential weighting factor, which determines the weight ratio of historical background and current moment; as time goes by, the background farther away from the current moment has less impact.

4. The vehicle identification method based on laser radar and visual information fusion according to claim 1 is characterized in that: The specific steps in S2 include: Step S21, by associating the spatiotemporal information of adjacent sampling points in the trajectory and applying the uniform motion model, the basic information of the driving direction and speed of each sampling point is calculated; Step S22, based on the speed and driving direction of the trajectory sampling points, using a feature extraction algorithm, extracting high-order feature factors that uniformly describe the motion characteristics of the target vehicle; Step S23: standardize and combine the extracted high-order feature factors to construct a target feature vector with uniform dimension.

5. The vehicle identification method based on laser radar and visual information fusion according to claim 4 is characterized in that: The specific steps of extracting and processing the high-order characteristic factors include: Define the target trajectory as Z = [z1, z2, ..., z N ], where z i (i=1,2,…,N) represents the i-th sampling point on the trajectory, and N is the trajectory length. The speed and direction of travel corresponding to the trajectory Z are described in vector form as V=[v1,v2,…,v N ] and H = [h1, h2, ..., h N ]; The average velocity is defined as the average of all velocities in the velocity vector V, and the formula is: The speed standard deviation is used to describe the speed change characteristics of the target. The formula is: The uncertainty of the target spatial position is described by using the deviation of the driving direction deflection angle. The formula is: The maneuverability factor is used to quantitatively describe the maneuverability of the target. The formula is: By calculating the relative difference in driving direction between each adjacent sampling point in the trajectory, the quantitative description of the trajectory fluctuation characteristics by the fluctuation factor is obtained. The formula is: The trajectory motion feature description factor is used to construct the target motion feature vector. The formula is: S=[v mean ,v std ,h std ,h mean ,s,ξ].

6. The vehicle identification method based on laser radar and visual information fusion according to claim 1 is characterized in that: The specific steps in S3 include: Step S31, using the trajectory motion feature description factor to construct a feature vector reflecting the target motion characteristics; Step S32: normalize all trajectory features to ensure that the maneuverability has a unified description standard among different trajectories; Step S33: using the constructed feature vector, combined with the supervised learning method and the random forest model, identify the type of the target vehicle and output the identification result.

Images