Intelligent vehicle multi-sensor information efficient automatic classification fusion method

Abstract

The application relates to a kind of intelligent vehicle multi-sensor information efficient automatic classification fusion methods, belong to intelligent transportation field, including: using multiple sensors to collect different information, obtain the information rate of each sensor;Automatic classification is carried out on sensor data, if the frequency value of its information rate is greater than the set frequency classification decision threshold C, it is considered as high-speed sensor, and is classified into high-speed sensor data class, otherwise, it is classified into low-speed sensor data class;The data of high-speed sensor is fused;The data of low-speed sensor is fused;The data after fusion of high-speed sensor is adjusted with the data after fusion of low-speed sensor Soft alignment data stacking is carried out.The application can make full use of the information of multiple sensors, reduce the limitation of single sensor data, improve the fusion accuracy of vehicle information;Different frequency sensor data is fused respectively, to avoid data loss and precision decline caused by too large difference in data frequency in traditional method.

Description

TECHNICAL FIELD

[0001] The application relates to an intelligent vehicle multi-sensor information efficient automatic classification fusion method and belongs to the technical field of intelligent transportation. BACKGROUND

[0002] An intelligent vehicle is an important application of an outdoor mobile robot in the field of transportation and is an important component of an intelligent transportation system. The intelligent vehicle identifies the environment and state of the vehicle through sensor technology, so that the vehicle can be automatically operated. However, as the information processing capacity of the intelligent vehicle is enhanced, some problems may occur when the multi-sensor information collection technology is applied to the vehicle body. As the amount of collected information increases, the information collected by different sensor devices is greatly different in information rate, data type and representative meaning. Due to the difference, the data fusion process in the later stage will encounter great difficulty, resulting in the reduction of the amount of effective information data. Therefore, how to ensure the maximum superposition fusion of the multi-sensor collected information of the intelligent vehicle and increase the amount of effective information data after fusion has become a difficult problem. SUMMARY

[0003] In order to solve the problems in the prior art, the application provides an intelligent vehicle multi-sensor information efficient automatic classification fusion method, which can fully utilize the information of multiple sensors, reduce the limitation of single sensor data and improve the fusion precision of vehicle information. The sensor data of different frequencies are fused respectively, so that the data loss and precision reduction caused by the great difference in data frequency in the traditional method are avoided.

[0004] The technical scheme of the application is as follows:

[0005] Parameter declaration:

[0006] The frequency classification decision threshold is C, which is an adjustable parameter and is used to distinguish high-speed sensors from low-speed sensors. Since the information rate of commonly used high-speed sensors such as IMU and encoder is more than 100 times that of commonly used low-speed sensors such as laser radar and camera, the parameter can be set to 1 / 5 of the information rate of the high-speed sensor with the lowest information rate or 5 times the information rate of the low-speed sensor with the highest information rate. However, the parameter should be adjusted according to the characteristics of the used sensors, so that the two types of sensors can be effectively distinguished.

[0007] The soft alignment decision time threshold is D.

[0008] The reference data frequency is f.

[0009] The target obstacle occupies the picture area S.

[0010] The distance between the target obstacle and the vehicle is L.

[0011] Obstacle coefficient: k, the coefficient is measured according to the size of the obstacle;

[0012] The calibration method of k:

[0013] Each obstacle is placed 1 meter in front of the vehicle camera, and the area S is calibrated by the obstacle recognition frame, and then S=k*(1 2 ), the value of k is obtained (that is, S is equal to the value of k at this time), and the k value is calibrated, and the picture area S is solved by using the k value in subsequent calculation;

[0014] Distance judgment deviation tolerance threshold: l, l is an autonomous parameter, and the smaller l is, the more strict the matching is.

[0015] The application adopts the following technical scheme:

[0016] An efficient automatic classification and fusion method of intelligent vehicle multi-sensor information, comprising:

[0017] Step 1: collect different information using multiple sensors, and sample the information to obtain the information rate of each sensor;

[0018] Step 2: automatically classify the sensor data according to the information rate, if the frequency value of the information rate of the sensor data is greater than the set frequency classification decision threshold C, it is considered that the sensor is a high-speed sensor, and is classified into a high-speed sensor data class, otherwise (that is, the frequency value of the information rate of the sensor data is less than or equal to the set frequency classification decision threshold C, it is considered that the sensor is a low-speed sensor), and is classified into a low-speed sensor data class, and after classification, the high-speed sensor data and the low-speed sensor data are respectively waiting for the next fusion processing;

[0019] Step 3: fuse the data of the high-speed sensor;

[0020] Step 4: fuse the data of the low-speed sensor;

[0021] Step 5: stack the fused data of the high-speed sensor and the fused data of the low-speed sensor by adjustable soft alignment, and thus the data fusion is completed, and the final fused data is output.

[0022] Preferably, in step 2, the frequency classification decision threshold C is set to 1 / 5 of the information rate of the high-speed sensor with the lowest information rate, or 5 times the information rate of the low-speed sensor with the highest information rate.

[0023] Preferably, in step 3, the high-speed sensor generally has a simple data structure and a single representative meaning, so it is important to fuse the information of different sensors. The application proposes an automatic classification and fusion method for high-speed sensor data:

[0024] First, data screening is carried out, and data of unknown data type is excluded. Users can also add blacklists or whitelists of data types, thereby ensuring the credibility of data fusion.

[0025] Then, according to the data type classification, the data types that can be fused are soft-aligned (according to different information fusion algorithms, the requirements for data types are different. If the data type meets the requirements of a certain information fusion algorithm, it is considered to be a data type that can be fused), and they are matched into different information fusion algorithms, such as Kalman algorithm, extended Kalman algorithm or mean algorithm (these algorithms are classic data fusion mathematical algorithms, which need to be added by users according to their own needs). Different information fusion algorithms have the characteristics of inputting multiple same data types and outputting one data of this data type. Finally, the fused data is stacked by adjustable soft alignment.

[0026] Preferably, in step 3, the specific steps of soft alignment are as follows:

[0027] The data with the lowest information rate among all the input sensor data is set as the reference data, and the frequency value is set as the reference data frequency f. The time stamps of other data with higher information rates are judged. If the minimum time difference between the other sensor data and the reference data is less than the threshold value D, it is judged as the same time, and the soft alignment is ended. If the minimum time difference between the other sensor data and the reference data is greater than or equal to the threshold value D, it is judged as unreliable data, which is discarded, and the next data in the reference data is selected and judged.

[0028] Preferably, the automatic performance of soft alignment is the automatic selection of threshold value D, that is, the automatic assignment of 10 times the inverse of the reference data information rate frequency value, that is, D = 1 / 10f. This parameter can also be manually adjusted, that is, the user determines the value of the threshold value D.

[0029] Preferably, the adjustable soft alignment data stacking is a time fusion method, which includes soft alignment and stacking fusion. The stacking fusion is to open a new storage space, encapsulate and combine different data types into a new data type, form a chain table-like structure, and transmit the new data type.

[0030] Preferably, in step 4, the data structure of the low-speed sensor is generally complex, and the meaning is rich. If all the data is to be preserved and processed, it will greatly increase the space, time cost and logical complexity of the operation. However, blind discarding will cause a large loss of information. Therefore, how to extract the information of each data and fuse the information is very important. In view of the fact that laser radar and camera are the most commonly used and most representative low-speed sensors, a new obstacle detection calibration information fusion method for laser radar and camera data is proposed for the fusion of radar data and camera data. Specifically, the method is as follows:

[0031] 4.1: Soft alignment of data from each low-speed sensor;

[0032] 4.2: Input laser radar data, filter and cluster the laser radar (the related algorithm is already very mature), obtain the radar graph of obstacle coordinate points, but cannot distinguish obstacle color and other information;

[0033] 4.3: Input camera data, target extraction of target obstacles (the related algorithm is also mature), frame the extracted obstacles, record the center coordinate position and area information S, because:

[0034] S = k * L 2

[0035] The relative distance L between the obstacle and the vehicle is calculated, and the line between the vehicle and the center coordinate position is the direction; this step is repeated to calculate the relative distance L and the direction of all extracted obstacles;

[0036] 4.4: Radar coordinate calibration of target obstacles and the vehicle using the relative distance L and the direction of the target obstacles and the vehicle extracted by the camera data in step 4.3, specifically: input the relative position information of the target obstacles and the vehicle in turn, on the radar graph, determine the origin according to the relative distance L and the direction of the vehicle and the obstacle obtained in 4.3, create a circular range with a radius of l (l is a self-set parameter, and the smaller l is, the more strict the matching is), and the target point in the circle is the matching obstacle coordinate of the obstacle, and the obstacle in the radar graph is classified as the same type of obstacle; all points of all categories are calibrated in this step to obtain the calibrated radar graph, which can distinguish the type of obstacle target point; laser radar data and camera data fusion is completed.

[0037] Preferably, if there is other low-speed sensor data, then the data after laser radar data and camera data fusion and other low-speed sensor data are stacked after adjustable soft alignment, to obtain the final data fusion.

[0038] The present application can realize automatic classification and fusion of sensors during operation, and can overcome the problem of changing the fusion framework every time a sensor is added, deleted or changed. Moreover, the present application proposes a method of separate fusion of high-speed and low-speed sensors. The traditional method directly fuses all sensors, and the output information rate depends on the lowest speed sensor, which can cause serious loss of high-speed sensor data, and in intelligent driving, high-speed sensor data needs to be input into the predictive algorithm (Kalman filter) with memory function for processing, which affects the accuracy. The present application classifies high-speed and low-speed sensors, which greatly reduces the problem of high-speed sensor information loss.

[0039] The details of the present application can be referred to the prior art.

[0040] The beneficial effects of the present application are:

[0041] 1. High precision: The present application can fully utilize the information of multiple sensors, reduce the limitations of single sensor data, and improve the fusion precision of vehicle information. Different frequency sensor data is fused respectively to avoid data loss and precision decline caused by large differences in data frequency in traditional methods.

[0042] 2. Reduce the impact of clock error on the system: The approximate time soft alignment method is easier to handle clock error than the precise time alignment method, because they do not need to synchronize time at each data transmission.

[0043] 3. High reliability: The present method uses corresponding preprocessing methods for different sensor data, which can effectively eliminate noise and outliers in sensor data, and reduces the direct discarding of data, and improves the accuracy of predictive algorithms with memory function such as Kalman filter.

[0044] 4. Strong adaptability: The method can adaptively handle the differences between different sensor data, and is suitable for a variety of different types of vehicles and driving scenes in different environments. And for the addition and discard of sensors in the middle, it can automatically adapt and output data without the need to rebuild the data fusion framework.

[0045] 5. Reduce system cost: Use monocular camera, lower cost, and lower computing power required for data processing, can be deployed on lower computing power computing platform. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings accompanying the specification of this application form a part thereof, serve to further provide a further understanding of the application, and together with the description of the exemplary embodiments of the application, serve to explain the application, and do not constitute an improper limitation on the application.

[0047] Figure 1 The intelligent vehicle multi-sensor information efficient automatic classification fusion method flow chart of the present application;

[0048] Figure 2 The data fusion flow chart of high-speed sensor;

[0049] Figure 3 The data fusion flow chart of low-speed sensor;

[0050] Figure 4 The frame marking diagram of the extracted obstacle;

[0051] Figure 5 The schematic diagram of creating a circular range with a radius of 1 in the radar chart. DETAILED DESCRIPTION

[0052] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the embodiments of the specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the specification, but not limited to this, the present application is not described in detail, and is according to the conventional technology in the art.

[0053] Embodiment 1

[0054] An intelligent vehicle multi-sensor information efficient automatic classification fusion method, as shown in Figure 1 , comprising:

[0055] Step 1: Collect different information using multiple sensors, and sample them to obtain the information rate of each sensor;

[0056] Step 2: According to the information rate, the sensor data is automatically classified, if the frequency value of the information rate of the sensor data is greater than the set frequency classification decision threshold C, it is considered that the sensor is a high-speed sensor, and is classified into the high-speed sensor data class, otherwise, it is classified into the low-speed sensor data class, and after classification, it is respectively waiting for the next fusion processing;

[0057] The frequency classification decision threshold C is set to 1 / 5 of the information rate of the lowest speed sensor or 5 times the information rate of the highest speed sensor.

[0058] Step 3: Fusion of high-speed sensor data

[0059] The general data structure of high-speed sensor is relatively simple, and the representative meaning is relatively single, so it is important to fuse the information of different sensors. The present application proposes an automatic classification fusion method for high-speed sensor data:

[0060] As shown in Figure 2 , first, data screening is performed, and data of unknown data type is removed, and the user can also add a black list or a white list of data type, so as to ensure the credibility of data fusion;

[0061] Then, according to the data type classification, the data types that can be fused are soft aligned (according to different information fusion algorithms, the requirements for data types are different, and the data types that meet the requirements of a certain information fusion algorithm are considered as a data type that can be fused), and they are matched into different information fusion algorithms, such as Kalman algorithm, extended Kalman algorithm or mean algorithm (these algorithms are classic data fusion mathematical algorithms, which need to be added by the user according to their own needs), different information fusion algorithms, the characteristics are inputting multiple same data types, and the output is a data of this data type; Finally, the fused data is stacked by adjustable soft alignment data.

[0062] The specific steps of soft alignment are:

[0063] Set the data with the lowest information rate among all input sensor data as the reference data, and set its frequency value as the reference data frequency f. Then, judge the time stamp of other data with higher information rate. If the minimum time difference between other sensor data and the reference data is less than the threshold value D, it is judged as the same time, and the soft alignment ends. If the minimum time difference between other sensor data and the reference data is greater than or equal to the threshold value D, it is judged as unreliable data, which is discarded, and the next data in the reference data is selected and judged.

[0064] The automatic performance of soft alignment is the automatic selection of the threshold value D, that is, the threshold value D is automatically assigned as 10 times the inverse of the frequency value of the reference data information rate, that is, D = 1 / 10f. This parameter can also be manually adjusted, that is, the user determines the value of the threshold value D.

[0065] The adjustable soft alignment data stacking is a time fusion method, which includes soft alignment and stacking fusion. The stacking fusion is to open a new storage space, encapsulate and combine different data types into a new data type, form a chain table-like structure, and transmit the new data type.

[0066] Step 4: Fusion of low-speed sensor data

[0067] Low-speed sensors generally have complex data structures and rich meanings. If all data is to be preserved and processed, it will greatly increase the space, time cost and logical complexity of the operation. However, blind discarding will cause a large amount of information loss. Therefore, how to extract the information of each data and fuse the information is crucial. Since laser radar and camera are the most commonly used and representative low-speed sensors, a new type of obstacle detection calibration information fusion method for laser radar and camera data is proposed for the fusion of radar data and camera data, as shown in Figure 3 , specifically:

[0068] 4.1: Soft alignment of data of each low-speed sensor;

[0069] 4.2: Input laser radar data, filter and cluster the laser radar (the related algorithm is already very mature, and will not be described here), and obtain the radar graph of the obstacle coordinate points, but cannot distinguish the obstacle color information;

[0070] 4.3: Input camera data, target extraction of target obstacles (the related algorithm is also mature), frame labeling of the extracted obstacles, and recording of the center coordinate position and area information S. Since:

[0071] S = k * L 2

[0072] The relative distance L between the obstacle and the vehicle and the direction of the line connecting the vehicle and the center coordinate position are calculated.

[0073] 4.4: The relative distance L between the target obstacle and the vehicle and the direction of the line connecting the vehicle and the center coordinate position are calculated using the camera data extracted in step 4.3. Specifically, the relative position information between the target obstacle and the vehicle is input in sequence, and in the radar chart, the origin is determined according to the relative distance L and the direction of the line connecting the vehicle and the obstacle obtained in step 4.3, a circular range with a radius of l (l is a parameter set by the autonomous system, and the smaller the l, the more strict the matching) is created, and the target point in the circle is the obstacle coordinate matching the current obstacle. The obstacles in the radar chart are classified as the current type of obstacles. The points of all categories are labeled in this step to obtain the labeled radar chart, which can distinguish the types of obstacle target points. The fusion of lidar data and camera data is completed.

[0074] If there is other low-speed sensor data, the data after the fusion of lidar data and camera data and other low-speed sensor data are subjected to adjustable soft alignment data stacking to obtain the final data fusion.

[0075] Step 5: The data after the fusion of high-speed sensors and the data after the fusion of low-speed sensors are subjected to adjustable soft alignment data stacking. Thus, the data fusion is completed, and the final fused data is output.

[0076] Embodiment 2

[0077] An intelligent vehicle multi-sensor information efficient automatic classification fusion method is provided, which assumes that a laser radar with a sampling rate of 10 Hz, a monocular camera with a sampling rate of 60 Hz, a wheeled odometer with a sampling rate of 400 Hz, an IMU inertial sensor with a sampling rate of 6000 Hz, a backup laser ranging module A with a sampling rate of 1800 Hz, and a backup laser ranging module B with a sampling rate of 4500 Hz are used.

[0078] According to the flowchart of the method described in step 2, the threshold value C = 200 Hz is set, and the laser radar and the monocular camera can be automatically classified as one category, and the wheeled odometer and the IMU inertial sensor are classified as one category.

[0079] According to step 3, in step 3, the input interface is set to ekf (extended Kalman filter) of sensor_msgs / Imu and nav_msgs / Odometry, and the mean algorithm (sum divided by the number of input data) of std_msgs / Float32. The data type of the wheeled odometry is nav_msgs / Odometry, and the data type of the IMU is sensor_msgs / Imu, which meets the input conditions of the ekf. Therefore, the two are automatically soft-aligned, with the wheeled odometry as the reference data, and the soft alignment decision time threshold D is automatically set to 1 / 4000s (seconds) = 0.25ms (milliseconds). After alignment, the input ekf algorithm is obtained, and a single nav_msgs / Odometry data is obtained without the need for fusion of other data, i.e. the fused data can be output, and the data rate is 400Hz.

[0080] According to step 4, the data of the laser radar and the monocular camera are processed. The data type of the laser radar is sensor_msgs / LaserScan, and the radar is filtered to remove noise and redundant data, and clustering is used to obtain the reliable coordinate points of the obstacles. The data type of the camera is sensor_msgs / lmage, and the images of different obstacles (in the example, red and blue cone barrels, extracted with black and white frames respectively) are extracted, as shown in Figure 4 ), the radar graph is calibrated, i.e. the origin is determined on the radar graph according to the relative distance L and the direction of the vehicle and the obstacle, a circle with a radius of l is drawn, the obstacles on the radar graph are determined, and a colored image is generated, which is fused into sensor_msgs / lmage data output, as shown in Figure 5 , and the data rate is 10Hz.

[0081] According to step 5, the nav_msgs / Odometry data fused by the high-speed sensor and the sensor_msgs / lmage data fused by the low-speed sensor are stacked in adjustable soft alignment (soft alignment: since the rate of sensor_msgs / lmage is low, it is set as the reference data, and the soft alignment decision time threshold D is automatically set to 1 / 100s = 10ms, the nearest neighbor nav_msgs / Odometry data is obtained, and since its rate 400Hz is greater than 1 / 10ms = 100Hz, the low-speed data will not be discarded, so the data is stacked, and a temporary data type is generated according to the two data types, i.e. a sensor_msgs / lmage and nav_msgs / Odometry linked list output is established, and the rate is 10Hz), and the output is a reliable fusion data type of nav_msgs / Odometry and sensor_msgs / lmage.

[0082] If the IMU inertial sensor is suddenly disconnected, the wheeled odometry is classified alone in step 2, and participates in the ekf alone in step 3 and generates nav_msgs / Odometry, and the subsequent process remains unchanged.

[0083] If the laser ranging module A and the laser ranging module B are suddenly added, the laser ranging module A, the laser ranging module B, the wheeled odometry and the IMU inertial sensor are automatically classified into one category in step 2, and in step 3, the wheeled odometry and the IMU inertial sensor data are automatically input into the ekf after soft alignment, while the laser ranging module A and the laser ranging module B as std_msgs / Float32 data types are automatically input into the mean algorithm after soft alignment, and the output is std_msgs / Float32, and then the ekf output nav_msgs / Odometry data and the mean output std_msgs / Float32 are stacked in adjustable soft alignment data, and the output is a linked list composed of nav_msgs / Odometry and std_msgs / Float32 data groups. In step 5, the linked list is stacked in adjustable soft alignment data with the sensor_msgs / image data generated in step 4, that is, the sensor_msgs / image and the elements in the linked list are recombined into a new linked list composed of sensor_msgs / image, nav_msgs / Odometry and std_msgs / Float32, and the new linked list is output.

[0084] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can also make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. An intelligent vehicle multi-sensor information efficient automatic classification fusion method, characterized in that, The application relates to a data fusion method and device. Step 1: collecting different information by using multiple sensors and sampling the information to obtain information rates of the sensors; Step 2: automatically classifying the sensor data according to the information rates, if the frequency value of the information rate of the sensor data is greater than a set frequency classification decision threshold C, the sensor is considered as a high-speed sensor and is classified into a high-speed sensor data class, otherwise, the sensor is classified into a low-speed sensor data class, and after classification, the high-speed sensor data and the low-speed sensor data are respectively waited for the next fusion processing; Step 3: fusing the data of the high-speed sensor; Step 4: fusing the data of the low-speed sensor; Step 5: performing adjustable soft alignment data stacking on the fused data of the high-speed sensor and the fused data of the low-speed sensor, and thus, the data fusion is completed, and the finally fused data is output; In step 3, first, data screening is performed to remove data of unknown data types; Then, according to the data type classification, the data types capable of being fused are soft-aligned and matched into different information fusion algorithms, and the different information fusion algorithms have the characteristics that multiple same data types are input, and one data type of the data is output; finally, the fused data is subjected to adjustable soft alignment data stacking; In step 3, the specific steps of the soft alignment are as follows: The data with the lowest information rate in all the input sensor data is set as reference data, the frequency value of which is set as a reference data frequency f, and the time stamp of other data with higher information rates is judged, if the minimum time difference between the other sensor data and the reference data is less than a threshold value D, it is judged that the time is the same, and the soft alignment is ended; If the minimum time difference between the other sensor data and the reference data is greater than or equal to the threshold value D, the data is judged as unreliable data and is discarded; The adjustable soft alignment data stacking includes soft alignment and stacking fusion, wherein the stacking fusion refers to opening a new storage space, encapsulating and combining different data types into a new data type, forming a chain table-like structure, and transmitting the new data type; Step 4 is specifically as follows: 4.1: soft aligning the data of each low-speed sensor; 4.2: inputting the laser radar data, filtering and clustering the laser radar to obtain a radar graph of obstacle coordinate points, but the color information of the obstacles cannot be distinguished; 4.3: inputting the camera data, extracting the target obstacle, framing and calibrating the extracted obstacle, recording the center coordinate position and area information S, and since: The relative distance between the obstacle and the vehicle is calculated as L, and the line between the vehicle and the center coordinate position is the direction; the step is circularly operated to calculate all the extracted relative distances L and directions; S = k*L 2 ​ 4.4: The relative distance L and the direction of the target obstacle extracted from the camera data in step 4.3 are used to calibrate the obstacle coordinates of the radar chart, specifically: in the radar chart, the origin is determined according to the relative distance L and the direction of the obstacle obtained in 4.3, a circular range with a radius of l is created, and the target point in the circle is the obstacle coordinate matched with the obstacle, and the obstacle in the radar chart is classified as the obstacle of this type; loop this step to calibrate all points of all categories to obtain the calibrated radar chart, which can distinguish the types of obstacle target points; laser radar data and camera data fusion is completed; If there is other low-speed sensor data, the laser radar data and camera data fusion data and other low-speed sensor data are subjected to adjustable soft alignment data stacking to obtain the final data fusion. 2.The intelligent vehicle multi-sensor information efficient automatic classification fusion method according to claim 1, characterized in that, In step 2, the frequency classification decision threshold C is set to 1 / 5 of the information rate of the high-speed sensor with the lowest information rate, or 5 times the information rate of the low-speed sensor with the highest information rate. 3.The intelligent vehicle multi-sensor information efficient automatic classification fusion method according to claim 2, characterized in that, The automation of soft alignment is the automatic selection of threshold D, that is, the automatic assignment of 10 times the reciprocal of the reference data information rate frequency value, that is, D = 1 / 10f.

Images