Multi-modal fusion method for automatic driving of 3D obstacles

Abstract

The invention provides a multi-modal fusion method for automatic driving of 3D obstacles. The multi-modal fusion method comprises the following steps: S1, acquiring driving data of a driving vehicle and information of obstacles around the vehicle by using a multi-modal sensor; s2, preprocessing the driving data of the autonomous vehicle and the data of obstacles around the vehicle, which are acquired by the multi-mode sensor in the step S1; s3, extracting features of the obstacles around the vehicle according to the preprocessed data of the obstacles around the vehicle; s4, a vehicle speed threshold value is set, when the vehicle speed is higher than the set threshold value, the collection frequency of the multi-mode sensor in the step S1 is increased, and when the vehicle speed is lower than the set threshold value, the collection frequency of the multi-mode sensor in the step S1 is reduced; and S5, according to the relationship between the vehicle speed and the speed threshold in the step S4, selecting a data layer fusion strategy to fuse the data or selecting a feature layer fusion strategy to fuse the data.

Description

Technical Field

[0001] The present invention relates to the field of 3D obstacle processing for autonomous driving, and in particular to a multimodal fusion method for 3D obstacles in autonomous driving. Background Art

[0002] With the continuous development of autonomous driving technology, the popularity of autonomous vehicles on the road continues to increase. However, one of the core issues to ensure the safe driving of autonomous vehicles is to effectively identify and track various obstacles in the surrounding environment, including other vehicles, pedestrians, buildings, etc. To achieve this goal, autonomous driving systems usually use multimodal sensors, such as lidar, cameras, millimeter-wave radar, etc., to obtain multi-dimensional information about the surrounding environment.

[0003] In the prior art, the method generally includes the following steps: acquiring surrounding environment features by transmitting signals through a multimodal sensor, fusing the acquired feature information to obtain feature information, and making safety decisions based on the environmental feature information. However, when the system is in use, the multimodal sensor generally transmits signals at a fixed frequency. In order to accurately identify features, the multimodal transmitter generally transmits signals at a higher frequency, and data fusion is also performed in the same manner. At this time, if the vehicle speed is slow and in a simple scenario, if a high-frequency signal transmission is maintained, the energy consumption of the multimodal sensor is high. If the multimodal sensor hardware is always in high-speed operation, the life of the hardware will also be reduced, and the high-frequency transmission signal will collect some unnecessary features, which increases the workload of the fusion module and the decision module in some unnecessary situations, and the fusion will always be performed in the same manner when fusing data, resulting in an increase in the workload of fusing data.

[0004] Therefore, it is necessary to provide a new multimodal fusion method for 3D obstacles in autonomous driving to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multimodal fusion method for 3D obstacles in autonomous driving.

[0006] The multimodal fusion method for 3D obstacles in autonomous driving provided by the present invention comprises the following steps:

[0007] S1. Collect driving data of the vehicle and information about obstacles around the vehicle using a multimodal sensor;

[0008] S2, preprocessing the driving data of the vehicle and the data of obstacles around the vehicle collected by the multimodal sensor in step S1 to obtain the driving speed of the vehicle and the characteristic data of the surrounding obstacles;

[0009] S3. Based on the pre-processed data of surrounding obstacles, a histogram is constructed through the lidar electric cloud features, the angle and distance relationship between each point and the neighboring points are counted in the histogram, and the coordinates of all points are calculated to confirm the position to obtain the obstacle features;

[0010] S4, setting a vehicle speed threshold, according to the driving vehicle speed obtained after the multimodal sensor preprocessing in step S2, when the vehicle speed is higher than the set threshold, increasing the multimodal sensor acquisition frequency in step S1, and when the vehicle speed is lower than the set threshold, reducing the multimodal sensor acquisition frequency in step S1;

[0011] S5. According to the relationship between the vehicle speed and the speed threshold in step S4, a data layer fusion strategy and a feature layer fusion strategy are selected to directly fuse the information of the obstacle feature data to quickly and accurately obtain the surrounding environment information.

[0012] Preferably, the multimodal sensor includes: a laser radar, a vehicle speed sensor, a camera and a millimeter wave radar.

[0013] Preferably, the pre-processing of step S2 comprises the following specific steps:

[0014] S201, collecting information, obtaining surrounding obstacle data based on the laser radar, camera and millimeter wave radar in the multimodal sensor;

[0015] S202, obtaining information, obtaining vehicle speed information according to a vehicle speed sensor in a multimodal sensor;

[0016] S203, denoising processing, using statistical filtering technology to remove the judged noise points from the data of surrounding obstacles obtained in step S201, and retain the required key points.

[0017] Preferably, the basis for determining the noise point is: if the distance from each point to the neighboring point calculated by statistical filtering technology is greater than a set value, the point is considered to be an outlier, that is, a noise point.

[0018] Preferably, the feature extraction in step S3 comprises the following specific steps:

[0019] S301, information call, call the data information pre-processed in step S2;

[0020] S302, extracting information, using laser radar point cloud feature extraction technology and millimeter wave radar feature extraction technology to perform data feature extraction;

[0021] S303, data conversion, the features extracted in step S302 are converted into computer language.

[0022] Preferably, step S4 specifically includes the following steps:

[0023] S401, setting a vehicle speed threshold;

[0024] S402, obtaining the vehicle speed according to step S202;

[0025] S403, adjusting the acquisition frequency of the multimodal sensor according to whether the vehicle speed is higher than a threshold or lower than a threshold;

[0026] S404. When the vehicle speed is less than a threshold, the multimodal sensor acquisition frequency in step S1 is reduced; when the vehicle speed is greater than the threshold, the multimodal sensor acquisition frequency in step S1 is increased.

[0027] Preferably, the fusion strategy of step S5 has the following specific steps:

[0028] S501, receiving the data converted in step S303;

[0029] S502, receiving the information of step S404, selecting to use a feature layer fusion strategy or a data layer fusion strategy to fuse the received data according to the relationship between the vehicle speed information and the speed threshold, directly and quickly fusing the data through the data layer fusion strategy, and accurately fusing the feature data through the feature layer fusion strategy to obtain feature information, and output the result;

[0030] S503: judge the output result and make a decision.

[0031] Compared with the related art, the multimodal fusion method for 3D obstacles in autonomous driving provided by the present invention has the following beneficial effects:

[0032] 1. The present invention measures the vehicle speed and the complexity of the vehicle's surroundings through a vehicle speed sensor. When the vehicle speed is less than a preset threshold, the frequency of multimodal sensor acquisition is reduced, the hardware pressure of the multimodal sensor is reduced, the service life of the multimodal sensor is extended, and the amount of data of the multimodal sensor in scenarios with relatively simple vehicle driving conditions is reduced. By using a data layer fusion strategy to fuse the feature extracted data, the surrounding environment features are more accurately acquired, the data quality in various vehicle driving conditions is improved, the pressure on data storage hardware is reduced, and the safety of vehicle driving is ensured.

[0033] 2. When the vehicle determines that the driving situation is more complicated and the vehicle speed is faster, the data collection frequency of the multimodal sensor is increased. At this time, the data collection frequency is high frequency, which enables accurate capture of information during vehicle driving. At this time, the feature extracted data is quickly fused using the data layer fusion strategy, which improves the system's rapid decision-making and improves the safety of the system.

[0034] 3. The present invention constructs a histogram by using lidar point cloud feature technology, counts the angle and distance relationship between each point and the neighboring points in the histogram, confirms the position of the obstacle by calculating the coordinates of all points, and obtains the characteristics of the obstacle. This method makes it possible to obtain the obstacle characteristics more accurately, and this method is not affected by ambient light and can provide three-dimensional spatial information, helping the system to better perceive the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The overall flow chart of the multimodal fusion method for 3D obstacles in autonomous driving provided by the present invention;

[0036] Figure 2 A flow chart of step S2 provided by the present invention;

[0037] Figure 3 A flowchart of step S3 provided by the present invention;

[0038] Figure 4 A flow chart of step S4 provided by the present invention;

[0039] Figure 5 This is a flow chart of step S5 provided by the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0041] Please refer to Figure 1 , Figure 2 , Figure 3 , Figure 4 as well as Figure 5 in, Figure 1 The overall flow chart of the multimodal fusion method for 3D obstacles in autonomous driving provided by the present invention; Figure 2 A flow chart of step S2 provided by the present invention; Figure 3 A flowchart of step S3 provided by the present invention; Figure 4 A flow chart of step S4 provided by the present invention; Figure 5 This is a flow chart of step S5 provided by the present invention.

[0042] In the specific implementation process, Figure 1-5 As shown, the multimodal fusion method for 3D obstacles in autonomous driving provided by the present invention comprises the following steps:

[0043] S1. Collect driving data of the vehicle and information about obstacles around the vehicle using a multimodal sensor;

[0044] S2, pre-processing the driving data of the vehicle and the obstacles around the vehicle collected by the multimodal sensor in step S1 to obtain the driving speed of the vehicle and the characteristic data of the surrounding obstacles;

[0045] The preprocessing of step S2 includes the following specific steps:

[0046] S201, collecting information, obtaining surrounding obstacle data based on the laser radar, camera and millimeter wave radar in the multimodal sensor;

[0047] S202, obtaining information, obtaining vehicle speed information according to a vehicle speed sensor in a multimodal sensor;

[0048] S203: De-noising the data of the surrounding obstacles obtained in step S201 by using statistical filtering technology to obtain the data of the surrounding obstacles.

[0049] It should be noted that the statistical filtering technology here mainly determines whether the point is a noise point by calculating the distance statistics of each point to its neighborhood points. Specifically, for each point in the point cloud, a neighborhood range can be determined by setting the neighborhood radius or the number of neighborhood points, and then the distance from the point to other points in the neighborhood is calculated, and the mean and standard deviation of these distances are counted;

[0050] The judgment criterion is that if the distance from a point to its neighboring points is greater than a certain value, which is usually set as the distance mean plus several times the standard deviation, the point is considered to be an outlier, that is, a noise point, and is removed.

[0051] S3, extracting features of obstacles around the vehicle based on the pre-processed data of the surrounding obstacles;

[0052] The feature extraction of step S3 is specifically performed as follows:

[0053] S301, information call, call the data information pre-processed in step S2;

[0054] S302, extracting information, using laser radar point cloud feature extraction technology and millimeter wave radar feature extraction technology to perform data feature extraction;

[0055] It should be noted that the above-mentioned lidar point cloud feature extraction technology mainly extracts the geometric features of obstacles. The lidar electric cloud feature extraction technology constructs a histogram, considers the geometric relationship of the normal angle and distance between each point and its neighboring points, and counts these relationships into a high-dimensional histogram. The center of gravity position is calculated by weighted averaging the coordinates of all points to obtain the geometric features of the obstacle.

[0056] S303, data conversion, the features extracted in step S302 are converted into computer language

[0057] S4, setting a vehicle speed threshold, according to the driving vehicle speed obtained after the multimodal sensor preprocessing in step S2, when the vehicle speed is higher than the set threshold, increasing the multimodal sensor acquisition frequency in step S1, and when the vehicle speed is lower than the set threshold, reducing the multimodal sensor acquisition frequency in step S1;

[0058] Step S4 specifically includes the following steps:

[0059] S401, setting a vehicle speed threshold;

[0060] It should be noted that the above vehicle speed threshold is set to P;

[0061] S402, obtaining the vehicle speed according to step S202;

[0062] It should be noted that in step S202, the vehicle speed can be directly measured by a vehicle speed sensor. The vehicle speed sensor mainly measures the wheel speed of the vehicle. The wheel speed is set to w, and the tire radius r of the vehicle is measured in advance. Here, the tire radius r of the vehicle is a fixed value, which is calculated by the formula:

[0063] v = wr

[0064] Calculate the speed of the vehicle at this time;

[0065] S403, judging whether the vehicle speed is higher than a threshold or lower than a threshold according to the vehicle speed in step S402, and adjusting the acquisition frequency of the multi-modal sensor;

[0066] S404. When the vehicle speed is less than a threshold, the multimodal sensor acquisition frequency in step S1 is reduced; when the vehicle speed is greater than the threshold, the multimodal sensor acquisition frequency in step S1 is increased.

[0067] It should be noted that the specific operation here is to compare the vehicle speed v with the set vehicle speed threshold P through the vehicle speed in step S402. When the vehicle speed v is greater than the set vehicle speed threshold P, it is judged that the vehicle is in a high-speed driving state and more features of surrounding obstacles need to be collected. At this time, the acquisition frequency of the multimodal sensor in step S1 is increased. At this time, for the multimodal sensor, more samples are collected for feature extraction by increasing the working frequency of various multimodal sensors, so as to ensure that the vehicle can obtain more surrounding information and ensure the safety of vehicle driving.

[0068] When the vehicle speed v is less than the set speed threshold P, it is determined that the vehicle is in a low-speed driving state. At this time, there is no need to collect too much information about the surrounding environment. At this time, the collection frequency of the multimodal sensor in step S1 is reduced. By reducing the working frequency of the multimodal sensor, fewer samples are obtained for feature extraction. At this time, step S1 reduces the collected samples to ensure that the collected feature data is not too redundant, and does not cause excessive pressure in subsequent steps due to collecting too much data. The hardware pressure of step S1 is reduced, the service life of the multimodal sensor is increased, and the pressure on data processing in steps S2, S3 and S5 is reduced, so that these steps can process more useful data when processing data, thereby improving the performance and safety of the system, avoiding system errors due to excessive data redundancy, and improving the quality and utilization of data.

[0069] S5. According to the relationship between the vehicle speed and the speed threshold in step S4, a data layer fusion strategy is selected to fuse the data or a feature layer fusion strategy is selected to fuse the data.

[0070] Multimodal sensors include: LiDAR, vehicle speed sensor, camera, and millimeter wave radar.

[0071] The specific steps of feature fusion in step S5 are as follows:

[0072] S501, receiving the data converted in step S303;

[0073] S502, receiving the information of step S404, selecting to use a feature layer fusion strategy or a data layer fusion strategy to fuse the received data according to the relationship between the vehicle speed information and the speed threshold, and outputting the result;

[0074] It should be noted that at this time, when the information of step S404 is received, when it is received that the vehicle speed is high speed in S404, the data layer fusion strategy is selected to fuse the feature data. In the high-speed driving scenario, the vehicle needs to respond quickly to changes in the surrounding environment. The data layer fusion directly fuses the original data without complex feature data fusion processing. It can complete the fusion in a short time and provide comprehensive information, thereby ensuring the safety of the vehicle under fast driving and making decisions in a shorter time.

[0075] When the information received in step S404 is that the vehicle speed is low, the feature layer fusion strategy is adopted. The feature layer fusion strategy can more accurately judge the obstacle information by fusing the various features of the extracted surrounding obstacles, and can more accurately identify the obstacles and the surrounding environment, so as to make more accurate decisions. In step S404, the collection frequency of step S1 is reduced to reduce the amount of data processed in step S502, thereby reducing the system pressure and the amount of useless data. The feature layer fusion strategy is used to make the judgment more accurate, thereby reducing the system pressure and ensuring safety, and avoiding the problem that the system cannot accurately identify the vehicle speed due to heavy traffic.

[0076] S503: judge the output result and make a decision.

[0077] The working principle provided by the present invention is as follows:

[0078] When the vehicle is in the process of automatic driving, the multimodal sensor installed on the vehicle is used to collect the driving conditions of the vehicle itself and the environmental information around the vehicle. After the information is collected, the collected environmental information is pre-processed by statistical filtering technology to remove noise, making the data information more reliable and accurate. After the data is pre-processed, the features of the data are extracted, mainly the geometric features and motion features of the data, and then the vehicle speed is calculated based on the vehicle speed information. After the vehicle speed is obtained, the vehicle speed is compared with the set speed threshold;

[0079] When the vehicle speed is greater than the set speed threshold, the acquisition frequency of the multimodal sensor is increased, and the data layer fusion strategy is used to fuse the data. At this time, the data layer fusion strategy directly uses the data information obtained after preprocessing to fuse the data, judge the information of obstacles around the vehicle, and make decisions to ensure the requirement of rapid judgment and decision-making of the vehicle at high speeds. Increasing the acquisition frequency of the multimodal sensor and the information after preprocessing ensure the accuracy of the data, so that the system ensures the safety of the vehicle at high speeds and rapid decision-making;

[0080] When the vehicle speed is less than the set speed threshold, the acquisition frequency of the multimodal sensor is reduced, and the feature layer fusion strategy is used to fuse the data. At this time, the feature data after feature extraction is used for fusion. Because the acquisition frequency of the multimodal sensor is reduced, the pressure on the system for feature extraction, preprocessing and fusion will be reduced. At this time, the system will not collect redundant useless data, and the feature layer fusion strategy ensures a more accurate fusion judgment of these less data, ensuring that the vehicle will not collect too much useless information when driving at low speed, causing excessive system pressure. The feature layer fusion strategy ensures the accuracy of judgment under less data conditions;

[0081] After the data is fused, a decision is made. During use, the system adopts different fusion strategies at high or low speeds, and uses different acquisition frequencies to make the system more efficient. The system processes data more accurately and extends the service life of the vehicle hardware. It reduces the problem of system errors caused by excessive pressure from too much data while ensuring the safety and reliability of the system.

[0082] The circuits and controls involved in the present invention are all prior art and will not be described in detail here.

[0083] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal fusion method for 3D obstacles in autonomous driving, characterized in that: The steps include: S1. Collect driving data of the vehicle and information about obstacles around the vehicle using a multimodal sensor; S2, preprocessing the driving data of the vehicle and the data of obstacles around the vehicle collected by the multimodal sensor in step S1 to obtain the driving speed of the vehicle and the characteristic data of the surrounding obstacles; S3. Based on the pre-processed data of surrounding obstacles, a histogram is constructed through the lidar electric cloud features, the angle and distance relationship between each point and the neighboring points are counted in the histogram, and the coordinates of all points are calculated to confirm the position to obtain the obstacle features; S4, setting a vehicle speed threshold, according to the driving vehicle speed obtained after the multimodal sensor preprocessing in step S2, when the vehicle speed is higher than the set threshold, increasing the multimodal sensor acquisition frequency in step S1, and when the vehicle speed is lower than the set threshold, reducing the multimodal sensor acquisition frequency in step S1; S5. According to the relationship between the vehicle speed and the speed threshold in step S4, a data layer fusion strategy and a feature layer fusion strategy are selected to directly fuse the information of the obstacle feature data to quickly and accurately obtain the surrounding environment information.

2. The multimodal fusion method for 3D obstacles in autonomous driving according to claim 1, characterized in that: The multimodal sensor includes: a laser radar, a vehicle speed sensor, a camera and a millimeter wave radar.

3. The multimodal fusion method for 3D obstacles in autonomous driving according to claim 1, characterized in that: The pre-processing of step S2 includes the following specific steps: S201, collecting information, obtaining surrounding obstacle data based on the laser radar, camera and millimeter wave radar in the multimodal sensor; S202, obtaining information, obtaining vehicle speed information according to a vehicle speed sensor in a multimodal sensor; S203, denoising processing, using statistical filtering technology to remove the judged noise points from the data of surrounding obstacles obtained in step S201, and retain the required key points.

4. The multimodal fusion method for 3D obstacles in autonomous driving according to claim 3, characterized in that: The basis for judging the noise point is: if the distance from each point to the neighboring point calculated by statistical filtering technology is greater than a set value, the point is considered to be an outlier, that is, a noise point.

5. The multimodal fusion method for 3D obstacles in autonomous driving according to claim 1, characterized in that: The feature extraction of step S3 includes the following specific steps: S301, information call, call the data information pre-processed in step S2; S302, extracting information, using laser radar point cloud feature extraction technology and millimeter wave radar feature extraction technology to perform data feature extraction; S303, data conversion, the features extracted in step S302 are converted into computer language.

6. The multimodal fusion method for 3D obstacles in autonomous driving according to claim 1, characterized in that: The S4 step specifically includes the following steps: S401, setting a vehicle speed threshold; S402, obtaining the vehicle speed according to step S202; S403, adjusting the acquisition frequency of the multimodal sensor according to whether the vehicle speed is higher than a threshold or lower than a threshold; S404. When the vehicle speed is less than a threshold, the multimodal sensor acquisition frequency in step S1 is reduced; when the vehicle speed is greater than the threshold, the multimodal sensor acquisition frequency in step S1 is increased.

7. The multimodal fusion method for 3D obstacles in autonomous driving according to claim 1, characterized in that: The specific steps of the fusion strategy of step S5 are as follows: S501, receiving the data converted in step S303; S502, receiving the information of step S404, selecting to use a feature layer fusion strategy or a data layer fusion strategy to fuse the received data according to the relationship between the vehicle speed information and the speed threshold, directly and quickly fusing the data through the data layer fusion strategy, and accurately fusing the feature data through the feature layer fusion strategy to obtain feature information, and output the result; S503: judge the output result and make a decision.

Images