A vehicle collision warning method based on multi-sensor fusion

Abstract

The present invention discloses a vehicle collision warning method based on multi-sensor fusion, comprising collecting raw data based on millimeter-wave radar and preprocessing it to obtain a target data set; acquiring image data based on a camera and performing enhancement processing in the image preprocessing stage based on an image enhancement algorithm to obtain an image data set; performing data fusion processing based on the obtained target data set and image data set; and, upon detecting an obstacle target ahead in real time, making a decision based on a collision warning control strategy to assist the driver in identifying the danger ahead in real time through appropriate means. The present invention uses a camera and millimeter-wave radar for obstacle recognition and distance perception. The collision warning based on multi-sensor fusion is more sensitive and efficient, capable of providing early warning before a collision, effectively reducing development costs and false alarms while shortening the development cycle.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned mining vehicle collision warning technology, and in particular to a vehicle collision warning method based on multi-sensor fusion. Background Art

[0002] Among the many factors that contribute to vehicle accidents in open-pit mines, rear-end collisions caused by vehicles following too closely, slow braking response, and side collisions caused by vehicles in adjacent lanes merging into the vehicle's lane account for a significant proportion. Therefore, early warning research on vehicle collisions is crucial for preventing and reducing accidents.

[0003] Domestic and international researchers have conducted extensive research in the field of vehicle collision avoidance and warning. These research can be categorized into collision avoidance technologies based on lidar and millimeter-wave radar, and collision avoidance algorithms based on machine learning. Lidar and millimeter-wave radar require corresponding physical equipment, resulting in high equipment costs and significant labor and resource consumption for ongoing maintenance and updates. In comparison, vehicle collision avoidance algorithms based on machine vision have the advantages of being able to detect and identify static obstacles, offering strong algorithm adaptability, low equipment costs, and ease of maintenance.

[0004] To sum up, the problems existing in current research mainly focus on the insufficient accuracy and reliability of the collision warning model based on a single sensor, the inability to effectively identify dynamic objects, and the poor adaptability of the algorithm in different traffic scenarios. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art. To achieve the above purpose, a vehicle collision warning method based on multi-sensor fusion is adopted to solve the problems raised in the above background technology.

[0006] A vehicle collision warning method based on multi-sensor fusion includes the following steps:

[0007] Step S1: pre-processing the raw data collected by the millimeter-wave radar to obtain a target data set;

[0008] Step S2: acquiring image data based on a camera, and performing enhancement processing in an image preprocessing stage based on an image enhancement algorithm to obtain an image data set;

[0009] Step S3: performing data fusion processing based on the obtained target data set and image data set;

[0010] Step S4: When an obstacle target is detected in real time, a decision is made according to the collision warning control strategy to assist the driver in identifying the danger ahead in real time through appropriate means.

[0011] As a further solution of the present invention: the specific steps in step S1 include:

[0012] Based on the raw data collected by millimeter-wave radar;

[0013] The original data is then filtered according to the filtering conditions to obtain a target data set close to the actual situation, where the filtering conditions include lateral distance and speed threshold.

[0014] As a further solution of the present invention: the specific steps in step S2 include:

[0015] Extract the gradient map of the image in the Y direction, and superimpose the extracted image with the original image as the guide image for guided filtering;

[0016] Perform guided filtering and noise reduction on the original image;

[0017] Then the guided filter is used for smooth denoising and Laplace is used to sharpen the knuckle pattern detail texture. The expression of the sharpened image is:

[0018]

[0019] As a further solution of the present invention: the specific steps in step S3 include:

[0020] Use the information of the target object at the previous moment to predict the information at the current moment, and then compare the prediction result with the detection result at the current moment to obtain the measurement value;

[0021] By leveraging the complementary advantages of millimeter-wave radar and camera images, effective targets can be identified through fusion of the two.

[0022] Among them, Kalman filtering is used to predict the target of the next cycle, and the selected state is:

[0023]

[0024] Where x(k), v x (k), y(k), v y (k), a x (k) are the relative distance in the x-direction, relative speed in the x-direction, relative distance in the y-direction, relative speed in the y-direction, and relative acceleration in the x-direction of the target in the k-th detection cycle;

[0025] Then the state equation and observation equation are:

[0026]

[0027] Where, the matrices are k+1 state matrix, state transfer matrix, k state matrix, and noise matrix in order; T is the millimeter wave radar detection period; U(k) is the noise;

[0028]

[0029] Where, the matrices on the right are the observation matrix and the k-state matrix respectively; V(k) is the observation noise;

[0030] The data of the two sensors are fused, and the fusion state vector and state covariance are calculated as follows:

[0031]

[0032] Where: X1 and X2 are the state vectors of the millimeter-wave radar and camera sensors respectively; P1 and P2 are the predicted state error covariances of the millimeter-wave radar and camera respectively.

[0033] As a further solution of the present invention: the specific steps in step S4 include:

[0034] When the millimeter-wave radar and camera sensors detect obstacles ahead in real time, they make decisions based on the collision warning control strategy, assisting the driver in timely detecting potential dangers ahead through appropriate means and issuing warning signals.

[0035] Compared with the prior art, the present invention has the following technical effects:

[0036] The above-mentioned technical solution utilizes multiple sensors, including cameras and millimeter-wave radar, for obstacle recognition and distance perception. After preprocessing the acquired data, collision warning based on multi-sensor fusion is more sensitive and efficient, providing early warning before a collision occurs, effectively reducing development costs and false alarms while shortening the development cycle. This also enables the recognition of dynamic targets, while improving the accuracy of collision prediction and the adaptability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings:

[0038] Figure 1 A schematic diagram of the steps of the vehicle collision warning method according to the embodiment disclosed in this application;

[0039] Figure 2 This is a schematic diagram of the overall framework of heterogeneous perception fusion collision warning according to the embodiment disclosed in this application. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a vehicle collision warning method based on multi-sensor fusion includes the following steps:

[0042] Step S1: Preprocessing the raw data collected by the millimeter-wave radar to obtain a target data set. The specific steps include:

[0043] In this embodiment, since the raw data collected by the millimeter-wave radar contains a large number of false targets, if not processed, it will not only interfere with valid targets but also increase the software calculation amount and processing time. Therefore:

[0044] Based on the raw data collected by millimeter-wave radar;

[0045] The original data is then filtered according to the filtering conditions to obtain a target data set close to the actual situation, where the filtering conditions include lateral distance, speed threshold and other filtering conditions.

[0046] Step S2: acquiring image data based on the camera, and performing enhancement processing based on the image enhancement algorithm in the image preprocessing stage to obtain an image data set. The specific steps include:

[0047] In this embodiment, in order to improve the quality of images captured by the vehicle-mounted camera, enhancement processing needs to be performed in the image preprocessing stage, for example, to minimize the adverse effects of uneven external lighting and environmental noise on feature extraction.

[0048] Extract the gradient map of the image in the Y direction, and superimpose the extracted image with the original image as the guide image for guided filtering;

[0049] Perform guided filtering and noise reduction on the original image;

[0050] Guided filtering is used for smoothing and denoising, while Laplacian is used to sharpen the knuckle texture details.

[0051] For the current pixel (x, y), get the grayscale values ​​of its eight neighboring pixels: f(x-1, y-1), f(x-1, y), f(x-1, y+1), f(x, y-1), f(x-1, y+1), f(x+1, y-1), f(x+1, y-1), f(x+1, y+1);

[0052] Laplace's expression is as follows:

[0053]

[0054] According to the above expression, the expression of the sharpened image is:

[0055]

[0056] Step S3: performing data fusion processing based on the obtained target data set and image data set. The specific steps include:

[0057] In this embodiment, the data fusion processing includes spatial fusion, temporal fusion, and decision-level fusion;

[0058] Spatial Fusion

[0059] Unify the coordinates of multiple sensors into a single coordinate system, enabling conversions between the millimeter-wave radar coordinate system, the 3D world coordinate system, and the camera coordinate system. First, convert the coordinates of the millimeter-wave radar coordinate system to the 3D world coordinate system, and then convert the coordinates of the 3D world coordinate system to the camera coordinate system.

[0060] Time Fusion

[0061] After spatial fusion, millimeter-wave radar and camera sensors also need temporal fusion. Because the target update frequency of the millimeter-wave radar and the acquisition frequency and time interval of the camera sensor are inconsistent, sensors with faster sampling rates are backwards compatible with sensors with slower sampling rates, enabling multi-sensor temporal fusion.

[0062] Decision-level fusion

[0063] Leveraging the respective strengths of millimeter-wave radar and cameras, the camera can output target types with higher confidence, while the millimeter-wave radar outputs target dynamic information. Detection results from these two sensors are combined for target matching, with data points projected onto the image through spatial transformation. The camera uses intelligent algorithms to identify obstacles, forming regions of interest (ROIs) in the image. These regions are then matched with the radar data points using the intersection-over-union (IoU) method.

[0064] The specific steps are:

[0065] Use the information of the target object at the previous moment to predict the information at the current moment, and then compare the prediction result with the detection result at the current moment to obtain the measurement value;

[0066] By leveraging the complementary advantages of millimeter-wave radar and camera images, effective targets can be identified through fusion of the two.

[0067] Among them, Kalman filtering is used to predict the target of the next cycle, and the selected state is:

[0068]

[0069] Where x(k), v x (k), y(k), v y (k), a x (k) are the relative distance in the x-direction, relative speed in the x-direction, relative distance in the y-direction, relative speed in the y-direction, and relative acceleration in the x-direction of the target in the k-th detection cycle;

[0070] Then the state equation and observation equation are:

[0071]

[0072] Where, the matrices are k+1 state matrix, state transfer matrix, k state matrix, and noise matrix in order; T is the millimeter wave radar detection period; U(k) is the noise;

[0073]

[0074] Where, the matrices on the right are the observation matrix and the k-state matrix respectively; V(k) is the observation noise;

[0075] The data of the two sensors are fused, and the fusion state vector and state covariance are calculated as follows:

[0076]

[0077] Where: X1 and X2 are the state vectors of the millimeter-wave radar and camera sensors respectively; P1 and P2 are the predicted state error covariances of the millimeter-wave radar and camera respectively.

[0078] Step S4: When an obstacle ahead is detected in real time, a decision is made based on the collision warning control strategy to assist the driver in identifying the danger ahead in real time through appropriate means. The specific steps include:

[0079] When the millimeter-wave radar and camera sensors detect obstacles ahead in real time, they make decisions based on the collision warning control strategy, assisting the driver in timely detecting potential dangers ahead through appropriate means and issuing warning signals.

[0080] Beneficial effects of the present invention:

[0081] First, image data is preprocessed to improve image quality and mitigate the effects of uneven lighting and ambient noise on feature extraction. This image enhancement algorithm mitigates the effects of uneven lighting while highlighting detailed features, making it suitable for image preprocessing. The enhanced image retains more detailed information, facilitating subsequent feature extraction and improving image recognition accuracy.

[0082] Since the raw data from the camera and millimeter-wave radar contain false detections and missed detections that seriously affect the system's perception results, the ID of each target object is not fixed. When tracking a fixed target ID, the output distance of the target object changes greatly and frequently. Sensor false detections and missed detections are caused by interference factors such as buildings, trees, and other objects in the vehicle's driving environment. The method related to this patent can eliminate false targets and improve the accuracy of the detected targets. The information of the target at the previous moment is used to predict the information at the current moment, and then the prediction result is compared with the detection result at the current moment, thereby obtaining a more reliable measurement value. By complementing the advantages of millimeter-wave radar and camera images, effective target identification can be achieved through the fusion of the two.

[0083] The algorithm uses the intersection-over-union (IoU) principle to achieve decision-level fusion of millimeter-wave radar and vision. Finally, a forward collision warning strategy is proposed based on a minimum safe distance model. Real-vehicle verification results in various scenarios demonstrate that the proposed algorithm is more accurate and robust than single-sensor algorithms.

[0084] Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A vehicle collision warning method based on multi-sensor fusion, characterized in that: The following steps are involved: Step S1: pre-processing the raw data collected by the millimeter-wave radar to obtain a target data set; Step S2: acquiring image data based on the camera, and performing enhancement processing based on the image enhancement algorithm in the image preprocessing stage to obtain an image data set. The specific steps include: Extract the gradient map of the image in the Y direction, and superimpose the extracted image with the original image as the guide image for guided filtering; Perform guided filtering and noise reduction on the original image; Then the guided filter is used for smooth denoising and Laplace is used to sharpen the knuckle pattern detail texture. The expression of the sharpened image is: ； Step S3: performing data fusion processing based on the obtained target data set and image data set; Step S4: When an obstacle target is detected in real time, a decision is made according to the collision warning control strategy to assist the driver in identifying the danger ahead in real time through appropriate means.

2. The vehicle collision warning method based on multi-sensor fusion according to claim 1, characterized in that: The specific steps in step S1 include: Based on the raw data collected by millimeter-wave radar; The original data is then filtered according to the filtering conditions to obtain a target data set close to the actual situation, where the filtering conditions include lateral distance and speed threshold.

3. The vehicle collision warning method based on multi-sensor fusion according to claim 1, characterized in that: The specific steps in step S3 include: Use the information of the target object at the previous moment to predict the information at the current moment, and then compare the prediction result with the detection result at the current moment to obtain the measurement value; By leveraging the complementary advantages of millimeter-wave radar and camera images, effective targets can be identified through fusion of the two. Among them, Kalman filtering is used to predict the target of the next cycle, and the selected state is: Where x(k), , y(k), (k) are the relative distance in the x direction, relative speed in the x direction, relative distance in the y direction, relative speed in the y direction, and relative acceleration in the x direction of the target in the kth detection cycle respectively; Then the state equation and observation equation are: Where, the matrices are k+1 state matrix, state transfer matrix, k state matrix, and noise matrix in order; T is the millimeter wave radar detection period; U(k) is the noise; Where, the matrices on the right are the observation matrix and the k-state matrix respectively; V(k) is the observation noise; The data of the two sensors are fused, and the fusion state vector and state covariance are calculated as follows: Where: and are the state vectors of the millimeter-wave radar and camera sensors respectively; and are the predicted state error covariances of the millimeter-wave radar and camera, respectively.

4. The vehicle collision warning method based on multi-sensor fusion according to claim 1, characterized in that: The specific steps in step S4 include: When the millimeter-wave radar and camera sensors detect obstacles ahead in real time, they make decisions based on the collision warning control strategy, assisting the driver in timely detecting potential dangers ahead through appropriate means and issuing warning signals.

Images