Vehicle motion state identification method based on radar video fusion technology

Abstract

The invention relates to the technical field of radar and video signal processing and machine learning, in particular to high-precision identification and analysis of a vehicle motion state based on a radar video fusion technology, and the method comprises the steps: collecting radar signals and video images around a vehicle; the collected radar signals and video images are preprocessed; vehicle operation information is extracted from the preprocessed radar signals and video images; a McMaster algorithm is used to divide the vehicle driving state; and fusing the extracted radar feature information and video feature information by using Kalman filtering. The algorithm precision is improved, and misjudgment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of radar and video signal processing and machine learning technology, and in particular to high-precision recognition and analysis of vehicle motion states based on radar and video fusion technology. Background Art

[0002] Traffic risk warning and control rely on real-time and accurate traffic flow data. With the development of sensor technology, traffic flow data collection technology tends to be diversified. In the field of intelligent transportation systems and urban monitoring, accurate and efficient identification of vehicle motion status is crucial for applications such as traffic flow management, accident prevention and autonomous driving. Traditional vehicle motion status identification methods mainly rely on data from a single sensor, which is often limited by environmental conditions and the limitations of the sensor itself.

[0003] With the improvement of sensor technology and data processing capabilities, multi-sensor fusion technology has gradually become an effective means to improve the accuracy and robustness of vehicle motion state recognition. As an active sensor, radar has the advantages of not being restricted by lighting conditions and being able to penetrate bad weather such as haze, rain and snow, but its spatial resolution is low and it is difficult to accurately identify the type and detailed movements of vehicles. Video cameras can provide high-resolution image information, which helps to identify the appearance features and subtle movements of vehicles, but are limited by lighting and occlusion problems. Therefore, we propose a fusion technology for radar and video data, which can make full use of the complementary advantages of the two and improve the accuracy and reliability of vehicle motion state recognition. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a method for identifying the motion state of a vehicle based on radar video fusion technology, which can improve the detection capability in complex environments and provide a richer and more reliable data source for the decision support system.

[0005] The present invention proposes a method for identifying the motion state of a vehicle based on radar video fusion technology, comprising the following steps:

[0006] S1. Collect radar signals and video images around the vehicle in real time by installing radar equipment and cameras on the vehicle or roadside;

[0007] S2. Preprocess the collected radar signals and video images, including filtering, denoising, correction and other operations, to improve data quality and accuracy;

[0008] S3, extracting vehicle operation information from pre-processed radar signals and video images, using a variety of algorithms and technologies to analyze and determine the vehicle operation status, filling in the lost path data of the vehicle operation track in bad weather, and restoring the vehicle driving information;

[0009] S4. Based on the obtained traffic flow and occupancy rate, the McMaster algorithm is used to divide the vehicle driving status into four categories: smooth traffic, occasional traffic congestion, blockage and frequent traffic congestion, and the traffic congestion probability function is introduced based on the traffic flow data set;

[0010] S5. Use Kalman filtering to fuse the extracted radar feature information and video feature information, estimate the target's position, velocity, acceleration and other state parameters, form comprehensive vehicle motion state information, and reduce the occurrence of misjudgment;

[0011] In S1, a timestamp is assigned to each radar data packet and video frame. The timestamp is in nanoseconds or microseconds, generated by the system clock or an external high-precision timing device, and attached to the data packet and video frame to achieve time synchronization between radar signals and video images, ensuring the accuracy of fusion and the effectiveness of subsequent analysis.

[0012] In S1, the radar device and the camera can be fixedly installed beside the road or installed on a moving vehicle. The installation height and angle of the radar device are determined according to its minimum detection distance and detection area to ensure the best monitoring effect.

[0013] In the above S2, the collected radar signals and video images are preprocessed, and the preprocessing steps include:

[0014] S21, suppressing background clutter on radar signals, including spatial filtering, time filtering, and frequency filtering, to separate target signals from radar echo signals, so as to improve signal quality and target detection accuracy;

[0015] S22. Doppler radar technology is used to improve target recognition and tracking performance. Moving target indication (MTI) technology is used to eliminate the influence of static background clutter and improve the signal-to-noise ratio between target signal and clutter. MTI uses the Doppler frequency shift of moving target echoes to distinguish moving targets from fixed targets. In pulse radar systems, this Doppler frequency shift manifests itself as a phase change in the echo signals between successively returning radar pulses.

[0016] In the above-mentioned S3, the collected information includes: extracting the vehicle's position, speed, acceleration and other information from the radar data, which can be obtained by analyzing the Doppler effect, time delay and signal strength and other characteristics of the radar echo signal; extracting the vehicle's outline, size, color and license plate number and other visual features from the video image, which are helpful to identify the vehicle and distinguish different targets.

[0017] In S4, the McMaster algorithm is used to classify the vehicle driving state, and the traffic flow state classification method based on flow and occupancy (Q represents traffic flow, O represents occupancy Q free ,Qrecurrent ,Q blocking ,O free ,O recurrent ,O blocking is a preset threshold), as follows:

[0018] S41. When the traffic flow is low and the occupancy rate is also low, the vehicle driving state is considered to be unobstructed, Q free And O <O free ;

[0019] S42: When the traffic flow is high and the occupancy rate is moderate, but not blocked, the vehicle driving state is considered to be occasional traffic congestion. free ≤Q recurrent And O free ≤O <O recurrent ;

[0020] S43. When the traffic flow and occupancy are very high, making it almost impossible for vehicles to move, the vehicle driving state is considered to be blocked, Q ≥ Q blocking And O≥O blocking ;

[0021] S44. When the traffic flow and occupancy are both at an intermediate level and not caused by peak hours, the vehicle driving state is considered to be regular traffic congestion, Q recurrent ≤Q blocking And O recurrent ≤O <O blocking ;

[0022] S45, traffic congestion probability function is expressed as

[0023] Where θ0 is the intercept term, θ1 is the coefficient of traffic flow Q, θ2 is the coefficient of occupancy O, and the model parameters θ are learned from the training data set by the maximum likelihood estimation method.

[0024] In the above S5, the extracted radar feature information and video feature information are fused using Kalman filtering:

[0025] S51, extracting feature information from radar and video sensors, obtaining the position, velocity and acceleration of the target, and standardizing the extracted information for subsequent fusion;

[0026] S52, determine the state space model, including state variables and observation variables, define the state transfer matrix A, control input model B, observation matrix H and process noise covariance matrix Q, and the prediction formula is:

[0027]

[0028] ​​​The update formula is:

[0029]

[0030] in, is the state estimate at the kth moment, A is the state transfer matrix, u k is the control signal, B is the control input model, P k∣k is the covariance matrix of the estimation error, Q is the covariance matrix of the process noise, K k is the Kalman gain, H is the observation matrix, z k is the observed value, R is the observation noise covariance matrix, and I is the identity matrix;

[0031] The state transfer matrix A(A) is a core component in the state space model. It describes the natural evolution of the system state when there is no external input or disturbance. The state equation is:

[0032] X i =FX i -1 +Wu i +Zw i (4);

[0033] Where X is the state vector, u is the control input vector, and Z is the process noise vector;

[0034] The control input model matrix B describes how the control input vector u affects the change of the system state in the state space model. The B matrix is ​​multiplied by the control signal u to represent the influence of the control input considered when predicting the next state. The dimension of this matrix is ​​usually n×m, where n is the dimension of the state space and m is the dimension of the control input.

[0035] S53, set initial state estimation and the covariance matrix P of the initial estimation error 0∣0 ;

[0036] S54, prediction stage, using the state transfer matrix A and the control input model B to perform state prediction, and calculate the covariance matrix P of the estimated error of the predicted state k∣k-1 ;

[0037] S55, update phase, calculate Kalman gain K k , combined with the observed value z k and the predicted value, update the state estimate Update the covariance matrix P of the estimated error k∣k .

[0038] Compared with the existing technology, the present invention aims to analyze the movement state of the vehicle and monitor the movement trajectory of the vehicle. The radar video fusion method can avoid the occurrence of misjudgment caused by a single system failure in bad weather, improve the accuracy of the algorithm, and the changes in the vehicle's movement state can timely detect potential dangers and prevent traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Flowchart of the vehicle driving state detection method of the present invention DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention are described below. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0041] A method for identifying vehicle motion state based on radar video fusion technology includes the following steps:

[0042] In S1, according to the actual requirements of road traffic monitoring, millimeter wave radars and video cameras can be fixedly installed on poles or other supporting structures beside the road. The installation height of the radar equipment should be determined according to its minimum detection distance to ensure that vehicles on the road can be effectively detected. The installation angle of the radar equipment should be adjusted according to the width of its detection area and the direction of the road to achieve the best monitoring effect. The installation position and angle of the video camera should be consistent with the radar for effective data fusion and target matching.

[0043] In S2, the collected radar signals and video images are preprocessed, and the preprocessing steps include:

[0044] S21. Suppress background clutter on radar signals, including spatial filtering, temporal filtering, and frequency filtering. First, the radar system collects raw echo data, which contains target signals and clutter signals. The raw data is preprocessed, including amplitude correction, denoising, and smoothing. Statistical analysis is performed using radar echo data without targets to estimate clutter characteristics. A filter is designed based on the estimated result of the clutter. The designed filter is applied to the raw radar data to suppress clutter signals and enhance target signals. The filtered signal is more easily detected as a target. The signal quality after filtering and the accuracy of target detection are evaluated to ensure that clutter is effectively suppressed and the target signal is retained.

[0045] S22. Doppler radar technology is used to improve target recognition and tracking performance. Moving target indication (MTI) technology is used to eliminate the influence of static background clutter and improve the signal-to-noise ratio between target signal and clutter. MTI uses the Doppler frequency shift of moving target echoes to distinguish moving targets from fixed targets. In pulse radar systems, this Doppler frequency shift manifests itself as a phase change in the echo signals between successively returning radar pulses.

[0046] In S3, the collected information includes: extracting the vehicle's position, speed, acceleration and other information from radar data, which can be obtained by analyzing the Doppler effect, time delay and signal strength of the radar echo signal; extracting the vehicle's outline, size, color and license plate number and other visual features from the video image, which help to identify the vehicle and distinguish different targets.

[0047] In S4, the McMaster algorithm is used to classify vehicle driving states, and the traffic flow state classification method based on flow and occupancy (Q represents traffic flow, O represents occupancy Q free ,Q recurrent ,Q blocking ,O free ,O recurrent ,O blocking is a pre-set threshold), as follows:

[0048] S41. When the traffic flow is low and the occupancy rate is also low, the vehicle driving state is considered to be unobstructed, Q <O free And O <O free ;

[0049] S42: When the traffic flow is high and the occupancy rate is moderate, but not blocked, the vehicle driving state is considered to be occasional traffic congestion. free ≤Q recurrent And O free ≤O <O recurrent ;

[0050] S43. When the traffic flow and occupancy are very high, making it almost impossible for vehicles to move, the vehicle driving state is considered to be blocked, Q ≥ Q blocking And O≥O blocking ;

[0051] S44. When the traffic flow and occupancy are both at an intermediate level and not caused by peak hours, the vehicle driving state is considered to be regular traffic congestion, Q recurrent ≤Q blocking And O recurrent ≤O <O blocking ;

[0052] S45, traffic congestion probability function is expressed as ​​

[0053] Where θ0 is the intercept term, θ1 is the coefficient of traffic flow Q, θ2 is the coefficient of occupancy O, and the model parameters θ are learned from the training data set by the maximum likelihood estimation method.

[0054] In S5, Kalman filtering is used to fuse the extracted radar feature information and video feature information:

[0055] S51, extract feature information from radar and video sensors, read the target distance and Doppler shift in the radar echo, convert them into actual velocity values, estimate the acceleration of the target using time series analysis methods through continuous radar measurement data, detect the target in each frame of the image using image recognition technology, obtain the target's bounding box, and calculate the target's position in the real world based on the target's position in the image and the camera's internal and external parameters;

[0056] S52, determine the state space model, including state variables and observation variables, define the state transfer matrix A, control input model B, observation matrix H and process noise covariance matrix Q, and the prediction formula is:

[0057]

[0058] The update formula is:

[0059]

[0060] in, is the state estimate at the kth moment, A is the state transfer matrix, u k is the control signal, B is the control input model, P k∣k is the covariance matrix of the estimation error, Q is the covariance matrix of the process noise, K k is the Kalman gain, H is the observation matrix, z k is the observed value, R is the observation noise covariance matrix, and I is the identity matrix;

[0061] The state transfer matrix A(A) is a core component in the state space model. It describes the natural evolution of the system state when there is no external input or disturbance. The state equation is:

[0062] X i =FX i -1 +Wu i +Zw i (4);

[0063] Where X is the state vector, u is the control input vector, and Z is the process noise vector;

[0064] The control input model matrix B describes how the control input vector u affects the change of the system state in the state space model. The B matrix is ​​multiplied by the control signal u to represent the influence of the control input considered when predicting the next state. The dimension of this matrix is ​​usually u×m, where n is the dimension of the state space and m is the dimension of the control input.

[0065] S53, set initial state estimation and the covariance matrix P of the initial estimation error 0∣0 ;

[0066] S54, prediction stage, using the state transfer matrix A and the control input model B to perform state prediction, and calculate the covariance matrix P of the estimated error of the predicted state k∣k-1 ;

[0067] S55, update phase, calculate Kalman gain K k , combined with the observed value z k and the predicted value, update the state estimate Update the covariance matrix P of the estimated error k∣k .

[0068] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for identifying vehicle motion status based on radar video fusion technology, characterized in that The following steps are involved: S1. Collect radar signals and video images around the vehicle in real time by installing radar equipment and cameras on the vehicle or roadside; S2. Preprocess the collected radar signals and video images, including filtering, denoising, correction and other operations, to improve data quality and accuracy; S3, extracting vehicle operation information from pre-processed radar signals and video images, using a variety of algorithms and technologies to analyze and determine the vehicle operation status, filling in the lost paths of vehicle operation tracks in bad weather, and restoring vehicle driving information; S4. Based on the obtained traffic flow and occupancy rate, the McMaster algorithm is used to divide the vehicle driving status into four categories: smooth traffic, occasional traffic congestion, blockage and frequent traffic congestion, and the traffic congestion probability function is introduced based on the traffic flow data set; S5. Use Kalman filtering to fuse the extracted radar feature information and video feature information, estimate the target's position, velocity, acceleration and other state parameters, form comprehensive vehicle motion state information, and reduce the occurrence of misjudgment.

2. The vehicle motion state recognition method based on radar video fusion technology according to claim 1 is characterized by: In S1, radar equipment and cameras can be fixedly installed beside the road or on a moving vehicle. The installation height and angle of the radar equipment are determined according to its minimum detection distance and detection area to ensure the best monitoring effect.

3. The vehicle motion state recognition method based on radar video fusion technology according to claim 1 is characterized in that: In S2, the collected radar signals and video images are preprocessed, and the preprocessing steps include: S21, suppressing background clutter on radar signals, including spatial filtering, time filtering, and frequency filtering, to separate target signals from radar echo signals, so as to improve signal quality and target detection accuracy; S22, use Doppler radar technology to improve target recognition and tracking performance, and use moving target indication (MTI) technology to eliminate the influence of static background clutter and improve the signal-to-noise ratio between target signal and clutter.

4. The vehicle motion state recognition method based on radar video fusion technology according to claim 3 is characterized by: In S4, the McMaster algorithm is used to classify vehicle driving states, and the traffic flow state classification method based on flow and occupancy (Q represents traffic flow, o represents occupancy Q free , Q recurrent , Q blocking , O free , o recurrent , O blocking is a pre-set threshold), as follows: S41. When the traffic flow is low and the occupancy rate is also low, the vehicle driving state is considered to be unobstructed, Q<Q free And O <O free ; S42: When the traffic flow is high and the occupancy rate is moderate, but not blocked, the vehicle driving state is considered to be occasional traffic congestion. free ≤Q recurrent And O free ≤O <O recurrent ;​ S43. When the traffic flow and occupancy are very high, making it almost impossible for vehicles to move, the vehicle driving state is considered to be blocked, Q ≥ Q blocking And O≥O blocking ; S44. When the traffic flow and occupancy are both at an intermediate level and not caused by peak hours, the vehicle driving state is considered to be regular traffic congestion, Q recurrent ≤Q blocking And o recurrent ≤o <o blocking ;​ S45, traffic congestion probability function is expressed as Where θ0 is the intercept term, θ1 is the coefficient of traffic flow Q, θ2 is the coefficient of occupancy o, and the model parameters θ are learned from the training data set by the maximum likelihood estimation method.

5. The vehicle motion state recognition method based on radar video fusion technology according to claim 4 is characterized in that: In S5, Kalman filtering is used to fuse the extracted radar feature information and video feature information: S51, extracting feature information from radar and video sensors, obtaining the position, velocity and acceleration of the target, and standardizing the extracted information for subsequent fusion; S52, determine the state space model, including state variables and observation variables, define the state transfer matrix A, control input model B, observation matrix H and process noise covariance matrix Q, and the prediction formula is: The update formula is: in, is the state estimate at the kth moment, A is the state transfer matrix, u k is the control signal, B is the control input model, P k|k is the covariance matrix of the estimation error, Q is the process noise covariance matrix, K k is the Kalman gain, H is the observation matrix, z k is the observed value, R is the observation noise covariance matrix, and I is the identity matrix; S53, set initial state estimation and the covariance matrix P of the initial estimation error 0|0 ; S54, prediction stage, using the state transfer matrix A and the control input model B to perform state prediction, and calculate the covariance matrix P of the estimated error of the predicted state k|k-1 ; S55, update phase, calculate Kalman gain K k , combined with the observed value z k and the predicted value, update the state estimate Update the covariance matrix P of the estimated error k|k .

Images