A highway radar and vision fusion data processing method

Abstract

The present application aims at overcoming the problems existing in radar track data, and using radar and vision fusion technology to carry out noise reduction processing on the radar track data. In order to achieve the above-mentioned purpose, the technical scheme of the present application is to provide a highway radar and vision fusion data processing method, characterized in that it comprises: obtaining highway radar and vision track data; coordinate conversion of video track data matrix; data fusion based on track similarity. The present application uses radar and vision fusion technology to carry out noise reduction on radar track data, improves the accuracy of radar perception data, and reduces the dependence on video verification assistance, and can be used for researches such as highway vehicle target tracking, high-precision traffic flow digital twin, short-term traffic prediction, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of expressway radar vision fusion data processing method, belong to radar, vision fusion data processing method technical field. BACKGROUND

[0002] With the transformation of expressway operation digitalization and the development of intelligent expressway construction, the demand for traffic perception data is increasingly high. In addition to high-precision digital twin and other scene applications, the accuracy and real-time performance of perception are also highly demanded. Advanced perception devices and detection technologies become the preferred choice for expressway reconstruction and intelligent expressway construction. However, the existing perception devices and detection technologies all have their own limitations, for example: video does not support all-weather, radar has inherent detection defects such as missing points and occlusion. High-precision perception data is the prerequisite for developing various intelligent applications of the system and the basis for subsequent analysis and evaluation, digital twin. The quality of perception data directly affects whether the transformation of expressway operation digitalization can be effectively implemented. SUMMARY

[0003] The purpose of the present application is to overcome the problems of radar trajectory data and to use radar vision fusion technology to process noise reduction of radar trajectory data.

[0004] To achieve the above purpose, the technical scheme of the present application provides an expressway radar vision fusion data processing method, characterized in that it comprises the following steps:

[0005] Step 1, periodically acquire expressway radar vision trajectory data, which includes radar trajectory data and video trajectory data, wherein: the radar trajectory data includes timestamp t, target ID, trajectory x coordinate, trajectory y coordinate, vehicle speed V, and lane l, and forms a radar trajectory data matrix R based on the same target ID ID ; the video trajectory data includes timestamp t, target ID, trajectory image u coordinate, trajectory image v coordinate, vehicle speed V, and lane l, and forms a video trajectory data matrix C ID based on the same target ID;

[0006] Step 2, determine whether the current time is night according to the timestamp in the expressway radar vision trajectory data, if not, go to step 3, if yes, go to step 7;

[0007] Step 3, pre-process the expressway radar vision trajectory data, eliminate invalid data, and fill in missing data;

[0008] Step 4, use the least common multiple of the sampling period of radar sensors and video sensors as the sampling period of the system to realize the time fusion of radar trajectory data and video trajectory data;

[0009] Step 5, convert the track image coordinates (u, v) in the video track data matrix C ID to world coordinates (x w , y w ), the mapping relationship is shown in the following formula: w w

[0010]

[0011] In the formula, Z is a scale factor, and H is a homography matrix;

[0012] Then match the video world coordinates (x w , y w ) with the radar world coordinates (x, y), and the corresponding relationship is shown in the following formula:

[0013]

[0014] In the formula, Kx and Ky are scale factors, θ is a plane angle, (e x , e y ) is an error, and (Δx, Δy) is the horizontal and vertical distance between the video world coordinates and the corresponding radar world coordinates;

[0015] Finally, the obtained track image coordinates are converted to the video track data matrix C ID under the radar world coordinates;

[0016] Step 6, data fusion based on track similarity, including the following steps:

[0017] Step 601, measure the track similarity by calculating the Hausdorff distance H(R ID , C ID ) of the radar and video track data in the same section and time period, and the calculation formula is as follows:

[0018] H(R ID , C ID ) = max(h(R ID , C ID ), h(C ID , R ID ))

[0019]

[0020]

[0021] In the formula, a i is the row vector of the radar track data matrix R ID after removing the timestamp; b j is the row vector of the video track data matrix C ID after removing the timestamp; h(R ID , CID ) is R ID each row vector a i to the nearest C ID row vector b j in C ID , R ID is the maximum value of each row vector b ID in C j to the nearest R ID row vector a i ;

[0022] Step 602, taking k as the minimum target ID number in the radar trajectory data in the same time period of the same section, and the calculation formula is:

[0023] k = min(m, n)

[0024] In the formula, m is the target ID number of the radar trajectory data; n is the target ID number of the video trajectory data;

[0025] Step 603, arranging the Hausdorff distance in the same time period of the same section from small to large, obtaining the radar and video trajectory data target IDs corresponding to the first k Hausdorff distances, and obtaining the fused radar and video trajectory data;

[0026] Step 7, only the radar trajectory data is preprocessed, invalid data is removed, missing data is filled, and time conversion is performed to ensure that the sampling periods of the radar sensor and the video sensor are the same;

[0027] Step 8, inputting the radar trajectory data into a binary support vector machine model to determine whether the radar trajectory data is abnormal, and only retaining normal radar trajectory data.

[0028] Preferably, in step 1, the radar trajectory data matrix R ID is represented as:

[0029]

[0030] In the formula, t1, t2,..., t m represent m sampling time points of the target ID, x1, x2,..., x m , and y1, y2,..., y n represent the trajectory coordinates of the target ID at the m sampling time points, V1, V2,..., V m represent the speed of the target ID at the m sampling time points, l1, l2,..., l m represent the lane in which the target ID is located at the m sampling time points.

[0031] Preferably, in step 1, the video trajectory data matrix CID is represented as:

[0032]

[0033] wherein u1, u2,..., u m and v1, v2,..., v m represents the trajectory image coordinates of the target ID at m sampling time points.

[0034] Preferably, in steps 3 and 7, invalid data is determined based on the following formula:

[0035] X t < μ - 3σ or X t > μ + 3σ

[0036] wherein X t is the highway radar or radar trajectory data at time t, μ is the average value of the data, and σ is the standard deviation of the data

[0037] Preferably, in step 8, the binary classification support vector machine model is constructed and trained using the following method:

[0038] Step 801, extract historical daytime radar trajectory data, and determine whether each radar trajectory data is fused trajectory data or unfused trajectory data based on steps 3 to 6;

[0039] Step 802, set a corresponding trajectory data category for each radar trajectory data, then the corresponding trajectory data category of the radar trajectory data X i is Y i : if the radar trajectory data X i is fused trajectory data, then the radar trajectory data X i is normal trajectory data, Y i = 1; if the radar trajectory data X i is unfused trajectory data, then the radar trajectory data X i is abnormal trajectory data, Y i = -1;

[0040] Based on the labeled historical daytime radar trajectory data, training samples and test samples of radar trajectory data are formed;

[0041] Step 803, construct and train the classification hyperplane with the training samples, and the calculation formula is:

[0042]

[0043] s.t. Y i (w·X i +b)-1≥0 i = 1, 2,..., n

[0044] wherein w and b are hyperplane parameters, X i is radar trajectory data, Y i is trajectory data category;

[0045] Step 804, a binary classification support vector machine model is constructed, and the calculation formula is as follows:

[0046] f(X) = sign(w·X + b)

[0047] wherein sign() is a sign function;

[0048] Step 805, a test sample is input to a decision function, and the classification accuracy is verified by a root mean square error (RMSE), and the calculation formula is as follows:

[0049]

[0050] wherein Y i is the i th sample classification obtained by the binary classification support vector machine model; Y ci is the i th sample classification obtained by video observation.

[0051] Preferably, in step 8, the data is input to the constructed binary classification support vector machine model to determine whether the radar trajectory data X i is abnormal: when f(X i ) = -1, the radar trajectory data X i is removed; when f(X i ) = 1, the radar trajectory data X i is output.

[0052] The present application uses radar and video fusion technology to reduce noise of radar trajectory data, improve radar perception data accuracy, and reduce dependence on video verification assistance, and can be used for researches such as highway vehicle target tracking, high-precision traffic flow digital twin, and short-term traffic prediction. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a schematic diagram of a highway radar and video fusion data processing method;

[0054] Figure 2 is a flowchart of binary classification support vector machine model construction and accuracy verification. DETAILED DESCRIPTION

[0055] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. Furthermore, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0056] In combination Figure 1 The application provides a highway radar and vision fusion data processing method, comprising the following steps:

[0057] Step S-1: acquiring highway radar and vision trajectory data every 5 seconds, wherein the highway radar and vision trajectory data comprises radar trajectory data and video trajectory data.

[0058] The radar trajectory data comprises a timestamp t, a target ID, a trajectory x coordinate, a trajectory y coordinate, a vehicle driving speed V and a lane l. In the embodiment, a radar trajectory data matrix R formed by any target ID is represented as: ID

[0059]

[0060] In the formula, t1, t2,..., t m m sampling time points of the target ID, x1, x2,..., x m and y1, y2,..., y m trajectory coordinates of the target ID at the m sampling time points, V1, V2,..., V m speeds of the target ID at the m sampling time points, l1, l2,..., l m lanes where the target ID is located at the m sampling time points.

[0061] The video trajectory data comprises a timestamp t, a target ID, a trajectory image u coordinate, a trajectory image v coordinate, a vehicle driving speed V and a lane l. In the embodiment, a video trajectory data matrix C formed by any target ID is represented as: ID

[0062]

[0063] In the formula, u1, u2,..., u m and v1, v2,..., v m trajectory image coordinates of the target ID at the m sampling time points.

[0064] Step S-2: determining whether it is night according to the timestamp, if not, turning to step S-2.1, and if yes, turning to step S-3.1.

[0065] Step S-2.1: pre-processing the radar and vision trajectory data, including data cleaning, eliminating invalid data and garbled data. In the embodiment, time segments with missing data and abnormal data are identified, and the missing data and abnormal data are filled by using a historical average method, and the calculation formula is:

[0066] ​​

[0067] In the formula, X t is the radar trajectory data at time t, is the radar trajectory data at time t on the i-th day, and N is the total number of effective days.

[0068] The determination condition of the abnormal data is:

[0069] X t <μ-3σor X t >μ+3σ

[0070] In the formula, μ is the average value of the data, and σ is the standard deviation of the data.

[0071] Step S-2.2: The least common multiple of the sampling periods of the two sensors is selected as the sampling period of the system to realize time fusion.

[0072] Step S-2.3: The trajectory image coordinates (u, v) obtained by the video are converted into world coordinates (x w , y w ), and the mapping relationship is:

[0073]

[0074] In the formula: Z is a proportional factor; H is a homography matrix, which can be obtained by inversely calculating the mapping relationship of the calibration object, constructing a vertical grid on the real road by referring to the lane line standard, obtaining the prior information of the relative position of the lane line in space, and estimating the world coordinates of the calibration object.

[0075] The video world coordinates (x w , y w ) are matched with the radar world coordinates (x, y), and the corresponding relationship is:

[0076]

[0077] In the formula: Kx and Ky are proportional factors; θ is a plane angle; (e x , e y ) is an error; (Δx, Δy) is the horizontal and vertical distance between the video world coordinates and the corresponding radar world coordinates. The above-mentioned parameters can be obtained by inversely calculating the video world coordinates and the corresponding radar world coordinates of the calibration object.

[0078] The finally obtained video trajectory data matrix C ID is:

[0079]

[0080] Step S-2.4: Data fusion is performed based on the similarity of the trajectories, including the following steps:

[0081] Step S-2.4.1: Hausdorff distance H(R ID , C ID ) is calculated by calculating the Hausdorff distance of the radar and video trajectory data in the same section and the same time period, and the calculation formula is as follows:

[0082] H(R ID , C ID ) = max(h(R ID , C ID ), h(C ID , R ID ))

[0083]

[0084]

[0085] In the formula, a i is the row vector of the radar trajectory data matrix R ID after removing the timestamp; b j is the row vector of the video trajectory data matrix C ID after removing the timestamp; h(R ID , C ID ) is the maximum value of each row vector a ID in R i to the row vector b ID in C j closest to this row vector; h(C ID , R ID ) is the maximum value of each row vector b ID in C j to the row vector a ID in R i closest to this row vector.

[0086] Step S-2.4.2: k is the minimum target ID number in the radar and video trajectory data in the same section and the same time period, and the calculation formula is as follows:

[0087] k = min(m, n)

[0088] In the formula, m is the target ID number of the radar trajectory data; and n is the target ID number of the video trajectory data.

[0089] Step S-2.4.3: the Hausdorff distances in the same section and the same time period are arranged from small to large, and the target IDs corresponding to the first k Hausdorff distances are obtained, that is, the radar and video trajectory data fusion is completed.

[0090] Step S-2.5.1: output the fused radar and video trajectory data;

[0091] Step S-2.5.2: Store data into the history database to build a binary classification support vector machine model, including the following steps;

[0092] Step S-2.5.2.1: Extract the daytime radar trajectory data from the history database every two weeks;

[0093] Step S-2.5.2.2: Set the corresponding trajectory data category for each radar trajectory data, then the radar trajectory data X i The corresponding trajectory data category is Y i : If the radar trajectory data X i is fused trajectory data, then the radar trajectory data X i is normal trajectory data, Y i = 1; if the radar trajectory data X i is non-fused trajectory data, then the radar trajectory data X i is abnormal trajectory data, Y i = -1. Based on the labeled historical data, form the training samples {(X1, Y1), (X2, Y2),..., (X n , Y n )} and test samples {(X1', Y1'), (X2', Y2'),..., (X n ', Y n ')} of the radar trajectory data.

[0094] Step S-2.5.2.3: Build and train the classification hyperplane with the training samples, and the calculation formula is:

[0095]

[0096] s.t. Y i (w·X i +b)-1≥0 i = 1, 2,..., n

[0097] In the formula: w, b are hyperplane parameters; X i is the radar trajectory data; Y i is the trajectory data category.

[0098] Step S-2.5.2.4: Build a binary classification support vector machine model, and the calculation formula is:

[0099] f(X) = sign(w·X + b)

[0100] In the formula, sign() is the sign function.

[0101] Step S-2.5.2.5: Input the test samples into the decision function, and verify the classification accuracy according to the daytime video observation with the root mean square error RMSE, and the calculation formula is:

[0102]

[0103] where Y i is the i-th sample classification obtained by the binary classification support vector machine model; Y ci is the i-th sample classification obtained by the video observation.

[0104] Step S-2.5.2.6: Save the radar trajectory data for the next iteration.

[0105] Step S-3.1: Preprocess the radar trajectory data according to step S-2.1, including data cleaning, missing data filling, and abnormal data processing.

[0106] Step S-3.2: Perform time conversion according to step S-2.2 to ensure the same sampling period.

[0107] Step S-3.3: Input the data into the constructed binary classification support vector machine model to determine whether the radar trajectory data X i is abnormal: when f(X i ) = -1, the radar trajectory data X i is removed; when f(X i ) = 1, the radar trajectory data X i is output.

Claims

1. A method for processing radar-visual fusion data on highways, characterized in that, Includes the following steps: Step 1: Periodically acquire highway radar trajectory data, which includes radar trajectory data and video trajectory data. The radar trajectory data includes a timestamp t, target ID, trajectory x-coordinate, trajectory y-coordinate, vehicle speed V, and lane number l. A radar trajectory data matrix R is formed based on the same target ID. ID The video trajectory data includes timestamp t, target ID, trajectory image u coordinates, trajectory image v coordinates, vehicle speed V, and lane position l, forming a video trajectory data matrix C based on the same target ID. ID ; Step 2: Determine whether the current time is nighttime based on the timestamp in the highway radar trajectory data. If not, proceed to step 3; if so, proceed to step 7. Step 3: Preprocess the highway radar sight trajectory data, remove invalid data, and fill in missing data; Step 4: Use the least common multiple of the sampling periods of the radar sensor and the video sensor as the system's sampling period to achieve temporal fusion of radar trajectory data and video trajectory data; Step 5: Convert the video trajectory data matrix C ID Transform the trajectory image coordinates (u, v) into world coordinates (x, v). w y w The mapping relationship is shown in the following formula: In the formula, Z is the scaling factor and H is the homography matrix; Then set the video world coordinates (x) w y w The coordinates (x, y) are matched with the radar world coordinates, and the correspondence is shown in the following formula: In the formula, Kx and Ky are scaling factors, θ is the plane angle, and (e x e y () represents the error, and (Δx, Δy) represents the horizontal and vertical distances between the video world coordinates and the corresponding radar world coordinates. The final obtained trajectory image coordinates are transformed into a video trajectory data matrix C in radar world coordinates. ID ; Step 6: Perform data fusion based on trajectory similarity, including the following steps: Step 601: Calculate the Hausdorff distance H(R) of the radar sight trajectory data within the same time period on the same cross section. ID C ID The similarity of trajectories is measured using the following formula: H(R ID ,C ID )=max(h(R ID ,C ID ),h(C ID ,R ID )) In the formula, a i For radar trajectory data matrix R ID Row vector excluding timestamps; b j For video trajectory data matrix C ID The row vector after removing the timestamp; h(R) ID C ID ) is R ID Each row vector a in i To the nearest C to this row vector ID middle row vector b j The maximum value of h(C); ID R ID ) is C ID Each row vector b in j To the nearest R to this row vector ID middle row vector a i The maximum value; Step 602: Take k as the minimum number of target IDs in the radar sight trajectory data within the same cross-section and time period. The calculation formula is: k = min(m, n) In the formula, m is the number of target IDs in the radar trajectory data; n is the number of target IDs in the video trajectory data; Step 603: Arrange the Hausdorff distances in the same section and time period from smallest to largest, obtain the target IDs of the radar trajectory data corresponding to the first k Hausdorff distances, and obtain the fused radar trajectory data. Step 7: Preprocess only the radar trajectory data, remove invalid data, fill in missing data, and then perform time conversion to ensure that the sampling period of the radar sensor and the video sensor is the same. Step 8: Input the radar trajectory data into the binary classification support vector machine model to determine whether the radar trajectory data is abnormal, and retain only the normal radar trajectory data.

2. The method for processing highway radar-visual fusion data as described in claim 1, characterized in that, In step 1, the radar trajectory data matrix R ID Represented as: In the formula, t1, t2, ..., t m Let x1, x2, ..., xm be the m sampling times of the target ID. m And y1, y2, ..., y n Let V1, V2, ..., V represent the trajectory coordinates of the target ID at m sampling times. m Let l1, l2, ..., l represent the velocity of target ID at m sampling times. m This indicates the lane where the target ID is located at m sampling times.

3. The method for processing highway radar-visual fusion data as described in claim 2, characterized in that, In step 1, the video trajectory data matrix C ID Represented as: In the formula, u1, u2, ..., u m And v1, v2, ..., v m This represents the trajectory image coordinates of the target ID at m sampling times.

4. The method for processing highway radar-visual fusion data as described in claim 1, characterized in that, In steps 3 and 7, invalid data is determined based on the following formula: X t <μ-3σ or X t >μ+3σ In the formula, X t Let μ be the highway radar trajectory data or radar trajectory data at time t, μ be the data mean, and σ be the data standard deviation.

5. The method for processing highway radar-visual fusion data as described in claim 1, characterized in that, In step 8, the binary classification support vector machine model is constructed and trained using the following method: Step 801: Extract historical daytime radar trajectory data, and determine whether each radar trajectory data is fused or unfused based on steps 3 to 6. Step 802: Set a corresponding trajectory data category for each radar trajectory data, then the radar trajectory data X i The corresponding trajectory data category is Y. i If radar trajectory data X i For the fused trajectory data, then the radar trajectory data X i For normal trajectory data, Y i =1; if radar trajectory data X i If the trajectory data is unfused, then the radar trajectory data X i For abnormal trajectory data, Y i =-1; Training and testing samples of radar trajectory data are generated based on labeled historical daytime radar trajectory data. Step 803: Construct and train a classification hyperplane using training samples. The calculation formula is as follows: s.t.Y i (w·X i +b)-1≥0 i=1,2,…,n In the formula, w and b are the hyperplane parameters, and X... i For radar trajectory data, Y i For trajectory data categories; Step 804: Construct a binary classification support vector machine model. The calculation formula is as follows: f(X) = sign(w·X+b) In the formula, sign() is the sign function; Step 805: Input the test samples into the decision function and verify the classification accuracy using the root mean square error (RMSE). The calculation formula is as follows: In the formula, Y i Y represents the classification of the i-th sample obtained from the binary support vector machine model. ci Classify the i-th sample obtained from video observation.

6. The highway radar-visual fusion data processing method as described in claim 5, characterized in that, In step 8, the data is input into the constructed binary classification support vector machine model to determine the radar trajectory data X. i Is it abnormal: when f(X) i When ) = -1, then the radar trajectory data X is discarded. i When f(X) i When ) = 1, the radar trajectory data X is output. i .

Images