Multimodal aerial video vehicle tracking method integrating scene prior knowledge

Abstract

The present invention provides a multimodal aerial video vehicle target tracking method that integrates scene prior knowledge, including the steps of associated region of interest extraction, local image edge detection, local image main color extraction, appearance weight vector calculation, scene prior information library construction, next frame target state variable prediction and target association result output. The method fully exploits the multimodal small target features, fully considers the temporal and spatial constraints between frames, and makes full use of scene prior information. The obtained target tracking result is more accurate, solves the difficulty of target tracking, namely the ID confusion problem, realizes stable and accurate aerial video vehicle target tracking, and is used to construct a real and accurate vehicle driving trajectory, extract traffic parameters from the trajectory, provide real and effective data support for intelligent vehicle research, lay the foundation for data-driven intelligent vehicle research methods, and support research in the fields of intelligent vehicle perception, decision planning, control, testing, etc.

Description

Technical Field

[0001] The present invention relates to a method for tracking vehicle targets in aerial video, and in particular to a method for tracking vehicle targets in aerial video that combines appearance features and motion features with multimodal fusion of scene prior knowledge. Background Art

[0002] In recent years, advances in computer technology have spurred the development of intelligent transportation systems and intelligent vehicle research. Key research areas such as vehicle trajectory prediction, traffic flow modeling, and driver behavior modeling have extensively leveraged computer vision techniques. Data-driven research methods, due to their efficiency, flexibility, and reliability, have become crucial tools in intelligent vehicle research. Accurate and efficient data collection methods and effective preprocessing techniques are key challenges.

[0003] With the rapid development of drone technology, the data it collects offers advantages such as high resolution, high lateral and vertical accuracy, and unobstructed viewing, making it a popular method for collecting vehicle driving data. Extracting traffic parameters from aerial video involves several steps: video preprocessing, target detection, target tracking, trajectory extraction, and traffic parameter extraction. Target tracking is a crucial step. Based on the target detection results, it assigns a unique ID to each target in each frame. Its accuracy directly affects trajectory accuracy and is crucial for obtaining accurate traffic parameters and in-depth driving research.

[0004] Existing aerial video target tracking methods have the following problems:

[0005] 1. Targets in aerial video are small, feature-rich, and often have similar vehicle models, which can easily cause ID confusion and make target tracking difficult. Directly applying existing general target tracking methods to small vehicle targets in aerial video yields poor results.

[0006] 2. Existing aerial video object tracking methods fail to adequately consider the spatiotemporal constraints between frames. Due to the fixed video frame rate and the kinematic laws of vehicles, a vehicle ID in one frame can only be associated with a target within a specific range in the next frame. Most existing research fails to specifically consider this spatiotemporal constraint, potentially causing an ID in one frame to be associated with a distant target in the next frame, leading to persistent ID association errors and reduced tracking accuracy.

[0007] 3. Existing aerial video target tracking methods do not fully utilize prior information. Aerial video capture is accompanied by a large amount of rich scene prior information. Most existing studies do not use this prior information to improve tracking accuracy, resulting in a waste of prior information. Summary of the Invention

[0008] In order to solve the above technical problems, in the process of tracking vehicle targets in aerial video, we should fully exploit the characteristics of small targets, solve the ID confusion problem caused by similar lighting and vehicle models, and integrate multimodal features; we should fully consider the temporal and spatial constraints between frames; and we should make full use of the abundant scene prior information during the aerial video shooting process to improve the target tracking accuracy. In order to solve the above technical problems, the present invention provides a multimodal aerial video vehicle target tracking method that integrates scene prior knowledge, including the following steps:

[0009] Step 1: Extraction of associated regions of interest:

[0010] Convert the aerial video into video frames, use the target detection algorithm to obtain the vehicle target detection frame in each frame, count the number of target detection frames, obtain the horizontal and vertical coordinates of the center of all target detection frames, the detection frame width, and the detection frame aspect ratio; save the global image with the detection frame of each frame, crop all vehicle targets in each frame according to the target detection frame, save it as a local image according to the frame number, obtain the detection frames of the front vehicle, rear vehicle, left front vehicle, left vehicle, left rear vehicle, right front vehicle, right vehicle, and right rear vehicle of the target, define the minimum enclosing rectangle of these 8 detection frames as the associated region of interest, and record the location information of the associated region of interest.

[0011] Step 2: Local image edge detection:

[0012] Calculate the Sobel operator for all local images to obtain a binary edge map, remove the pseudo-edges, and obtain a more accurate edge map. Extract the front windshield edge, sunroof edge, and rear windshield edge of all local images from the edge map, determine the front windshield area, sunroof area, and rear windshield area, calculate the front windshield area s1, sunroof area s2, and rear windshield area s3, and calculate the area eigenvalue ε s , the formula is as follows:

[0013] ε s =s1:s2:s3.

[0014] Step 3: Extract the main color of the local image:

[0015] The sampling area is determined based on the front windshield area, sunroof area, and rear windshield area in step 2, and the sampling area is defined as all pixels in the area outside the front windshield area, sunroof area, and rear windshield area removed from the local image; uniform sampling is performed within the sampling area to obtain several sampling pixels, and the R, G, and B values ​​of all the sampling pixels are averaged to obtain The RGB space is converted to the HSV space to obtain the hue H value and saturation S value to form the color feature vector [H, S] of the local image. The HSV model can classify various colors using only the hue H and saturation S that reflect the essential characteristics of color, removing the brightness and grayscale information from the color, thereby removing the influence of lighting and processing color and brightness separately, which has stronger robustness.

[0016] Step 4: Calculate the appearance weight vector:

[0017] Read the next frame and define all targets within the associated ROI in step 1 as associated ROIs, including targets partially within the ROI. Repeat steps 2 and 3 for all associated ROIs, obtain their color feature vectors [H, S], calculate their cosine distances with the color feature vectors of the local image, obtain 9 cosine distances, record them as a nine-dimensional vector, and perform normalization. Define this nine-dimensional vector as the appearance weight vector λ a :

[0018] λ a =[l1,l2,l3,l4,l5,l6,l7,l8,l9]

[0019] Where l1, l2, l3, l4, l5, l6, l7, l8, l9 are the cosine distances between the color feature vectors of the associated target of interest and the local image;

[0020] If a surrounding vehicle does not exist, the cosine distance of this item is recorded as 0.

[0021] Step 5: Build a scene prior information database:

[0022] Aerial video capture is accompanied by a wealth of prior information. Most current research fails to utilize this prior information to improve tracking accuracy, resulting in a waste of prior information. This paper considers three types of prior information suitable for aerial vehicle scenes before using the Kalman filter method for target tracking: road structure, traffic density, and weather conditions. Geometric constraints are established based on road structure, and speed constraints are constructed based on traffic density and weather conditions. These constraints are then quantified to improve the Kalman filter's target tracking accuracy.

[0023] Step 6: Prediction of target state variables for the next frame:

[0024] Based on the output of the scene prior information library in step 5, a fusion-constrained Kalman filter is constructed to predict the target state variables of the next frame based on the target state variables of the current frame;

[0025] Under the aerial photography perspective with a fixed hovering height, the aspect ratio change rate and width change rate of different vehicle targets are not obvious, so the state variables in the new Kalman filter designed by the present invention are:

[0026] x=[cx,cy,γ,h,ε s ,v cx ,v cy ] T

[0027] Where cx is the horizontal coordinate of the center of the detection frame, cy is the vertical coordinate of the center of the detection frame, γ is the aspect ratio of the detection frame, h is the width of the detection frame, and ε s is the area eigenvalue, v cx is the speed of change of the horizontal coordinate of the center of the detection frame, v cy is the speed of change of the vertical coordinate of the center of the detection frame;

[0028] Step 7: Output of target association results:

[0029] Based on the next frame target state variable obtained in step 6, calculate the Mahalanobis distance d(i, j) between it and the state variable of the associated target of interest:

[0030]

[0031] Where i is the target state variable of the next frame, j is the associated target state variable of interest, and ∑ is the covariance matrix of the multidimensional random variable.

[0032] Combined with the appearance weight vector in step 4, calculate the multimodal association weight:

[0033] α i,j =λ a d(i,j)

[0034] Where, α i,j is the multimodal association weight, λ a is the appearance weight vector;

[0035] Select the associated target of interest with the highest multimodal association weight as the association result, and output the association result;

[0036] Repeat step 1 to update the associated region of interest, repeat steps 2 to 7 to continue associating vehicle targets in the remaining videos, and output the association results of all targets.

[0037] Furthermore, the conversion formula from RGB space to HSV space in step 3 is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] Δ=Mm

[0044]

[0045]

[0046] Where, are the average values ​​of the brightness of the R, G, and B channels of the sampled pixels, They are Normalized brightness value, M is The maximum value among the three, m is The minimum value of the three, Δ is the difference between M and m, H is the hue in the HSV color space, and S is the saturation in the HSV color space.

[0047] Furthermore, step five of constructing a scene prior information database includes the following steps:

[0048] (1) In terms of road structure, three types of road structures are considered: urban expressway without ramps, urban expressway with ramps, and roundabouts. Pixel coordinates are converted into real coordinates, and a road structure expression is established: a linear polynomial is fitted for the straight line segment of the expressway; for the ramp segment of the expressway, a horizontal arc curve is used to fit the entrance and exit, and a transition curve is used to fit the middle section of the ramp; for the roundabout, an inner circle curve and an outer circle curve are fitted according to the inner circle radius and the outer circle radius;

[0049] After obtaining the mathematical expression of the road structure, the vehicle position coordinates (c x ,c y ) is constrained and the mean square error method is used to establish road geometry constraints to improve the accuracy and robustness of state estimation.

[0050] (2) In terms of traffic flow density, when the number of vehicles on the road increases, the time required for starting and braking will accumulate for each vehicle due to the decrease in the longitudinal distance between vehicles, resulting in reduced traffic efficiency and a decrease in the average speed. By counting the number of detection boxes n in the global image, the traffic flow density influencing factor is defined to obtain the speed constraint in target tracking. The formula is shown below:

[0051] δ n =c1n

[0052] Where, δ nis the traffic flow influencing factor, c1 is the constant balance coefficient of speed constraint, and n is the number of detection boxes;

[0053] (3) Weather conditions: In bad weather, vehicles travel at lower speeds and tend to maintain a larger headway with the vehicle ahead. The present invention quantifies the impact of weather conditions on vehicle travel as visibility and road adhesion coefficient, defines visibility influencing factors and adhesion coefficient influencing factors, and defines weather influencing factors accordingly:

[0054]

[0055] Where, δ w is the weather influence factor, c2 is the constant balance coefficient of the weather influence factor, ρ is the visibility influence factor, is the factor affecting the adhesion coefficient;

[0056] Combining the traffic flow influencing factors and weather influencing factors, the speed impact weight expression is obtained:

[0057] δ v =δ n δ w .

[0058] Where, δ v The speed influence weight.

[0059] Furthermore, in step 6, the process of constructing a fusion-constrained Kalman filter based on the scene prior information library is as follows:

[0060] (1) The state transfer equation is: X k+1 =FX k +ω k , initialized to:

[0061]

[0062] Where, X k+1 is the predicted value of the next frame state variable, F is the state transfer matrix, X k is the current frame state variable, ω k is the process error vector;

[0063] (2) The measurement equation is: L k =A k X k +R, initialized to:

[0064]

[0065] Where, L k is the observed variable, A k is the observation matrix, X kis the current frame state variable, R is the measurement error vector;

[0066] (3) The road structure constraint formula is: BX k -D=0

[0067] Where B is the design matrix and D is a constant vector, which is approximated based on the road structure expression described in step 5;

[0068] (4) The revised predicted state value is:

[0069] Where, is the correction value of the predicted value of the state variable in the next frame, For X k+1 , B is the design matrix, and D is the constant vector;

[0070] (5) The covariance equation of the predicted state vector is:

[0071] Where, is the predicted state vector covariance, F is the state transfer matrix, is the covariance of the current frame state vector, δ v is the speed influence weight, which is used to incorporate speed constraints, Q k is the process noise covariance matrix, which is used to represent the error between the state estimation process and the actual process;

[0072] (6) Kalman gain is:

[0073] Where, is the predicted state vector covariance, A k is the observation matrix;

[0074] (7) The state update equation is:

[0075] Where, is the posterior state estimate, is the prior state estimate, K k is the Kalman gain, A k is the observation matrix, L k is the measured value;

[0076] (8) The covariance equation of the shape update vector is:

[0077]

[0078] Where K k is the Kalman gain, A k is the observation matrix, is the state vector covariance, ∑k+1 is the covariance;

[0079] Based on the current frame target state variable, the next frame target state variable is predicted through the above steps.

[0080] Beneficial effects of the present invention:

[0081] The present invention provides a multimodal aerial video vehicle target tracking method that integrates scene prior knowledge. It is aimed at the application of aerial vehicle video, fully explores the multimodal small target characteristics, fully considers the temporal and spatial constraints between frames, and makes full use of scene prior information. The obtained target tracking results are more accurate, and the difficulty of target tracking, namely the ID confusion problem, is solved. Stable and accurate aerial video vehicle target tracking is achieved to construct a real and accurate vehicle driving trajectory, extract traffic parameters from the trajectory, provide real and effective data support for intelligent vehicle research, lay the foundation for data-driven intelligent vehicle research methods, and support research in the fields of intelligent vehicle perception, decision planning, control, testing, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0083] Figure 2 Schematic diagram of the associated region of interest of the present invention. DETAILED DESCRIPTION

[0084] See Figure 1 As shown, the present invention provides a multimodal aerial video vehicle target tracking method that integrates scene prior knowledge, based on a region of interest extraction and update module, a morphological processing module, a priori scene library construction module, a fusion constraint Kalman filter prediction module, and a target association module, including the following steps:

[0085] Step 1: In the ROI Extraction and Update module, perform associated ROI extraction:

[0086] Convert the aerial video into video frames, use the target detection algorithm to obtain the vehicle target detection frame in each frame, count the number of target detection frames, obtain the horizontal and vertical coordinates of the center of all target detection frames, the detection frame width, and the detection frame aspect ratio; save the global image with the detection frame of each frame, crop all vehicle targets in each frame according to the target detection frame, save it as a local image according to the frame number, obtain the detection frames of the front vehicle, rear vehicle, left front vehicle, left vehicle, left rear vehicle, right front vehicle, right vehicle, and right rear vehicle of the target, define the minimum enclosing rectangle of these 8 detection frames as the associated region of interest, and record the location information of the associated region of interest.

[0087] Step 2: Perform local image edge detection in the edge detection submodule of the morphological processing module:

[0088] The Sobel operator is calculated for all local images to obtain a binary edge map. Morphological closing, opening, and connected region analysis are then used to remove pseudo edges to obtain a more accurate edge map. The front windshield edge, sunroof edge, and rear windshield edge of all local images are extracted from the edge map. The front windshield area, sunroof area, and rear windshield area are determined. The front windshield area s1, sunroof area s2, and rear windshield area s3 are calculated, and the area eigenvalue ε is calculated. s , the formula is as follows:

[0089] ε s =s1:s2:s3.

[0090] Step 3: In the main color extraction submodule of the morphological processing module, extract the main color of the local image:

[0091] The sampling area is determined based on the front windshield area, sunroof area, and rear windshield area in step 2. The sampling area is defined as all pixels in the area outside the front windshield area, sunroof area, and rear windshield area removed from the local image. Sample uniformly within the sampling area to obtain 100 sampling pixels, and take the average value of the R, G, and B values ​​of all sampling pixels to obtain Value, then convert the RGB space to HSV space, obtain the hue H value and saturation S value, and form the color feature vector [H, S] of the local image; the conversion formula from RGB space to HSV space is as follows:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] Δ=Mm

[0098]

[0099]

[0100] Where, are the average values ​​of the brightness of the R, G, and B channels of the sampled pixels, They are Normalized brightness value, M is The maximum value among the three, m is The minimum value of the three, Δ is the difference between M and m, H is the hue in the HSV color space, and S is the saturation in the HSV color space.

[0101] Step 4: Calculate the appearance weight vector in the morphological processing module:

[0102] Read the next frame and define all targets within the associated region of interest in step 1 as associated targets of interest, including targets partially within the region of interest, such as Figure 2 As shown, repeat steps 2 and 3 for all associated objects of interest to obtain their color feature vectors [H, S], calculate the cosine distance between them and the color feature vectors of the local image, obtain 9 cosine distances, record them as a nine-dimensional vector and perform normalization operation, and define this nine-dimensional vector as the appearance weight vector λ a :

[0103] λ a =[l1,l2,l3,l4,l5,l6,l7,l8,l9]

[0104] Where l1, l2, l3, l4, l5, l6, l7, l8, l9 are the cosine distances between the color feature vectors of the associated target of interest and the local image;

[0105] If a surrounding vehicle does not exist, the cosine distance of this item is recorded as 0.

[0106] Step 5: Build a scene prior information library in the prior scene library construction module:

[0107] The scene prior information includes: road structure, traffic flow density, and weather conditions. Geometric constraints are established based on the road structure, speed constraints are constructed based on the traffic flow density and weather conditions, and quantified to improve the target tracking accuracy of the Kalman filter.

[0108] (1) In terms of road structure, three types of road structures are considered: urban expressway without ramps, urban expressway with ramps, and roundabouts. Pixel coordinates are converted into real coordinates, and a road structure expression is established: a linear polynomial is fitted for the straight line segment of the expressway; for the ramp segment of the expressway, a horizontal arc curve is used to fit the entrance and exit, and a transition curve is used to fit the middle section of the ramp; for the roundabout, an inner circle curve and an outer circle curve are fitted according to the inner circle radius and the outer circle radius;

[0109] After obtaining the mathematical expression of the road structure, the vehicle position coordinates (c x ,c y ) is constrained and the mean square error method is used to establish road geometry constraints to improve the accuracy and robustness of state estimation.

[0110] (2) In terms of traffic flow density, when the number of vehicles on the road increases, the time required for starting and braking will accumulate for each vehicle due to the decrease in the longitudinal distance between vehicles, resulting in reduced traffic efficiency and a decrease in the average speed. By counting the number of detection boxes n in the global image, the traffic flow density influencing factor is defined to obtain the speed constraint in target tracking. The formula is shown below:

[0111] δ n =c1n

[0112] Where, δ n is the traffic flow influencing factor, c1 is the constant balance coefficient of speed constraint, and n is the number of detection boxes;

[0113] (3) The present invention quantifies the impact of weather conditions on vehicle driving as visibility and road adhesion coefficient;

[0114] This embodiment classifies road visibility levels according to the "Fog Forecast Level" and defines visibility influencing factors based on the classification results, as shown in the following table:

[0115]

[0116] According to the average adhesion coefficient on various road surfaces in Automobile Theory, the adhesion coefficient influencing factors are defined as shown in the following table:

[0117]

[0118] The weather influencing factor is defined based on the visibility influencing factor and the adhesion coefficient influencing factor:

[0119]

[0120] Where, δ w is the weather influence factor, c2 is the constant balance coefficient of the weather influence factor, ρ is the visibility influence factor, is the factor affecting the adhesion coefficient;

[0121] Combining the traffic flow influencing factors and weather influencing factors, the speed impact weight expression is obtained:

[0122] δ v =δ n δ w .

[0123] Where, δ v The speed influence weight.

[0124] Step 6: Predict the next frame target state variable in the fusion constraint Kalman filter prediction module:

[0125] The state variables in the Kalman filter of this embodiment are:

[0126] x=[cx,cy,γ,h,ε s ,v cx ,v cy ] T

[0127] Where cx is the horizontal coordinate of the center of the detection frame, cy is the vertical coordinate of the center of the detection frame, γ is the aspect ratio of the detection frame, h is the width of the detection frame, and ε s is the area eigenvalue, v cx is the speed of change of the horizontal coordinate of the center of the detection frame, v cy is the speed of change of the vertical coordinate of the center of the detection frame;

[0128] Based on the output of the scene prior information library in step 5, a Kalman filter with fusion constraints is constructed. The process is as follows:

[0129] (1) The state transfer equation is: X k+1 =FX k +ω k , initialized to:

[0130]

[0131] Where, X k+1 is the predicted value of the next frame state variable, F is the state transfer matrix, X k is the current frame state variable, ω k is the process error vector;

[0132] (2) The measurement equation is: L k =A k X k +R, initialized to:

[0133]

[0134] Where, L k is the observed variable, A k is the observation matrix, X k is the current frame state variable, R is the measurement error vector;

[0135] (3) The road structure constraint formula is: BX k -D=0

[0136] Where B is the design matrix and D is a constant vector, which is approximated based on the road structure expression described in step 5;

[0137] (4) The revised predicted state value is:

[0138] Where, is the correction value of the predicted value of the state variable in the next frame, For Xk+1 , B is the design matrix, and D is the constant vector;

[0139] (5) The covariance equation of the predicted state vector is:

[0140] Where, is the predicted state vector covariance, F is the state transfer matrix, is the covariance of the current frame state vector, δ v is the speed influence weight, which is used to incorporate speed constraints, Q k is the process noise covariance matrix, which is used to represent the error between the state estimation process and the actual process;

[0141] (6) Kalman gain is:

[0142] Where, is the predicted state vector covariance, A k is the observation matrix;

[0143] (7) The state update equation is:

[0144] Where, is the posterior state estimate, is the prior state estimate, K k is the Kalman gain, A k is the observation matrix, L k is the measured value;

[0145] (8) The state update vector covariance equation is:

[0146]

[0147] Where K k is the Kalman gain, A k is the observation matrix, is the state vector covariance, ∑ k+1 is the covariance;

[0148] Based on the current frame target state variable, predict the next frame target state variable;

[0149] Step 7: Determine the target association result in the target association module and output it:

[0150] Based on the next frame target state variable obtained in step 6, calculate the Mahalanobis distance d(i, j) between it and the state variable of the associated target of interest:

[0151]

[0152] Where i is the target state variable of the next frame, j is the associated target state variable of interest, and ∑ is the covariance matrix of the multidimensional random variable;

[0153] Combined with the appearance weight vector in step 4, calculate the multimodal association weight:

[0154] α i,j =λ a d(i,j)

[0155] Where, α i,j is the multimodal association weight, λ a is the appearance weight vector;

[0156] Select the associated target of interest with the highest multimodal association weight as the association result, and output the association result;

[0157] Repeat step 1 to update the associated region of interest, repeat steps 2 to 7 to continue associating vehicle targets in the remaining videos, and output the association results of all targets.

Claims

1. A multimodal aerial video vehicle target tracking method integrating scene prior knowledge, characterized in that: The following steps are involved: Step 1: Extraction of associated regions of interest: Convert the aerial video into video frames, use the target detection algorithm to obtain the vehicle target detection frame in each frame, count the number of target detection frames, and obtain the horizontal and vertical coordinates of the center of all target detection frames, the detection frame width, and the detection frame aspect ratio; Save the global image with the detection frame of each frame, crop all vehicle targets in each frame according to the target detection frame, save them as local images according to the frame number, obtain the detection frames of the front vehicle, rear vehicle, left front vehicle, left vehicle, left rear vehicle, right front vehicle, right vehicle, and right rear vehicle of the target, define the minimum circumscribed rectangle of these 8 detection frames as the associated region of interest, and record the location information of the associated region of interest; Step 2: Local image edge detection: Calculate for all local images Operator, after obtaining the binary edge map, remove the pseudo edge to obtain a more accurate edge map, extract the front windshield edge, sunroof edge, and rear windshield edge of all local images from the edge map, determine the front windshield area, sunroof area, and rear windshield area, and calculate the front windshield area , skylight area , rear windshield area , calculate the area eigenvalue , the formula is as follows: ， Step 3: Extract the main color of the local image: The sampling area is determined based on the front windshield area, sunroof area, and rear windshield area in step 2, and the sampling area is defined as all pixel points in the area other than the front windshield area, sunroof area, and rear windshield area removed from the local image; uniform sampling is performed within the sampling area to obtain several sampling pixel points, and all sampling pixel points are uniformly sampled. The values ​​are averaged to obtain value, and then Space conversion to Space, get hue Value and Saturation Value, the color feature vector that constitutes the local image ; Step 4: Calculate the appearance weight vector: Read the next frame and define all targets within the associated region of interest in step 1 as associated targets of interest, including some targets within the region of interest. Repeat steps 2 and 3 for all associated targets of interest to obtain their color feature vectors. , calculate the cosine distance between it and the color feature vector of the local image, obtain 9 cosine distances, record them as a nine-dimensional vector and perform normalization operation, and define this nine-dimensional vector as the appearance weight vector : ， Where, are the cosine distances between the color feature vectors of the associated target of interest and the local image; If a surrounding vehicle does not exist, the cosine distance of this item is recorded as 0; Step 5: Build a scene prior information database: The vehicle scenario prior information includes: road structure, traffic flow density, and weather conditions. Geometric constraints are established based on the road structure, and speed constraints are constructed based on the traffic flow density and weather conditions, and are quantified. Step 6: Prediction of target state variables for the next frame: Based on the output of the scene prior information library in step 5, a fusion-constrained Kalman filter is constructed to predict the target state variables of the next frame based on the target state variables of the current frame; The state variables of the Kalman filter are: ， Where, is the horizontal coordinate of the center of the detection frame, is the vertical coordinate of the center of the detection frame, is the detection frame aspect ratio, is the detection frame width, is the area eigenvalue, is the speed of change of the horizontal coordinate of the center of the detection frame, is the speed of change of the vertical coordinate of the center of the detection frame; Step 7: Output of target association results: Based on the next frame target state variable obtained in step 6, calculate the Mahalanobis distance between it and the state variable of the associated target of interest : ， Where, is the target state variable for the next frame, To associate the target state variable of interest, is the covariance matrix of the multidimensional random variable; Combined with the appearance weight vector in step 4, calculate the multimodal association weight: ， Where, is the multimodal association weight, is the appearance weight vector; Select the associated target of interest with the highest multimodal association weight as the association result, and output the association result; Repeat step 1 to update the associated region of interest, repeat steps 2 to 7 to continue associating vehicle targets in the remaining videos, and output the association results of all targets.

2. The multimodal aerial video vehicle target tracking method integrating scene prior knowledge according to claim 1 is characterized by: In step three Space conversion to The space conversion formula is as follows: ， ， ， Where, are the average values ​​of the brightness of the R, G, and B channels of the sampled pixels, They are Normalized brightness value, The maximum of the three, The minimum of the three, The difference, Hue in color space, Saturation in color space.

3. The multimodal aerial video vehicle target tracking method integrating scene prior knowledge according to claim 1 is characterized by: Step 5: Constructing a scene prior information database includes the following steps: (1) In terms of road structure, three types of road structures are considered: urban expressway without ramps, urban expressway with ramps, and roundabouts. Pixel coordinates are converted into real coordinates, and a road structure expression is established: a linear polynomial is fitted for the straight line segment of the expressway; for the ramp segment of the expressway, a horizontal arc curve is used to fit the entrance and exit, and a transition curve is used to fit the middle section of the ramp; for the roundabout, an inner circle curve and an outer circle curve are fitted according to the inner circle radius and the outer circle radius; After obtaining the mathematical expression of the road structure, the vehicle position coordinates Constraints are performed and road geometry constraints are established using the mean square error method; (2) In terms of traffic flow density, when the number of vehicles on the road increases, the longitudinal distance between vehicles decreases, and the time required for vehicle starting and braking will accumulate from vehicle to vehicle, resulting in reduced traffic efficiency and lower average speed. By counting the number of detection boxes in the global image, , define the traffic flow density influencing factor and obtain the speed constraint in target tracking. The formula is shown as follows: ， Where, is the traffic flow influencing factor, is the constant balance coefficient of the velocity constraint, is the number of detection boxes; (3) In terms of weather conditions, the impact of weather conditions on vehicle driving is quantified as visibility and road adhesion coefficient, and the visibility influencing factor and adhesion coefficient influencing factor are defined. The weather influencing factor is defined accordingly: ， Where, is the weather factor, is the constant balance coefficient of the weather influence factor, is the visibility influencing factor, is the factor affecting the adhesion coefficient; Combining the traffic flow density influencing factor and the weather influencing factor, the speed impact weight expression is obtained: ， Where, The speed influence weight.

4. The multimodal aerial video vehicle target tracking method integrating scene prior knowledge according to claim 1 is characterized by: In step 6, the process of constructing a fusion-constrained Kalman filter based on the scene prior information library is as follows: (1) The state transfer equation is: , initialized to: ， Where, is the predicted value of the state variable of the next frame, is the state transition matrix, is the current frame state variable, is the process error vector; (2) The measurement equation is: , initialized to: ， Where, is the observed variable, is the observation matrix, is the current frame state variable, is the measurement error vector; (3) The road structure constraint formula is: , Where B is the design matrix and D is a constant vector; (4) The revised predicted state value is: , Where, is the correction value of the predicted value of the state variable in the next frame, , B is the design matrix, and D is the constant vector; (5) The covariance equation of the predicted state vector is: , Where, is the predicted state vector covariance, F is the state transfer matrix, is the covariance of the current frame state vector, is the speed influence weight, used to incorporate speed constraints, is the process noise covariance matrix, which is used to represent the error between the state estimation process and the actual process; (6) Kalman gain is: , Where, is the predicted state vector covariance, is the observation matrix; (7) The state update equation is: , Where, is the posterior state estimate, is the prior state estimate, is the Kalman gain, is the observation matrix, is the measured value; (8) The state update vector covariance equation is: ， Where, is the Kalman gain, is the observation matrix, is the state vector covariance, is the covariance; Based on the current frame target state variable, the next frame target state variable is predicted through the above steps.

Images