A multi-vehicle target tracking method based on deep learning
Through deep learning technology, combined with convolutional neural network and Kalman filtering algorithm, stable tracking of multi-vehicle targets in complex road environments is achieved, and the tracking difficulties in occlusion situations is solved, and the accuracy of multi-vehicle target tracking is improved. reliability.
Patent Information
- Application Number
- CN202210441387.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-25
AI Technical Summary
In complex road environments, multi-vehicle target tracking technology has difficulty maintaining stability and accuracy in occlusion, especially during peak hours and dense traffic flow, resulting in target tracking frame drifting and tracking failure.
Using a multi-vehicle target tracking method based on deep learning, the vehicle appearance features are extracted using a convolutional neural network, combined with the Kalman filtering algorithm to predict the motion state, and using a twin network to perform real-time single-object tracking under occlusion, using the average peak response as confidence The degree indicator detects the degree of occlusion to ensure the stability and accuracy of tracking.
Multi-vehicle target tracking when the vehicle is blocked and unobstructed is realized, which reduces tracking errors, improves tracking accuracy and reliability, and solves the problem of target identity jump and reset.
Smart Images

Figure CN114862910B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent vehicle autonomous driving environment perception technology, and in particular to a multi-vehicle target tracking method based on deep learning. Background Art
[0002] In recent years, with the rise of artificial intelligence technology, autonomous driving technology has entered the public eye, gradually achieved certain results, and has become one of the cutting-edge hotspots in the current automotive engineering field.
[0003] The autonomous driving system is mainly composed of three modules: environmental perception, path planning, and decision execution. Environmental perception is the primary task. As the basis of autonomous driving, it has an important impact on the safety and accuracy of autonomous driving. Environmental perception refers to the accurate perception of the road environment in which the vehicle is located, determining the type, location and movement trajectory of the target in time and space. It is specifically divided into target detection, target tracking and trajectory prediction, which can provide an information basis for subsequent path planning and decision execution.
[0004] Multi-vehicle target tracking is an important part of the environmental perception module. It can provide data information of multiple spatiotemporal points and positions of multiple targets, obtain the motion trajectory of each target, and provide important data information for target scene understanding, such as behavior feature analysis and prediction. However, due to the dense traffic flow during peak hours and road sections and the diverse weather conditions, the current road environment is complex, and multi-vehicle target tracking technology is very challenging.
[0005] In multi-vehicle target tracking tasks, solving the target occlusion problem is an important challenge. During driving, vehicles, pedestrians, and trees around the road may block the target vehicle, especially during peak hours, where vehicles may even be completely blocked from each other, causing the target to disappear. When the target is blocked, the tracking frame will drift, tracking will fail, and subsequent tasks will also be affected. Therefore, a multi-vehicle target tracking method is needed to ensure the stability and reliability of vehicle tracking tasks under target occlusion. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a multi-vehicle target tracking method based on deep learning in view of the defects involved in the background technology.
[0007] The present invention adopts the following technical solutions to solve the above technical problems:
[0008] A multi-vehicle target tracking method based on deep learning includes the following steps:
[0009] Step 1), obtaining detection data of M target vehicles ahead, and using a convolutional neural network to extract appearance features of the M target vehicles;
[0010] Step 2), based on the acquired detection data of the M target vehicles, the average peak response APR of each target vehicle is calculated, and the motion state of each target vehicle is predicted and updated using the Kalman filter algorithm;
[0011] Step 3), for each target vehicle, the average peak response APR is used as a confidence index to detect its occlusion degree;
[0012] Step 4), when the target vehicle is not blocked, tracking is performed based on the appearance characteristics and motion state of the target vehicle;
[0013] Step 5), when the target vehicle is blocked, the twin network is used to perform real-time single target tracking on the target vehicle.
[0014] As a further optimization scheme of the multi-vehicle target tracking method based on deep learning of the present invention, the specific steps of extracting the appearance features of the detected vehicle by using a convolutional neural network in step 1) are as follows:
[0015] The convolutional neural network model comprises a convolution module, a classification module and an orthogonal module; the convolution module comprises a plurality of convolution branches, each of which comprises a convolution layer, a pooling layer and a flattening layer; the classification module comprises a concatenation layer and a classification output layer; the orthogonal module comprises an orthogonal layer and an orthogonal output layer;
[0016] Step 1.1), determine the classification loss function;
[0017] In the classification module, the prediction probability of each category in the neural network is mapped by the ratio of the output values of each neuron in the output layer, that is: Where P n,m is the ratio of the output value of the mth neuron to the output values of all n neurons in the classification module; H m is the output value of the mth neuron in the classification output layer; n is the total number of neurons in the classification output layer;
[0018] The classification loss function reflects the degree of deviation between the predicted probability and the true probability of each category. Its function value is Where P m is the true probability of the mth neuron in the classification output layer of the neural network;
[0019] Step 1.2), determine the orthogonal loss function;
[0020] In a convolutional neural network, the convolutional layer extracts the feature vector Where I is the input sample; a and b are the height and width of I respectively; c and d are the height and width of the convolution kernel respectively; e and r are the number of steps the convolution kernel moves in the horizontal and vertical directions respectively; K q is the convolution kernel of the qth layer; u q is the bias term of the qth convolutional layer; σ is the activation function;
[0021] The output of the orthogonal layer is Where O is the output value of the orthogonal layer; R is any constant greater than 1; L is the total number of convolution branches;
[0022] The orthogonal loss function expression of the orthogonal output layer is:
[0023]
[0024] Where G2 is the orthogonal loss function value of the orthogonal output layer;
[0025] Step 1.3), determine the comprehensive loss function;
[0026] The comprehensive loss function of the convolutional neural network model is:
[0027] G=αG1+βG2
[0028] Where G is the comprehensive loss function value of the neural network model; α and β are the coefficients of the classification loss function and the orthogonal loss function respectively;
[0029] In step 1.4), the detection data of the target vehicle in front is input, and the convolutional neural network model is trained with a comprehensive loss function. The orthogonal feature vectors, i.e., the appearance features of the target vehicle, are extracted by multiple convolution branches.
[0030] As a further optimization scheme of the multi-vehicle target tracking method based on deep learning of the present invention, the specific steps of the Kalman filter algorithm in step 2) are as follows:
[0031] Step 2.1), predict the motion state and estimate two state parameters of the target tracking box, namely the mean and covariance;
[0032] The mean value x of the target tracking frame is [cx, cy, h, k, vx, vy, vk, vh], where cx and cy are the coordinates of the center point of the target tracking frame, h is the height of the target tracking frame, k is the ratio of the width to the height of the target tracking frame, vx is the speed change value of the horizontal coordinate of the center point of the target tracking frame, vy is the speed change value of the vertical coordinate of the center point of the target tracking frame, vk is the speed change value of the ratio of the width to the height of the target tracking frame, and vh is the speed change value of the height of the target tracking frame.
[0033] Estimate the mean: x t =Fx t-1, where x t and x t-1 are the means of the target tracking frames at time t and time t-1 respectively; F is the state transfer matrix;
[0034] Estimate the covariance: Y t =FY t-1 F T +M, where Y t and Y t-1 are the covariance of the target tracking frame at time t and time t-1 respectively; M is the system noise matrix;
[0035] Step 2.2), update the motion state, the update process is as follows:
[0036] Δ=z-Qx t-1
[0037] S=QY t-1 Q T +N
[0038] λ=Y t-1 Q T S -1
[0039] x t =x t-1 +λΔ
[0040] Y t =(I-λQ)Y t-1
[0041] Where Δ is the mean error between the target detection box and the target prediction box; z is the mean vector of the target detection box; Q is the measurement matrix; S is the state transfer matrix; N is the noise matrix of the detector; λ is the Kalman gain; and I is the unit matrix.
[0042] As a further optimization scheme of the multi-vehicle target tracking method based on deep learning of the present invention, the calculation formula of the confidence index APR in step 3) is:
[0043]
[0044] In the formula, J max , J min , J w,h They are the maximum value, minimum value, and w-row and h-column elements of the W×H correlation response matrix in target vehicle detection; when the APR fluctuates violently, the target vehicle is obscured.
[0045] As a further optimization scheme of the multi-vehicle target tracking method based on deep learning of the present invention, the specific steps of tracking the target vehicle according to the appearance characteristics and motion state in step 4) are as follows:
[0046] Step 4.1), calculate the matching degree of the target vehicle appearance features, the calculation formula is In the formula, f i,j is the matching degree of the target vehicle appearance features; γ j is the orthogonality feature vector extracted from the jth target detection box, and ||γ j ||=1;γ k is any orthogonal eigenvector;
[0047] Step 4.2), calculate the matching degree of the target vehicle's motion state, the calculation formula is In the formula, g i,j is the matching degree of the target vehicle’s motion state; x i and x j are the means of the i-th and j-th target detection boxes respectively; Y i is the covariance of the i-th target detection box;
[0048] Step 4.3), weight the appearance features and motion state of the target vehicle, and match the target vehicle with all vehicles in the previous cycle, which is expressed as follows:
[0049] c i,j =ξf i,j +(1-ξ)g i,j
[0050] In the formula, c i,j is the final matching degree of the target vehicle; ξ is the influencing factor of the appearance feature, and the influence of the appearance feature and the motion state on the data association is adjusted by controlling ξ;
[0051] Step 4.4), when the matching metric c i,j If the appearance feature and the motion state matching threshold are within the intersection, the target detection box D t Associate with the successfully matched vehicle to achieve target tracking and update the target trajectory τ t .
[0052] As a further optimization scheme of the multi-vehicle target tracking method based on deep learning of the present invention, the specific steps of using the twin network to perform real-time single target tracking on the target vehicle in step 5) are as follows:
[0053] Step 5.1), create a static template;
[0054] The static template is used as the reference template for target vehicle tracking. It is a square area cut with the target vehicle position as the center. Where w′=w+ε×(w+h), h′=h+ε×(w+h); w and h are the width and height of the target vehicle respectively; ε is the influencing factor of the background information;
[0055] Step 5.2), update the dynamic template;
[0056] The confidence APR of the current frame and the maximum response peak J max Update the dynamic template for the indicator:
[0057]
[0058] Where κ and θ are the update frequencies of the control dynamic template; is the historical mean of confidence; is the historical mean of the maximum peak value of the response;
[0059] Step 5.3), learning the apparent changes and background suppression of the target vehicle;
[0060] During the tracking process, the apparent change of the target vehicle is defined as a smooth change, and the depth feature f(O t-1 ) and static template depth feature f(O s ) and dynamic template deep features f(O d ) is similar to solving the apparent change V of the target vehicle. s and V d :
[0061] V s =arg min||V*f(O s )-f(O t-1 )|| 2 +λ V ||V|| 2
[0062] V d =arg min||V*f(O d )-f(O t-1 )|| 2 +λ V ||V|| 2 Where V is the apparent change of the target vehicle learned during the tracking process; λ V is the regularization parameter;
[0063] At frame t-1, a region G that is the same as the search region is cropped with the predicted position as the center t-1 , through G t-1 The deep feature f(G t-1 ) is the deep feature after fusion with the target likelihood probability map of the region Similarly, solve for background suppression W t-1 :
[0064] Where W is the background suppression of the target vehicle learned during the tracking process; W is the regularization parameter.
[0065] Step 5.4), positioning the target vehicle;
[0066] During tracking, the change in the target vehicle’s appearance V s and V d With the template image feature f(O s ) and f(O d ) performs a circular convolution operation to suppress the background W t-1 and the search area feature f(Z t ) performs a circular convolution operation; performs similarity calculation and weighted fusion on the features extracted by the template branch and the search branch to obtain the final target position of the current frame t:
[0067] R s =corr(V s *f(O s ),W t-1 *f(Z t ))
[0068] R d =corr(V d *f(O d ),W t-1 *f(Z t ))
[0069]
[0070] In the formula, R s , R d They are the response graphs obtained by calculating the similarity of static template and dynamic template features; R t It is the response map after weighted fusion, and the highest response position is the final position of the predicted target; is a hyperparameter that controls the weight;
[0071] Step 5.5), update the target scale;
[0072] After obtaining the final position of the target vehicle, multi-scale sampling is performed with the predicted position as the center, and the target scale is updated:
[0073] (w,h)=ν(w t-1 ,h t-1 )+(1-ν)(w t ,h t )
[0074] Where (w,h) represents the final updated target scale; (w t ,h t) represents the maximum scale of the t-frame response; (w t-1 ,h t-1 ) represents the target scale of frame t-1; ν is the changing speed of the target scale.
[0075] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0076] The present invention takes into account the situation where vehicles are obscured during target tracking, proposes a tracking framework that combines a single target tracker with a multi-target tracker, and uses deep learning to achieve multi-vehicle target tracking, thereby ensuring the real-time and accuracy of target tracking. This solves the problem of identity jump and reset, and target loss caused by similar vehicles being difficult to distinguish due to partial occlusion when continuously tracking multiple vehicles in front, and achieves multi-vehicle target tracking in both obscured and unobstructed situations, thereby reducing errors in the vehicle tracking process, ensuring the reliability of vehicle tracking, and improving the accuracy of vehicle tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] Figure 1 A flowchart of a multi-vehicle target tracking method of the present invention;
[0078] Figure 2 It is a structural diagram of the convolutional neural network model of the present invention;
[0079] Figure 3 It is a flow chart of the single target tracking algorithm of the present invention. DETAILED DESCRIPTION
[0080] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:
[0081] The present invention can be implemented in many different forms and should not be considered to be limited to the embodiments described herein. On the contrary, these embodiments are provided to make this disclosure thorough and complete, and will fully express the scope of the present invention to those skilled in the art. In the accompanying drawings, components are enlarged for clarity.
[0082] like Figure 1 As shown, the present invention discloses a multi-vehicle target tracking method based on deep learning, comprising the following steps:
[0083] Step 1), obtaining the detection data of M target vehicles in front, and using a convolutional neural network to extract the appearance features of the M target vehicles;
[0084] Step 2), based on the acquired detection data of the M target vehicles, the average peak response APR of each target vehicle is calculated, and the motion state of each target vehicle is predicted and updated using the Kalman filter algorithm;
[0085] Step 3), for each target vehicle, the average peak response APR is used as a confidence index to detect its occlusion degree;
[0086] Step 4), when the target vehicle is not blocked, tracking is performed based on the appearance characteristics and motion state of the target vehicle;
[0087] Step 5), when the target vehicle is blocked, the twin network is used to perform real-time single target tracking on the target vehicle.
[0088] The specific steps of using a convolutional neural network to extract the appearance features of the detected vehicle in step 1) are as follows:
[0089] like Figure 2 As shown, the convolutional neural network model includes a convolution module, a classification module and an orthogonal module; the convolution module includes several convolution branches, each of which includes a convolution layer, a pooling layer and a flattening layer; the classification module includes a splicing layer and a classification output layer; the orthogonal module includes an orthogonal layer and an orthogonal output layer;
[0090] Step 1.1), determine the classification loss function;
[0091] In the classification module, the prediction probability of each category in the neural network is mapped by the ratio of the output values of each neuron in the output layer, that is: Where P n,m is the ratio of the output value of the mth neuron to the output values of all n neurons in the classification module; H m is the output value of the mth neuron in the classification output layer; n is the total number of neurons in the classification output layer;
[0092] The classification loss function reflects the degree of deviation between the predicted probability and the true probability of each category. Where P m is the true probability of the mth neuron in the classification output layer of the neural network;
[0093] Step 1.2), determine the orthogonal loss function;
[0094] In a convolutional neural network, the convolutional layer extracts the feature vector Where I is the input sample; a and b are the height and width of I respectively; c and d are the height and width of the convolution kernel respectively; e and r are the number of steps the convolution kernel moves in the horizontal and vertical directions respectively; K q is the convolution kernel of the qth layer; u q is the bias term of the qth convolutional layer; σ is the activation function;
[0095] The output of the orthogonal layer is Where O is the output value of the orthogonal layer; R is any constant greater than 1; L is the total number of convolution branches;
[0096] The orthogonal loss function expression of the orthogonal output layer is:
[0097]
[0098] Where G2 is the orthogonal loss function value of the orthogonal output layer;
[0099] Step 1.3), determine the comprehensive loss function;
[0100] The comprehensive loss function of the convolutional neural network model is:
[0101] G=αG1+βG2
[0102] Where G is the comprehensive loss function value of the neural network model; α and β are the coefficients of the classification loss function and the orthogonal loss function respectively;
[0103] In step 1.4), the detection data of the target vehicle in front is input, and the convolutional neural network model is trained with a comprehensive loss function. The orthogonal feature vectors, i.e., the appearance features of the target vehicle, are extracted by multiple convolution branches.
[0104] The specific steps of the Kalman filter algorithm in step 2) are as follows:
[0105] Step 2.1), predict the motion state and estimate two state parameters of the target tracking box, namely the mean and covariance;
[0106] The mean value x of the target tracking frame is [cx, cy, h, k, vx, vy, vk, vh], where cx and cy are the coordinates of the center point of the target tracking frame, h is the height of the target tracking frame, k is the ratio of the width to the height of the target tracking frame, vx is the speed change value of the horizontal coordinate of the center point of the target tracking frame, vy is the speed change value of the vertical coordinate of the center point of the target tracking frame, vk is the speed change value of the ratio of the width to the height of the target tracking frame, and vh is the speed change value of the height of the target tracking frame.
[0107] Estimate the mean: x t =Fx t-1 , where x t and x t-1 are the means of the target tracking frames at time t and time t-1 respectively; F is the state transfer matrix;
[0108] Estimate the covariance: Y t =FY t-1 F T +M, where Y t and Y t-1 are the covariance of the target tracking frame at time t and time t-1 respectively; M is the system noise matrix;
[0109] Step 2.2), update the motion state, the update process is as follows:
[0110] Δ=z-Qx t-1
[0111] S=QY t-1 Q T +N
[0112] λ=Y t-1 Q T S -1
[0113] x t =x t-1 +λΔ
[0114] Y t =(I-λQ)Y t-1
[0115] Where Δ is the mean error between the target detection box and the target prediction box; z is the mean vector of the target detection box; Q is the measurement matrix; S is the state transfer matrix; N is the noise matrix of the detector; λ is the Kalman gain; and I is the unit matrix.
[0116] The calculation formula of the confidence index APR in step 3) is:
[0117]
[0118] In the formula, J max , J min , J w,h They are the maximum value, minimum value, and w-row and h-column elements of the W×H correlation response matrix in target vehicle detection; when the APR fluctuates violently, the target vehicle is obscured.
[0119] The specific steps of tracking the target vehicle according to the appearance characteristics and motion state in step 4) are as follows:
[0120] Step 4.1), calculate the matching degree of the target vehicle appearance features, the calculation formula is In the formula, f i,j is the matching degree of the target vehicle appearance features; γ j is the orthogonality feature vector extracted from the jth target detection box, and ||γ j ||=1;γ k is any orthogonal eigenvector;
[0121] Step 4.2), calculate the matching degree of the target vehicle's motion state, the calculation formula is In the formula, g i,j is the matching degree of the target vehicle’s motion state; x i and x jare the means of the i-th and j-th target detection boxes respectively; Y i is the covariance of the i-th target detection box;
[0122] Step 4.3), weight the appearance features and motion state of the target vehicle, and match the target vehicle with all vehicles in the previous cycle, which is expressed as follows:
[0123] c i,j =ξf i,j +(1-ξ)g i,j
[0124] In the formula, c i,j is the final matching degree of the target vehicle; ξ is the influencing factor of the appearance feature, and the influence of the appearance feature and the motion state on the data association is adjusted by controlling ξ;
[0125] Step 4.4), when the matching metric c i,j If the appearance feature and the motion state matching threshold are within the intersection, the target detection box D t Associate with the successfully matched vehicle to achieve target tracking and update the target trajectory τ t .
[0126] like Figure 3 As shown, the specific steps of using the twin network to perform real-time single target tracking on the target vehicle in step 5) are as follows:
[0127] Step 5.1), create a static template;
[0128] The static template is used as the reference template for target vehicle tracking. It is a square area cut with the target vehicle position as the center. Where w′=w+ε×(w+h), h′=h+ε×(w+h); w and h are the width and height of the target vehicle respectively; ε is the influencing factor of the background information;
[0129] Step 5.2), update the dynamic template;
[0130] The confidence APR of the current frame and the maximum response peak J max Update the dynamic template for the indicator:
[0131]
[0132] Where κ and θ are the update frequencies of the control dynamic template; is the historical mean of confidence; is the historical mean of the maximum peak value of the response;
[0133] Step 5.3), learning the apparent changes and background suppression of the target vehicle;
[0134] During the tracking process, the apparent change of the target vehicle is defined as a smooth change, and the depth feature f(O t-1 ) and static template depth feature f(O s ) and dynamic template deep features f(O d ) is similar to solving the apparent change V of the target vehicle. s and V d :
[0135] V s =arg min||V*f(O s )-f(O t-1 )|| 2 +λ V ||V|| 2
[0136] V d =arg min||V*f(O d )-f(O t-1 )|| 2 +λ V ||V|| 2 Where V is the apparent change of the target vehicle learned during the tracking process; λ V is the regularization parameter;
[0137] At frame t-1, a region G that is the same as the search region is cropped with the predicted position as the center t-1 , through G t-1 The deep feature f(G t-1 ) is the deep feature after fusion with the target likelihood probability map of the region Similarly, solve for background suppression W t-1 :
[0138] Where W is the background suppression of the target vehicle learned during the tracking process; W is the regularization parameter.
[0139] Step 5.4), positioning the target vehicle;
[0140] During tracking, the change in the target vehicle’s appearance V s and V d With the template image feature f(O s ) and f(O d ) performs a circular convolution operation to suppress the background W t-1 and the search area feature f(Z t ) performs a circular convolution operation; performs similarity calculation and weighted fusion on the features extracted by the template branch and the search branch to obtain the final target position of the current frame t:
[0141] R s =corr(V s *f(O s ),W t-1 *f(Z t ))
[0142] R d =corr(V d *f(O d ),W t-1 *f(Z t ))
[0143]
[0144] In the formula, R s , R d They are the response graphs obtained by calculating the similarity of static template and dynamic template features; R t It is the response map after weighted fusion, and the highest response position is the final position of the predicted target; is a hyperparameter that controls the weight;
[0145] Step 5.5), update the target scale;
[0146] After obtaining the final position of the target vehicle, multi-scale sampling is performed with the predicted position as the center, and the target scale is updated:
[0147] (w,h)=ν(w t-1 ,h t-1 )+(1-ν)(w t ,h t )
[0148] Where (w,h) represents the final updated target scale; (w t ,h t ) represents the maximum scale of the t-frame response; (w t-1 ,h t-1 ) represents the target scale of frame t-1; ν is the changing speed of the target scale.
[0149] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0150] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A multi-vehicle target tracking method based on deep learning, characterized in that: The following steps are involved: Step 1), obtaining detection data of M target vehicles ahead, and using a convolutional neural network to extract appearance features of the M target vehicles; The convolutional neural network model comprises a convolution module, a classification module and an orthogonal module; the convolution module comprises a plurality of convolution branches, each of which comprises a convolution layer, a pooling layer and a flattening layer; the classification module comprises a concatenation layer and a classification output layer; the orthogonal module comprises an orthogonal layer and an orthogonal output layer; Step 1.1), determine the classification loss function; In the classification module, the prediction probability of each category in the neural network is mapped by the ratio of the output values of each neuron in the output layer, that is: Where P n,m is the ratio of the output value of the mth neuron to the output values of all n neurons in the classification module; H m is the output value of the mth neuron in the classification output layer; n is the total number of neurons in the classification output layer; The classification loss function reflects the degree of deviation between the predicted probability and the true probability of each category. Where P m is the true probability of the mth neuron in the classification output layer of the neural network; Step 1.2), determine the orthogonal loss function; In a convolutional neural network, the convolutional layer extracts the feature vector Where I is the input sample; a and b are the height and width of I respectively; c and d are the height and width of the convolution kernel respectively; e and r are the number of steps the convolution kernel moves in the horizontal and vertical directions respectively; K q is the convolution kernel of the qth layer; u q is the bias term of the qth convolutional layer; σ is the activation function; The output of the orthogonal layer is Where O is the output value of the orthogonal layer; R is any constant greater than 1; L is the total number of convolution branches; The orthogonal loss function expression of the orthogonal output layer is: Where G2 is the orthogonal loss function value of the orthogonal output layer; Step 1.3), determine the comprehensive loss function; The comprehensive loss function of the convolutional neural network model is: G=αG1+βG2 Where G is the comprehensive loss function value of the neural network model; α and β are the coefficients of the classification loss function and the orthogonal loss function respectively; Step 1.4), input the detection data of the target vehicle in front, and train the convolutional neural network model with a comprehensive loss function. The orthogonal feature vectors, i.e., the appearance features of the target vehicle, are extracted from multiple convolution branches; Step 2), based on the acquired detection data of the M target vehicles, the average peak response APR of each target vehicle is calculated, and the motion state of each target vehicle is predicted and updated using the Kalman filter algorithm; the specific steps of the Kalman filter algorithm are as follows: Step 2.1), predict the motion state and estimate two state parameters of the target tracking box, namely the mean and covariance; The mean value x of the target tracking frame is [cx, cy, h, k, vx, vy, vk, vh], where cx and cy are the coordinates of the center point of the target tracking frame, h is the height of the target tracking frame, k is the ratio of the width to the height of the target tracking frame, vx is the speed change value of the horizontal coordinate of the center point of the target tracking frame, vy is the speed change value of the vertical coordinate of the center point of the target tracking frame, vk is the speed change value of the ratio of the width to the height of the target tracking frame, and vh is the speed change value of the height of the target tracking frame. Estimate the mean: x t =Fx t-1 , where x t and x t-1 are the means of the target tracking frames at time t and time t-1 respectively; F is the state transfer matrix; Estimate the covariance: Y t =FY t-1 F T +M, where Y t and Y t-1 are the covariance of the target tracking frame at time t and time t-1 respectively; M is the system noise matrix; Step 2.2), update the motion state, the update process is as follows: Δ=z-Qx t-1 S=QY t-1 Q T +N λ=Y t-1 Q T S -1 x t =x t-1 +λΔ Y t =(I-λQ)Y t-1 Where Δ is the mean error between the target detection box and the target prediction box; z is the mean vector of the target detection box; Q is the measurement matrix; S is the state transfer matrix; N is the noise matrix of the detector; λ is the Kalman gain; I is the identity matrix; Step 3), for each target vehicle, the average peak response APR is used as a confidence index to detect its degree of occlusion; the calculation formula of the confidence index APR is: In the formula, J max , J min , J w,h They are the maximum value, minimum value, and w-row and h-column elements of the W×H-sized correlation response matrix in target vehicle detection; When the APR fluctuates violently, the target vehicle is obscured; Step 4), when the target vehicle is not blocked, tracking is performed based on the appearance characteristics and motion state of the target vehicle; Step 4.1), calculate the matching degree of the target vehicle appearance features, the calculation formula is In the formula, f i,j is the matching degree of the target vehicle’s appearance features; γ j is the orthogonality feature vector extracted from the jth target detection box, and ||γ j ||=1;γ k is any orthogonal eigenvector; Step 4.2), calculate the matching degree of the target vehicle's motion state, the calculation formula is g i,j =(x j -x i ) T Y i -1 (x j -x i ), where g i,j is the matching degree of the target vehicle’s motion state; x i and x j are the means of the i-th and j-th target detection boxes respectively; Y i is the covariance of the i-th target detection box; Step 4.3), weight the appearance features and motion state of the target vehicle, and match the target vehicle with all vehicles in the previous cycle, which is expressed as follows: c i,j =ξf i,j +(1-ξ)g i,j In the formula, c i,j is the final matching degree of the target vehicle; ξ is the influencing factor of the appearance feature, and the influence of the appearance feature and the motion state on the data association is adjusted by controlling ξ; Step 4.4), when the matching metric c i,j If the appearance feature and the motion state matching threshold are within the intersection, the target detection box D t Associate with the successfully matched vehicle to achieve target tracking and update the target trajectory τ t ; Step 5), when the target vehicle is blocked, the twin network is used to perform real-time single target tracking on the target vehicle; Step 5.1), create a static template; The static template is used as the reference template for target vehicle tracking. It is a square area cut with the target vehicle position as the center. Where, w′=w+ε×w+h), h′=h+ε×(w+h); w and h are the width and height of the target vehicle respectively; ε is the influence factor of background information; Step 5.2), update the dynamic template; The confidence APR of the current frame and the maximum response peak J max Update the dynamic template for the indicator: Where κ and θ are the update frequencies of the control dynamic template; is the historical mean of confidence; is the historical mean of the maximum peak value of the response; Step 5.3), learning the apparent changes and background suppression of the target vehicle; During the tracking process, the apparent change of the target vehicle is defined as a smooth change, and the depth feature f(O t-1 ) and static template depth feature f(O s ) and dynamic template deep features f(O d ) is similar to solving the apparent change V of the target vehicle. s and V d : V s =arg min||V*f(O s )-f(O t-1 )|| 2 +λ V ||V|| 2 V d =arg min||V*f(O d )-f(O t-1 )|| 2 +λ V ||V|| 2 Where V is the apparent change of the target vehicle learned during the tracking process; λ V is the regularization parameter; At frame t-1, a region G that is the same as the search region is cropped with the predicted position as the center t-1 , through G t-1 The deep feature f(G t-1 ) is the deep feature after fusion with the target likelihood probability map of the region Similarly, solve for background suppression W t-1 : Where W is the background suppression of the target vehicle learned during the tracking process; W is the regularization parameter; Step 5.4), positioning the target vehicle; During tracking, the change in the target vehicle’s appearance V s and V d With the template image feature f(O s ) and f(O d ) performs a circular convolution operation to suppress the background W t-1 and the search area feature f(Z t ) performs a circular convolution operation; performs similarity calculation and weighted fusion on the features extracted by the template branch and the search branch to obtain the final target position of the current frame t: R s =corr(V s *f(O s ),W t-1 *f(Z t )) R d =corr(V d *f(O d ),W t-1 *f(Z t )) In the formula, R s , R d They are the response graphs obtained by calculating the similarity of static template and dynamic template features; R t It is the response map after weighted fusion, and the highest response position is the final position of the predicted target; is a hyperparameter that controls the weight; Step 5.5), update the target scale; After obtaining the final position of the target vehicle, multi-scale sampling is performed with the predicted position as the center, and the target scale is updated: (w,h)=v(w t-1 ,h t-1 )+(1-v)(w t ,h t ) Where (w,h) represents the final updated target scale; (w t ,h t ) represents the maximum scale of the t-frame response; (w t-1 ,h t-1 ) represents the target scale of frame t-1; v is the speed of change of the target scale.
