A method for tracking multiple vehicles in video

Abstract

The present invention discloses a method for tracking multiple vehicles in a video, comprising constructing a deep neural network model that integrates vehicle detection with self-supervised visual representation learning, constructing a target feature loss function and a target detection loss function for dual comparison of instances and dimensions, training the network model on a target detection dataset without identity labels, and constructing a visual similarity association method with a placeholder mechanism; converting the video data to be tracked for multiple vehicles into images frame by frame, sequentially feeding them into a trained neural network for target detection and feature extraction of the vehicles, performing correlation matching on all vehicle features between two adjacent frames, and obtaining tracking results for all vehicles within the detection area after traversing all frame images. The present invention reduces additional computational overhead, improves model calculation speed, greatly reduces the cost of constructing a target tracking dataset, avoids tracking instability caused by vehicle targets disappearing and reappearing due to temporary occlusion, and makes multi-vehicle tracking more accurate.

Description

Technical Field

[0001] The invention belongs to the technical field of pattern recognition and relates to a multi-vehicle tracking method in a video. Background Art

[0002] The multi-vehicle tracking task involves three modules: target detection, visual representation, and data association. Existing multi-vehicle tracking methods often employ a two-stage model: first, target detection is performed in the target area, followed by visual representation and data association of the detected target images or extracted target features. Model training often requires specialized datasets containing target location, size, and identity information, requiring extensive manual annotation. Furthermore, multi-vehicle tracking often results in tracking loss due to occlusion and subsequent reappearance of vehicles. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a method for tracking multiple vehicles in video, which reduces additional computing overhead, improves the model calculation speed, greatly reduces the cost of constructing a target tracking data set, effectively records disappearing and newly appearing vehicles, avoids tracking instability caused by vehicle targets disappearing and reappearing due to temporary occlusion, and makes multi-vehicle tracking more accurate.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] A method for tracking multiple vehicles in a video comprises the following steps:

[0006] Perform random image transformation on the collected images to form a data enhancement algorithm set

[0007] The image dataset is divided into several instance sets Z by random sampling. i , each instance set has a corresponding label set E i , i∈{1,2,…T}, T is the number of instance sets;

[0008] From the data enhancement algorithm collection Randomly select an algorithm For each instance Z i ,E i Perform the transformation to obtain the sample set Z' i ,E' i ;

[0009] According to the sample set, a backbone network model of a deep neural network model integrating vehicle detection and self-supervised visual representation learning is constructed, and two sample sets Z are extracted in pairs. i and Z' i The characteristic matrix ofi and h' i ;

[0010] According to the feature matrix of the sample set, a branch network with the same structure is constructed. i and h i ′ is predicted and the heat map F is obtained i and F' i , center offset O i and O' i , target size S i and S' i ;

[0011] Based on the branch network with the same structure, the target feature representation network model and the projection layer network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning are constructed. The two feature matrices h i and h' i The target feature representation H is obtained through the target feature representation network i and H' i , target feature representation H i and H' i The target feature matrix is obtained by projection, denoted as L i and L' i ;

[0012] The target feature matrix L i and L' i Divide along the row and column vectors respectively to obtain instance features and dimension features, and construct positive and negative instance feature pairs and related and irrelevant dimension feature pairs;

[0013] The feature relationship between instance feature pairs is calculated using the cosine distance measurement function, and the feature relationship between dimension feature pairs is calculated using the correlation coefficient measurement function. A dual contrast loss function of instance features and dimension features is constructed as the target feature loss function.

[0014] Use Gaussian model to label E i ,E' i The ground truth bounding box information is mapped to a heat map And calculate the label target center offset and target size

[0015] Calculate the sample thermal feature map F through the Focal Loss loss function i and F' i and The relationship between The loss function calculates the sample center point offset O i and O'i and The relationship between the target sample size S i and S' i and According to the calculated loss function, the target detection loss function is constructed, and the total loss function of the target detection and feature calculation of the model is formed together with the sample dimension double comparison loss function;

[0016] The deep neural network model that integrates vehicle detection and self-supervised visual representation learning is trained based on the total loss function of target detection and feature calculation;

[0017] Video Dataset Contains N frames of images, the first frame image I 1 Send it into the trained deep neural network model to get the corresponding center point position P 1 , bounding box size S 1 And the target feature matrix H 1 , initialize the identity information and trajectory information of all targets;

[0018] The feature matrix of two consecutive frames is used for correlation algorithm to obtain the correlation relationship between the targets in the frames, the trajectory of the same target is updated, and the identity and trajectory of the newly appeared target are initialized;

[0019] The deep neural network model finally obtains the tracking results of the video data after traversing all frame images

[0020] Optionally, random image transformations include random horizontal flipping, random grayscale, color jitter, Gaussian blur, Gaussian noise, sharpening, and overexposure.

[0021] Optionally, the image dataset is a target detection dataset, that is, each image contains label information of the target position and size; for a target k in the image, its label information is is the horizontal and vertical coordinate value of the upper left corner of the target bounding box, is the width and height of the target bounding box.

[0022] Optionally, the backbone network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning is an encoder f(·). The encoder f(·) is a network model composed of a multi-layer convolutional neural network. The encoder f(·) is used to extract the embedded representation of the sample set. The calculation formula is as follows:

[0023]

[0024] B is the number of samples in the sample set, D is the size of the embedding vector, represents the set of real numbers.

[0025] Optionally, the encoder f(·) includes DarkNet-53, ResNet-50, and DLA-34.

[0026] Optionally, a branch network with the same structure is the prediction head p j (·), j = {1, 2, 3}, used to predict the heat map, center offset vector and target size respectively;

[0027] Each prediction head p j (·) Perform convolution calculation with a kernel size of 3×3 on the feature map. After passing through the batch normalization layer and nonlinear activation function, the prediction matrix of the sample set is obtained by 1×1 convolution calculation. It is used to predict the embedded representation of the sample set. The calculation formula is as follows:

[0028]

[0029] I is the size of the sample thermal signature;

[0030]

[0031] J is the size of the sample target center point offset vector prediction;

[0032]

[0033] K is the size of the sample target size prediction.

[0034] Optionally, the target feature characterization network model is the target feature characterizer c(·), and the projection layer network model is the projection head g(·);

[0035] The target feature characterizer c(·) includes a convolutional neural network and a feature channel set network. The target feature is represented as the set of all channels at the target center point coordinates of the feature matrix obtained by convolution calculation. The calculation formula is as follows:

[0036]

[0037] N is the number of sample target center points, and D is the number of channels of the feature matrix obtained by convolution calculation;

[0038] The projection head g(·) consists of a three-layer fully connected network. The first two layers are followed by a batch normalization layer and a nonlinear activation function to calculate the feature matrix of the sample set. The calculation formula is as follows:

[0039]

[0040] M is the size of the sample output target feature matrix.

[0041] Optionally, the two feature matrices L i and L' i Divide by row to get the instance representation set t represents the image enhancement algorithm, where 1 means no image enhancement algorithm is used, i is the instance index, and the instance pair is constructed from the set S. is a positive pair, i≠j is a negative pair; the two feature matrices L i and L' i Divide by column to get the feature dimension set p is the dimension index, constructing the dimension pair from the set D are related pairs, and p≠k are unrelated pairs.

[0042] Optionally, the similarity between instance feature pairs is calculated using the cosine distance metric function. The formula for the cosine distance metric function is:

[0043]

[0044] |·| means finding the modulus of the sample feature;

[0045] The correlation coefficient measurement function is used to calculate the correlation between dimension feature pairs. The correlation coefficient measurement function formula is:

[0046]

[0047] and is the average value of all elements in the dimension feature;

[0048] The instance feature contrast loss function is:

[0049]

[0050] The dimensional feature contrast loss function is:

[0051]

[0052] τ1, τ2 are hyperparameters of the loss function, representing temperature parameters;

[0053] Combined with instance feature contrast loss function Compared with the dimension feature loss function The target feature loss function is obtained as follows:

[0054]

[0055] Where λ is a parameter that controls the loss balance.

[0056] Optionally, the label information representing the target position and size in the sample data set is (x1, y1, x2, y2), and its center point Round down to get

[0057] Use the Gaussian model to map the true bounding box information in the label to a heat map The calculation formula is:

[0058]

[0059] Y xy is the heat map mapping value, indicating the heat map The value of the pixel at (x,y) coordinate, is the variance related to the bounding box size. If the Gaussian distributions of the two center points overlap, the maximum value is taken.

[0060] Calculate the label target center point offset and target size The formula is as follows:

[0061]

[0062] p i ,p' i Z i ,Z' i The target center point coordinates;

[0063]

[0064] Optionally, the sample thermal feature map F is calculated using the Focal Loss function i and F' i and The relationship between, Focal Loss loss function formula is:

[0065]

[0066] Represents the thermal characteristic diagram F i ,F' i The estimated value of the pixel at the coordinate (x, y) above, c is the number of categories of the target to be detected; α and β are hyperparameters used to balance the number of difficult and easy samples; Y xyc Representing thermal characteristics The label value of the pixel at the coordinate (x, y) above; N is the number of sample target center points;

[0067] pass The loss function calculates the sample center point offset O i and O' i and The relationship between them, the loss function calculation formula is:

[0068]

[0069] pass Loss function calculation sample target size S i and S' i and The relationship between them, the loss function calculation formula is:

[0070]

[0071] The loss functions of the three branches are added together by fixing the weights to obtain the overall loss of the detection target. The loss function calculation formula is as follows:

[0072]

[0073] λ1,λ2 are fixed weights;

[0074] The target detection loss function is combined with the target feature loss function to construct the overall loss function. The calculation formula is as follows:

[0075]

[0076] Where w1, w2 are learnable parameters that balance the two tasks.

[0077] Optionally, transform the sample set Z' i and label set E' i , and the original sample set Z i and the original label set E i The samples are input into the deep neural network model in pairs, and the feature matrix is calculated to obtain the corresponding predicted heat map, center offset and target size. The feature matrix is then calculated to obtain the corresponding target feature matrix. The label set is calculated to obtain the corresponding label heat map, center offset and target size. The results of the predicted heat map, center offset and target size, target feature matrix, label heat map, center offset and target size are substituted into the total loss function of target detection and feature calculation for calculation. The deep neural network model is back-propagated according to the total loss function to automatically adjust and optimize the model parameters.

[0078] Optionally, the video dataset The first image in I 1 Input the trained deep neural network model to obtain the initialized identity information and trajectory information. The calculation formula is as follows:

[0079]

[0080] H 1 =c(f(I 1 ));

[0081] From the heat map F 1 and the center point offset O 1 The target center point position can be calculated using the following formula:

[0082] P 1 =q(F 1 ,O 1 ).

[0083] Optionally, the video dataset All images in the image are fed into the trained deep neural network model for calculation and the trajectory of the tracking target is recorded. in represents the trajectory information of target i in frame t, is the center point coordinate, is the bounding box size, is the feature representation of the target, ID i is the target’s identity, l i is the survival time parameter;

[0084] The association relationship between targets in two consecutive image frames is obtained by the data association algorithm AFRS based on feature representation similarity. The calculation formula is:

[0085]

[0086] Similarity Matrix N t-1 ,N t is the number of targets in the t-1,t frame image. If target i and target j have the same identity, then S i,j >0, otherwise S i,j <0;

[0087] In the similarity matrix S i,j Add a column vector P with a fixed value as a placeholder to form a new matrix The value of P is the similarity matrix S i,j The average value of all elements except the diagonal in , the placeholder is used to match the disappeared target or the newly appeared target;

[0088] The similarity matrix S'i,j is calculated along the row vector using the Softmax function with a temperature coefficient to obtain the correlation matrix M in which each row contains the similarity results between the current target and all targets in the two frames:

[0089]

[0090] T is the temperature parameter,

[0091] Use adaptive temperature regulation T:

[0092] T=2log(N t-1 +N t +1);

[0093] Record the detection results of all targets where N t represents the number of targets in the t-th frame image,

[0094] Perform AFRS association algorithm on the feature matrix of two consecutive frames of images. If the target i exists in the current frame and is successfully associated with the previous frame, the detection result is used. Update the target trajectory and set the center point coordinates Bounding box size Identity ID i And the current frame number t is saved to the tracking result If the association fails, the trajectory information of the previous frame remains unchanged and the survival time l i Add 1, if l i >30, the target is considered to have left the tracking screen.

[0095] Optionally, the video dataset is fed into the trained deep neural network model frame by frame to obtain the corresponding target detection prediction and target feature matrix, which is then fed into the AFRS association algorithm to associate the feature relationships between the targets and record all target trajectories and identity information in the tracking results. middle.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] The present invention provides a method for tracking multiple vehicles in videos. It constructs a tracking framework that integrates vehicle detection and self-supervised visual representation. It combines the target detection and visual representation modules of the traditional two-stage vehicle tracking model. It simultaneously performs target detection and learns highly discriminative visual representations, reducing additional computational overhead and improving model calculation speed.

[0098] This method uses a data augmentation algorithm to obtain different view samples of the same image and maximizes the relationship between the same object in different view samples based on the instance and dimension dual contrast loss function, enabling the model to extract accurate target features. Compared with traditional supervised learning-based target tracking algorithms, this method does not rely on identity labels in the dataset during training but learns the target's own information. It can be trained on traditional target detection datasets, eliminating the need to build dedicated target tracking datasets, greatly reducing the cost of datasets that require manual annotation.

[0099] The present invention designs a visual similarity association algorithm (AFRS) with an added placeholder mechanism. Based on target feature representation, it calculates the similarity of target pairs between frames and constructs an association matrix, making it easier to identify disappearing and newly appearing targets. In multi-vehicle scenarios, it can still accurately track all vehicle targets, temporarily save the features of disappeared targets, and set a survival time to determine whether the target has left the tracking screen, greatly alleviating the problem of tracking instability caused by target occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 A flowchart of a deep neural network model training process integrating vehicle detection and self-supervised visual representation learning according to an embodiment of the present invention;

[0101] Figure 2 These are some examples of the UA-DETRAC dataset in the embodiment of the present invention;

[0102] Figure 3 This is a data enhancement transformation picture according to an embodiment of the present invention;

[0103] Figure 4 A flow chart of a visual similarity association method with a placeholder mechanism added to an embodiment of the present invention;

[0104] Figure 5 This is a flowchart of multi-vehicle tracking according to an embodiment of the present invention;

[0105] Figure 6 The initial frame image and the final frame image of the road monitoring video data to be subjected to multi-vehicle tracking in an embodiment of the present invention;

[0106] Figure 7 This is a display of multi-vehicle tracking results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0107] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0108] The endpoints of the ranges disclosed herein and any values are not limited to the exact ranges or values, and these ranges or values should be understood to include values close to these ranges or values. For numerical ranges, the endpoints of each range, the endpoints of each range and individual point values, and the individual point values can be combined with each other to obtain one or more new numerical ranges, which should be considered to be specifically disclosed herein.

[0109] For the purposes of this specification and the appended claims, unless otherwise indicated, all numbers expressing amounts, percentages or ratios and other numerical values used in this specification and the appended claims are to be understood as being modified in all instances by the term "about." In addition, all ranges disclosed herein are inclusive and independently combinable.

[0110] Example 1

[0111] Figure 2 This is a demonstration of some examples from the UA-DETRAC dataset, which collects more than 140,000 frames of road surveillance video images from 24 different locations in the real world, and annotates the identities of 8,250 vehicles. The dataset also annotates more than 1.21 million labeled object bounding boxes.

[0112] like Figures 1 to 7 As shown, a method for tracking multiple vehicles in a video includes the following steps:

[0113] S1, randomly transform the image to form a data enhancement algorithm set

[0114] The image is randomly transformed into random horizontal flipping, random grayscale or color jittering. The specific parameters of each method follow the general scheme of current self-supervised training.

[0115] S2, using random sampling method to divide the image dataset into several instance sets Z i , each instance set has a corresponding label set E i , i∈{1,2,…T}, T is the number of instance sets;

[0116] The image dataset is a target detection dataset, that is, each image contains label information of the target position and size; for a target k in the image, its label information is is the horizontal and vertical coordinate value of the upper left corner of the target bounding box, is the width and height of the target bounding box;

[0117] S3, from the data enhancement algorithm collection Randomly select an algorithm For each instance Z i ,E iPerform the transformation to obtain the sample set Z' i ,E' i ;

[0118] Instance Z i After data enhancement algorithm After the transformation, its label information also undergoes the same transformation to ensure that the transformed label information E' i With Z' i correspond;

[0119] S4, based on the sample set, builds the backbone network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning, and extracts two sample sets Z in pairs i and Z' i The characteristic matrix of i and h' i ;

[0120] The backbone network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning is the encoder f(·);

[0121] The encoder f(·) is a DLA-34 network model. The encoder f(·) is used to extract the embedding representation of the sample set. The calculation formula is as follows:

[0122] B is the number of samples in the sample set, D is the size of the embedding vector, represents the set of real numbers;

[0123] S5, construct three branch networks with the same structure according to the feature matrix of the sample set, respectively i and h i ′ is predicted and the heat map F is obtained i and F' i , center offset O i and O' i , target size S i and S' i ;

[0124] Three branch networks with exactly the same structure are the prediction head p j (·), j = {1, 2, 3}, used to predict the heat map, center offset vector and target size respectively;

[0125] Each prediction head p j (·) The feature map is first convolved with a kernel size of 3×3, then passed through a batch normalization layer and a nonlinear activation function, and finally a 1×1 convolution is performed to obtain the prediction matrix of the sample set, which is used to predict the embedded representation of the sample set in S4. The calculation formula is as follows:

[0126] I is the size of the sample thermal signature;

[0127] J is the size of the sample target center point offset vector prediction;

[0128] K is the size of the sample target size prediction.

[0129] S6, based on the branch network with the same structure, constructs the target feature representation network model and projection layer network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning. The two feature matrices h i and h' i The target feature representation H is obtained through the target feature representation network i and H' i , target feature representation H i and H' i The target feature matrix is obtained by projection, denoted as L i and L' i ;

[0130] The target feature representation network model is the target feature characterizer c(·), and the projection layer network model is the projection head g(·);

[0131] The target feature characterizer c(·) consists of a two-layer convolutional neural network and a feature channel set network. The target feature is represented as the set of all channels at the target center point coordinates of the feature matrix obtained by convolution calculation. The calculation formula is as follows:

[0132] N is the number of sample target center points, and D is the number of channels of the feature matrix obtained by convolution calculation;

[0133] The projection head g(·) consists of a three-layer fully connected network. The first two layers are followed by a batch normalization layer and a nonlinear activation function to calculate the feature matrix of the sample set. The calculation formula is as follows:

[0134] M is the size of the sample output target feature matrix;

[0135] S7, the target feature matrix L i and L' i Divide along the row and column vectors respectively to obtain instance features and dimension features, and construct positive and negative instance feature pairs and related and irrelevant dimension feature pairs;

[0136] Two feature matrices L i and L' i Divide by row to get the instance representation set t represents the image enhancement algorithm, where 1 means no image enhancement algorithm is used, i is the instance index, and the instance pair is constructed from the set S. is a positive pair, i≠j is a negative pair; the two feature matrices L i and L' i Divide by column to get the feature dimension set p is the dimension index, constructing the dimension pair from the set D are related pairs, and p≠k are unrelated pairs;

[0137] S8, calculates the similarity between instance feature pairs using the cosine distance measurement function, calculates the correlation between dimension feature pairs using the correlation coefficient measurement function, and constructs a dual contrast loss function of instance features and dimension features as the target feature loss function;

[0138] The cosine distance metric function formula is:

[0139] ||·|| means finding the modulus of the sample feature;

[0140] The correlation coefficient measurement function formula is:

[0141] and is the average value of all elements in the dimension feature;

[0142] The instance feature contrast loss function is:

[0143]

[0144] The dimensional feature contrast loss function is:

[0145]

[0146] τ1, τ2 are hyperparameters of the loss function, representing temperature parameters;

[0147] Combined with instance feature contrast loss function Compared with the dimension feature loss function The target feature loss function is obtained as follows:

[0148] Where λ is the parameter that controls the loss balance;

[0149] S9, using Gaussian model to label E i ,E' i The ground truth bounding box information is mapped to a heat map And calculate the label target center offset and target size

[0150] For the label information (x1, y1, x2, y2) representing the target position and size in the sample data set, the center point Round down to get

[0151] Use the Gaussian model to map the true bounding box information in the label to a heat map The calculation formula is:

[0152] Y xy is the heat map mapping value, indicating the heat map The value of the pixel at (x,y) coordinate, is the variance related to the bounding box size. If the Gaussian distributions of the two center points overlap, the maximum value is taken.

[0153] Calculate the label target center point offset and target size The formula is as follows:

[0154] p i ,p' i Z i ,Z' i The target center point coordinates;

[0155]

[0156] S10, calculate the sample thermal feature map F through the Focal Loss function i and F' i and The relationship between The loss function calculates the sample center point offset O i and O' i and The relationship between the target sample size S i and S' i and The relationship between the three loss functions is combined to construct the target detection loss function, which is then combined with the sample dimension double comparison loss function to form the total loss function of the model's target detection and feature calculation.

[0157] The Focal Loss loss function formula is:

[0158] Represents the thermal characteristic diagram F i ,F' iThe estimated value of the pixel at the coordinate (x, y) above, c is the number of categories of the target to be detected; α and β are hyperparameters used to balance the number of difficult and easy samples; Y xyc Representing thermal characteristics The label value of the pixel at the coordinate (x, y) above; N is the number of sample target center points;

[0159]

[0160] pass The loss function calculates the sample center point offset O i and O' i and The relationship between them, the loss function calculation formula is:

[0161]

[0162] pass Loss function calculation sample target size S i and S' i and The relationship between them, the loss function calculation formula is:

[0163]

[0164] The loss functions of the three branches are added together by fixing the weights to obtain the overall loss of the detection target. The loss function calculation formula is as follows:

[0165] λ1,λ2 are fixed weights;

[0166] Combine the target detection loss function with the target feature loss function in step (8) to construct the overall loss function. The calculation formula is as follows:

[0167] Where w1, w2 are learnable parameters that balance the two tasks.

[0168] S11, training the deep neural network model that integrates vehicle detection and self-supervised visual representation learning based on the total loss function of target detection and feature calculation;

[0169] Set Z in S3 i and label set E i Transformed sample set Z' i and label set E' i , and the original sample set Z i and the original label set E iThe deep neural network model is input in pairs. The sample set is calculated by S4 to obtain a feature matrix. The feature matrix is calculated by S5 to obtain the corresponding prediction heat map, center offset and target size. The feature matrix is then calculated by S6 to obtain the corresponding target feature matrix. The label set is calculated by S9 to obtain the corresponding label heat map, center offset and target size. The results of S5, S6 and S9 are substituted into the total loss function of target detection and feature calculation in S10 for calculation. The deep neural network model is back-propagated according to the total loss function to automatically adjust and optimize the model parameters.

[0170] S12, video dataset Contains N frames of images, the first frame image I 1 Send it into the trained deep neural network model to get the corresponding center point position P 1 , bounding box size S 1 And the target feature matrix H 1 , initialize the identity information and trajectory information of all targets;

[0171] The video dataset The first image in I 1 Input the trained deep neural network model to obtain the initialized identity information and trajectory information. The calculation formula is as follows:

[0172]

[0173] H 1 =c(f(I 1 ));

[0174] From the heat map F 1 and the center point offset O 1 The target center point position can be calculated using the following formula:

[0175] P 1 =q(F 1 ,O 1 );

[0176] S13, performing an association algorithm on the feature matrices of two consecutive frames to obtain the association relationship between the targets in the frames, updating the trajectory of the same target, and initializing the identity and trajectory of the newly appeared target;

[0177] like Figure 4 , the video dataset Perform S12 calculation on all images in the image and record the trajectory of the tracking target. in represents the trajectory information of target i in frame t, is the center point coordinate, is the bounding box size, is the feature representation of the target, ID i is the target’s identity, l i is the survival time parameter;

[0178] The association relationship between targets in two consecutive image frames is obtained by the data association algorithm AFRS based on feature representation similarity. The calculation formula is:

[0179] Similarity Matrix N t-1 ,N t is the number of targets in the t-1,t frame image. If target i and target j have the same identity, then S i,j >0, otherwise S i,j <0;

[0180] In the similarity matrix S i,j Add a column vector P with a fixed value as a placeholder to form a new matrix The value of P is the similarity matrix S i,j The average value of all elements except the diagonal in , the placeholder is used to match the disappeared target or the newly appeared target;

[0181] The similarity matrix S'i,j is calculated along the row vector using the Softmax function with a temperature coefficient to obtain the correlation matrix M in which each row contains the similarity results between the current target and all targets in the two frames:

[0182] T is the temperature parameter, and adaptive temperature is used to adjust T:

[0183] T=2log(N t-1 +N t +1);

[0184] Record the detection results of all targets where N t represents the number of targets in the t-th frame image,

[0185] Perform AFRS association algorithm on the feature matrix of two consecutive frames of images. If the target i exists in the current frame and is successfully associated with the previous frame, the detection result is used. Update the target trajectory and set the center point coordinates Bounding box size Identity ID i And the current frame number t is saved to the tracking result If the association fails, the trajectory information of the previous frame remains unchanged and the survival time l iAdd 1, if l i >30, the target is considered to have left the tracking screen;

[0186] S14, after the deep neural network model traverses all frame images, it finally obtains the tracking results of the video data.

[0187] like Figure 5 , the video dataset of S12 is input into the trained deep neural network model frame by frame to obtain the corresponding target box prediction and target feature matrix, which is then substituted into the AFRS association algorithm in S13 to associate the feature relationships between the targets and record all target trajectories and identity information in the tracking results middle.

[0188] Figure 6 is a frame of image in the road monitoring video data for vehicle multi-target tracking, where Figure 6 (a) is the initial frame image of the road monitoring video, Figure 6 (b) is the last frame image of the road monitoring video; Figure 7 This is the result of multi-target vehicle tracking, where each curve represents the motion trajectory of a vehicle in the detection area. Through the deep neural network calculation of the present invention, all vehicles in the detection area are accurately identified and tracked and their motion trajectories are plotted.

[0189] Example 2

[0190] like Figures 1 to 7 As shown, the image is randomly transformed to form a data enhancement algorithm set

[0191] The image is randomly transformed into Gaussian blur or Gaussian noise, and the specific parameters of each method are set according to the general scheme of current self-supervised training.

[0192] According to the sample set, a backbone network model of a deep neural network model integrating vehicle detection and self-supervised visual representation learning is constructed, and two sample sets Z are extracted in pairs. i and Z' i The characteristic matrix of i and h' i ;

[0193] The backbone network model of the deep neural network model constructed to integrate vehicle detection and self-supervised visual representation learning is the encoder f(·); the encoder f(·) is the DarkNet-53 network model.

[0194] Example 3

[0195] like Figures 1 to 7 As shown, the image is randomly transformed to form a data enhancement algorithm set

[0196] The images are randomly transformed to be sharpened or overexposed, and the specific parameters of each method are set according to the general scheme of current self-supervised training.

[0197] According to the sample set, a backbone network model of a deep neural network model integrating vehicle detection and self-supervised visual representation learning is constructed, and two sample sets Z are extracted in pairs. i and Z' i The characteristic matrix of i and h' i ;

[0198] The backbone network model of the deep neural network model constructed to integrate vehicle detection and self-supervised visual representation learning is the encoder f(·); the encoder f(·) is a ResNet-50 network model.

[0199] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for tracking multiple vehicles in a video, characterized in that: The steps include: Perform random image transformation on the collected images to form a data enhancement algorithm set The image dataset is divided into several instance sets Z by random sampling. i , each instance set has a corresponding label set E i , i∈{1,2,…T}, T is the number of instance sets; From the data enhancement algorithm collection Randomly select an algorithm For each instance Z i ,E i Perform the transformation to obtain the sample set Z′ i ,E′ i ; According to the sample set, the backbone network model of the deep neural network model integrating vehicle detection and self-supervised visual representation learning is constructed, and two sample sets Z are extracted in pairs. i and Z' i The characteristic matrix of i and h' i ; According to the feature matrix of the sample set, a branch network with the same structure is constructed. i and h i ′ is used to make predictions and obtain the heat map F i and F' i , center offset O i and O' i , target size S i and S' i ; Based on the branch network with the same structure, the target feature representation network model and the projection layer network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning are constructed. The two feature matrices h i and h' i The target feature representation H is obtained through the target feature representation network i and H' i , target feature representation H i and H' i The target feature matrix is obtained by projection, denoted as L i and L' i ; The target feature matrix L i and L' i Divide along the row and column vectors respectively to obtain instance features and dimension features, and construct positive and negative instance feature pairs and related and irrelevant dimension feature pairs; The feature relationship between instance feature pairs is calculated using the cosine distance measurement function, and the feature relationship between dimension feature pairs is calculated using the correlation coefficient measurement function. A dual contrast loss function of instance features and dimension features is constructed as the target feature loss function. Use Gaussian model to label E i ,E' i The ground truth bounding box information is mapped to a heat map And calculate the label target center offset and target size Calculate the sample thermal feature map F through the Focal Loss loss function i and F' i and The relationship between The loss function calculates the sample center point offset O i and O' i and The relationship between the target sample size S i and S' i and According to the calculated loss function, the target detection loss function is constructed, and the total loss function of the target detection and feature calculation of the model is formed together with the sample dimension double comparison loss function; The deep neural network model that integrates vehicle detection and self-supervised visual representation learning is trained based on the total loss function of target detection and feature calculation; Video Dataset Contains N frames of images, the first frame image I 1 Send it into the trained deep neural network model to get the corresponding center point position P 1 , bounding box size S 1 And the target feature matrix H 1 , initialize the identity information and trajectory information of all targets; The feature matrix of two consecutive frames is used for correlation algorithm to obtain the correlation relationship between the targets in the frames, the trajectory of the same target is updated, and the identity and trajectory of the newly appeared target are initialized; The deep neural network model finally obtains the tracking results of the video data after traversing all frame images 2. The method for tracking multiple vehicles in a video according to claim 1, wherein: The backbone network model of the deep neural network model that integrates vehicle detection and self-supervised visual representation learning is the encoder f(·). The encoder f(·) is a network model composed of a multi-layer convolutional neural network. The encoder f(·) is used to extract the embedded representation of the sample set. The calculation formula is as follows: B is the number of samples in the sample set, D is the size of the embedding vector, represents the set of real numbers.

3. The method for tracking multiple vehicles in a video according to claim 1, wherein: The branch network with the same structure is the prediction head p j (·), j = {1, 2, 3}, used to predict the heat map, center offset vector and target size respectively; Each prediction head p j (·) Perform convolution calculation with a kernel size of 3×3 on the feature map. After passing through the batch normalization layer and nonlinear activation function, the prediction matrix of the sample set is obtained by 1×1 convolution calculation. It is used to predict the embedded representation of the sample set. The calculation formula is as follows: I is the size of the sample thermal signature; J is the size of the sample target center point offset vector prediction; K is the size of the sample target size prediction.

4. The method for tracking multiple vehicles in a video according to claim 1, wherein: The target feature representation network model is the target feature characterizer c(·), and the projection layer network model is the projection head g(·); The target feature characterizer c(·) includes a convolutional neural network and a feature channel set network. The target feature is represented as the set of all channels at the target center point coordinates of the feature matrix obtained by convolution calculation. The calculation formula is as follows: N is the number of sample target center points, and D is the number of channels of the feature matrix obtained by convolution calculation; The projection head g(·) consists of a three-layer fully connected network. The first two layers are followed by a batch normalization layer and a nonlinear activation function to calculate the feature matrix of the sample set. The calculation formula is as follows: M is the size of the sample output target feature matrix.

5. The method for tracking multiple vehicles in a video according to claim 1, wherein: The two feature matrices L i and L' i Divide by row to get the instance representation set t represents the image enhancement algorithm, where 1 means no image enhancement algorithm is used, i is the instance index, and the instance pair is constructed from the set S. i=j is a positive pair, i≠j is a negative pair; the two feature matrices L i and L' i Divide by column to get the feature dimension set p is the dimension index, constructing the dimension pair from the set D p=k is a related pair, and p≠k is an unrelated pair.

6. The method for tracking multiple vehicles in a video according to claim 1, wherein: The similarity between instance feature pairs is calculated using the cosine distance metric function. The formula for the cosine distance metric function is: ||·|| means finding the modulus length of the sample feature; The correlation coefficient measurement function is used to calculate the correlation between dimension feature pairs. The correlation coefficient measurement function formula is: and is the average value of all elements in the dimension feature; The instance feature contrast loss function is: The dimensional feature contrast loss function is: τ1, τ2 are hyperparameters of the loss function, representing temperature parameters; Combined with instance feature contrast loss function Compared with the dimension feature loss function The target feature loss function is obtained as follows: Where λ is a parameter that controls the loss balance.

7. The method for tracking multiple vehicles in a video according to claim 1, wherein: The label information representing the target position and size in the sample data set is (x1, y1, x2, y2), and its center point is Round down to get Use the Gaussian model to map the true bounding box information in the label to a heat map The calculation formula is: Y xy is the heat map mapping value, indicating the heat map The value of the pixel at (x,y) coordinate, is the variance related to the bounding box size. If the Gaussian distributions of the two center points overlap, the maximum value is taken. Calculate the label target center point offset and target size The formula is as follows: p i ,p' i Z i ,Z' i The target center point coordinates; 8. The method for tracking multiple vehicles in a video according to claim 1, wherein: Calculate the sample thermal feature map F through the Focal Loss loss function i and F' i and The relationship between, Focal Loss loss function formula is: Represents the thermal characteristic diagram F i ,F' i The estimated value of the pixel at the coordinate (x, y) above, c is the number of categories of the target to be detected; α and β are hyperparameters used to balance the number of difficult and easy samples; Y xyc Representing thermal characteristics The label value of the pixel at the coordinate (x, y) above; N is the number of sample target center points; pass The loss function calculates the sample center point offset O i and O' i and The relationship between them, the loss function calculation formula is: pass Loss function calculation sample target size S i and S' i and The relationship between them, the loss function calculation formula is: The loss functions of the three branches are added together by fixing the weights to obtain the overall loss of the detection target. The loss function calculation formula is as follows: λ1,λ2 are fixed weights; The target detection loss function is combined with the target feature loss function to construct the overall loss function. The calculation formula is as follows: Where w1, w2 are learnable parameters that balance the two tasks.

9. The method for tracking multiple vehicles in a video according to claim 1, wherein: The video dataset The first image in I 1 Input the trained deep neural network model to obtain the initialized identity information and trajectory information. The calculation formula is as follows: H 1 =c(f(I 1 )); From the heat map F 1 and the center point offset O 1 The target center point position can be calculated using the following formula: P 1 =q(F 1 ,O 1 )。 10. The method for tracking multiple vehicles in a video according to claim 1, wherein: The video dataset All images in the image are fed into the trained deep neural network model for calculation and the trajectory of the tracking target is recorded. in represents the trajectory information of target i in frame t, is the center point coordinate, is the bounding box size, is the feature representation of the target, ID i is the target’s identity, l i is the survival time parameter; The association relationship between targets in two consecutive image frames is obtained by the data association algorithm AFRS based on feature representation similarity. The calculation formula is: Similarity Matrix N t-1 ,N t is the number of targets in the t-1,t frame image. If target i and target j have the same identity, then S i,j >0, otherwise S i,j <0; In the similarity matrix S i,j Add a column vector P with a fixed value as a placeholder to form a new matrix The value of P is the similarity matrix S i,j The average value of all elements except the diagonal in , the placeholder is used to match the disappeared target or the newly appeared target; The similarity matrix S' i,j Calculate the Softmax function with a temperature coefficient along the row vector to obtain the correlation matrix M in which each row contains the similarity results between the current target and all targets in the two frames: T is the temperature parameter, Use adaptive temperature regulation T: T=2log(N t-1 +N t +1); Record the detection results of all targets where N t represents the number of targets in the t-th frame image, Perform AFRS association algorithm on the feature matrix of two consecutive frames of images. If the target i exists in the current frame and is successfully associated with the previous frame, the detection result is used. Update the target trajectory and set the center point coordinates Bounding box size Identity ID i And the current frame number t is saved to the tracking result If the association fails, the trajectory information of the previous frame remains unchanged and the survival time l i Add 1, if l i >30, the target is considered to have left the tracking screen.

Images