A vehicle re-identification method based on spatio-temporal contrast unsupervised learning

Abstract

The present application relates to the field of road monitoring video vehicle detection, and particularly relates to a vehicle re-identification method based on space-time comparison unsupervised learning. By using a visual time hole feature extraction model, space-time information such as camera number, time stamp and visual information of a picture itself are fused, more representative fused features are constructed, effective use of camera identity number and time stamp is realized, and the accuracy of the model is improved by constraining the unsupervised vehicle re-identification model through space-time information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of road monitoring video vehicle detection, and particularly relates to a vehicle re-identification method based on unsupervised learning of space-time comparison. BACKGROUND

[0002] In order to guarantee road traffic efficiency and operation safety, highways usually invest a lot in the construction of monitoring systems. The monitoring system has important practical significance in assisting the regulatory department to make decisions and respond to emergencies. In order to ensure the effective operation of the monitoring system, early reliance on a large number of human resources for manual search of monitoring video, but with the rapid development of computer computing power, computer vision technology represented by vehicle re-identification algorithm for searching monitoring video by using artificial intelligence has also developed rapidly.

[0003] Vehicle re-identification is to detect, identify and match the same vehicle photographed by different cameras at different times, so as to realize long-time and long-distance tracking of vehicles under the linkage of multi-camera view.

[0004] The initial vehicle re-identification uses license plate information to match the vehicle identity, but due to the shielding, pollution and behavior of fake license plate vehicles, a vehicle re-identification method using vehicle image information to train a deep feature extraction model has been developed. However, using only vehicle image information for re-identification requires providing a large number of labeled data sets during training of the vehicle re-identification model, so unsupervised vehicle re-identification has received more attention.

[0005] Common unsupervised vehicle re-identification can be divided into unsupervised vehicle re-identification without manual labeling of data sets and domain adaptation vehicle re-identification by pre-training on a labeled source data set and then migrating to a target domain data set without labels. Existing vehicle re-identification methods generally focus on the development and utilization of image information, ignoring the high space-time constraints of vehicles driving on the road network, and ignoring the analysis of space-time information such as camera identity number and time stamp. Moreover, the accuracy of existing unsupervised vehicle re-identification is much lower than that of supervised vehicle re-identification. SUMMARY

[0006] In view of the problems in the prior art, the purpose of the present application is to provide a vehicle re-identification method based on unsupervised learning of space-time comparison, which does not require manual labeling of data sets, uses the time stamp and camera identity information automatically labeled by the camera to train the unsupervised vehicle re-identification model, and obtains an unsupervised vehicle re-identification model with high recognition accuracy, thereby identifying vehicles.

[0007] In order to achieve the above purpose, the technical scheme of the present application is adopted.

[0008] A vehicle re-identification method based on spatio-temporal contrast unsupervised learning, comprising the following steps:

[0009] Step 1, a plurality of vehicle images with camera number labels and time stamp labels are extracted from road monitoring videos to constitute a training set;

[0010] Step 2, a fusion feature extraction model is established to extract visual spatio-temporal fusion features and spatio-temporal features of the training set pictures; the spatio-temporal features are input into a spatio-temporal interval classifier to obtain spatio-temporal scores;

[0011] Step 3, the Jacard distance between each two pictures in the training set is calculated according to the visual spatio-temporal fusion features to obtain a MxM Jacard distance matrix, wherein M is the number of pictures in the training set;

[0012] Step 4, according to the Jacard distance matrix, a density clustering algorithm DBSCAN is used to cluster the visual spatio-temporal fusion features of the training set pictures to assign pseudo labels to the training set pictures;

[0013] Step 5, a fusion feature dictionary is constructed;

[0014] Step 6, the contrast loss of the fusion features of the training set pictures and the fusion feature dictionary is calculated, and the cross-entropy loss of the spatio-temporal scores of the training set pictures and the spatio-temporal interval is calculated; the fusion feature extraction model is trained by using the sum of the contrast loss and the cross-entropy loss;

[0015] Step 7, the trained fusion feature extraction model is used to extract the visual spatio-temporal fusion features of the vehicle images to be identified with camera number labels and time stamp labels from the road monitoring videos; the trained fusion feature extraction model is used to extract all vehicle images with camera number labels and time stamp labels from the road monitoring videos, and the visual spatio-temporal fusion features of each vehicle are extracted respectively;

[0016] Step 8, the Euclidean distance between the visual spatio-temporal fusion features of the vehicle to be identified and the visual spatio-temporal fusion features of each vehicle is calculated respectively; the smaller the Euclidean distance, the more likely the vehicle corresponding to the visual spatio-temporal fusion features is the same vehicle as the vehicle to be identified.

[0017] Compared with the prior art, the beneficial effects of the present application are: by using the visual spatio-temporal feature extraction model, the camera number, time stamp and other spatio-temporal information and the visual information of the picture are fused to construct more representative fusion features, the effective use of the camera identity number and the time stamp is realized, the unsupervised vehicle re-identification model is constrained by the spatio-temporal information, and the precision of the model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0019] Figure 1 A flow chart of the vehicle re-identification method based on spatio-temporal contrast unsupervised learning of the present application;

[0020] Figure 2 A flow chart of the step of extracting visual spatio-temporal fusion features by the fusion feature extraction model of the present application. DETAILED DESCRIPTION

[0021] The embodiments of the present application will be described in detail below with reference to the examples, but those skilled in the art will understand that the following examples are only for illustration of the present application and should not be regarded as limiting the scope of the present application.

[0022] REFERENCE Figure 1 A vehicle re-identification method based on spatio-temporal contrast unsupervised learning, comprising the following steps:

[0023] Step 1. Extracting multiple vehicle images with camera number labels and time stamp labels from road monitoring videos to form a training set;

[0024] Step 2. Establishing a fusion feature extraction model to extract visual spatio-temporal fusion features and spatio-temporal features of the training set pictures; inputting the spatio-temporal features into a spatio-temporal interval classifier to obtain a spatio-temporal score;

[0025] Specifically, the fusion feature extraction model comprises a deep residual network ResNet50 and a multi-layer perceptron MLP; the network input of the deep residual network ResNet50 is an RGB picture with a size of HxWx3, and the network output is a visual feature; the network input of the multi-layer perceptron MLP is a camera number and a time stamp, and the network output is a spatio-temporal feature; the network output of the deep residual network ResNet50 and the network output of the multi-layer perceptron MLP are residual fused to obtain a visual spatio-temporal fusion feature, which is the output of the fusion feature extraction model.

[0026] REFERENCE Figure 2 The sub-steps of extracting the visual spatio-temporal fusion features of the training set pictures are as follows:

[0027] Sub-step 2.1. Converting the pictures in the training set into RGB pictures with a size of HxWx3;

[0028] Sub-step 2.2. Inputting the RGB pictures into the deep residual network ResNet50 to obtain the visual features of the RGB pictures; inputting the camera number and the time stamp in the training set pictures into the multi-layer perceptron MLP, the multi-layer perceptron MLP extracts the spatial features and the temporal features of the pictures, and the spatial features and the temporal features are residual connected to obtain the spatio-temporal features of the training set pictures;

[0029] Sub-step 2.3, residual fusion is performed on the visual features and the spatio-temporal features of each picture in the training set to obtain visual spatio-temporal fusion features.

[0030] Step 3, the Jacard distance between each two pictures in the training set is calculated according to the visual spatio-temporal fusion features to obtain a MxM Jacard distance matrix, where M is the number of pictures in the training set;

[0031] Step 4, according to the Jacard distance matrix, the density clustering algorithm DBSCAN is used to cluster the visual spatio-temporal fusion features of the pictures in the training set to assign pseudo labels to the pictures in the training set;

[0032] Step 5, a fusion feature dictionary is constructed;

[0033] Specifically, first, the camera number and time stamp of all pictures in the training set are counted, and the pictures under each camera are divided into time intervals;

[0034] Secondly, the distance spatio-temporal fusion features of a picture are randomly extracted in each spatio-temporal interval of each pseudo label category in the training set, and the average value of the distance spatio-temporal fusion features of all spatio-temporal intervals in each pseudo label category in the training set is calculated;

[0035] Finally, the average value is taken as the representative feature of the corresponding pseudo label category to construct a fusion feature dictionary with a size of CxL; where C is the number of pseudo label categories, and L is the dimension size of the feature.

[0036] Step 6, the contrast loss of the fusion features of the pictures in the training set and the fusion feature dictionary is calculated, and the cross-entropy loss of the spatio-temporal score of the pictures in the training set and the spatio-temporal interval is calculated; the fusion feature extraction model is trained by using the sum of the contrast loss and the cross-entropy loss;

[0037] Specifically, the calculation formula of the contrast loss L1 is as follows:

[0038]

[0039] In the formula, F represents the visual spatio-temporal fusion features of the pictures in the training set; f s represents the visual spatio-temporal fusion features stored in the fusion feature dictionary which are of the same pseudo label as the pictures in the training set; F s represents the set of visual spatio-temporal fusion features stored in the fusion feature dictionary; and τ represents a temperature parameter, which is a constant with a value of 0.1.

[0040] The role of the contrast loss L1 is to increase the distance between the training sample and the feature of the corresponding category stored in the fusion feature dictionary, and at the same time, to increase the distance between the training sample and the features of other categories stored in the visual feature dictionary as much as possible, so that the model obtains the vehicle re-identification ability.

[0041] The calculation formula of the cross-entropy loss L2 is as follows:

[0042] L2 = ∑ -y * log(p / τ)

[0043] In the formula, y represents the space-time interval of the training set picture, p represents the space-time score of the training set picture, and τ represents a temperature parameter, which is a constant with a value of 0.1.

[0044] The cross-entropy loss L2 is used to train the space-time feature extraction part, that is, to accelerate the convergence of the multi-layer perceptron MLP with a residual structure, so that the model can better utilize the space-time features.

[0045] Step 7, using the trained fusion feature extraction model, extracting the vehicle image to be identified with camera number label and time stamp label from the road monitoring video, extracting the visual space-time fusion feature of the vehicle to be identified; using the trained fusion feature extraction model, extracting all vehicle images with camera number label and time stamp label from the road monitoring video, and extracting the visual space-time fusion feature of each vehicle respectively;

[0046] Step 8, respectively calculating the Euclidean distance between the visual space-time fusion feature of the vehicle to be identified and the visual space-time fusion feature of each vehicle; the smaller the Euclidean distance, the more likely it is that the vehicle corresponding to the visual space-time fusion feature is the same vehicle as the vehicle to be identified.

[0047] The application will be further described through simulation tests.

[0048] Experiments were conducted on the VeRi dataset, and the adam optimizer was used in the experiments, the learning rate was set to 0.00015, the number of training rounds was set to 60, and 32 images were used for training each time.

[0049] Method 1: traditional method without using space-time information;

[0050] Method 2: the method of the application using space-time information;

[0051] Methods 1 and 2 were used to verify the same test set respectively, and the test results are as follows.

[0052]

[0053] As shown in the above table, compared with the traditional method without using spatio-temporal information, the method of the present application using spatio-temporal information has an increase of 1.7% in the mAP index, and the Rank-1, Rank-5 and Rank-10 are also improved, indicating that the method of the present application has higher accuracy and effectiveness. Among them, the mAP index is a common index for measuring detection accuracy by calculating the average correct rate of detection of all categories by weighted average. The n-bit hit rate index Rank-n is the probability that the top n pictures in the search result have correct results.

[0054] Although the present application has been described in detail with general description and specific embodiments in the specification, some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application, all belong to the scope of protection claimed by the present application.

Claims

1. A vehicle re-identification method based on spatiotemporal contrastive unsupervised learning, characterized in that, Includes the following steps: Step 1: Extract multiple vehicle images with camera number tags and timestamp tags from road surveillance videos to form a training set; Step 2: Establish a fusion feature extraction model to extract visual spatiotemporal fusion features and spatiotemporal features from the training set images; input the spatiotemporal features into the spatiotemporal interval classifier to obtain spatiotemporal scores; The fusion feature extraction model includes a deep residual network ResNet50 and a multilayer perceptron MLP. The input of the deep residual network ResNet50 is an RGB image of size H×W×3, and the network output is visual features. The input of the multilayer perceptron MLP is the camera number and timestamp, and the network output is spatiotemporal features. The network outputs of the deep residual network ResNet50 and the network outputs of the multilayer perceptron MLP are fused residually to obtain visual spatiotemporal fusion features. Step 3: Calculate the Jaccard distance between two images in the training set based on the visual spatiotemporal fusion features to obtain an M×M Jaccard distance matrix, where M is the number of images in the training set; Step 4: Based on the Jaccard distance matrix, use the density clustering algorithm DBSCAN to cluster the visual spatiotemporal fusion features of the training set images and assign pseudo-labels to the training set images. Step 5: Construct a fusion feature dictionary; Specifically, first, the camera ID and timestamp of all images in the training set are counted, and the images under each camera are divided into time intervals; Secondly, the distance-temporal fusion feature of an image is randomly selected from each spatiotemporal interval of each pseudo-label category in the training set, and the average value of the distance-temporal fusion feature of all spatiotemporal intervals under each pseudo-label category in the training set is calculated. Finally, the average value is used as the representative feature of the corresponding pseudo-label category to construct a fusion feature dictionary of size C×L; Where C is the number of pseudo-label categories, and L is the dimension of the feature; Step 6: Calculate the contrastive loss of the fusion features of the training set images and the fusion feature dictionary; calculate the spatiotemporal score of the training set images and the cross-entropy loss of the spatiotemporal interval; train the fusion feature extraction model using the sum of the contrastive loss and the cross-entropy loss. Step 7: Using the trained fusion feature extraction model, extract the images of the vehicles to be identified from the road surveillance video, which have camera number labels and timestamp labels, and extract the visual spatiotemporal fusion features of the vehicles to be identified. Using a trained fusion feature extraction model, extract all vehicle images with camera number labels and timestamp labels from road surveillance videos, and extract the visual spatiotemporal fusion features of each vehicle. Step 8: Calculate the Euclidean distance between the visual spatiotemporal fusion feature of the vehicle to be identified and the visual spatiotemporal fusion feature of each vehicle; the smaller the Euclidean distance, the more likely the vehicle corresponding to the visual spatiotemporal fusion feature is to be the same vehicle as the vehicle to be identified.

2. The vehicle re-identification method based on spatiotemporal contrast unsupervised learning according to claim 1, characterized in that, The sub-steps for extracting the visual-spatiotemporal fusion features of the training set images are as follows: Sub-step 2.1: Convert the images in the training set into RGB images of size H×W×3; Sub-step 2.2: Input the RGB image into the deep residual network ResNet50 to obtain the visual features of the RGB image; input the camera number and timestamp in the training set image into the multilayer perceptron MLP to obtain the spatiotemporal features of the training set image; Sub-step 2.3 involves performing residual fusion on the visual and spatiotemporal features of each image in the training set to obtain visual-spatiotemporal fusion features.

3. The vehicle re-identification method based on spatiotemporal contrast unsupervised learning according to claim 1, characterized in that, The contrast loss in step 6, specifically, the contrast loss... L The formula for calculating 1 is as follows: In the formula, F represents the visual spatiotemporal fusion features of the training set images; f s F represents the spatiotemporal fusion features stored in the fusion feature dictionary that share the same pseudo-label as the training set images; S This represents the set of visual spatiotemporal fusion features stored in the fusion feature dictionary; τ represents the temperature parameter, which is a constant with a value of 0.

1.

4. The vehicle re-identification method based on spatiotemporal contrast unsupervised learning according to claim 1, characterized in that, The cross-entropy loss in step 6, specifically, the cross-entropy loss... L The formula for calculating 2 is as follows: In the formula, y represents the spatiotemporal interval of the training set images; p represents the spatiotemporal score of the training set images; and τ represents the temperature parameter, which is a constant with a value of 0.1.

Images