An unsupervised vehicle re-identification method for aerial images

Through the combination of cyclic consistency learning and self-supervised loss function, unsupervised vehicle re-identification is realized, the problem of the existing technology's dependence on manual labeled data is solved, the vehicle feature representation and re-identification performance is improved, and the vehicle re-identification technology is simplified extension.

CN111950367BActive Publication Date: 2024-11-01UNIV OF CHINESE ACAD OF SCI
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202010653505.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-08
Publication Date
2024-11-01
Estimated Expiration
2040-07-08

AI Technical Summary

Technical Problem

Existing vehicle recognition technology relies on a large amount of manual labeling data, lacks unsupervised vehicle recognition algorithms, especially when the distribution of source data and target data is large, performance is difficult to improve.

Method used

Using cyclic consistency learning method and self-supervised triple loss function, unsupervised learning of vehicle tracking and positioning model from the original video, and correct the tracking and positioning results through the vehicle detection box, automatically label the vehicle ID, combined with automatic labeling and manual labeling data Network training.

Benefits of technology

Unsupervised learning of vehicle re-identification features is achieved, reducing dependence on manual labeling data, improving vehicle feature representation ability and re-identification performance, and being able to automatically label vehicle IDs from the original video, simplifying the expansion and implementation of re-identification technology. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111950367B_ABST
    Figure CN111950367B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of computer vision accessory devices, and in particular to an unsupervised vehicle re-identification method for aerial images, which can directly learn vehicle re-identification feature representations from the collected original video in an unsupervised manner, and includes the following steps: (1) randomly selecting 200 frames from the original video to annotate the vehicle frame, and then training the vehicle detection model; (2) obtaining the position and bounding box parameters of the vehicle in all video frames based on the trained vehicle detection model. (3) training a vehicle tracking and positioning model based on the target vehicle bounding box parameters and a cycle consistency method. (4) correlating the corresponding vehicles in the video frames and annotating the corresponding vehicle ID (identification number) based on the vehicle tracking and positioning model. (5) training a neural network based on the automatically annotated vehicle data and updating the parameters, and then extracting and retrieving vehicle features on a test set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision accessory devices, and in particular to an unsupervised vehicle re-identification method for aerial images. Background Art

[0002] Target re-identification technology is the basis for wide-area target retrieval and long-term tracking, trajectory prediction, as well as applications such as smart cities and safe engineering, and has important research significance and application value. Vehicle re-identification is one of the important technologies for achieving efficient traffic monitoring and smart city monitoring. Given a query target vehicle image, vehicle re-identification requires searching for images belonging to the same vehicle in a large-scale vehicle image library collected from surveillance cameras or the Internet. Current vehicle re-identification methods are mainly divided into three categories: global feature optimization based on metric learning; high-discriminative feature learning by integrating local details; and perspective-robust feature learning based on perspective information. Although vehicle re-identification has made great progress, these methods all rely on training with a large amount of manually annotated data.

[0003] At present, there is no related work on unsupervised vehicle re-identification algorithms, but unsupervised re-identification algorithms for pedestrians have been widely concerned and studied, and can be divided into two categories: reducing the distribution difference between the source dataset and the target dataset, and then directly applying the model learned on the source dataset to the target dataset; clustering the target dataset using the model learned from the source dataset and then annotating pseudo-labels, and fine-tuning the network parameters according to the pseudo-labels obtained. These unsupervised re-identification methods will use one or more source datasets to improve the generalization performance of the network model on the target dataset. However, when the distribution difference between the source data and the target data is too large, it is difficult for such transfer learning methods and clustering methods to achieve relatively good performance. If it is possible to automatically annotate the vehicle ID (identification) directly from the original video without relying on any other related datasets and labels and learn re-identification features, it will greatly facilitate the expansion and implementation of re-identification technology. Using unsupervised tracking technology to automatically annotate the vehicle ID in the video can reduce the reliance of re-identification technology on a large amount of manually annotated data. However, previous work did not have a similar solution to the problem. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an unsupervised vehicle re-identification method for aerial images, which can directly learn vehicle re-identification feature representation from the collected original video in an unsupervised manner.

[0005] The present invention provides an unsupervised vehicle re-identification method for aerial images, comprising the following steps:

[0006] (i) Randomly select 200 frames from the original video to mark the vehicle frame, and then train the vehicle detection model;

[0007] (ii) Obtain the position and bounding box parameters of the vehicles in all video frames based on the trained vehicle detection model.

[0008] (iii) Train the vehicle tracking and positioning model based on the target vehicle bounding box parameters and the cycle consistency method.

[0009] (iv) According to the vehicle tracking and positioning model, the corresponding vehicles in the video frames are associated and the corresponding vehicle IDs (identification numbers) are marked.

[0010] (V) Train the neural network and update the parameters based on the automatically labeled vehicle data, and then extract and retrieve vehicle features on the test set.

[0011] The present invention provides an unsupervised vehicle re-identification method for aerial images. In the step (a), when training a vehicle detection model, the SSD widely used in target detection tasks is selected as a basic model for vehicle detection.

[0012] The present invention provides an unsupervised vehicle re-identification method for aerial images, in the step (ii), the bounding box parameters are the relative positions of the four vertices of the vehicle target in the frame.

[0013] The present invention provides an unsupervised vehicle re-identification method for aerial images. In the step (iv), in the process of associating corresponding vehicles and marking corresponding vehicle IDs, a vehicle detection frame is used to replace the vehicle positioning area to timely correct the tracking and positioning results.

[0014] The present invention provides an unsupervised vehicle re-identification method for aerial images. In the step (iii), a self-supervised triplet loss function is introduced to assist vehicle feature representation learning.

[0015] The present invention provides an unsupervised vehicle re-identification method for aerial images. In the step (five), the automatically labeled vehicle data is combined with the manually labeled vehicle data for network training to further improve the representation capability of vehicle features.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: a cycle consistency learning method is designed to learn a vehicle tracking and positioning module in an unsupervised manner, and a triplet loss function is introduced to optimize the training of the tracking and positioning module, while accelerating network convergence; the intrinsic correspondence of a large amount of unlabeled video data can be mined in the training phase, and the position of the target vehicle in the next frame can be inferred according to the position of the target vehicle in the previous frame in the test phase; the problem of automatically assigning labels to unlabeled vehicle data in the video is solved by using the learned vehicle tracking and positioning module; the ID information of each vehicle in the video can be automatically annotated according to the massive amount of unlabeled video data, reducing the dependence of vehicle re-identification technology on a large amount of manually annotated data; in the process of automatically annotating the vehicle data set, the vehicle detection frame is used to replace the vehicle positioning area to correct the tracking in time Positioning results; making the vehicle images extracted and labeled from the video frames more accurate, thereby obtaining better vehicle feature representation capabilities; after automatically labeling the vehicle data set, using the feature distance relationship between the same label to remove some noise images; making the automatically labeled vehicle data set cleaner and more accurate, thereby facilitating better vehicle feature representation capabilities; combining automatically labeled vehicle data with manually labeled vehicle data for model training; first pre-training the network using automatically labeled vehicle data, and then fine-tuning on manually labeled vehicle data, can greatly improve the vehicle feature representation capabilities; compared to the previous unsupervised re-identification algorithms based on transfer learning or clustering that rely on related source data sets to assist training, this method can directly learn vehicle re-identification feature representations from the collected original video without supervision. The lack of related auxiliary data sets can greatly facilitate the expansion and implementation of the re-identification system. In addition, the cycle consistency method is used to associate the image sequences of each vehicle in the original video, and then automatically obtain the identity label information of each vehicle, which can greatly alleviate the dependence of the re-identification task on manually labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the present invention;

[0018] Figure 2 It is a diagram of the cycle consistency learning algorithm apparatus;

[0019] Figure 3 It is the structural diagram of the positioning module;

[0020] Figure 4 This is the result diagram of the effect of the amount of training data on the re-identification performance;

[0021] Figure 5 This is a display diagram of the automatic annotation results of the vehicle sequence;

[0022] Figure 6 This is the impact of triplet loss on the final vehicle re-identification performance;

[0023] Figure 7 It is a table showing the influence of using a vehicle detection frame to replace the vehicle positioning area to timely correct the tracking and positioning results on the final vehicle re-identification performance;

[0024] Figure 8 It is a performance display table combining automatically labeled data and supervised data;

[0025] Markings in the accompanying drawings: DETAILED DESCRIPTION

[0026] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0027] Example:

[0028] Depend on Figure 1 It can be seen that the following steps are included:

[0029] 1. Randomly select 200 frames from the original video to annotate the vehicle frame, and then train the vehicle detection model. Specifically, when training the vehicle detection model in step 1, select SSD, which is widely used in target detection tasks, as the basic model for vehicle detection. Take 200 frames of images randomly extracted from the training video and the corresponding vehicle frame annotation files as input (the input image size is scaled to 512*512), train the detection model SSD, and save the model parameters after the model converges.

[0030] 2. According to the trained vehicle detection model, the position and bounding box parameters θ of the vehicle in all video frames are obtained. The bounding box parameters are the relative positions of the four vertices of the vehicle target in the frame. For example, the coordinates are normalized with the upper left corner of the frame as (0,0) and the lower right corner as (1,1).

[0031] 3. Train the vehicle tracking and localization model LM (Localization Model) based on the target vehicle bounding box parameters and the cycle consistency method. During training, a triplet is selected in each frame in an unsupervised manner to construct a ternary loss function, thereby assisting the vehicle tracking and localization model to converge better and faster.

[0032] Depend on Figure 2 and Figure 3 It can be seen that, specifically, during forward tracking and positioning, according to the two consecutive frame images It, It+1 and the corresponding vehicle position parameter θt in the image It, a vehicle position is randomly selected from It to extract the corresponding target vehicle image Tt, and the corresponding search area Pt and Pt+1 in It, It+1 with four times the size of Tt centered on Tt. Input Tt, Pt and Pt+1 Figure 3The feature map is obtained from the network with shared parameters and then the similarity matrix between the two feature vectors is calculated:

[0033] Where and are both bi-norm normalized in the c dimension. A convolutional layer and a fully connected layer are added after H, and finally the transformation parameters with three values ​​are output. These three values ​​represent the translation and rotation parameters of the target T t corresponding to the search area P t+1, which means that the most similar feature area can be found in , which also means that the pixel area T t+1 that is most similar to T t can be found in P t+1. When tracking and locating in reverse, the predicted area T t+1 is used as the initial target and P t as the search area, and then the position transformation parameters can be obtained after passing through the positioning module in the same way.

[0034] Based on and , we can calculate the cycle consistency loss Lθ (for specific operations, refer to

[12] ). The smaller Lθ is, the better our tracking and positioning model is learned. Based on and , we can calculate the feature similarity loss L sim (for specific operations, refer to

[13] ). The smaller Lsim is, the more similar and , that is, the more discriminative the feature is. Self-supervised ternary loss L tri (fa, fp, fn) = max (0, d (fa, fp) - d (fa, fn) + m), where fa is Figure 2 The feature representation of the initial target in is, fp is Figure 2 The feature representation of the positive example (obtained by rotating and scaling the initial target in a small range), fn is Figure 2 The most difficult feature representation of the negative example (select the negative feature closest to the initial target feature), m is a distance threshold constant. The final loss function can be expressed as:

[0035] L=Lθ+λ1L sim+λ2L tri

[0036] Among them, λ1 and λ2 are weight coefficients.

[0037] 4. According to the vehicle tracking and positioning model LM, the corresponding vehicles in the video frame are associated and the corresponding vehicle ID (identification) is marked. In the process of associating the corresponding vehicles and marking the corresponding vehicle ID, the vehicle detection frame is used to replace the vehicle positioning area to timely correct the tracking and positioning results. At the same time, after automatically marking the vehicle data set, some noise images are removed by using the feature distance relationship between the same labels, making the automatically marked vehicle data set cleaner and more accurate.

[0038] Specifically, because the learned LM can locate the position of the target vehicle in the next frame based on the position of the target vehicle in the current frame, we can use LM continuously to obtain all the sequence images of the target vehicle in the subsequent frames, and then annotate them with the same ID. However, LM may have errors, and continuous use may amplify the errors, so using the vehicle detection frame near the positioning result to correct the tracking and positioning results of LM can make the automatically obtained vehicle sequence more accurate. When there is no detection frame near the positioning result or the closest vehicle detection frame is incorrect, interference images with inaccurate vehicle frames or incorrect ID annotations will appear in the automatically annotated sequence. If these interference images can be removed and the network is retrained, it will be more conducive to improving the performance of vehicle re-identification. First, a vehicle feature encoding network is trained using the automatically annotated vehicle sequence, and then the features of all vehicle images in the sequence are extracted. For vehicle sequences with the same ID, the feature distances of all images in the sequence to the initial target (the first vehicle image in the sequence) are calculated, and those exceeding a certain threshold are removed from the sequence.

[0039] 5. Train the neural network and update the parameters based on the automatically labeled vehicle data, then extract and retrieve vehicle features on the test set. Combined with a small amount of manually labeled data, the vehicle re-identification performance can be further significantly improved.

[0040] The experimental results in this embodiment are obtained by testing on a vehicle image database taken by a drone (UAV-VeRI). The table reflecting the technical effect obtained through the experiment is as follows:

[0041] Figure 6 is the impact of triplet loss on the final vehicle re-identification performance, where:

[0042] CCLM-w / o-TL represents the method without using triplet loss;

[0043] CCLM-w / -TL represents the method using triplet loss.

[0044] from Figure 6 It can be seen that using triplet loss can improve the final vehicle re-identification performance.

[0045] Figure 7 In order to use the vehicle detection frame to replace the vehicle positioning area, the impact of the tracking and positioning results on the final vehicle re-identification performance is corrected in time, including:

[0046] SLLM-w / o-RVS stands for the method without correction of vehicle tracking sequence;

[0047] SLLM-w / -RVS stands for the revised vehicle tracking sequence method.

[0048] from Figure 7 It can be seen that using the vehicle detection frame instead of the vehicle positioning area to timely correct the tracking and positioning results can improve the final vehicle re-identification performance.

[0049] Depend on Figure 4 It can be seen that as the number of training frames increases, the performance gradually improves. Since these training video frames do not need to be manually labeled with IDs, it shows that our method is very easy to expand.

[0050] Depend on Figure 5 It can be seen that each line is a vehicle image with the same ID in 10 consecutive frames that are automatically annotated. The first 5 images are the results of correcting the vehicle tracking sequence without using the vehicle detection frame, and the last 5 images are the results of correcting the vehicle tracking sequence with the vehicle detection frame. It can be seen that our method is effective.

[0051] Figure 8 To combine automatically annotated data and supervised data, joint training is performed to improve the re-identification performance, where:

[0052] Fully Supervised represents a method that uses only labeled data to train the neural network and then tests it on the test set;

[0053] Weakly Supervised represents a method that uses only automatically labeled data to train the neural network and then tests it on the test set;

[0054] S+W represents the method of training the neural network by mixing the automatically annotated data with the labeled data, and then testing it on the test set;

[0055] S->W represents the method of first training the neural network with labeled data and then fine-tuning the network with automatically labeled data;

[0056] W->S represents the method of first training the neural network with automatically labeled data and then fine-tuning the network with labeled data.

[0057] from Figure 8 It can be seen that W->S has greatly improved performance compared to Fully Supervised. This shows that the unsupervised re-identification method we proposed can not only achieve the good performance of the supervised method, but also assist the supervised data to further improve the performance.

[0058] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An unsupervised vehicle re-identification method for aerial images, characterized in that: The following steps are involved: (i) Randomly select 200 frames from the original video to mark the vehicle frame, and then train the vehicle detection model; (ii) obtaining the position and bounding box parameters of the vehicles in all video frames according to the trained vehicle detection model; (3) Train the vehicle tracking and positioning model based on the target vehicle bounding box parameters and the cycle consistency method; (iv) according to the vehicle tracking and positioning model, the corresponding vehicles in the video frame are associated and the corresponding vehicle ID (identification number) is marked; (V) Train the neural network and update the parameters based on the automatically labeled vehicle data, and then extract and retrieve vehicle features on the test set; In the step (a), when training the vehicle detection model, a target detection algorithm such as SSD, which is widely used in target detection tasks, is selected as the basic model for vehicle detection; In the step (ii), the bounding box parameters are the relative positions of the four vertices of the vehicle target in the frame; In the step (iv), the learned vehicle tracking and positioning model can locate the position of the target vehicle in the next frame according to the position of the target vehicle in the current frame, so we can continuously use the vehicle tracking and positioning model to obtain all sequence images of the target vehicle in the subsequent frames, and then mark them with the same ID. However, the vehicle tracking and positioning model may have errors, and continuous use may amplify the errors. Therefore, using the vehicle detection frame near the positioning result to correct the tracking and positioning result of the vehicle tracking and positioning model can make the automatically obtained vehicle sequence more accurate. When there is no detection frame near the positioning result or the closest vehicle detection frame is incorrect, interference images with inaccurate vehicle frames or incorrect ID annotations will appear in the automatically marked sequence. If these interference images can be removed and the network is retrained, it will be more conducive to improving the performance of vehicle re-identification. First, a vehicle feature encoding network is trained using the automatically marked vehicle sequence, and then the features of all vehicle images in the sequence are extracted. For vehicle sequences with the same ID, the feature distances of all images in the sequence to the initial target (the first vehicle image in the sequence) are calculated, and those exceeding a certain threshold are removed from the sequence.

2. The unsupervised vehicle re-identification method for aerial images as claimed in claim 1, characterized in that: In the step (iv), in the process of associating the corresponding vehicles and marking the corresponding vehicle IDs, the vehicle detection frame is used to replace the vehicle positioning area to timely correct the tracking and positioning results.

3. The unsupervised vehicle re-identification method for aerial images as claimed in claim 2, characterized in that: In the step (iii), a self-supervised triplet loss function is introduced to assist vehicle feature representation learning.

4. The unsupervised vehicle re-identification method for aerial images as claimed in claim 3, characterized in that: In the step (five), the automatically labeled vehicle data is combined with the manually labeled vehicle data for network training to further improve the representation capability of vehicle features.

Citation Information

Patent Citations

  • Anodes for electrochemical purposes

    EP0002017A1

  • Real-time vehicle flow detecting and tracking method based on aerial photography data

    CN108921875A

  • Vehicle target segmentation method and device and communication equipment

    CN109766828A

  • Vehicle brand multi-angle identification method based on YOLOv3 model

    CN110119726A

  • Checkpoint vehicle retrieval method and system based on multi-target area feature aggregation

    CN111078946A