A feature post-fusion vehicle re-identification method based on pure vision

Abstract

The application is a pure vision-based feature post-fusion vehicle re-identification method, which improves the vehicle fine-grained feature and global feature extraction model structure, and proposes a new measurement method for vehicle orientation constraint. First, a classification model is designed for the vehicle model information, including a UNet structure for extracting fine-grained features and a convolution model for image coding, and the classification results of the model are used for sample pre-screening. For the global feature of the vehicle, a model with convolutional neural network and Transformer structure is used for coding, and the features are post-fused. At the same time, a local center loss function based on clustering algorithm is introduced to constrain the intra-class distance of the same orientation target, and finally the sum of cross entropy loss, triplet loss and local center loss is used to represent the total loss function. The application achieves mAP accuracy of 0.799 and 0.699 on VeRi776 and VERI-WILD (large) data sets respectively, and the overall performance is better than most existing pure vision-based vehicle re-identification methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image processing, and particularly relates to a vehicle re-identification algorithm using a feature fusion model with multiple network branches and a clustering algorithm to constrain the vehicle orientation, as shown in Figure 1 The present application relates to the field of computer vision and image processing, and particularly relates to a vehicle re-identification algorithm using a feature fusion model with multiple network branches and a clustering algorithm to constrain the vehicle orientation, as shown in BACKGROUND

[0002] Vehicle Re-identification (Vehicle ReID) is a technology that uses computer vision to retrieve specific vehicles from large-scale surveillance videos, and is one of the sub-tasks in the field of Content-based image retrieval (CBIR). This technology plays an important role in current intelligent transportation systems and intelligent security fields. In real-world scenarios, it may not be possible to obtain license plate information due to lighting conditions and image shooting angles, or the vehicle to be located and retrieved may not have a license plate hanging. Therefore, vehicle retrieval based on appearance features without using license plate information is increasingly attracting the attention of computer vision researchers. Unlike vehicle classification, vehicle re-identification requires a retrieval system to implement an instance-level search task based on vehicle images, which is extremely challenging.

[0003] In recent years, with the development of deep neural networks, many related researches have shown that convolutional neural networks and similar structures have good effects on the extraction of vehicle global features in most cases. In order to distinguish vehicles with similar appearances, feature extraction models need to extract distinguishable and robust local and global features. However, when the photographed vehicle images have similar colors, models, and poses, the overall appearance features of the vehicles are difficult to distinguish. The main challenges faced by the vehicle re-identification task include local occlusion of images, vehicle perspective changes, lighting conditions, and image similarity differences. Most recent vehicle re-identification task-related works have basically taken breaking the above difficulties as the main goal. According to the specific problems and solutions in the research, the research background of this task is introduced in the following order.

[0004] (1) Feature representation-based method

[0005] Encoding the whole image and forming a feature representation that can describe the image features is one of the key works of most CBIR tasks. In the early related research of vehicle re-identification tasks, pre-designed operators are usually used for manual feature extraction. After the CNN was proposed, deep learning began to be popular in various fields of computer vision, which made up for the shortcomings of the traditional manual feature expression, and was more suitable for image recognition and retrieval tasks. In 2016, Liu et al. proposed using VGGNet model to extract the global information of vehicle image. (Reference 1: Liu H, Tian Y, Yang Y, et al. Deep relative distance learning: Tell the difference between similar vehicles [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2167-2175.) In 2017, Cui et al. used different deep neural networks trained with multiple targets to extract image features. (Reference 2: Cui C, Sang N, Gao C, et al. Vehicle re-identification by fusing multiple deep neural networks [C] / / 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 2017: 1-6.) In 2019, Chen et al. divided the vehicle image into multiple parts by the height channel and the width channel in the deep feature map to extract the fine-grained features of the vehicle respectively. (Reference 3: Chen H, Lagadec B, Bremond F. Partition and reunion: A two-branch neural network for vehicle re-identification [C] / / CVPR Workshops. 2019: 184-192.)

[0006] In recent years, models based on the Transformer structure have achieved outstanding results in the field of computer vision. Various network structures based on the self-attention mechanism of the Transformer have reached the state-of-the-art level in various computer vision tasks, and have gradually replaced CNN to become the most commonly used benchmark model in some tasks. Dosovitskiy et al. first drew on the research results in the field of NLP and proposed the ViT model, which completely uses the Transformer-based model to encode image information. (Reference 4: Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [J]. arXiv preprint arXiv:2010.11929, 2020.) In the vehicle re-identification task, He et al. first proposed a ReID network based on the pure Transformer structure in 2021, and also proposed the use of the Jigsaw Patch Module (JPM) and the Side Information Embeddings (SIE) method to further enhance the robustness of the model in extracting features (Reference 5: He S, Luo H, Wang P, et al. Transreid: Transformer-based object re-identification [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 15013-15022.). At the same time, Luo et al. also used the Transformer model as the backbone network for unsupervised domain adaptation training, and achieved the best result in the second track of the AI CITY 2021 public competition (Reference 6: Luo H, Chen W, Xu X, et al. An empirical study of vehicle re-identification on the AI City Challenge [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 4095-4102.)

[0007] (2) Methods based on view angle constraints

[0008] In real-world traffic monitoring systems, the camera perspectives at different locations are different, and the different directions of the photographed vehicle bodies result in great differences in the appearance of the same vehicle, which brings great challenges to vehicle re-identification. The re-identification bias problem caused by perspective changes can be further subdivided into intra-class bias caused by perspective changes of the same vehicle and inter-class similarity caused by the same model of different vehicles under the same perspective. There are mainly two ways to solve this problem in deep learning, one is to use additional auxiliary information, such as predicting the appearance features of a vehicle at a certain perspective, predicting the current perspective according to the vehicle key points, or using a pre-trained model to predict the local posture of the vehicle to obtain the vehicle features under the known perspective; the second is to design a reasonable network structure and loss function to minimize the impact of perspective changes on re-identification bias, so that vehicles with the same class are more compact in the feature space. In 2017, Wang et al. proposed a method to mark 20 special positions on the vehicle body surface that are recognizable and divide them into four categories: front, back, left, and right. A stacked hourglass network model is used to predict the 20 key points to form a key point mapping diagram, which is then accumulated according to the face it belongs to to form a key region mapping diagram of the four faces. (Reference 7: WANG Z, TANG L, LIU X. Orientation invariant feature embedding and spatial temporal regularization for vehicle re-identification [C] / / The IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2017: 379-387.) Lin et al. proposed a multi-view learning (MRL) method, which uses the K-means clustering algorithm to obtain the perspective label of the vehicle, and divides the vehicle re-identification task into two sub-tasks: vehicle re-identification under the same perspective and cross-perspective. Under the same perspective, let the vehicles in the feature space close to the perspective center point and push away the negative sample vehicles; under the cross-perspective, let the vehicle samples in the same cluster in the feature space close to each other and ensure a certain interval between different clusters.(Reference 8: Lin W, Li Y, Yang X, et al. Multi-view learning for vehicle re-identification [C] / / 2019 IEEE international conference on multimedia and expo (ICME). IEEE, 2019: 832-837.) Meng et al. used a pre-trained image segmentation model to divide the vehicle target into multiple different parts, and established the alignment of the perception view elements by comparing the segmentation results of different samples to constrain the intra-class distance caused by shooting from different angles. (Reference 9: Meng D, Li L, Liu X, et al. Parsing-based view-aware embedding network for vehicle re-identification [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 7103-7112.)

[0009] (3) Method based on metric learning

[0010] Metric learning is a commonly used method in computer vision tasks such as image retrieval, vehicle re-identification, and face recognition. Generally speaking, metric learning aims to learn a feature embedding space in which samples belonging to the same category have a smaller distance, while samples from different categories have a larger distance. Vehicle re-identification technology uses the similarity between features to estimate the probability that a vehicle belongs to the same category, and selecting an appropriate metric learning method can effectively improve the discriminability and expressiveness of the features. The research goal of metric learning is to make the distance between samples belonging to the same class in a high-dimensional embedding space smaller than the distance between samples from different classes. In deep learning, the key to implementing metric learning lies in how to design the corresponding loss function.

[0011] In the image recognition and retrieval tasks, in order to compare the similarity between the same class samples and different class samples, it is usually necessary to directly compare the feature vectors of different class samples. In 2015, relevant scholars proposed a triplet loss function in the research of face recognition, which became a typical metric loss method and was widely used in related tasks of face recognition, pedestrian re-identification and vehicle re-identification. The main idea of the triplet loss function is: in the training data set, a sample (Anchor) is randomly selected, and samples with the same class (Positive) and different classes (Negative) are selected at the same time to form a triplet sample group, and the same backbone model is used to extract the feature vectors of the samples, and the loss function is defined to increase the distance between the features of Anchor and Negative samples and to reduce the distance between the features of Anchor and Positive samples in the updating process, as shown in formula (1).

[0012] L Triplet = max(0,||f(x a )-f(x p )||-||f(x a )-f(x n )||+m) (1)

[0013] Wherein, m represents a preset margin, f(x a ), f(x p ) and f(x n ) represent the feature vectors of Anchor, Positive and Negative samples respectively, and ||·|| represents L2 norm.

[0014] In 2016, Wen et al. proposed to apply the center loss function to the face recognition task (reference document 10: Wen Y, Zhang K, Li Z, et al. A discriminative feature learning approach for deep face recognition [C] / / European conference on computer vision. Springer, Cham, 2016: 499-515.). The center loss learns the center of the deep features of each class and calculates the distance between the sample embedding feature and the corresponding class center as the loss function. This method makes up for the shortcoming of the triplet loss that can only randomly sample a pair of positive and negative samples at a time, and its calculation formula is shown in formula (2).

[0015]

[0016] wherein, represents the depth feature vector of the jth sample, y j is the label corresponding to the jth sample in the small batch, represents the center of the depth feature of the y j class, and B is the size of the batch.

[0017] In summary, the current vehicle re-identification algorithm still mainly uses CNN and similar structures to generate feature expression of images, and the model based on Transformer only has some preliminary attempts without further research. At the same time, for the feature deviation caused by cross-view, most current methods need additional data labeling or the help of pre-training models of other tasks to achieve this, which also causes additional costs for the training of the vehicle re-identification model. The present application mainly starts from the above problems to further improve the accuracy of the vehicle re-identification algorithm. SUMMARY

[0018] The present application aims to improve the accuracy and robustness of the vehicle re-identification algorithm from the perspective of pure vision. The main difficulty problem to be solved is to improve the reliability of the deep neural network model for global feature extraction of vehicle images and the constraint on the appearance changes of images under different views. At the same time, the present application uses the training strategy of the current advanced visual model and makes certain degree of innovation and optimization according to the particularity of the vehicle re-identification task.

[0019] Next, the main content of the present application will be described in detail, which specifically includes the following steps:

[0020] Step 1: Designing a fine-grained feature extraction network based on vehicle type recognition task

[0021] The current mainstream public data set will provide additional visual information including color and vehicle type. Appropriately using these information for auxiliary model training can effectively improve the accuracy of the final re-identification result. Compared with the color information of the vehicle which is often concentrated in a complete area of the image, the vehicle type information needs more fine-grained features for description. The appearance difference between images of different vehicle types is often scattered in multiple parts of the vehicle.

[0022] The vehicle type information of the vehicle can be used as a classification task. This classification task can usually directly use a CNN network to extract image features, and obtain the classification result predicted by the model after using a fully connected network. However, for vehicle targets, directly using a network with common convolution structure (such as ResNet50) for feature extraction will make the model difficult to focus on fine-grained features of the vehicle, and cause the inability to distinguish vehicle samples with similar scenes and colors.

[0023] To extract the fine-grained features of the vehicle, the network based on the UNet structure (as shown in Figure 2 ) is used in the present application to generate the weight mask feature map of the original image, and the input of ResNet50 is composed of the original image and the weight mask feature map, and the overall model structure is as shown in Figure 3 The UNet network structure is a commonly used backbone network in the field of medical image segmentation and semantic segmentation, and the U-shaped structure can capture various features from semantic information to fine-grained features in the image. Since the receptive field of ResNet50 will gradually expand in the process of forward propagation, it is possible to lose the key details of the image in the down-sampling process. In the present application, a pre-trained UNet structure model is used to generate the attention distribution map, the purpose being to enable ResNet50 to focus more on the information related to the vehicle type when extracting features. The UNet generates a mask map with global attention, and the size of the mask map is the same as that of the original image. A linear combination of the original image and the UNet output feature map is calculated, and the linear combination is input into ResNet50 to obtain the global encoding features of the image. Finally, the output result of ResNet50 is transmitted into the classifier model of the fully connected layer to generate the classification result. Similar to other classification tasks, the Cross Entropy Loss is used for the training of the model.

[0024] In the test process of the vehicle type classification model, all samples will be pre-screened first: whether there is a same vehicle type as the sample to be searched in the two vehicle types with the highest probabilities in the classification result generated by the vehicle type classification model, if there is no same vehicle type, the test set sample will be directly determined as a negative example, and no subsequent global feature extraction will be performed.

[0025] Finally, the vehicle type recognition model proposed in the present application is compared with related methods in experiments on two large vehicle re-identification datasets VeRi776 and VERI-WILD, and it is proved that the method has good performance in the results of vehicle type classification. At the same time, the ablation experiment verifies that the pre-screening process using the vehicle type classification model can improve the final accuracy of vehicle re-identification, and verifies the effectiveness of the method.

[0026] Step two: design and training of the global feature extraction network

[0027] In this step, the structure of the image global feature extraction model is designed, and a targeted training method is proposed. It is divided into the following two parts:

[0028] 2.1 Design of network structure

[0029] The extraction process of the image global feature is as shown in Figure 1As shown, the model is mainly divided into two independent branches, namely the CNN-based network structure and the Transformer-based network structure, which respectively use the two branches to encode the image and perform post-fusion processing on the extracted features.

[0030] The Swin Transformer model (referred to as Swin model) has been proven to be superior to most current methods in a wide range of visual tasks such as object detection and semantic segmentation. The Swin model uses a sliding window strategy (Shifted Window) to subsample the image. The windows between different layers of the network have different receptive fields, which can introduce local aggregation information, so that the obtained image embedding features contain richer global semantic information.

[0031] The present application aims to generate a robust feature representation of a vehicle image: the CNN-based network structure (such as the ResNet50 used in the present application) has stronger ability to obtain local features of the image because it gradually expands the receptive field by encoding local feature maps and subsampling; while the Transformer-based network (such as the Swin Transformer used in the present application) performs global self-attention perception on the image, and each local part is related and influenced by each other, and the result pays more attention to the understanding of the semantic information of the image. In the task of vehicle re-identification, only focusing on the semantic features of the image cannot distinguish vehicles with similar appearances, and paying too much attention to the local information of the vehicle cannot make the model train stable feature extraction effect, so the two types of deep neural networks with different structures are used in the present application to obtain stable and accurate feature representation of the image.

[0032] In the training process of the global feature extraction model, the two branches use the same data set for independent training and produce different feature representations of the same sample; in the inference stage, the two different feature vectors are post-processed by vector splicing to take advantage of the respective advantages of the Swin model and the Resnet50 model, and an L2 normalization is performed to facilitate the calculation of the similarity between different samples.

[0033] 2.2 Network training strategy

[0034] During the training process of the model, some common model training techniques are used, and some improvements and optimizations are made for the particularity of the vehicle re-identification task. The specific training process and training strategies used are as follows.

[0035] The warm-up training of the model and the label smoothing in the classification task are currently commonly used methods for improving the training effect of the model: in the initial stage of model training, due to the randomness of the initialization parameters of the model, the prediction result of the model is often greatly different from the real result, and using a normal learning rate for training will cause large fluctuations in the updating of the model parameters, which is not conducive to the training of the model. The warm-up training strategy is used to initialize the learning rate with a smaller value in the initial training stage, so that the parameters of the model are smoothly transitioned to the normal training stage, so as to improve the network performance. The label smoothing processing is used to replace the real label value in the cross-entropy loss function in the classification task, which helps to alleviate the overfitting problem of the model.

[0036] The data augmentation method is one of the commonly used methods for effectively improving the sample quantity and richness in computer vision tasks. Some data augmentation methods (such as image geometric transformation, image blur processing, color jitter, Gaussian noise interference, etc.) can be directly applied to most vision-related tasks, and for different tasks, targeted data augmentation methods can also effectively optimize the training results of the model. Among them, the random erasing augmentation method of the image is to generate new samples by covering part of the area of the image. In most current methods, the random erasing is a strategy of erasing a large area of the image in a single area. In the related experiments of the present application, it is found that using the REA strategy in the early stage is not conducive to the convergence of the loss function in the Swin model training, because the large-area area shielding will destroy the self-attention mechanism of the Swin model to the key information, thereby affecting the understanding of the overall semantics of the image. Moreover, the key information that can distinguish similar vehicles is often not distributed in a single area, and the simple REA strategy cannot effectively solve this problem. Therefore, the present application proposes to disperse the erasing area in REA to the whole image and perform local scattered point erasing sampling: first, limit the erasing area of a single area to not more than 10% of the whole image, and randomly generate 0 to 3 erasing areas in the whole image to replace the original whole area. Through this simple optimization measure, the training effect of the Swin model can be effectively improved.

[0037] The present application uses the BNNeck structure to further process the global features generated by the model. The feature vectors generated by the ResNet50 and Swin models are directly used for similarity comparison between different samples, and the original feature vectors are normalized before calculating the loss function of the classification task. The BNNeck structure effectively improves the inconsistency of the embedding space constraints between the loss function of the classification task and the similarity comparison loss function.

[0038] For the training of Swin model, a two-stage training process is used. In the first stage, only the optimization method of cross-entropy loss function and Triplet loss function with label smoothing processing is used to make the model converge quickly. In the second stage, more constraints and data augmentation methods are used to further optimize the model: the application of local center loss is proposed to constrain the model training process of vehicle viewpoint change. At the same time, the improved REA method and label smoothing strategy are used to reduce the overfitting of the model in the training process.

[0039] Some comparative experiments and ablation experiments for fused features on VeRi776 and VERI-WILD datasets verify the effectiveness of the global feature extraction model structure and related training strategies.

[0040] Step three: loss function design based on vehicle orientation

[0041] For the task related to re-identification, it is a common method to use cross-entropy loss function for vehicle label prediction and triplet loss function for similarity comparison according to the class at the same time. The triplet loss function itself can use the features of different samples to simultaneously constrain the intra-class distance and inter-class distance, which is very consistent with the final goal of the re-identification task, and its schematic diagram is shown in Figure 4 However, due to the randomness of the process of generating triplets, using the triplet loss function alone will usually slow down the convergence speed of the model, and will also cause the final model to not reach the optimal result. Cross-entropy loss function can indirectly affect the extraction ability of the model for image features by comparing the predicted label of the model with the real label to optimize the model parameters. In the task of vehicle re-identification, adding cross-entropy loss function to the triplet loss function can make the training process of the model more stable, and is more conducive to optimizing the model to the optimal result. Formula (3) is the calculation formula of cross-entropy loss function.

[0042]

[0043] wherein, N i represents the number of images in a batch, N id represents the number of classes in the entire training set, y represents the real label of the input image, and x[j] represents the output result of the jth sample after a fully connected layer.

[0044] The present application uses the strategy shared by cross-entropy loss and triplet loss, and also adds a local center loss function with weight. The local center loss function is an optimization method for the center loss function on the vehicle re-identification task. The center loss function itself is based on the assumption that the feature vectors of each class are clustered at a center point. However, in the vehicle re-identification task, due to the change of the orientation of the vehicle and the image shooting angle, the features of the same class samples cannot be clustered into the same center point, and their distribution in the feature space is simultaneously affected by the class and the orientation. In the feature space, vehicles with the same ID are clustered into 4 to 5 clusters due to the cross-view problem. Based on this assumption, the present application proposes a local center loss function more suitable for the vehicle task. After each iteration of the model training process, the K-means clustering method is used to cluster the samples of each class, the clustering centers of all training samples in the feature space are obtained, and the local center loss function is calculated using the formula shown in formula (4).

[0045]

[0046] wherein B represents the number of samples in a batch, f j is the feature vector corresponding to the jth sample, represents the feature vector of the Nth cluster center corresponding to the jth sample.

[0047] As Figure 5 shown, different shades of points represent the distribution of two classes of samples in the feature space. Figure (a) shows the assumption of the distribution of different classes of samples by the center loss function. Figure (b) shows the assumption of the distribution of vehicle samples in the feature space by the local center loss function. The goal of the local center loss is to push the sample corresponding to the asterisk to the nearest local cluster center (solid arrow), rather than the global cluster center of the class (dashed arrow). Like the center loss function, the local center loss function uses the information of all samples in its formula to reduce the randomness generated during the training process, and assumes that the feature distribution formed by the multi-view vehicle image is four cluster clusters, and then through experimental verification, it is proved that it is more suitable for the vehicle re-identification task than the center loss.

[0048] The present application finally uses the sum of cross-entropy loss, triplet loss and local center loss to represent the total loss function, and optimizes the global feature extraction model, as shown in formula (5), wherein λ is the weight parameter corresponding to the local center loss function.

[0049] L=L ID +L Tripelt +λL LocalCenter (5)

[0050] In the model inference stage, the image to be queried can be better matched to the vehicle sample with the same perspective, and the mis-matching of different categories with similar perspectives is reduced, further verifying the effectiveness of the use of the local center loss function for limiting the cross-perspective feature offset. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the network structure of the global feature extraction model and the output result.

[0052] Figure 2 is a UNet network structure diagram.

[0053] Figure 3 is a vehicle model recognition model overall structure diagram.

[0054] Figure 4 is a ternary loss function optimization process diagram.

[0055] Figure 5 is a local center loss function optimization process diagram.

[0056] Figure 6 is a VeRi776 and VERI-WILD dataset sample example.

[0057] Figure 7 is a vehicle re-identification result diagram of the present application. DETAILED DESCRIPTION

[0058] The technical solutions, experimental methods and test results of the present application will be further described in detail below in combination with the drawings and specific experimental embodiments.

[0059] The present application relates to the vehicle re-identification subject in the field of computer vision, and proposes a feature post-fusion vehicle re-identification method based on pure vision, which mainly optimizes the model structure and metric learning method and the like. The method includes three main steps, i.e. constructing a vehicle model classification model, performing feature fusion based on different structure feature extraction networks, and designing a loss function based on vehicle orientation.

[0060] The experimental steps will be described in detail below.

[0061] Step one: use the fine-grained feature extraction capability of the UNet model to assist ResNet50 to constitute a vehicle model recognition model, and use the vehicle images and vehicle model labels in the dataset to train the vehicle model recognition model.

[0062] Step two: use the optimized training strategy to train two branch image global feature extraction models with different structures respectively, and perform post-fusion processing on the extracted features. The related loss function proposed in the present application is used in the training process of the model.

[0063] Step three: performance test of the model on the test set data, i.e. feature extraction is performed for the image to be detected (Query) and the candidate set (Gallery) image respectively, and the candidate set image is returned in the order of high to low similarity of the features of the image to be detected, and the various indicators for evaluating the performance of the model are calculated according to the results.

[0064] The experimental conditions and conclusions of the present application are described in detail below.

[0065] (1) Experimental results of the vehicle type classification model

[0066] In order to verify the effectiveness of the vehicle type recognition model proposed in the present application, related experiments were performed on the VeRi776 dataset, and the experimental results of two other models were compared, as shown in Table 1. The experimental results show that the method proposed in the present application obtains the highest accuracy in the vehicle type recognition experiment.

[0067] Table 1 Comparison of performance of three different networks for vehicle type recognition task

[0068] Model Method description Accuracy (%) Model one GoogLeNet 90.5 Model two ResNet50 93.8 Model three ResNet50+UNet (the present method) 95.3

[0069] (2) Vehicle re-identification dataset and main evaluation indicators

[0070] Main dataset: The following introduces the datasets and evaluation indicators used for related experiments on the vehicle re-identification task. The present application mainly performs detailed experiments on two large public datasets, VeRi776 and VERI Wild, as shown in the sample Figure 6 The VeRi776 dataset is an improved and enhanced dataset of the VeRi dataset proposed by Liu et al. in 2018, which contains 37781 images of 576 vehicles in the training set and 11579 images of 200 vehicles in the test set. The annotation information includes the color, model and camera ID of the vehicle. The VERI Wild dataset is a dataset collected by Lou et al. in 2019. It contains about 400000 vehicle images and 40000 different vehicle samples. Compared with VeRi776, VERI Wild has larger data volume, richer shooting scenes and more sample categories.

[0071] Common evaluation indicators: for the evaluation of vehicle re-identification model, usually use CMC curve (Cumulative Matching Characteristics) and mAP (mean Average Precision) as two common evaluation standards. In the test process of vehicle re-identification task, first need to give the sample to be detected (Query), through the calculation of vehicle re-identification model, the images in the candidate set (Gallery) are sorted according to the similarity with the sample to be detected from high to low, and the re-identification result is obtained. Among them, the most commonly used in CMC curve is rank-1, rank-5 index, which respectively represents the probability that the first item in the returned sorting result is the correct result and the probability that the correct result is contained in the first five items. While the mAP index calculates the comprehensive level of the accuracy and recall rate of the returned result. The three indexes of rank-1, rank-5 and mAP can indirectly indicate the accuracy of the returned result of the model, and the higher the index value is, the better the overall performance of the model is.

[0072] (3) Experimental details and parameter configuration

[0073] In the experiment, the ResNet network used by the application is a network with ResNet50 structure, the size of the input image is 256x256, and the dimension of the output feature vector is 2048; while the Swin Transformer model uses a Swin model with window size of 7 and patch size of 4, the size of the input image is 224x224, and the dimension of the output feature vector is 1024.

[0074] For the training process of ResNet50 model, set its basic learning rate to 5e-4, including 5 rounds of warm-up training. In the 40th and 80th generation of training, the learning rate is reduced to 5e-5 and 5e-6 respectively. Use Adam optimizer to directly optimize ResNet50.

[0075] For the training of Swin model, a two-stage training strategy is adopted. In the first stage, use a simple training strategy: set its basic learning rate to 1.25 and directly train for 30 generations. In the second stage, use cosine learning rate and improved REA method, and use local center loss function to constrain model update, and train for 120 generations. Because the training result of Swin model is greatly affected by the optimizer, using Adam optimizer will lead to suboptimal result, so AdamW optimizer is used in the whole training process of Swin model.

[0076] (4) Re-identification network experimental results

[0077] Based on the above evaluation indexes and experimental details, the application carries out comparative experiments based on two data sets of VeRi776 and VERI-WILD, and the corresponding experimental results are shown in Tables 2 and 3, and the visual results are shown in Figs. 2 and 3. Figure 7 The method of the application is compared with other currently more advanced network models in the table.

[0078] Table 2: Performance comparison of experimental results on VeRi776 dataset and other methods

[0079] Method name mAP Rank-1 Rank-5 GoogLeNet 0.170 0.498 0.712 Siamese+Path 0.583 0.835 0.900 RAM 0.615 0.886 0.940 AAVER 0.612 0.890 0.947 PAMTRI 0.719 0.929 0.970 PRN 0.743 0.943 0.989 SAVER 0.796 0.964 0.986 Ours 0.799 0.968 0.985

[0080] Table 3: Performance comparison of experimental results on VERI-WILD dataset and other methods (large (L), medium (M), small (S))

[0081]

[0082] In order to verify the effectiveness of the design of each part in the application, a series of ablation experiments are carried out on the VeRi776 dataset, and a plurality of sets of experimental results are obtained, which compare the final effects of the model under the use of different strategies from different angles, and verify the effectiveness of each part.

[0083] (a) For the effectiveness of the vehicle recognition model, an experiment on whether the addition of vehicle type information in preprocessing affects the performance of the final model is designed. The results in Table 4 show that after adding the vehicle type recognition model as a sample pre-screening, the mAP value of the final result of re-identification increases from 0.790 to 0.799.

[0084] Table 4: Experimental results for verifying the effectiveness of vehicle type recognition (VeRi776 dataset)

[0085] Method setting mAP Rank-1 Rank-5 Without adding a vehicle type recognition model 0.790 0.966 0.980 With adding a vehicle type recognition model 0.799 0.968 0.985

[0086] (b) For the structure of the global feature extraction model, a group of experiments are designed to compare the performance of global feature extraction networks with different structures, in order to verify the effectiveness of the backbone network of the application and the effectiveness of the feature post-fusion method. The experimental results are shown in Table 5. It can be observed that the post-fusion result of the features of the Swin model and the features of the ResNet50 using the method proposed in the application reaches the best effect in this group of experiments on the VeRi776 and VERI-Wild datasets.

[0087] Table 5: Performance comparison of global feature extraction networks with different structures

[0088]

[0089] (c) The influence of different loss functions on the training results of the model was designed to design an ablation experiment to verify the effectiveness of the local loss function, and the experimental results are shown in Table 6. The combination of cross-entropy loss and triplet loss function is widely used in the re-identification task, and it is regarded as the baseline method. Through the experimental results of VeRi776 dataset, it can be found that the increase of center loss function and local center loss function respectively improves the final performance of the model, and the effect of the model reaches the optimal after using the local center loss.

[0090] Table 6 Verification of the effectiveness of the local center loss function on the optimization of model training (VeRi776 dataset)

[0091] Method mAP Rank-1 Rank-5 Cross-entropy+triplet 0.788 0.954 0.982 Cross-entropy+triplet+center 0.795 0.960 0.986 Cross-entropy+triplet+local center 0.799 0.968 0.985

[0092] In summary, the application proposes a vehicle re-identification method based on pure vision. First, the attention module based on Unet is used for weight allocation of feature extraction of ResNet50, and a vehicle model recognition model is realized. In addition, a method of fusing Swin model and ResNet50 model is used to obtain the robust deep feature expression of the vehicle image. At the same time, the application proposes to use the local center loss function as a constraint for the cross-view deviation of the vehicle to participate in the training process of the model. Through the comparative experiment and ablation experiment of the model on VeRi-776 dataset and VERI-WILD dataset, the effectiveness of each part designed in the application is verified.

Claims

1. A vehicle re-identification method based on pure vision feature fusion, characterized in that: A vehicle model recognition model is implemented through fine-grained perception using the Unet network structure for sample pre-screening; global image features are extracted through feature fusion of two deep neural networks with different structures; and the parameters of the training model are constrained by three parts, including the center loss function, in the loss function. The implementation steps are as follows: S1. Structural design of the vehicle model recognition model: The vehicle model recognition model is used to identify vehicle models in candidate images using vehicle model labels, and determines whether the sample enters the subsequent global image matching process based on whether it matches. The vehicle model recognition model's network structure includes a ResNet50 network for encoding global image features and an attention module of a UNet structure for generating adaptive weights. The UNet network generates a weight allocation layer of the same size as the original image, and then uses the encoded result and the original image as input to the ResNet50 network for feature extraction, obtaining the image's feature vector representation. Finally, a fully connected layer is added to the model to generate the classification result. The vehicle model recognition model uses the vehicle's model label as the evaluation criterion and uses the cross-entropy loss function for model training. During the testing phase of the model, for each sample to be detected, the vehicle model is first identified for all samples in the candidate set, and the results are compared with the actual vehicle model of the sample to be detected. It is determined whether there is a vehicle model with the same probability as the sample to be detected among the two vehicle models with the highest probability in the vehicle model identification results. If there is no vehicle model with the same probability, the sample is marked as a negative class and is not included in the subsequent global feature extraction process. S2, Global Feature Extraction Network Based on Feature Fusion: The global feature extraction model is used to generate global visual information representations of images and to determine the similarity between images through relevant vector similarity calculation methods. The feature extraction models based on CNN structure and Transformer structure each have their advantages, and are respectively suitable for understanding local information and understanding global semantic information. The post-fusion method is adopted to use the two vectors together as the encoding result of the original image to participate in the similarity calculation, forming a two-branch global feature extraction network. The specific structures of the two-branch network are ResNet50 and Swing Transformer network, respectively. During training, the two branch networks are trained separately, and the model's encoding results are trained using the BNNeck structure: the feature vectors are directly used to calculate the loss function for comparing the similarity between samples, and the feature vectors are normalized and then connected to the fully connected layer for calculating the loss function for the classification type; during the model's inference process, the feature vectors generated by the two networks are concatenated to generate high-dimensional features, which are used for calculating the similarity between samples. S3. Design of metric learning methods: In the training task of the global feature extraction model, a constraint function that can be applied to both vehicle sample images and re-identification tasks is needed as the loss function for model training. Specifically, the three loss functions of cross-entropy loss, triplet loss and local center loss are used in joint training. The calculation formula is shown in Equation (1): L=L ID +L Tripelt +λL LocalCenter (1) Among them, L ID L Tripelt and L LocalCenter Let represent the cross-entropy loss, triplet loss, and local center loss, respectively, and λ be the weight parameter corresponding to the local center loss; The local center loss function is an optimization of the center loss function for vehicle samples: to avoid the inability of vehicle samples of the same category to cluster to the same center point due to cross-viewpoint bias, which would cause the distribution of samples in the feature space to be affected by both category and orientation, the local center loss function uses K-means clustering to cluster the samples of each class after each iteration of the model training process, resulting in 4 clusters, and obtains the cluster center of all training samples in the feature space to express the actual feature distribution of vehicle samples of the same class under multiple different viewpoints. Its specific calculation formula is shown in Equation (2): Where B represents the number of samples in a batch, f j c is the feature vector corresponding to sample j. jN The feature vector representing the cluster center of the Nth category corresponding to sample j; In the two branches of the global feature network, the Swing Transformer network only incorporates local centroid loss for parameter training during the second stage of training to avoid model training instability.

Images