A vehicle re-identification method based on multi-loss joint learning

Abstract

The present invention proposes a vehicle re-identification method based on multi-loss joint learning, comprising: extracting vehicle key features in vehicle image samples using a ResNet50 network model; mapping the extracted vehicle key features to a feature space, and using the distance of vehicle image samples in the feature space to reflect the correlation between images; continuously training and learning the loss network so that vehicle samples with the same ID form clusters in the feature space; using a BN layer to normalize the generated global features; and using a batch difficulty triple loss function L t 、ID loss function L ID And the circle loss function L c The present invention can solve the weak constraint of the traditional triplet loss and the technical problem that similar vehicles with different IDs but belonging to the same model are difficult to distinguish.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to a vehicle re-identification method based on multi-loss joint learning. Background Art

[0002] Vehicle recognition is not only the core of smart city transportation, but also one of the hot topics in computer vision research. It has been widely used in various fields such as vehicle detection and vehicle classification. Vehicle re-identification, as a new challenging and special topic in this field, has attracted more and more researchers in recent years. Vehicle re-identification means that given a vehicle image, the same vehicle that appears in the camera or other photos can be identified. It can be divided into sub-problems of image retrieval. Vehicle re-identification can be applied to real traffic systems in reality to locate and track target vehicles, thereby achieving functions such as combating crime. However, in the vehicle re-identification task, different camera positions will produce changes in illumination, perspective and resolution, which will lead to intra-class differences in the same vehicle under different perspectives or inter-class similarities between different vehicles due to the same model, which makes researchers in the vehicle re-identification task face huge challenges.

[0003] According to the difference in the objective function of the training network, the vehicle re-identification methods based on deep learning can be divided into two categories: representation learning-based methods and metric learning-based methods. The representation learning-based methods do not directly consider the similarity of the images when training the network, but treat the vehicle re-identification task as a classification problem or a verification problem; the metric learning-based methods aim to learn the similarity of two images through the network, which can be regarded as a clustering problem. There is a lot of research work on representation learning and metric learning in the field of vehicle re-identification, and they also have their own advantages and disadvantages. After a period of development, the most commonly used method in the field of vehicle re-identification is to use representation learning loss and metric learning loss to jointly optimize the network, thereby integrating the advantages of the two methods. However, although the triplet loss function constantly narrows the distance between the same type and pushes the distance between different types, it does not make a clear measurement of the absolute distance between classes. That is, it focuses too much on the local, which makes it difficult to train and takes a long time to converge. In addition, there is a special difficulty in vehicle re-identification. Since vehicles of the same model are very similar to each other, it is very difficult to distinguish vehicles of the same model with different IDs even with the human eye. Therefore, vehicle re-identification needs to more robustly distinguish the difference between targets and non-targets. Summary of the invention

[0004] Technical problem to be solved: In view of the weak constraint of the aforementioned traditional triplet loss and the technical problem that similar vehicles with different IDs but belonging to the same model are difficult to distinguish, the present invention proposes a vehicle re-identification method based on multi-loss joint learning.

[0005] Technical solution:

[0006] A vehicle re-identification method based on multi-loss joint learning, the vehicle re-identification method comprising the following steps:

[0007] S1, preprocessing the imported vehicle traffic image, cropping it to obtain vehicle images of the same size, and randomly erasing the cropped vehicle images to obtain vehicle image samples;

[0008] Build a vehicle re-identification model, which includes a ResNet50 network model, a loss network, a BN layer, and a fully connected layer;

[0009] S2, using the ResNet50 network model to extract key vehicle features in vehicle image samples;

[0010] S3, maps the extracted key features of the vehicle to the feature space, and the distance of the vehicle image samples in the feature space is used to reflect the correlation between the images; the loss network is continuously trained and learned so that the vehicle samples with the same ID form clusters in the feature space; among them, by selecting vehicle samples of different styles with the same color in the randomly generated small batch samples as difficult triplets; the similarity metric is learned using the triplet loss to minimize the distance between the positive sample pairs and maximize the distance between the negative sample pairs; the corresponding batch difficult triplet loss L is calculated t ;

[0011] S4, use the BN layer to normalize and reduce the latitudinal size of the generated global features to extract the vehicle features f i , the vehicle feature f i Send to the fully connected layer;

[0012] S5, using the difficult triple loss function L t 、ID loss function L ID And the circle loss function L c To jointly learn the feature vectors, enhance the vehicle re-identification model's ability to identify similar samples, expand the feature differences of different types of vehicles, and extract more distinguishing and discriminative global features; the ID loss function is used to learn the feature vectors of vehicle image samples with the same or different IDs; the circle loss function is used to learn category discriminative information;

[0013] S6, uses the test set to test the vehicle re-identification model.

[0014] Furthermore, the vehicle re-identification method further comprises the following steps:

[0015] S7, importing the vehicle image to be identified into the vehicle re-identification model, and extracting the corresponding feature vector;

[0016] The feature vector of the vehicle image to be identified is compared with the feature vector of the vehicle in the video image within the effective geographical area. The vehicle target image with the highest similarity is searched using the similarity calculation formula, and the final recognition result is output.

[0017] Furthermore, in step S1, the imported vehicle traffic image is preprocessed, vehicle images of the same size are cropped therefrom, and the cropped vehicle images are randomly erased to obtain vehicle image samples, and the process includes the following steps:

[0018] S11, performs geometric transformation, horizontal flipping, and mirror source processing on the signature image to make the data more diverse;

[0019] S12, cropping and extracting the vehicle part from the entire image into the same size;

[0020] S13, randomly erases the extracted vehicle image, and removes a rectangular block of a predefined range in the original image for different epochs.

[0021] Furthermore, in step S2, the process of extracting key features of vehicles in vehicle image samples using the ResNet50 network model includes the following steps:

[0022] S21, load the ResNet50 network model, and use the parameters of the network model as initial parameters for training;

[0023] S22, using the trained ResNet50 network model to extract the key features of the vehicle.

[0024] Furthermore, in step S3, generalized average pooling is used to aggregate the features generated by the ResNet50 network model into global features.

[0025] Furthermore, in step S3, the Euclidean distance between vehicle image sample pairs is calculated according to the following formula:

[0026]

[0027] In the formula, d(x,y) refers to the Euclidean distance between the feature vectors of two sample pairs, x and y refer to the feature vectors extracted from the two sample pairs, num is the number of elements in the feature vector, k is the index of each element in the feature vector, and x is the distance between the feature vectors. k andk are the values ​​of the elements with index k in the feature vectors of the two samples respectively.

[0028] Further, in step S3, the batch difficulty triple loss L t The formula is as follows:

[0029] L t =(d a,p +Ad a,n ) +

[0030] In the formula, (·) + is the hinge function, A is an added margin, is the Euclidean distance interval between positive and negative samples, and d(a,n) is the anchor point X a and class X n The distance, d a,p is the anchor point X a and class X p The distance satisfies d(a,n) <d(a,p)。

[0031] Furthermore, in step S5, the ID loss refers to the input being a pair of vehicle images x 1 and x 2 ; y indicates whether the images of two cars belong to the same ID. If y = 1, it means that the two cars belong to the same label, which is called a positive sample; y = 0 means that the cars come from different IDs, which is called a negative sample; By fitting the loss function, the inter-class distance is also increased to a certain extent;

[0032] The formula of the ID loss function is as follows:

[0033]

[0034] In the formula, represents the Euclidean distance calculated for the input image after two feature extractions, N is the total number of vehicle samples in each batch, y represents the label of the sample, the positive class is 1, and the negative class is 0. f(·) is the feature extractor, β represents the βth feature index, n represents the number of features, and markin is the threshold parameter set to measure the degree of distance difference between the feature vectors of positive and negative samples.

[0035] Furthermore, in step S5, the circle loss is used to learn the class discrimination information, that is, to maximize the intra-class similarity and minimize the inter-class similarity;

[0036] Assuming that there are K intra-class similarity scores and L inter-class similarity scores associated with the sample space x, the formula of the circle loss objective function is as follows:

[0037]

[0038] Among them, S n and S P They are the inter-class similarity and intra-class similarity about the sample space x, respectively. The cosine similarity is used to calculate the intra-class and inter-class similarity scores; and are non-negative integers, and The weight, Δ n and Δ p are the inter-class and intra-class intervals respectively, and satisfy and γ is the scale factor.

[0039] Furthermore, the total loss function L of the vehicle re-identification network is all It is expressed as:

[0040] L all =L ID +a′L t +b′L c

[0041] Among them, a′ and b′ are the weights of triplet loss and circle loss respectively.

[0042] Beneficial effects:

[0043] 1. In view of the weak constraints of the traditional triplet loss, the difficult triplet loss is proposed, which not only solves the problem that many triplets become "useless" triplets as the training continues to deepen, but also solves the problem of time-consuming training.

[0044] 2. Jointly learn the ID loss and the difficult triplet loss to strengthen their constraints.

[0045] 3. In view of the inconsistent distribution of features optimized by ID loss and Triplet loss in the embedding space, the present invention introduces Circle loss joint learning. Circle loss is a deep learning method based on a pair of similarity optimization to maximize intra-class similarity and minimize inter-class similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the steps of a vehicle re-identification method based on multi-loss joint learning according to an embodiment of the present invention;

[0047] Figure 2 This is a flow chart of a vehicle re-identification method based on multi-loss joint learning according to an embodiment of the present invention;

[0048] Figure 3 A network structure diagram corresponding to a vehicle re-identification method based on multi-loss joint learning according to an embodiment of the present invention;

[0049] Figure 4 An example diagram of vehicle differences according to an embodiment of the present invention; Figure 4 (a) is a schematic diagram of intra-class differences. Figure 4 (b) is a schematic diagram of inter-class similarity. DETAILED DESCRIPTION

[0050] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0051] See also Figure 1 and Figure 2 The embodiment of the present invention proposes a vehicle re-identification method based on multi-loss joint learning, and the vehicle re-identification method comprises the following steps:

[0052] S1, preprocessing the imported vehicle traffic image, cropping it to obtain vehicle images of the same size, and randomly erasing the cropped vehicle images to obtain vehicle image samples.

[0053] Build a vehicle re-identification model, which includes a ResNet50 network model, a loss network, a BN layer, and a fully connected layer.

[0054] S2, uses the ResNet50 network model to extract the key features of the vehicle in the vehicle image samples.

[0055] S3, maps the extracted key features of the vehicle to the feature space, and the distance of the vehicle image samples in the feature space is used to reflect the correlation between the images; the loss network is continuously trained and learned so that the vehicle samples with the same ID form clusters in the feature space; among them, by selecting vehicle samples of different styles with the same color in the randomly generated small batch samples as difficult triplets; the similarity metric is learned using the triplet loss to minimize the distance between the positive sample pairs and maximize the distance between the negative sample pairs; the corresponding batch difficult triplet loss L is calculated t .

[0056] S4, use the BN layer to normalize and reduce the generated global features to extract the vehicle features f i , the vehicle feature f i Feed it into the fully connected layer.

[0057] S5, using the difficult triple loss function L t 、ID loss function L ID And the circle loss function L cTo jointly learn the feature vectors, enhance the vehicle re-identification model's ability to identify similar samples, expand the feature differences of different types of vehicles, and extract more distinguishing and discriminative global features; the ID loss function is used to learn the feature vectors of input vehicle image samples with the same ID or different IDs; the circle loss function is used to learn category discriminative information.

[0058] S6, uses the test set to test the vehicle re-identification model.

[0059] Step (1): preprocess the vehicle image

[0060] Specifically, step (1) includes the following sub-steps:

[0061] (1.1) Perform geometric transformation on the signature image to make the data more diverse by mirroring the source image through horizontal flipping.

[0062] (1.2) Crop the vehicle part into the same size from the entire image and extract it.

[0063] (1.3) Random erasing is performed on the cropped vehicle image, that is, a rectangular block of a predefined size range is removed from the original image in different epochs.

[0064] Step (2), feature extraction

[0065] Specifically, step (2) includes the following sub-steps:

[0066] (2.1) Load the ResNet50 network model and use the parameters of the network model as the initial parameters for training. ResNet50 is one of the versions in the ResNet series.

[0067] (2.2) Extract the key features of the vehicle.

[0068] Step (3), aggregation operation

[0069] Specifically, step (3) includes the following sub-steps:

[0070] (3.1) The present embodiment uses generalized-mean (GeM) Pooling, which is a learnable pooling layer. As a fine-grained instance retrieval, it can capture the distinctive features of a specific domain. The purpose is to reduce the spatial resolution of the feature map, thereby achieving spatial invariance to input distortion and translation. In this embodiment, generalized average pooling is used to aggregate the features generated by the backbone network into global features.

[0071] (3.2) The key vehicle information obtained in step (2.2) is mapped into the feature space. The distance in a specific space can reflect the correlation between images.

[0072] (3.3) Calculate triplet loss. When using triplet loss to learn similarity metrics, the goal is to make samples with the same label as close as possible and as far away as possible from other samples that do not belong to the same label. First, let's briefly introduce the triplet. It is constructed as follows: randomly select a sample from the training data set, which is called an anchor point, and then randomly select an anchor point (denoted as X a ) belongs to the same class and the samples of different classes. These two samples are called positive samples (denoted as X p ) and negative samples (denoted as X n ), thus forming a (anchor point, positive sample, negative sample) triple. The purpose of triple loss is to make X a and X p The distance between feature expressions is as small as possible, and X a and X n The distance between feature expressions and the distance between feature expressions are as large as possible, and let X a and X p With X a and X n The distance between and the minimum interval between.

[0073] After obtaining the Euclidean distance between vehicle image sample pairs, the distance formula is as follows:

[0074]

[0075] Where d(x,y) refers to the Euclidean distance between the feature vectors of two sample pairs, x and y refer to the feature vectors extracted from the two sample pairs, num is the number of elements in the feature vector, k is the index of the element in the feature vector, and x is the distance between the feature vectors. k and k are the values ​​of the elements with index k in the feature vectors of the two samples respectively.

[0076] Through continuous training and learning of the loss network, vehicle samples with the same ID eventually form clusters in the feature space, thereby completing the vehicle re-identification task. When using triplet loss to learn similarity metrics, the goal is to make samples with the same label as close as possible and as far away as possible from other samples that do not belong to the same label. Through continuous training and learning of the loss network, vehicle samples with the same ID eventually form clusters in the feature space, thereby completing the vehicle re-identification task. Triplet loss aims to separate positive and negative samples at a certain distance, that is, to minimize the distance between positive sample pairs while maximizing the distance between negative sample pairs. The formula is as follows:

[0077] L t =(d a,p +Ad a,n ) +

[0078] in() + is the hinge function. This loss ensures that, for a given anchor point X a Similar to class X p X a and dissimilar class X n Closer. a is an added margin, which is the Euclidean distance interval between positive and negative samples. In this way, after training, the feature points of the same vehicle are closer than the feature points of different vehicles. Then, the image with a smaller distance to the query image has a higher probability of having the same vehicle ID.

[0079] Although triple loss helps to train convolutional neural networks to have certain discriminative capabilities, it also has certain limitations. Triple loss randomly selects triplets. The larger the data set, the larger the number of triplets may be. It is impractical to give a lot of time to train. What's worse is that some triplets can learn correctly relatively quickly, resulting in a large number of "useless" triplets. Therefore, selecting difficult triplets will become the focus of learning. In fact, repeatedly learning that vehicles of different colors are not the same car, for convolutional neural networks, no key information is learned. Vehicles of different styles with the same color (hard negative) help to understand the concept of "the same vehicle". The so-called hard triplet: At this time, the negative is closer to the anchor than the positive, that is, d(a,n) <d(a,p)。

[0080] Since the batch difficult triple loss only selects the most difficult positive and negative samples in a small batch, it can select triplets more efficiently, which not only solves the problem of non-convergence of traditional triple loss training, but also solves the problem of time-consuming training.

[0081] Step (4), normalization

[0082] Specifically, step (4) includes the following sub-steps:

[0083] (4.1) BN is used to normalize and reduce the dimension of the generated global features. The purpose is to make the model focus on more and larger areas, obtain the features that can be used to construct the loss function and finally complete the learning.

[0084] (4.2) Extract vehicle features f i .

[0085] (4.3) The vehicle features are sent to the fully connected layer for image classification.

[0086] Step (5): Joint learning of multiple loss functions

[0087] Step (5) is as follows: Finally, the improved Triplet loss is jointly learned with ID loss and Circle loss, so that the convolutional network can better extract similar features of vehicles of the same type, significantly expand the feature differences of vehicles of different types, enhance the model's ability to identify similar samples, and extract more distinguishing and higher discriminative global features.

[0088] ID loss refers to the input is a pair of vehicle images x1 and x2. These two vehicle images may have the same ID or different IDs. y indicates whether the two vehicle images belong to the same ID. If y = 1, it means that the two vehicles belong to the same label, which we call a positive sample; y = 0 means that the vehicles come from different IDs, which is a negative sample. The formula of the ID loss function is shown in formula (1). Where d x1,x2 By x1,x2 =||f x1 -f x2 || 2 It is obtained, which represents the Euclidean distance calculated for the input image after two feature extractions, that is, calculating their similarity. Where f(·) is the feature extractor, and mark in is the set threshold parameter.

[0089]

[0090] The circular loss aims to learn the class discrimination information, that is, to maximize the intra-class similarity and minimize the inter-class similarity. Assuming that there are K intra-class similarity scores related to x and L inter-class similarity scores, the circular loss objective function is as shown in the formula:

[0091]

[0092] Among them, S n and S P They are the inter-class similarity and intra-class similarity about the sample space x, respectively. The cosine similarity is used to calculate the intra-class and inter-class similarity scores; and are non-negative integers, and The weight, Δ n and Δ p are the inter-class and intra-class intervals respectively, and satisfy and γ is the scale factor.

[0093] The total loss function L of the vehicle re-identification network all It can be expressed as:

[0094] Lall =L ID +a′L t +b′L c

[0095] where a′ and b′ are the weights of triplet loss and circle loss respectively.

[0096] The triplet loss function, the circle loss function and the ID loss function are used to jointly learn the feature vectors, so that the convolutional network can better extract similar features of the same type of vehicles and significantly expand the feature differences of different types of vehicles.

[0097] Step (6): Test on the test set

[0098] Step (6) is as follows: randomly select two vehicle images from the test set and input them into the network model for prediction. Check the final output of the classifier. If the final output of the network is 1, it proves that the reference vehicle image and the image to be detected are from the same vehicle; if the final output of the network is 0, it proves that the reference vehicle image and the image to be detected are from different vehicles.

[0099] Step (7), image recognition

[0100] like Figure 3 As shown, the feature vector of the image to be identified is compared with the feature vector of the vehicle in the video image within the effective geographical area, and the vehicle target image with the highest similarity is searched using the similarity calculation formula, and the final result is output. Figure 4 exemplifies vehicle differences using the vehicle re-identification method according to an embodiment of the present invention.

Claims

1. A vehicle re-identification method based on multi-loss joint learning, characterized in that: The vehicle re-identification method comprises the following steps: S1, preprocessing the imported vehicle traffic image, cropping it to obtain vehicle images of the same size, and randomly erasing the cropped vehicle images to obtain vehicle image samples; Build a vehicle re-identification model, which includes a ResNet50 network model, a loss network, a BN layer, and a fully connected layer; S2, using the ResNet50 network model to extract key vehicle features in vehicle image samples; S3, maps the extracted key features of the vehicle to the feature space, and the distance of the vehicle image samples in the feature space is used to reflect the correlation between the images; the loss network is continuously trained and learned so that the vehicle samples with the same ID form clusters in the feature space; among them, by selecting vehicle samples of different styles with the same color in the randomly generated small batch samples as difficult triplets; the similarity metric is learned using the triplet loss to minimize the distance between the positive sample pairs and maximize the distance between the negative sample pairs; the corresponding batch difficult triplet loss L is calculated t ; S4, use the BN layer to normalize and reduce the latitudinal size of the generated global features to extract the vehicle features f i , the vehicle feature f i Send to the fully connected layer; S5, using the difficult triple loss function L t 、ID loss function L ID And the circle loss function L c To jointly learn the feature vectors, enhance the vehicle re-identification model's ability to identify similar samples, expand the feature differences of different types of vehicles, and extract more distinguishing and discriminative global features; the ID loss function is used to learn the feature vectors of vehicle image samples with the same or different IDs; the circle loss function is used to learn category discriminative information; S6, uses the test set to test the vehicle re-identification model.

2. The vehicle re-identification method based on multi-loss joint learning according to claim 1 is characterized in that: The vehicle re-identification method further comprises the following steps: S7, importing the vehicle image to be identified into the vehicle re-identification model, and extracting the corresponding feature vector; The feature vector of the vehicle image to be identified is compared with the feature vector of the vehicle in the video image within the effective geographical area. The vehicle target image with the highest similarity is searched using the similarity calculation formula, and the final recognition result is output.

3. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S1, the imported vehicle traffic image is preprocessed, vehicle images of the same size are cropped therefrom, and the cropped vehicle images are randomly erased to obtain vehicle image samples, and the process includes the following steps: S11, performs geometric transformation, horizontal flipping, and mirror source processing on vehicle images to make the data more diverse; S12, cropping and extracting the vehicle part from the entire image into the same size; S13, randomly erases the extracted vehicle image, and removes a rectangular block of a predefined range in the original image for different epochs.

4. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S2, the process of extracting key features of vehicles in vehicle image samples using the ResNet50 network model includes the following steps: S21, load the ResNet50 network model, and use the parameters of the network model as initial parameters for training; S22, using the trained ResNet50 network model to extract the key features of the vehicle.

5. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S3, generalized average pooling is used to aggregate the features generated by the ResNet50 network model into global features.

6. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S3, the Euclidean distance between vehicle image sample pairs is calculated according to the following formula: In the formula, d(x,y) refers to the Euclidean distance between the feature vectors of two sample pairs, x and y refer to the feature vectors extracted from the two sample pairs, num is the number of elements in the feature vector, k is the index of each element in the feature vector, and x is the distance between the feature vectors. k and k are the values ​​of the elements with index k in the feature vectors of the two samples respectively.

7. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S3, the batch difficulty triple loss L t The formula is as follows: L t =(d a,p +A-d a,n ) + In the formula, (·) + is the hinge function, A is an added margin, is the Euclidean distance interval between positive and negative samples, and d(a,n) is the anchor point X a and class X n The distance, d a,p is anchor point X a and class X p The distance satisfies d(a,n) <d(a,p)。 8. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S5, ID loss means that the input is a pair of vehicle images x1 and x2; y indicates whether the images of the two vehicles belong to the same ID. If y=1, it means that the two vehicles belong to the same label, which is called a positive sample; y=0 means that the vehicles come from different IDs, which is called a negative sample. The formula of the ID loss function is as follows: In the formula, represents the Euclidean distance calculated for the input image after two feature extractions, N is the total number of vehicle samples in each batch, y represents the label of the sample, the positive class is 1, and the negative class is 0. f(·) is the feature extractor, β represents the βth feature index, n represents the number of features, and margin is the threshold parameter set to measure the degree of distance difference between the feature vectors of positive and negative samples.

9. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: In step S5, the circle loss is used to learn the class discrimination information, that is, to maximize the intra-class similarity and minimize the inter-class similarity; Assuming that there are K intra-class similarity scores and L inter-class similarity scores associated with the sample space x, the formula of the circle loss objective function is as follows: Among them, S n and S P They are the inter-class similarity and intra-class similarity about the sample space x, respectively. The cosine similarity is used to calculate the intra-class and inter-class similarity scores; and are non-negative integers, and The weight, Δ n and Δ p are the inter-class and intra-class intervals respectively, and satisfy and γ is the scale factor.

10. The vehicle re-identification method based on multi-loss joint learning according to claim 1, characterized in that: The total loss function L of the vehicle re-identification network all It is expressed as: L all =L ID +a′L t +b′L c Among them, a′ and b′ are the weights of the hard triplet loss and circle loss, respectively.

Images