A Vehicle Re-identification Method and System Based on ViT Dual-Scale Feature Fusion Network

Abstract

This invention discloses a vehicle re-identification method and system based on a ViT dual-scale feature fusion network. The method includes the following steps: batch images are sequentially input into a linear projection layer and a ViT-Base network to obtain an image feature sequence; the image feature sequence is sequentially input into a multi-scale module and a multi-scale feature fusion module to obtain two fused features; the global image features and the two fused features are respectively input into a classifier to obtain the probability of each image category and calculate the cross-entropy loss function; the cross-entropy loss function and the triplet loss function are added together to obtain the overall loss function; the trained network extracts the global features and the two fused features, and after concatenation, the Euclidean distance between the query image features and the reference image features is calculated; the Euclidean distances are sorted from smallest to largest to obtain the vehicle re-identification result. This invention enhances the network's ability to extract fine-grained features, improves the coupling degree of features at different scales, and enhances the robustness of the features.

Description

Technical Field

[0001] This invention relates to the field of vehicle re-identification technology, specifically to a vehicle re-identification method and system based on a ViT dual-scale feature fusion network. Background Technology

[0002] Vehicle re-identification is a vehicle search method that involves inputting an image of a query vehicle and searching a database for vehicles with the same identity. Vehicle re-identification is commonly used in large-scale urban video surveillance networks, helping traffic management departments quickly, accurately, and conveniently discover, locate, and track target vehicles within massive amounts of traffic monitoring data. Therefore, research on vehicle re-identification technology is of great significance.

[0003] Existing vehicle re-identification methods are mainly divided into two categories. One is a vehicle re-identification technology that utilizes the uniqueness of license plates by first identifying the license plate and then matching it. The other is a vehicle re-identification technology that utilizes the image feature extraction capabilities of convolutional neural networks by first using convolutional neural networks to extract vehicle feature vectors, then calculating the distance between different vehicle feature vectors, and finally returning the image with the smallest distance.

[0004] While the above methods improve the performance of vehicle re-identification to some extent, they suffer from problems such as license plate occlusion and loss of fine-grained features, which reduce their practicality and application value. Summary of the Invention

[0005] To overcome the defects and shortcomings of existing technologies, this invention provides a vehicle re-identification method and system based on a ViT dual-scale feature fusion network. This invention uses a ViT-Base network to extract image features, avoiding the problem of fine-grained feature loss caused by multiple downsampling in convolutional networks; it uses a multi-scale module to increase the feature scale, enhancing the network's ability to extract fine-grained features; and it fuses multi-scale features based on the multi-scale feature fusion module, improving the coupling degree of features at different scales and enhancing the robustness of the features.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a vehicle re-identification method based on a ViT dual-scale feature fusion network, comprising the following steps:

[0008] Construct a linear projection layer, which is used to divide the image into blocks;

[0009] A ViT-Base network is constructed, which is used to extract single-scale vehicle features from the segmented image.

[0010] A multi-scale module is constructed, which is used to expand the single-scale features of the vehicle into multi-scale features, and output the original scale features and double-scale features respectively.

[0011] Construct a multi-scale feature fusion module;

[0012] Construct a classifier for vehicle classification;

[0013] Construct batch images and input them sequentially into a linear projection layer and a ViT-Base network to obtain image feature sequences;

[0014] The image feature sequence is sequentially input into the multi-scale module and the multi-scale feature fusion module to obtain the original scale fusion feature and the double scale fusion feature;

[0015] The first image feature vector in the image feature sequence is selected as the global image feature. The global image feature, the original scale fusion feature and the double scale fusion feature are respectively input into the classifier to obtain the probability of each image category and calculate the cross-entropy loss function.

[0016] The triplet loss function is calculated based on global image features, original scale fusion features, and double scale fusion features, respectively.

[0017] The overall loss function is obtained by adding the cross-entropy loss function and the triplet loss function.

[0018] The network weights are updated based on the gradient of the overall loss function, and the current network weights are saved after training is completed.

[0019] A query image and multiple reference images are input into the trained network. The trained network outputs global image features, original scale fusion features, and double scale fusion features. The global image features, original scale fusion features, and double scale fusion features are concatenated and the Euclidean distance between the query image features and the reference image features is calculated. The reference images sorted based on the Euclidean distance are the vehicle re-identification results.

[0020] As a preferred technical solution, the linear projection layer is used to divide the image into blocks, specifically including:

[0021] The linear projection layer uses a single convolutional layer, and the output, after flattening, is a feature sequence x of image patches of length N. p Specifically, it is expressed as:

[0022] x p =Flatten(F proj (x))=[x 1 ;x 2 ;...;x N ]

[0023]

[0024] in, The function indicates rounding down, while Flatten(·) is a function that flattens the first two dimensions of an array into one dimension in row order.

[0025] As a preferred technical solution, the ViT-Base network is used to extract single-scale vehicle features from the segmented image, specifically including:

[0026] The input z0 of the ViT-Base network is the vector x output by the category image patch. 0 and the image patch feature sequence x output by the linear projection layer p After concatenation, it is combined with the learnable positional code E pos The summation yields the image feature sequence z. L Specifically, it is expressed as:

[0027]

[0028] z0 = x patches +E pos ;

[0029]

[0030] Where, x 0 It is a vector with an initial value of 0. Indicates the concatenation operation, z L This represents the output of the ViT-Base network.

[0031] As a preferred technical solution, the multi-scale module is used to expand vehicle single-scale features into multi-scale features, specifically including:

[0032] The multi-scale module uses transposed convolution to expand the multi-scale, and the expanded multi-scale is the original scale and the scale after upsampling by 2 times.

[0033] Obtain the image feature vector output by the ViT-Base network, delete the first image feature vector, and input the remaining image feature vector into the multi-scale module to output the original scale feature and the double scale feature respectively.

[0034] The original scale features are represented as follows:

[0035]

[0036] Among them, f 1× LN represents the original scale features, Conv represents convolution, and Reshape represents the dimension-changing operation. This represents the remaining image feature vectors output by the ViT-Base network after deleting the first image feature vector;

[0037] And / or,

[0038] The double-scale feature is represented as:

[0039]

[0040] Among them, f 2× represents a double-scale feature, and ConvTranspose represents the upsampling of features by the transposed convolution.

[0041] As a preferred technical solution, the construction of the multi-scale feature fusion module specifically includes:

[0042] The multi-scale feature fusion module consists of two cross-attention layers. The first image feature vector from the image feature sequence is input into the multi-scale feature fusion module along with the original scale fusion feature, and the original scale fusion feature is output. The first image feature vector from the image feature sequence is input into the multi-scale feature fusion module along with the double-scale fusion feature, and the double-scale fusion feature is output. Specifically, it is expressed as follows:

[0043]

[0044]

[0045]

[0046]

[0047] Among them, W q W k and W v These represent the weights of the fully connected layers for the query sequence Q, key sequence K, and value sequence V, respectively. Flatten(·) flattens the first two dimensions of the array into one dimension. Indicates the concatenation operation, K T This represents the transpose of K, and D represents the number of channels. n represents the first image feature vector in the image feature sequence. H This represents the number of MSA heads in the ViT-Base network, and Softmax(·) represents the normalized exponential function. When i = 1, f fi× The original scale fusion feature f represents f1× When i = 2, f fi× f represents the double-scale fusion feature f2× .

[0048] As a preferred technical solution, the cross-entropy loss function is specifically expressed as follows:

[0049]

[0050]

[0051] p = Softmax(fE) C ),

[0052] Among them, E C b represents the weights of the classifier. s y′ represents the number of images in each batch, M represents the total number of vehicle training set categories, and y′ represents the number of images in each batch. ic p represents the probability of the i-th image in each batch with respect to the c-th class after label smoothing and regularization. ic y represents the predicted probability of the i-th image in each batch with respect to the c-th class. ic Let f represent the true probability of the i-th image in each batch with respect to the c-th class, and let ε represent a constant, which is determined when f takes a certain value. When, the calculated L ce For L ce_0 When f takes f f1× When, the calculated L ce For L ce_1× When f takes f f2× When, the calculated L ce For L ce_2× .

[0053] As a preferred technical solution, the triplet loss function is specifically expressed as follows:

[0054]

[0055] Among them, P M This represents the number of vehicle categories in each batch, and K represents the number of images for each vehicle category in each batch. This represents the features of the a-th image of the i-th vehicle class, where the superscripts i and j represent the vehicle class indices, and the subscripts a, p, and n represent the anchor point, positive sample point, and negative sample point of the triple, respectively. m represents the minimum margin between positive and negative sample pairs, and Dist(·) represents the Euclidean distance. When f takes... When, the calculated L tr For L tr_0 When f takes f f1× When, the calculated L tr For L tr_1× When f takes f f2× When, the calculated L tr For L tr_2× .

[0056] As a preferred technical solution, the Euclidean distance between the query image features and the reference image features is calculated by concatenating the global image features, the original scale fusion features, and the double scale fusion features. Specifically, it is expressed as follows:

[0057]

[0058] D m ={d ij}

[0059]

[0060] In the formula, d ij Represents the distance between the i-th query image and the j-th reference image, ||·|| 2 This indicates finding the L2 norm of a vector. Indicates the concatenation operation, f i N represents the features of the images in the query dataset. Q f represents the number of images in the query dataset. j N represents the features of the images in the reference dataset. G This indicates the number of images in the reference dataset.

[0061] As a preferred technical solution, the method further includes a data augmentation step, in which the batch images are augmented and then input into a linear projection layer. The data augmentation includes adjusting image size, zero-value filling, random cropping, random brightness variation, random rotation, normalization, and random erasure.

[0062] The present invention also provides a vehicle re-identification system based on a ViT dual-scale feature fusion network, comprising: a linear projection layer construction unit, a ViT-Base network construction unit, a multi-scale module construction unit, a multi-scale feature fusion module construction unit, a classifier construction unit, a batch image construction unit, an image feature sequence generation unit, a loss function construction unit, a network training and update unit, and a vehicle re-identification unit;

[0063] The linear projection layer construction unit is used to construct a linear projection layer, which is used to divide the image into blocks;

[0064] The ViT-Base network building unit is used to build the ViT-Base network, which is used to extract single-scale vehicle features from the segmented image.

[0065] The multi-scale module construction unit is used to construct a multi-scale module, which is used to expand the single-scale features of the vehicle into multi-scale features, and output the original scale features and double-scale features respectively.

[0066] The multi-scale feature fusion module construction unit is used to construct the multi-scale feature fusion module;

[0067] The classifier construction unit is used to construct a classifier, which is used for vehicle classification.

[0068] The batch image construction unit is used to construct batch images;

[0069] The image feature sequence generation unit is used to sequentially input batch images into a linear projection layer and a ViT-Base network to obtain image feature sequences;

[0070] The image feature sequence is sequentially input into the multi-scale module and the multi-scale feature fusion module to obtain the original scale fusion feature and the double scale fusion feature;

[0071] The loss function construction unit is used to construct the overall network loss function. It selects the first image feature vector in the image feature sequence as the global image feature, and inputs the global image feature, the original scale fusion feature and the double scale fusion feature into the classifier to obtain the probability of each image category and calculate the cross-entropy loss function.

[0072] The triplet loss function is calculated based on global image features, original scale fusion features, and double scale fusion features, respectively.

[0073] The overall loss function is obtained by adding the cross-entropy loss function and the triplet loss function.

[0074] The network training update unit is used to update the network weights according to the gradient value of the overall loss function, and saves the current network weights after training is completed;

[0075] The vehicle re-identification unit is used to output the vehicle re-identification result. It inputs a query image and multiple reference images into the trained network. The trained network outputs global image features, original scale fusion features and double scale fusion features. After concatenating the global image features, original scale fusion features and double scale fusion features, it calculates the Euclidean distance between the query image features and the reference image features. The reference images sorted based on the Euclidean distance are the vehicle re-identification results.

[0076] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0077] (1) This invention uses the ViT-Base network to extract global features of the image, which avoids the problem of loss of fine-grained features caused by multiple downsampling in the convolutional network. It preserves fine-grained features well, and the network extracts features in the global scope, so that fine-grained features that are far apart can also establish a connection. The extracted fine-grained features are richer and the robustness of the features is enhanced.

[0078] (2) The present invention adds scale to the global image features extracted by the ViT-Base network. The high-resolution scale features enhance the network’s ability to extract fine-grained features, the low-resolution scale features enhance the network’s ability to extract semantic features, and the use of multi-scale features enhances the robustness of the features.

[0079] (3) This invention adds feature scale to the image feature sequence output by the ViT-Base network without using intermediate layer features and without changing the ViT-Base network structure. Therefore, it can use the existing pre-trained weights and achieve the technical effect of achieving better multi-scale feature extraction capability with a simple structure.

[0080] (4) The present invention first transforms the features output by the ViT-Base network from a sequence to an image structure, and then uses transposed convolution to increase the feature scale, which further enhances the network’s ability to learn spatial features and enriches the spatial information of the ViT-Base network’s output features.

[0081] (5) This invention proposes a method for fusing multi-scale features using a cross-attention mechanism. By using the globally identical features of the image output by the ViT-Base network as the query sequence to focus on extracting feature value sequences at different scales, information interaction between features at different scales is realized, the coupling degree of features at different scales is improved, and the robustness of features is enhanced. Attached Figure Description

[0082] Figure 1 This is a flowchart illustrating the training process of the vehicle re-identification method based on the ViT dual-scale feature fusion network of this invention.

[0083] Figure 2 This is a test flowchart of the vehicle re-identification method based on the ViT dual-scale feature fusion network of the present invention;

[0084] Figure 3 This is a schematic diagram of the overall model structure of the vehicle re-identification method based on the ViT dual-scale feature fusion network of the present invention;

[0085] Figure 4 This is a schematic diagram of the multi-scale module MS structure of the vehicle re-identification method based on the ViT dual-scale feature fusion network of the present invention;

[0086] Figure 5 This is a schematic diagram of the cross-attention layer structure of the vehicle re-identification method based on the ViT dual-scale feature fusion network of the present invention;

[0087] Figure 6 This is a schematic diagram of the CMC curve of the present invention on the VeRi-776 dataset;

[0088] Figure 7This is a schematic diagram showing the correct samples among the top 10 results returned by some vehicles in the VeRi-776 query set according to the present invention. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0090] Example 1

[0091] In this embodiment, the Veri-776 dataset is used. The Veri-776 dataset was collected under the supervision of Liu Xinchen from Beijing University of Posts and Telecommunications. The images in the dataset are from urban surveillance videos; 20 cameras cover an area of ​​one square kilometer, with a monitoring duration of 24 hours. This dataset contains 776 vehicles and a total of 49,357 images, including 37,778 training images, 1,678 query images, and 11,579 test images. Detailed information about the dataset is shown in Table 1. This embodiment is mainly based on the PyTorch 1.7.1 framework. The graphics card used in the experiment is a Tesla V100, the system is Ubuntu 18.04, CUDA version 10.1, and cuDNN version 7.0.1.

[0092] Table 1. Detailed information on the vehicle re-identification dataset.

[0093] Dataset Total images / vehicles Training images / number of vehicles Query image / vehicle count Test images / number of vehicles Number of cameras Number of viewpoints Veri-776 49357 / 776 37778 / 579 1678 / 200 11579 / 200 20 6

[0094] like Figure 1 , Figure 2 As shown, this embodiment provides a vehicle re-identification method based on a ViT dual-scale feature fusion network, including the following steps:

[0095] S1. Construct a linear projection layer F for dividing the image into blocks. proj ;

[0096] In this embodiment, the linear projection layer F proj A single convolutional layer is used, with 3 input channels (C=3), 768 output channels (D=768), a block width and height (P=16), a kernel size (k=P=16), and a stride (s=P=16). The input to this layer is any image. Where H, W, and 3 represent the height, width, and number of channels of the image, respectively, and the output, after flattening, is a feature sequence of image patches of length N. i represents the index position, such as Figure 3 As shown, the linear projection layer F is obtained. proj The input and output are defined by the following formulas:

[0097] Fproj =Conv(C=3,D=768,k=16,s=16)

[0098]

[0099] x p =Flatten(F proj (x))=[x 1 ;x 2 ;...;x N ]

[0100] in, The function indicates rounding down, while Flatten(·) is a function that flattens the first two dimensions of an array into one dimension in row order.

[0101] S2. Construct the ViT network B for single-scale feature extraction of vehicles;

[0102] In this embodiment, the ViT network B adopts the ViT-Base network, where the number of ViT encoders L = 12 and the number of MSA (Multihead Self-Attention) heads is n. H =12, the hidden channel ratio of the MLP (Multilayer Perceptron) is 4, the normalization layer is LN (Layer Normalization), and the activation function is GELU (Gaussian Error Linear Unit). The input z0 of network B is x, which is the output of the class image block (class token). 0 and F proj Output x p After concatenation, it is combined with the learnable positional encoding (weight) E pos The summation yields the image feature sequence z. L ,like Figure 3 As shown, the input and output of the ViT-Base network are obtained. The vector containing numbers (0, 1, ..., 9) in the input of the figure is E. pos The input and output of the ViT-Base network are as follows:

[0103]

[0104]

[0105]

[0106] Where, x 0 It is a vector initialized to 0, used to extract global features from the image; the superscript 0 indicates x. 0 Located at xpatches The position with index 0 in the middle, This indicates a splicing operation. This represents the output of network B, where the superscript indicates the index position and the subscript L indicates z. L It is the output of the Lth encoder.

[0107] S3. Construct a multi-scale module MS that expands single-scale features into multi-scale features;

[0108] In this embodiment, transposed convolution is used to expand the multi-scale, resulting in a multi-scale that is twice the original scale and twice the scale after upsampling. For example... Figure 3 As shown, in the input and output of the multi-scale module MS, the input of MS is the network B after removing... The output after The output is the original scale feature f 1× and double-scale features f 2× . First, a reshape operation is performed to transform the one-dimensional sequence structure into a two-dimensional image structure. Then, through a combination of 1×1 convolution, LN, 3×3 convolution, and LN, the original scale features f are further learned. 1× ,like Figure 4 As shown, in the left branch of the MS structure diagram, the implementation process is represented as follows:

[0109]

[0110] The double-scale feature f is obtained by upsampling features using a transposed convolutional layer, followed by a combination of 1×1 convolution, LN, 3×3 convolution, and LN to learn the double-scale feature f. 2× ,like Figure 4 As shown, in the right branch of the MS structure diagram, the implementation process is represented as follows:

[0111]

[0112] S4. Construct the multi-scale feature fusion module MF;

[0113] In this embodiment, the multi-scale feature fusion module (MF) consists of two cross-attention layers, such as... Figure 3 As shown, in the cross-attention layer, when the input of this layer (such as...) Figure 3 The cross-attention layer on the left is and f 1× At that time, the output is the original scale fused feature f f1× When the input of this layer (such as...) Figure 3 The right-hand cross-attention layer is and f2× At that time, the output is the double-scale fused feature f f2× .like Figure 5 As shown, in the cross-attention layer structure, the query sequence Q(Query) is composed of... After passing through a fully connected layer, the key sequence K(Key) is obtained from... and f 1× (or f) 2× After concatenation, the result is obtained through a fully connected layer, and the value sequence V(Value) is obtained from... and f 1× (or f) 2× After concatenation, the data is passed through a fully connected layer to obtain the fused feature f. The Q, K, and V values ​​are then processed through a self-attention mechanism (a subsequent operation within the cross-attention layer). f1× (or f) f2× The specific formula is as follows:

[0114]

[0115]

[0116]

[0117]

[0118] Among them, W q W k and W v These represent the weights of the fully connected layers Q, K, and V, respectively. Flatten(·) flattens the first two dimensions of the array into one dimension. Indicates the concatenation operation, K T Let K be the transpose, D be the number of output channels in step S1, and n be the number of output channels. H This represents the number of MSA heads in network B during step S2, and Softmax(·) represents the normalized exponential function. When i = 1, f fi× It is the original scale fusion feature f f1× When i = 2, f fi× It is a two-scale fusion feature f f2× .

[0119] S5. Construct a classifier C for vehicle classification;

[0120] In this embodiment, the classifier C for vehicle classification is a single fully connected layer with D = 768 input channels and 579 output channels, which is the number of vehicles in the training set.

[0121] S6. Construct a batch of images;

[0122] P is randomly selected in each training batch MFor each class of vehicles, K images are randomly selected, for a total of B images. s =P M ×K images are then processed, and real-time data augmentation is performed. In this embodiment, P M Choose 16, K is 4, B s Take 64.

[0123] In this embodiment, the real-time data augmentation techniques during the training phase include image resizing, zero-filling, random cropping, random brightness variation, random rotation, normalization, and random erasure.

[0124] In this embodiment, the image resizing step scales the image to a fixed size using bilinear interpolation; the zero-filling step fills the four sides of the image with pixels of gray level 0, filling each side with a fixed number of pixels; the random cropping step uses a fixed-size cropping box to randomly crop within a specified range of the image; the random brightness variation step adjusts the brightness of the vehicle image using a random factor; the random rotation step rotates the image at random angles within a set angle threshold, using zero-fill for rotation; the normalization step subtracts a specified average value from each of the red, green, and blue channels of the image, and then divides the result by a specified standard deviation; and the random erasure step sets up erasure blocks to randomly erase various locations in the image.

[0125] First, the image size was uniformly adjusted to 256×256. Then, each of the four sides of the image was filled with 10 pixels with a gray level of 0, resulting in an image size of 276×276. Next, the image was randomly cropped using a 256×256 cropping frame. Then, a brightness difference factor of 0.2 was used to randomly change the brightness of the image. Then, an angle difference factor of 20 degrees was used to randomly rotate the image. Then, the red, green, and blue channels of the image were normalized separately, using parameters with average values ​​of 0.485, 0.456, and 0.406 and standard deviations of 0.229, 0.224, and 0.225, respectively. Finally, considering the actual situation where vehicles might be obscured, random erasure was used with an erasure probability of 0.5. The erasure areas appeared at various locations in the image, with the ratio of the erasure area to the image ranging from 0.02 to 0.33 and the aspect ratio of the erasure area ranging from 0.3 to 3.3. The erasure areas were filled with pixels with a gray level of 0. These data augmentation techniques reduce the overfitting of the model and enhance its generalization ability;

[0126] S7. Input the batch of images sequentially into the linear projection layer F. proj Image feature sequences are obtained from the ViT network B.

[0127] In this embodiment, the number of encoders L in the ViT network B is 12, the number of blocks N after image segmentation is 256, and the feature length D of each image block is 768.

[0128] S8. Transform the image feature sequence z L The multi-scale module MS and the multi-scale feature fusion module MF are input sequentially to obtain the original scale fused features. and double-scale fusion features

[0129] S9. Global features of the image (Global features of the image) It is z L The first image feature vector in the image and two fused features f f1× and f f2× The data are input into classifier C to obtain the probabilities of each class and the cross-entropy loss function L is calculated. ce_0 L ce_1× and L ce_2× Cross-entropy loss function L ce The specific formula is as follows:

[0130]

[0131]

[0132]

[0133] Among them, E C B represents the weights of classifier C. s y′ represents the number of images in each batch, M represents the total number of vehicle training set categories, and y′ represents the number of images in each batch. ic p represents the probability of the i-th image in each batch with respect to the c-th class after label smoothing and regularization. ic y represents the predicted probability of the i-th image in each batch with respect to the c-th class. ic Let f represent the true probability of the i-th image in each batch with respect to the c-th class, and ε represent a constant. When f takes... When, the calculated L ce For L ce_0 When f takes f f1× When, the calculated L ce For L ce_1× When f takes f f2× When, the calculated L ce For L ce_2× In this embodiment, the number of images in each batch is B. s The total number of categories in the vehicle training set is set to 64, M is set to 579, and ε is set to 0.1.

[0134] S10. Using global image features and two fusion features f f1× and f f2×Calculate the triplet loss function L separately. tr_0 L tr_1× and L tr_2× Batch triplet loss function L tr The specific formula is as follows:

[0135]

[0136] Among them, P M This represents the number of vehicle categories in each batch, and K represents the number of images for each vehicle category in each batch. Let f represent the features of the a-th image of the i-th vehicle class. The superscripts i and j represent the vehicle class indices, and the subscripts a, p, and n represent the anchor point, positive sample point, and negative sample point of the triple, respectively. m represents the minimum margin between positive and negative sample pairs, and Dist(·) represents the Euclidean distance. When f takes... When, the calculated L tr For L tr_0 When f takes f f1× When, the calculated L tr For L tr_1× When f takes f f2× When, the calculated L ta For L ta_2× In this embodiment, P M Choose 16, and K should be 4.

[0137] S11, the cross-entropy loss function L ce With triplet loss function L tr Adding them together, we get the overall loss function L, the specific formula of which is as follows:

[0138] L = L ce_0 +L ce_1× +L ce_2× +L tr_0 +L tr_1× +L tr_2×

[0139] S12. During the training phase, the network weights are updated based on the gradient value of the loss function L. After training is completed, the current network weights are saved.

[0140] In this embodiment, the training epoch threshold is set to e = 120, the SGD optimizer is used as the training optimizer, the initial learning rate is lr = 0.008, which decreases linearly to 0, and the weight decay is wd = 10. -4 In the initial training phase, the network weights trained on the ImageNet dataset are initialized.

[0141] S13. During the testing phase, a query image and multiple reference images are input into the trained network to obtain global image features. and two fusion features f f1× and f f2× After stitching, the Euclidean distance *d* between the features of the query image and the features of the reference image is calculated. The images with distances *d* are then sorted from smallest to largest. The sorted reference images are the query (vehicle re-identification) results. The higher the ranking of the reference image, the greater the probability that it is the same vehicle as the query image. The specific formula is as follows:

[0142]

[0143] D m ={d ij}

[0144] d ij =Dist(f i ,f j )=||f i -f j || 2 i∈{0,1,...,N Q},j∈{0,1,...,N G}

[0145] In the formula, d ij Represents the distance between the i-th query image and the j-th reference image, ||·|| 2 This indicates finding the L2 norm of a vector. Indicates the concatenation operation, f i N represents the features of the images in the query dataset. Q f represents the number of images in the query dataset. j N represents the features of the images in the reference dataset. G This represents the number of images in the reference dataset. In this embodiment, N... Q For 1678, N G The value is 11579.

[0146] The distance matrix can be calculated from the features of the query image set and the reference image set. According to the distance matrix D m The related label calculation model calculates the cumulative match characteristic curve (CMC) and mean average precision (mAP) of the query image set and the reference image set. For a query image, Top k indicates whether the first k results contain images with the same identity; if so, Top k equals 1, otherwise it is 0. The CMC is the mean curve of the Top k values ​​for all query images. The mAP calculation formula is as follows:

[0147]

[0148]

[0149] Where, N gt P(k) represents the number of correct samples that match the identity of the query image, P(k) represents the accuracy of sorting the k-th image, and gt(k) represents an indicator function that returns 1 if the identity of the k-th result matches the identity of the query image, and 0 otherwise.

[0150] This embodiment uses the ViT network to extract image features, avoiding the problem of fine-grained feature loss caused by multiple downsampling in convolutional networks; it uses multi-scale features to enhance the fine-grained and semantic information of the features; it increases the feature scale of the ViT network output without using intermediate layer features or changing the ViT network structure, ensuring the use of pre-trained weights; it uses transposed convolution to increase the feature scale, strengthening the network's ability to extract spatial information from the image; and it proposes to use a cross-attention mechanism to fuse multi-scale features, improving the coupling degree of features at different scales and enhancing the robustness of the features.

[0151] This embodiment was tested on the Veri-776 dataset and compared with the ResNet50 convolutional neural network. CMC and mAP were used to evaluate model performance. Higher CMC and mAP indicate better model performance and a better vehicle re-identification algorithm. Test results are shown in Table 2 and... Figure 6 As shown.

[0152] Table 2 Comparison of Cumulative Matching Curve (CMC) and Mean Accuracy (mAP) for Different Methods

[0153] Methods and Evaluation Indicators mAP(%) CMC Top 1 (%) CMC Top 5 (%) ResNet50 76.4 95.2 98.4 This embodiment 79.8 96.8 98.6

[0154] As can be seen from the experimental results in Table 2, this embodiment outperforms the ResNet50 method in both mAP and CMC Top 1, demonstrating better re-identification performance and verifying the effectiveness of the method in this embodiment. Figure 6 As shown, the CMC curves on the VeRi-776 dataset are obtained from... Figure 6 The CMC results show that k=10 is the inflection point of the curve, and 99.3% of queries get the correct match in the first 10 results. The larger the k is, the higher the accuracy and the smaller the slope, indicating that the model can better distinguish between simple and difficult samples.

[0155] like Figure 7As shown, this diagram displays some of the query images and the top 10 correctly matched images returned by the model. It can be seen that vehicles in reference images with similar perspectives to the query image are more likely to rank higher in the returned results, indicating that the model has learned fine-grained features well. Fine-grained features (such as the location of the vehicle inspection sticker) are crucial for distinguishing different vehicles from the same perspective. Furthermore, the correctly matched images also include vehicles with different perspectives, demonstrating that the model has learned semantic features well and can return matching vehicles based on common semantic features (such as sunroof location and vehicle color).

[0156] Example 2

[0157] This embodiment provides a vehicle re-identification system based on a ViT dual-scale feature fusion network, including: a linear projection layer construction unit, a ViT-Base network construction unit, a multi-scale module construction unit, a multi-scale feature fusion module construction unit, a classifier construction unit, a batch image construction unit, an image feature sequence generation unit, a loss function construction unit, a network training and update unit, and a vehicle re-identification unit;

[0158] In this embodiment, the linear projection layer construction unit is used to construct the linear projection layer, which is used to divide the image into blocks;

[0159] In this embodiment, the ViT-Base network building unit is used to build the ViT-Base network, which is used to extract single-scale vehicle features from the segmented image.

[0160] In this embodiment, the multi-scale module construction unit is used to construct a multi-scale module, which is used to expand the single-scale features of the vehicle into multi-scale features, and output the original scale features and double-scale features respectively.

[0161] In this embodiment, the multi-scale feature fusion module construction unit is used to construct the multi-scale feature fusion module;

[0162] In this embodiment, the classifier building unit is used to build a classifier for vehicle classification;

[0163] In this embodiment, the batch image construction unit is used to construct batch images;

[0164] In this embodiment, the image feature sequence generation unit is used to sequentially input batch images into a linear projection layer and a ViT-Base network to obtain image feature sequences;

[0165] The image feature sequence is sequentially input into the multi-scale module and the multi-scale feature fusion module to obtain the original scale fusion feature and the double scale fusion feature.

[0166] In this embodiment, the loss function construction unit is used to construct the overall network loss function. The first image feature vector in the image feature sequence is selected as the global image feature. The global image feature, the original scale fusion feature, and the double scale fusion feature are respectively input into the classifier to obtain the probability of each image category and calculate the cross-entropy loss function.

[0167] The triplet loss function is calculated based on global image features, original scale fusion features, and double scale fusion features, respectively.

[0168] The overall loss function is obtained by adding the cross-entropy loss function and the triplet loss function.

[0169] In this embodiment, the network training update unit is used to update the network weights according to the gradient value of the overall loss function, and save the current network weights after training is completed;

[0170] In this embodiment, the vehicle re-identification unit is used to output the vehicle re-identification result. A query image and multiple reference images are input into the trained network. The trained network outputs global image features, original scale fusion features, and double scale fusion features. The global image features, original scale fusion features, and double scale fusion features are concatenated and the Euclidean distance between the query image features and the reference image features is calculated. The reference images sorted based on the Euclidean distance are the vehicle re-identification results.

[0171] In this embodiment, a data enhancement unit is also provided. The data enhancement unit performs data enhancement on the batch images and then inputs them into the linear projection layer. The data enhancement in this embodiment includes adjusting the image size, zero-value filling, random cropping, random brightness variation, random rotation, normalization, and random erasure.

[0172] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A vehicle re-identification method based on a ViT dual-scale feature fusion network, characterized in that, Includes the following steps: Construct a linear projection layer, which is used to divide the image into blocks; A ViT-Base network is constructed, which is used to extract single-scale vehicle features from the segmented image. A multi-scale module is constructed, which is used to expand the single-scale features of the vehicle into multi-scale features, and output the original scale features and double-scale features respectively. Construct a multi-scale feature fusion module; Construct a classifier for vehicle classification; Construct batch images and input them sequentially into a linear projection layer and a ViT-Base network to obtain image feature sequences; The image feature sequence is sequentially input into the multi-scale module and the multi-scale feature fusion module to obtain the original scale fusion feature and the double scale fusion feature; The first image feature vector in the image feature sequence is selected as the global image feature. The global image feature, the original scale fusion feature and the double scale fusion feature are respectively input into the classifier to obtain the probability of each image category and calculate the cross-entropy loss function. The triplet loss function is calculated based on global image features, original scale fusion features, and double scale fusion features, respectively. The overall loss function is obtained by adding the cross-entropy loss function and the triplet loss function. The network weights are updated based on the gradient of the overall loss function, and the current network weights are saved after training is completed. A query image and multiple reference images are input into the trained network. The trained network outputs global image features, original scale fusion features, and double scale fusion features. The global image features, original scale fusion features, and double scale fusion features are concatenated and the Euclidean distance between the query image features and the reference image features is calculated. The reference images sorted based on the Euclidean distance are the vehicle re-identification results.

2. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The linear projection layer is used to divide the image into blocks, specifically including: The linear projection layer uses a single convolutional layer, and the output, after flattening, is a feature sequence x of image patches of length N. p Specifically, it is expressed as: x p =Flatten(F proj (x))=[x 1 ;x 2 ;...;x N ] in, F represents rounding down, Flatten(·) is a function that flattens the first two dimensions of an array into one dimension in row-major order. proj denoted as a linear projection layer, x represents the input image of the linear projection layer, N represents the length of the image patch feature sequence, H represents the height of the image, P represents the width and height of the patch, and W represents the width of the image.

3. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The ViT-Base network is used to extract single-scale vehicle features from the segmented image, specifically including: The input z0 of the ViT-Base network is the vector x output by the category image patch. 0 and the image patch feature sequence x output by the linear projection layer p After concatenation, it is combined with the learnable positional code E pos The summation yields the image feature sequence z. L Specifically, it is expressed as: z0 = x patches +E pos ; Where, x 0 It is a vector with an initial value of 0. Indicates the concatenation operation, z L This represents the output of the ViT-Base network.

4. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The multi-scale module is used to expand vehicle single-scale features into multi-scale features, specifically including: The multi-scale module uses transposed convolution to expand the multi-scale, and the expanded multi-scale is the original scale and the scale after upsampling by 2 times. Obtain the image feature vector output by the ViT-Base network, delete the first image feature vector, and input the remaining image feature vector into the multi-scale module to output the original scale feature and the double scale feature respectively. The original scale features are represented as follows: Among them, f 1× LN represents the original scale features, Conv represents convolution, and Reshape represents the dimension-changing operation. This represents the remaining image feature vectors output by the ViT-Base network after deleting the first image feature vector; And / or, The double-scale feature is represented as: Among them, f 2× represents a double-scale feature, and ConvTranspose represents the upsampling of features by the transposed convolution.

5. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The construction of the multi-scale feature fusion module specifically includes: The multi-scale feature fusion module consists of two cross-attention layers. The first image feature vector from the image feature sequence is input into the multi-scale feature fusion module along with the original scale fusion feature, and the original scale fusion feature is output. The first image feature vector from the image feature sequence is input into the multi-scale feature fusion module along with the double-scale fusion feature, and the double-scale fusion feature is output. Specifically, it is expressed as follows: Among them, W q W k and W v These represent the weights of the fully connected layers for the query sequence Q, key sequence K, and value sequence V, respectively. Flatten(·) flattens the first two dimensions of the array into one dimension. Indicates the concatenation operation, K T This represents the transpose of K, and D represents the number of channels. n represents the first image feature vector in the image feature sequence. H This represents the number of MSA heads in the ViT-Base network, and Softmax(·) represents the normalized exponential function. When i = 1, f fi× The original scale fusion feature f represents f1× When i = 2, f fi× f represents the double-scale fusion feature f2× .

6. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The cross-entropy loss function is specifically expressed as follows: Among them, E C B represents the weight of the classifier. s y′ represents the number of images in each batch, M represents the total number of vehicle training set categories, and y′ represents the number of images in each batch. ic p represents the probability of the i-th image in each batch with respect to the c-th class after label smoothing and regularization. ic y represents the predicted probability of the i-th image in each batch with respect to the c-th class. ic Let f represent the true probability of the i-th image in each batch with respect to the c-th class, and let ε represent a constant, which is determined when f takes a certain value. When, the calculated L ce For L ce_0 When f takes f f1× When, the calculated L ce For L ce_1× When f takes f f2× When, the calculated L ce For L ce_2× L ce Let f represent the cross-entropy loss function. f1× f represents the original scale fusion feature. f2× This indicates a two-scale fusion feature. It represents the global features of the image.

7. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The triplet loss function is specifically expressed as follows: Among them, P M This represents the number of vehicle categories in each batch, and K represents the number of images for each vehicle category in each batch. This represents the features of the a-th image of the i-th vehicle class, where the superscripts i and j represent the vehicle class indices, and the subscripts a, p, and n represent the anchor point, positive sample point, and negative sample point of the triple, respectively. m represents the minimum margin between positive and negative sample pairs, and Dist(·) represents the Euclidean distance. When f takes... When, the calculated L tr For L tr_0 When f takes f f1× When, the calculated L tr For L tr_1× When f takes f f2× When, the calculated L tr For L tr_2× .

8. The vehicle re-identification method based on ViT dual-scale feature fusion network according to claim 1, characterized in that, The Euclidean distance between the query image features and the reference image features is calculated by concatenating the global features, original-scale fused features, and double-scale fused features. Specifically, it is expressed as follows: D m ={d ij } d ij =Dist(f i ,f j )=||f i -f j || 2 ,i∈{0,1,...,N Q },j∈{0,1,...,N G } In the formula, f f1× f represents the original scale fusion feature. f2× This indicates a two-scale fusion feature. d represents the global features of the image. ij Represents the distance between the i-th query image and the j-th reference image, ‖·‖ 2 This indicates finding the L2 norm of a vector. Indicates the concatenation operation, f i N represents the features of the images in the query dataset. Q f represents the number of images in the query dataset. j N represents the features of the images in the reference dataset. G This indicates the number of images in the reference dataset.

9. The vehicle re-identification method based on ViT dual-scale feature fusion network according to any one of claims 1-8, characterized in that, It also includes a data augmentation step, in which the batch of images are augmented and then input into a linear projection layer. The data augmentation includes adjusting image size, zero-value filling, random cropping, random brightness variation, random rotation, normalization, and random erasure.

10. A vehicle re-identification system based on a ViT dual-scale feature fusion network, characterized in that, include: The system includes a linear projection layer construction unit, a ViT-Base network construction unit, a multi-scale module construction unit, a multi-scale feature fusion module construction unit, a classifier construction unit, a batch image construction unit, an image feature sequence generation unit, a loss function construction unit, a network training and update unit, and a vehicle re-identification unit. The linear projection layer construction unit is used to construct a linear projection layer, which is used to divide the image into blocks; The ViT-Base network building unit is used to build the ViT-Base network, which is used to extract single-scale vehicle features from the segmented image. The multi-scale module construction unit is used to construct a multi-scale module, which is used to expand the single-scale features of the vehicle into multi-scale features, and output the original scale features and double-scale features respectively. The multi-scale feature fusion module construction unit is used to construct the multi-scale feature fusion module; The classifier construction unit is used to construct a classifier, which is used for vehicle classification. The batch image construction unit is used to construct batch images; The image feature sequence generation unit is used to sequentially input batch images into a linear projection layer and a ViT-Base network to obtain image feature sequences; The image feature sequence is sequentially input into the multi-scale module and the multi-scale feature fusion module to obtain the original scale fusion feature and the double scale fusion feature; The loss function construction unit is used to construct the overall network loss function. It selects the first image feature vector in the image feature sequence as the global image feature, and inputs the global image feature, the original scale fusion feature and the double scale fusion feature into the classifier to obtain the probability of each image category and calculate the cross-entropy loss function. The triplet loss function is calculated based on global image features, original scale fusion features, and double scale fusion features, respectively. The overall loss function is obtained by adding the cross-entropy loss function and the triplet loss function. The network training update unit is used to update the network weights according to the gradient value of the overall loss function, and saves the current network weights after training is completed; The vehicle re-identification unit is used to output the vehicle re-identification result. It inputs a query image and multiple reference images into the trained network. The trained network outputs global image features, original scale fusion features and double scale fusion features. After concatenating the global image features, original scale fusion features and double scale fusion features, it calculates the Euclidean distance between the query image features and the reference image features. The reference images sorted based on the Euclidean distance are the vehicle re-identification results.

Images