A Mask-Based Vehicle Re-identification Method and System Based on Counterfactual Attention Mechanism

Abstract

This disclosure relates to the field of vehicle re-identification technology, proposing a mask-aware vehicle re-identification method and system based on a counterfactual attention mechanism. The method includes the following steps: acquiring and preprocessing an image of the vehicle to be identified; extracting spatial features from the preprocessed vehicle image; performing counterfactual attention learning on the extracted spatial features using a counterfactual attention causal reasoning method to obtain global features with counterfactual attention; extracting mask-aware features from the preprocessed vehicle image to obtain local features; concatenating and fusing the global and local features to obtain the total vehicle features; and obtaining the target vehicle re-identification result based on the total features. This disclosure, through counterfactual attention learning, enables the network to extract highly discriminative features from the image. The addition of mask-aware vehicle feature recognition allows the network to distinguish features in different regions of the vehicle, solving the problem of large differences in the same vehicle viewed from different perspectives.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle re-identification technology, specifically to a mask-based vehicle re-identification method and system based on a counterfactual attention mechanism. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Vehicle re-identification technology refers to the accurate retrieval of a specified target vehicle using artificial intelligence technology with minimal time and labor costs. Vehicle re-identification falls under the category of fine-grained identification tasks among similar targets, and with economic development, it has become a hot topic. However, the development of vehicle re-identification technology has consistently faced two major challenges: first, the same vehicle can appear significantly different in different views; second, different vehicles may share the same model and color, making subtle differences difficult to distinguish. These issues have become key challenges for vehicle re-identification technology. The inventors discovered that conventional vehicle re-identification primarily utilizes global vehicle features as the matching basis, lacking the recognition of detailed features. This makes it difficult to distinguish vehicles of the same model and color, significantly limiting the accuracy of vehicle re-identification.

[0004] To address the above issues, Ye Song et al., in their paper "View-Decision Based Compound Match Learning for Vehicle Re-identification in UAV Surveillance" published in the *Proceedings of the 39th Chinese Control Conference*, employed a multi-view compound matching learning strategy. This strategy extracts positive sample images from different views to perform multi-view matching learning on the target sample, thereby improving the model's generalization ability across different views. However, this method relies heavily on the selection of positive samples, introducing a degree of randomness. If suitable positive samples are not selected or if the image library lacks positive samples from a specific viewpoint, it cannot effectively solve the problem of excessive differences between different views of the same vehicle. Furthermore, relying solely on global features cannot resolve the difficulty in distinguishing different vehicles of the same type and color.

[0005] Local features can provide crucial clues and effectively distinguish between similar vehicles of different categories. In their paper "Destruction and Construction Learning for Fine-grained Image Recognition" published in *ComputerVision Foundation*, Yue Chen et al. proposed a method of destruction and reconstruction to improve the accuracy of fine-grained image recognition. Reconstructing local regions can effectively extract detailed features; however, reconstructing the image also introduces new noise into the network. Although adversarial learning is used to reduce the impact of this noise, it does not fundamentally solve the problem. Moreover, during image reconstruction, the original local information of the image is likely to be destroyed, causing much crucial information to be scattered across different local bounding boxes. This disrupts the continuity of key information, leading to distortion of the extracted local features. Summary of the Invention

[0006] To address the aforementioned problems, such as significant differences in the appearance of the same vehicle across different views, and the difficulty in distinguishing subtle differences between vehicles sharing the same model and color, this disclosure proposes a mask-based vehicle re-identification method and system. Through counterfactual attention learning, the network can extract high-discriminative features from the image. Simultaneously, the addition of mask-based vehicle feature recognition enables the network to distinguish features from different regions of the vehicle, increasing its ability to extract local features and resolving the issue of significant differences in the same vehicle across different views.

[0007] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0008] One or more embodiments provide a mask-based vehicle re-identification method based on a counterfactual attention mechanism, comprising the following steps:

[0009] Acquire images of the vehicles to be identified and perform preprocessing;

[0010] Spatial feature extraction is performed on the preprocessed vehicle images;

[0011] The extracted spatial features are subjected to counterfactual attention learning using a counterfactual attention causal reasoning method to obtain global features with counterfactual attention.

[0012] Mask-aware feature extraction is performed on the preprocessed vehicle image to obtain local mask-aware features;

[0013] The global and local features are fused together to obtain the total vehicle features, and the target vehicle re-identification result is obtained based on the total features.

[0014] One or more embodiments provide a mask-aware vehicle re-identification system based on a counterfactual attention mechanism, including:

[0015] Data acquisition module: configured to acquire and preprocess images of the vehicle to be identified;

[0016] Feature extraction module: configured to extract spatial features from preprocessed vehicle images;

[0017] Counterfactual attention learning module: configured to perform counterfactual attention learning on the extracted spatial features using a counterfactual attention causal reasoning method to obtain global features with counterfactual attention;

[0018] Mask perception module: configured to extract mask perception features from preprocessed vehicle images to obtain local mask perception features;

[0019] Fusion module: It is configured to concatenate and fuse global features and local features to obtain the total vehicle features, and obtain the target vehicle re-identification result based on the total features.

[0020] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps described in the above method.

[0021] A computer-readable storage medium is characterized in that it is used to store computer instructions, which, when executed by a processor, perform the steps described in the above method.

[0022] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0023] In this disclosure, counterfactual attention learning is used to extract high-discriminative features of vehicles in images. Simultaneously, the addition of mask perception enables the differentiation of features in different regions of the vehicle, increasing the ability to extract local features and addressing the issue of significant differences in the same vehicle viewed from different perspectives.

[0024] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0025] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0026] Figure 1 This is a flowchart of the vehicle re-identification method according to Embodiment 1 of this disclosure;

[0027] Figure 2This is a schematic diagram of the vehicle re-identification network structure of Embodiment 1 of this disclosure;

[0028] Figure 3 is a target vehicle attention map predicted by an example of Embodiment 1 of this disclosure, wherein Figure 3(a) is a general attention map and Figure 3(b) is a counterfactual attention map. Detailed Implementation

[0029] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0031] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0032] Example 1

[0033] In one or more of the technical solutions disclosed in the embodiments, such as Figure 1 Figure 3 shows a mask-based vehicle re-identification method based on counterfactual attention mechanism, which includes the following steps:

[0034] Step 1: Acquire the image of the vehicle to be identified and perform preprocessing;

[0035] Step 2: Extract spatial features from the preprocessed vehicle images;

[0036] Step 3: Use the counterfactual attention causal reasoning method to perform counterfactual attention learning on the extracted spatial features to obtain global features with counterfactual attention;

[0037] Step 4: Extract mask-aware features from the preprocessed vehicle image to obtain local mask-aware features;

[0038] Step 5: Concatenate and fuse the global features and local features to obtain the total vehicle features, and obtain the target vehicle re-identification result based on the total features.

[0039] In this embodiment, counterfactual attention learning enables the extraction of highly discriminative features of vehicles in the image. Simultaneously, the addition of mask awareness distinguishes features from different regions of the vehicle, increasing the ability to extract local features and resolving the issue of significant differences in the same vehicle across different views.

[0040] In step 1, the vehicle image is specifically sized and normalized.

[0041] In step 2, spatial features can be extracted using ResNet50 as the first backbone network. The extracted spatial features are then subjected to average pooling to obtain multi-dimensional feature vectors.

[0042] Step 3, counterfactual attention causal reasoning, includes the following steps:

[0043] Step 31: Perform original attention learning on the extracted image spatial features, and perform multi-channel convolution operation on the image features to obtain a multi-channel attention map for learning the spatial distribution of each part of the object;

[0044] Step 32: Softly weight the feature map based on the multi-channel attention map, and aggregate it through global average pooling to obtain the original attention feature vector;

[0045] Step 33: Apply counterfactual attention intervention to the extracted image feature maps to generate counterfactual attention feature maps;

[0046] Step 34: Softly weight the image feature map based on the counterfactual attention feature map, and aggregate it through global average pooling to obtain the counterfactual attention feature vector;

[0047] Step 35: Input the original attention feature vector and the counterfactual attention feature vector into the linear layer for classification processing to obtain the original classification result and the counterfactual classification result respectively;

[0048] Step 36: Subtract the counterfactual classification result from the original classification result to obtain the global feature with counterfactual attention.

[0049] Optional, such as Figure 2 As shown, counterfactual attention causal reasoning can be implemented by constructing a counterfactual attention learning network. This network includes a counterfactual attention generation network, a primary attention generation network, a counterfactual attention classification module, a primary attention classification module, and a cross-entropy loss calculation module (ClsLoss), etc. Figure 2 The counterfactual attention learning network part of the algorithm.

[0050] Specifically, the original attention classification module and the counterfactual attention classification module can each adopt linear layer classification.

[0051] This embodiment proposes a counterfactual attention learning mechanism based on causal reasoning, which can effectively extract fine-grained features between objects and greatly help improve the classification accuracy between similar objects. The counterfactual attention mechanism makes the attention learned by the network more accurate and effective, avoiding the network learning useless attention that has no effect on classification.

[0052] Step 4, mask-aware feature extraction, includes the following steps:

[0053] Step 41: Obtain preprocessed vehicle images from different shooting angles;

[0054] Step 42: Input the vehicle image into the second backbone network and extract the feature map of the vehicle image;

[0055] Specifically, the second backbone network can also use the ResNet50 backbone network, and the first and second backbone networks can share weights.

[0056] Step 43: Generate a mask for the target vehicle, and downsample the generated mask so that the mask image is the same size as the feature map;

[0057] Step 44: Use the downsampled mask image to extract the region of the feature map corresponding to the target vehicle. Based on the area of ​​the mask region, the extracted features are weighted accordingly to generate the final local features.

[0058] Mask-aware feature extraction can be implemented by constructing a mask-aware network. This mask-aware network includes a mask feature map generation module, a local mask feature extraction module, and a local mask feature fusion module, among others. Figure 2 The mask sensing network part of the algorithm.

[0059] In this embodiment, a mask perception network is used to extract local features of the vehicle. By matching features from different perspectives, the problem of matching the same vehicle under different views can be effectively solved.

[0060] In step 5, the global features and local features are fused together to obtain the total vehicle features.

[0061] The vehicle re-identification method in this embodiment can be implemented in a mask-aware vehicle re-identification network based on a counterfactual attention learning mechanism of causal reasoning, including a counterfactual attention learning network, a mask-aware network, and a fusion network.

[0062] The counterfactual attention learning network is used to learn counterfactual attention from extracted spatial features using the counterfactual attention causal reasoning method, resulting in global features with counterfactual attention.

[0063] Mask-aware networks are used to extract mask-aware features from preprocessed vehicle images to obtain local mask-aware features.

[0064] The fusion network is used to concatenate and fuse global and local features to obtain the total vehicle features, and the target vehicle re-identification result is obtained based on the total features.

[0065] Furthermore, it also includes a method for training the vehicle re-identification network, comprising the following steps:

[0066] Step S1: Obtain the vehicle image dataset.

[0067] To prepare the vehicle dataset, the VeRi-UAV dataset can be used. This dataset contains 17,516 complete images of 453 vehicles from different directions. The training set contains 10,243 images, the test set contains 6,122 images, and the query set contains 1,151 images.

[0068] Step S2: Training data processing for the mask perception network: Mask annotation is performed for different viewpoints of the vehicle image, and the training data for the mask perception network is obtained after data preprocessing.

[0069] To obtain the vehicle's features from different perspectives, we perform masking annotation on a vehicle image for different viewpoints. We divide an image into five distinct parts: background, front, rear, top, and sides. We denote these five parts as N, where N represents the number of different viewpoints in the original vehicle image (N = 5). We then map each part to one of five corresponding categories: {0, 1, 2, 3, 4}. Let i be one of these five viewpoint categories, then i ∈ {0, 1, 2, 3, 4} represents the vehicle viewpoint of the corresponding category.

[0070] We denote the vehicle image as X and the labeled mask information as M. M can be represented as:

[0071]

[0072]

[0073] Where cat(·) represents the chaining operation. M i This represents the mask information for category i of the view, where element a takes the value 0 or 1, where 0 indicates that the pixel at that position does not belong to category i, and 1 indicates that the pixel at that position belongs to category i. H and W represent the height and width of the image.

[0074] The vehicle image X and mask information M are subjected to operations such as size transformation, random flipping, and normalization, as follows:

[0075]

[0076]

[0077] RHF stands for Random Flip. Resize is a uniform scaling operation, where C, H, and W are the channels, height, and width of the image after scaling, respectively. Here, C = 3, H = W = 256. Normalize is a normalization operation, where the original image is normalized using a mean of (0.485, 0.456, 0.406) and a standard deviation of (0.229, 0.224, 0.225).

[0078] Step S3: Mask perception network training: Input the training data of the mask perception network into the mask perception network, extract feature maps, and use balanced cross-entropy loss for recognition training to obtain the trained mask perception network.

[0079] The processed vehicle image X from step S2 is input into the mask perception network M_Net(·), where SeResNeXt50 is used as the backbone network:

[0080]

[0081] Among them, P M The mask information map for the vehicle image X predicted by the mask perception network.

[0082] The predicted mask information P M The actual mask information M is input together into the balanced cross-entropy loss function for training and learning, as follows:

[0083] L M =L bce (P M ,M) (6)

[0084] L bce =-[M×log(σ(P)] M ))+(1-M)×log(1-σ(P M (7)

[0085]

[0086] Among them, L M L is the masking loss predicted by the network. bce To balance the cross-entropy loss function, σ(x) is the Sigmoid function.

[0087] The trained mask-aware network M_Net(·) is used to extract local mask features of the vehicle.

[0088] Step S4, Counterfactual Attention Learning Network Training, includes the following steps:

[0089] Step S41: Extract features from the vehicle images in the vehicle image dataset from step S1.

[0090] This embodiment proposes a counterfactual attention learning network to extract global features from vehicle images. Through counterfactual learning of the original attention, detailed vehicle features can be extracted. This embodiment uses ResNet50 as the backbone extraction network, feeding the vehicle image X into the feature extraction network to extract global image features. The explanatory formula for the feature extraction network is shown below:

[0091]

[0092] Where F is the feature map extracted from the input image X, Net(·) is the vehicle re-identification network with a counterfactual attention learning mechanism, and c, h, w are the channels, height and width of the feature map F, respectively, where c = 2048, h = w = 16;

[0093] Step S42: Perform original attention learning on the image features obtained in step S41, and perform multi-channel convolution operation on the image features to obtain a multi-channel attention map;

[0094] The original attention is generated by performing a convolution operation on the feature map F using a 1×1 convolution kernel. The original attention map, where d represents the number of multi-channel attention points. The calculation process is as follows:

[0095]

[0096] Among them, f 1×1 (·) represents a 1×1 convolution operation, and ReLU is a piecewise linear activation function. A is the original attention map, where d is the number of attention channels, and h and w are the height and width of the original attention map A. In this embodiment, d = 32 and h = w = 16 are set.

[0097] Step S43: Softly weight the feature map according to the multi-channel attention map, and aggregate it through global average pooling to obtain image feature data H with the original attention.

[0098] The feature map F is subjected to soft weighting of the original attention, which involves performing a dot product summation operation between the original attention map A and the feature map F, and then aggregating the sums using average pooling to obtain the feature vector with the original attention. The operation process is as follows:

[0099]

[0100] Among them, A i For the attention map of the i-th channel, h i The original attention feature vector is generated for the i-th channel. After obtaining the feature vector with the original attention, we concatenate the original attention feature vectors of all channels and then normalize them.

[0101]

[0102] Where cat(·) represents the concatenation operation, normalize is the normalization operation, and H is the original attention feature vector generated from all original attention channels.

[0103] Step S44: Compare and learn the image feature data H with the original attention and the counterfactual attention to generate a counterfactual attention feature map.

[0104] After obtaining the original attention feature vector H, it is compared and learned with the counterfactual attention. Here, a random uniform distribution is used to generate the counterfactual attention feature map. The size of the generated counterfactual attention feature map should be the same as the size of the original attention map A mentioned above. The method is as follows:

[0105]

[0106] Where uniform(0,2) represents generating uniformly distributed values ​​in the range [0,2]. This is the generated counterfactual attention graph.

[0107] Step S45: Based on the counterfactual attention map The feature maps are soft-weighted, aggregated using a global average pooling operation, concatenated across multiple channels, and normalized to obtain the counterfactual attention feature vector.

[0108] Apply counterfactual attention soft weighting to feature map F, and then apply the counterfactual attention map... The sum is obtained by performing a dot product operation with the feature map F, followed by average pooling. The calculation process is as follows:

[0109]

[0110] in For the counterfactual attention of the i-th channel, The counterfactual attention feature vector generated for the i-th channel; we will generate the counterfactual attention feature vectors for all channels. After concatenation, normalization is then performed:

[0111]

[0112] Where cat(·) represents the concatenation operation, and normalize is the normalization operation. Counterfactual attention feature vectors generated for all counterfactual attention channels.

[0113] Step S46, Counterfactual Causal Reasoning: Combine the obtained original attention feature vector H with the counterfactual attention feature vector H. The classification result is obtained by performing linear layer processing. The cross-entropy loss between the classification result and the classification label is calculated to iteratively train the network until the termination condition is met, thus obtaining the trained counterfactual attention learning network.

[0114] The linear layer is processed as follows:

[0115]

[0116]

[0117] Where Linear(·) is a linear layer operation, (c×d,n) cls ) represents the input and output parameters of the linear layer, n cls Let p be the number of vehicle categories, and p be the original classification result of the attention. The classification results are for counterfactual attention. Finally, both results are combined with the classification labels and subjected to cross-entropy loss.

[0118]

[0119]

[0120] y represents the category label, L ce y is the cross-entropy loss function. ic Let c be the sign function; it takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. ic Let N be the probability that sample i belongs to category c, and N be the number of samples.

[0121] By optimizing L cls In this function, we aim to achieve two things: 1) The attention model should improve predictions based on erroneous attention as much as possible, encouraging attention to discover the most discriminative regions and avoiding suboptimal results; 2) We penalize predictions based on erroneous attention, which forces the classifier to make decisions based more on main cues than biased cues and reduces the influence of biased training sets.

[0122] Step S5: Extract global vehicle features based on the trained counterfactual attention learning network, extract local vehicle features based on the trained mask perception network, fuse the global and local features to obtain the total features f of the vehicle image, and obtain the vehicle recognition result based on the total features.

[0123] The counterfactual attention learning network proposed in this embodiment enables the network to learn discriminative features. This means that the feature map F extracted by the network carries highly discriminative vehicle features. We perform average pooling on the feature map F to obtain the global features of the vehicle image:

[0124]

[0125] f g This represents the global features of the vehicle image. After obtaining the global vehicle features, we also need to generate a mask information map of the vehicle image using the mask perception network M_Net(·) trained above, and use it to extract the local features of the vehicle image:

[0126]

[0127] Here, Rescale represents bilinear interpolation used to scale the mask information map, N represents the five views of the vehicle, m represents the scaled mask information map generated by the mask perception network, and h and w represent the size of the scaled mask information map, which is scaled to the feature size. Figure 1 The general size is that h = w = 16.

[0128] Then, the scaled mask-generated image m is scored from four viewpoints (the background category is removed here), and local mask features from different viewpoints are extracted:

[0129]

[0130]

[0131] Where m i For the mask information graph of category i view, score i For the score of the i-th category view, f l i For local mask features of category i view.

[0132] Finally, the extracted local mask features for each view category are weighted by score and then averaged and pooled.

[0133]

[0134] f l This refers to the final local feature after all category views have been merged.

[0135] Then, the global and local features are fused and normalized to obtain the total feature f of the vehicle image, which is used to calculate the cosine similarity between samples:

[0136]

[0137]

[0138] Where cat(·) represents the chaining operation, f α For the total characteristic after cascading, f α (i) represents the total characteristic f α The value of the i-th element, f is f α The normalized total features. The normalized total features f are more conducive to calculating the cosine similarity distance.

[0139] Step S6: Triple metric learning based on difficult samples. The triple loss training method based on difficult positive and negative samples is adopted. The cross-entropy loss function and the difficult negative sample loss function are fused as the overall loss function to train the vehicle re-identification network as a whole.

[0140] This embodiment employs a triplet loss training method with difficult positive and negative samples to amplify the differences between similar objects and improve the model's accuracy in classifying similar objects.

[0141] Specifically, the triplet metric learning based on difficult samples, using a triplet loss training method with difficult positive and negative samples, includes the following steps:

[0142] Step S61: Assign the total features f of the extracted vehicle images. For each target vehicle, match it with vehicle images from different viewpoints to form positive sample pairs. Then, find the positive sample with the lowest similarity score to the target sample in the positive sample pairs and mark it as a difficult positive sample.

[0143] Step S62: In the training of a batch, calculate the similarity score between all positive samples in the batch and the target sample, exclude positive samples that are in the same category as the target sample, and keep only the samples that are not in the same category as the target sample, i.e., negative samples. Find the negative sample with the highest similarity score to the target sample and mark it as a difficult negative sample.

[0144] Step S63: Perform metric learning on the target sample, difficult positive sample, and difficult negative sample so that the score of the difficult positive sample is higher than the score of the difficult negative sample, that is, the score of the least similar positive sample is higher than the score of the most similar negative sample, so as to achieve the classification effect.

[0145] The following example will be used to explain in detail the learning process of triplet loss metric.

[0146] In this embodiment, a multivariate composite contrast loss function is designed for the target image X. i Match a positive sample image X + A difficult negative sample image X - Pairs of positive and negative samples are constructed, and then the cosine similarity score between them is calculated:

[0147]

[0148]

[0149] Wherein, S(X) i ,X + ) represents sample X i With X + Cosine similarity score between them This represents the total features of the target sample, positive sample, and negative sample. Since the total features f are normalized, the denominator of the score function is 1.

[0150] The training method employs a difficult negative sample training approach, which involves metric learning on the target sample, difficult positive samples, and difficult negative samples. The goal is to ensure that the score of the difficult positive sample is higher than that of the difficult negative sample; that is, the score of the least similar positive sample should be higher than that of the most similar negative sample, thus achieving the desired classification effect. The loss function is designed as follows:

[0151] L hard =ReLu{min[S(X i ,X + )]-max[S(X i ,X - )]+γ} (29)

[0152] Where S(x,y) represents the cosine similarity between samples, ReLU(x) is the activation function, ensuring the loss is greater than or equal to 0 to prevent network divergence, and X i X + and X - These represent the target sample, positive sample, and negative sample, respectively, with γ being a manually set expected distance parameter.

[0153] The overall loss function is a fusion of the cross-entropy loss function and the hard negative sample loss function. Combining the advantages of the cross-entropy loss function in extracting key attention points and the hard negative sample metric learning in distinguishing similar images, the overall loss function design for the training framework is as follows:

[0154] L=αL cls +βL hard (30)

[0155] Where α and β are manually set weights, and here α+β=1.

[0156] The training process of the vehicle re-identification network is as follows:

[0157] In this embodiment, 10 images are used as a batch, with each image paired with 4 positive samples of the same category, and all are input into the convolutional neural network. Except for the backbone feature extraction network ResNet50, which uses the provided ImageNet pre-trained weights and pre-trained parameters, all other parts are initialized using a Kaiming Gaussian distribution. The optimizer is SGD, with an initial learning rate of 0.001, weight decay of 0.0001, and momentum of 0.9. The loss function is set using formula (19). In this embodiment, the parameters are set to γ ​​= 0.6, α = 0.2, and β = 0.8. The entire re-identification network is trained for a total of 100 epochs on a single GPU.

[0158] To further illustrate the effectiveness of vehicle re-identification in this embodiment, the following experiments were conducted.

[0159] 1. Experimental conditions

[0160] The hardware requirements for all verification experiments of this invention are: Ubuntu 16.04L, CPU: Intel i9-9900X, RAM 64G, and one TITAN XP graphics card.

[0161] The software environment used in the experiment was: Python 3.7, PyTorch = 1.10.2, and TorchVision = 0.8.2.

[0162] The dataset used is the VeRi-UAV dataset, which contains aerial images of 453 vehicle IDs. This paper divides the dataset into a training set (image_train), a test set (image_test), and a query set (image_query). The training, test, and query sets are distributed in approximately a 7.5:5:1 ratio. The training set contains 9792 images from 273 vehicle IDs, the test set contains 6489 images from 180 vehicle IDs, and the query set contains 1235 images from 176 vehicle IDs.

[0163] 2. Experimental Content and Results

[0164] The vehicle images to be trained are input into the vehicle re-identification network based on the counterfactual attention learning mechanism of causal reasoning in this embodiment. The purpose is to effectively extract key information features of the vehicles and ignore irrelevant background information, thereby increasing the distinguishability of vehicles of the same type and color and improving the accuracy of vehicle re-identification. At the same time, the training strategy of difficult negative samples is combined to effectively expand the differentiation between classes.

[0165] Figures 3(a) and (b) show the attention heatmaps with and without counterfactual attention. It can be seen that the counterfactual attention mechanism enables the network to learn better vehicle representations, resulting in better vehicle re-identification performance.

[0166] The effectiveness and advancement of this experiment will be evaluated using evaluation criteria consistent with current state-of-the-art vehicle re-identification research. Current evaluation metrics for vehicle re-identification algorithms primarily include Top-k (some studies denote it as Rank-k or Hit@k, which have the same meaning) and mAP, as described below.

[0167] (1) Top-k: In the output similarity sequence, the hit rate of the top k samples containing correct samples is Top-k, which describes the accuracy of the query sample matching correctly in the retrieval set. In vehicle re-identification, it mainly describes the accuracy of query sample ID judgment.

[0168] (2) mAP (mean accuracy): mAP comprehensively reflects the algorithm's ability to retrieve all vehicle images with the same ID as the query target. It is often used to evaluate various algorithms in pattern recognition. In vehicle re-identification, mAP is the average AP (accuracy) of all samples in the test set, reflecting the accuracy of image retrieval in the image library. A higher mAP indicates fewer missed samples in the image library. Formulas (3-11) and (3-12) show the calculation method of mAP.

[0169]

[0170]

[0171] Where N is the total number of samples in the query set, AP(n) is the accuracy of the nth query sample, S is the number of positive samples of the nth query sample in the retrieval set, and k i This indicates the order of the i-th positive sample in the image similarity sequence of the retrieval set.

[0172] Table 1 Comparison of the method in this embodiment with other algorithms

[0173]

[0174]

[0175] In Table 1, PC-HSPML, AcNet, DCL, and HBPASM represent a metric learning method based on pose calibration and hard sample perception, an attention binary tree convolutional neural network method, a fine-grained classification method based on destruction and construction learning, and a hierarchical bilinear pooling fine-grained classification method based on aggregation relaxation mask, respectively. As can be seen from the table, the vehicle recognition accuracy of the method in this embodiment is significantly higher than that of the methods in the prior art, and the false negative rate is also the lowest.

[0176] Table 2 Comparison of results between counterfactual attention and general attention

[0177] method Top1 Top 5 Top 10 mAP Basline 97.2％ 98.3％ 98.4％ 87.6％ Basline + attention 97.7％ 98.3％ 98.8％ 89.7％ Basline + counterfactual 98.1％ 99％ 99％ 89.8％

[0178] Basline is a general method that does not employ attention; it is a method that only includes triplet metric learning.

[0179] Basline+attention is a multi-channel attention method, which adds a multi-channel convolutional attention method to the triplet metric learning method.

[0180] Basline+counterfactual is a method that incorporates counterfactual attention, that is, it adds counterfactual attention to the triplet metric learning method and the multichannel attention method.

[0181] As can be seen from Table 2, the use of the counterfactual attention learning algorithm can greatly improve the accuracy of vehicle recognition and enhance the effectiveness of the recognition system.

[0182] Example 2

[0183] Based on Embodiment 1, this embodiment provides a mask-sensing vehicle re-identification system based on a counterfactual attention mechanism, including:

[0184] Data acquisition module: configured to acquire and preprocess images of the vehicle to be identified;

[0185] Feature extraction module: configured to extract spatial features from preprocessed vehicle images;

[0186] Counterfactual attention learning module: configured to perform counterfactual attention learning on the extracted spatial features using a counterfactual attention causal reasoning method to obtain global features with counterfactual attention;

[0187] Mask perception module: configured to extract mask perception features from preprocessed vehicle images to obtain local mask perception features;

[0188] Fusion module: It is configured to concatenate and fuse global features and local features to obtain the total vehicle features, and obtain the target vehicle re-identification result based on the total features.

[0189] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0190] Example 3

[0191] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it performs the steps described in the method of Embodiment 1.

[0192] Example 4

[0193] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps described in the method of Embodiment 1.

[0194] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0195] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for vehicle re-identification based on a mask perception vehicle using a counterfactual attention mechanism, characterized in that, The method comprises the following steps: Obtain a vehicle image to be identified and perform preprocessing; Extract spatial features from the preprocessed vehicle image; Perform counterfactual attention learning on the extracted spatial features using a counterfactual attention causal reasoning method to obtain global features with counterfactual attention; The counterfactual attention causal reasoning comprises the following steps: Perform original attention learning on the extracted image spatial features, and perform multi-channel convolution operation on the image features to obtain a multi-channel attention map for learning the spatial distribution of each part of the object; Softly weight the feature map according to the multi-channel attention map, and aggregate it through global average pooling operation to obtain an original attention feature vector; Perform counterfactual attention intervention on the extracted image feature map to generate a counterfactual attention feature map; Softly weight the image feature map according to the counterfactual attention feature map, and aggregate it through global average pooling operation to obtain a counterfactual attention feature vector; Input the original attention feature vector and the counterfactual attention feature vector into a linear layer for classification processing to obtain original classification results and counterfactual classification results, respectively; Subtract the counterfactual classification results from the original classification results to obtain global features with counterfactual attention; Perform mask perception feature extraction on the preprocessed vehicle image to obtain local features perceived by the mask; The mask perception feature extraction comprises the following steps: Obtain preprocessed vehicle images at different shooting angles; Input the vehicle image into a second backbone network to extract a feature map of the vehicle image; Generate a mask for the target vehicle, and downsample the generated mask to make the mask image consistent in size with the feature map; Extract the region of the feature map corresponding to the target vehicle using the downsampled mask image, and generate the final local feature by weighting the extracted feature according to the area size of the mask region; Concatenate the global feature and the local feature to obtain vehicle total features, and obtain a target vehicle re-identification result based on the total features.

2. The method of claim 1, wherein the method is based on an anti-factual attention mechanism mask-aware vehicle re-identification method. It also includes constructing a vehicle re-identification network, including a counterfactual attention learning network, a mask perception network, and a fusion network; The counterfactual attention learning network is used to perform counterfactual attention learning on the extracted spatial features using a counterfactual attention causal reasoning method to obtain global features with counterfactual attention; The mask perception network is used to perform mask perception feature extraction on the preprocessed vehicle image to obtain local features perceived by the mask; The fusion network is used to concatenate the global feature and the local feature to obtain vehicle total features, and obtain a target vehicle re-identification result based on the total features.

3. The method of claim 2, wherein the method comprises: It also includes a method for training a vehicle re-identification network, comprising the following steps: Obtain a vehicle image dataset; Mask annotation is performed for different viewing angles of the vehicle image, and the training data of the mask perception network is obtained after data preprocessing; Input the training data of the mask perception network into the mask perception network to extract a feature map, and perform recognition training using balanced cross-entropy loss to obtain a trained mask perception network; Perform counterfactual attention learning network training; The global feature of the vehicle is extracted based on the trained counterfactual attention learning network, the local feature of the vehicle is extracted based on the trained mask perception network, the total feature of the vehicle picture is obtained by fusing the global feature and the local feature, and the recognition result of the vehicle is obtained based on the total feature; The triplet metric learning based on difficult samples is adopted, a triplet loss training method of difficult positive and negative samples is adopted, a cross-entropy loss function and a difficult negative sample loss function are fused as an overall loss function, and the vehicle re-identification network is trained as a whole.

4. The mask perception vehicle re-identification method based on the counterfactual attention mechanism according to claim 1, wherein: The counterfactual attention learning network training comprises the following steps: The vehicle image of the vehicle picture data set is subjected to feature extraction; The obtained image feature is subjected to original attention learning, and multi-channel convolution operation is performed on the basis of the image feature to obtain a multi-channel attention map; According to the multi-channel attention map, the feature map is soft weighted, and is aggregated through a global average pooling operation to obtain image feature data with original attention ; image feature data with original attention contrastive learning with counterfactual attention to generate counterfactual attention feature maps ; According to the counterfactual attention map The feature map is soft weighted, aggregated by a global average pooling operation, concatenated in multiple channels, and normalized to obtain a counterfactual attention feature vector ; The obtained original attention feature vector The counterfactual attention feature vector The linear layer processing is performed to obtain a classification result, and the cross-entropy loss of the classification result and a classification label is calculated to iteratively train the network until a termination condition is met to obtain a trained counterfactual attention learning network.

5. The mask-based vehicle re-identification method based on counterfactual attention mechanism as described in claim 3, characterized in that... The triplet metric learning based on difficult samples is adopted, a triplet loss training method of difficult positive and negative samples is adopted, and the vehicle re-identification network is trained as a whole. Total features of vehicle pictures The distribution is performed, each target vehicle matches vehicle pictures corresponding to different view angles, and a positive sample pair is formed; in the positive sample pair, a positive sample with the lowest similarity score with the target sample is found and marked as a difficult positive sample; In one batch of training, the similarity score calculation is performed between all positive samples in one batch and the target sample, the positive samples of the same category as the target sample are excluded, the samples of different categories from the target sample are retained, that is, negative samples, the negative sample with the highest similarity score with the target sample is found, and the negative sample is marked as a difficult negative sample; The target sample, the difficult positive sample and the difficult negative sample are subjected to metric learning, so that the score of the difficult positive sample is higher than that of the difficult negative sample, so as to achieve the classification effect.

6. A mask-aware vehicle re-identification system based on counterfactual attention mechanism, characterized in that, It comprises: A data acquisition module configured to acquire a vehicle image to be identified and perform preprocessing; A feature extraction module configured to perform spatial feature extraction on the preprocessed vehicle image; A counterfactual attention learning module configured to perform counterfactual attention learning on the extracted spatial feature by using a counterfactual attention causal reasoning method to obtain a global feature with counterfactual attention; The counterfactual attention causal reasoning comprises the following steps: The extracted image spatial feature is subjected to original attention learning, and multi-channel convolution operation is performed on the basis of the image feature to obtain a multi-channel attention map for learning the spatial distribution of each part of the object; The feature map is soft-weighted according to the multi-channel attention map, and is aggregated by a global average pooling operation to obtain an original attention feature vector; The extracted image feature map is subjected to counterfactual attention intervention to generate a counterfactual attention feature map; The image feature map is soft-weighted according to the counterfactual attention feature map, and is aggregated by a global average pooling operation to obtain a counterfactual attention feature vector; The original attention feature vector and the counterfactual attention feature vector are input into a linear layer for classification processing to obtain an original classification result and a counterfactual classification result, respectively; The global feature with counterfactual attention is obtained by subtracting the counterfactual classification result from the original classification result; A mask perception module configured to perform mask perception feature extraction on the preprocessed vehicle image to obtain a mask perception local feature; The mask-aware feature extraction comprises the following steps: Obtaining vehicle images at different shooting angles after preprocessing; Inputting the vehicle images into a second backbone network to extract feature maps of the vehicle images; Generating a mask for the target vehicle, and downsampling the generated mask to make the mask image consistent in size with the feature maps; Using the downsampled mask image to extract regions of the feature maps corresponding to the target vehicle, and generating final local features by weighting the extracted features according to the area sizes of the mask regions; A fusion module configured to concatenate and fuse the global features and the local features to obtain vehicle total features, and obtain a target vehicle re-identification result based on the total features.

7. An electronic device, comprising: A computer program product comprising a memory and a processor, and computer instructions stored on the memory and run on the processor, wherein when the computer instructions are run by the processor, the steps of the method of any one of claims 1-5 are completed.

8. A computer-readable storage medium, characterized in that, A computer program product for storing computer instructions, wherein when the computer instructions are executed by a processor, the steps of the method of any one of claims 1-5 are completed.

Images