A domain-adaptive person re-identification method based on interactive cascaded lightweight transformers

Abstract

The present invention discloses a domain adaptive pedestrian re-identification method based on interactive cascade lightweight transformers, which belongs to the cross-technical field of computer vision and pedestrian recognition. The lightweight transformer designed by the method includes a local lightweight transformer and a global lightweight transformer, which are composed of an embedding module, a multi-head self-attention module and a multi-head hybrid module. The lightweight transformer can capture long-distance dependencies and avoid the aggravation of the influence of noise labels. In the interactive cascade framework designed by the method, pixels are alternately assigned local information and global information, which enhances the interactive ability of local information modeling and global information modeling, thereby improving the feature expression ability of the model. This method realizes effective domain adaptive pedestrian re-identification.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, specifically the cross-technical field of computer vision and pedestrian recognition, and relates to a pedestrian re-identification method based on domain adaptation and interactive cascade lightweight transformers. Background Art

[0002] Since the beginning of the 21st century, intelligent monitoring equipment has developed rapidly, and hardware devices for storing massive data have continued to mature. In order to effectively improve public safety, a large number of surveillance cameras have been deployed in more and more places such as public places, traffic intersections, and public homes. The processing of surveillance videos has become an important research direction. Person re-identification aims to identify the specified query person from a large number of candidates captured by non-overlapping cameras. It is widely used in many fields such as video surveillance and intelligent security. Although supervised person re-identification methods have achieved good results, the heavy annotation burden makes it difficult to apply in the real world. Therefore, domain adaptive person re-identification has attracted widespread attention in recent years. In this task, the model is trained by using labeled source domain data and unlabeled target domain data to fit the unlabeled target domain. The main challenge of the domain adaptive person re-identification task is distribution drift. Distribution drift refers to the phenomenon that the model trained in the source domain will have serious performance degradation in the target domain due to the large difference between the feature distribution of the source domain and the feature distribution of the target domain.

[0003] There are two common approaches to address distribution drift: domain alignment strategies or assigning pseudo labels to target domain data. Domain alignment-based methods typically use generative adversarial networks (GANs) to generate additional images. For example, SBSGAN uses GANs to convert source domain images into background-free images and target domain style images, and then uses these two datasets as training sets (Huang Y, Wu Q, Xu JS, et al. SBSGAN: Suppression of inter-domain backgroundshift for person re-identification [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 9527-9536.). The performance of such methods depends largely on the quality of the generated images.

[0004] Pseudo-label based methods can effectively capture the target domain distribution. The challenge faced by such methods is how to improve the accuracy of pseudo labels. Some researchers have addressed this challenge by designing better pseudo label estimation algorithms. Ge et al. proposed a framework called "Synchronous Mean Teaching" to optimize pseudo labels, learning soft multi-labels for each unlabeled person by comparing with a set of known reference people in the auxiliary domain (Yixiao G, Dapeng C, Hongsheng L.Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification[C] / / Proceedings of the International Conference on Learning Representations.2018:79-88.). Other studies have improved the accuracy of pseudo labels by obtaining discriminative pedestrian features. PatchNet learns discriminative features from selected patches (Yang Q, Yu HX, Wu A, et al. Patch-based discriminative feature learning for unsupervised person re-identification[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 3633-3642.); Zhong et al. use three pseudo-labels based on multi-scale features to filter data and train the model, and propose a camera-invariant loss to reduce the influence of camera style (Zhang H, Cao H, Yang X, et al. Self-Training With ProgressiveRepresentation Enhancement for Unsupervised Cross-Domain Person Re-Identification[J]. IEEE Transactions on Image Processing, 2021, 30: 5287-5298.).

[0005] In the above work, how to extract discriminative pedestrian features is a key issue in cross-domain pedestrian re-identification. Most of the existing feature extraction works are based on convolutional neural networks (CNNs), so they are hindered by the limited receptive field of CNNs. Transformer (Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [C] / / Advances in Neural Information Processing SysteNS. 2017: 5998-6008.) can capture long-range interdependencies, but due to its weak generalization ability, transformer requires a lot of pre-training, large-scale datasets or complex hyperparameter adjustment (Touvron, H., Cord, M., Douze, M., et al. Training data-efficient image transformers & distillation through attention [C] / / International Conference on Machine Learning. 2021: 10347-10357.), which makes it difficult to directly apply to domain adaptive pedestrian re-identification tasks. Therefore, the present invention proposes a lightweight transformer with fewer parameters to overcome the problem of limited receptive field and extract identifiable pedestrian features.

[0006] On the other hand, in the person re-identification task, the interaction between local information modeling and global information modeling also helps to extract identifiable pedestrian features. Usually, there is a connection between local information and global information. For example, people wearing skirts or boots are more likely to be women with long hair, and people carrying black backpacks are more likely to be men with short hair. Therefore, the global attribute dependency generated by global information modeling can improve the accuracy of local information, while the local information from local information modeling is conducive to the establishment of global dependency. The interaction between these two modeling processes helps to enhance feature expression.

[0007] However, existing visual transformers cannot utilize this interaction. Typical structures of visual transformers include self-attention-based non-hierarchical structures, naive hierarchical structures, pyramid structures, and cross-attention structures. The self-attention-based non-hierarchical structure only performs global information modeling; the naive hierarchical structure performs multi-scale information modeling; the pyramid structure integrates multi-scale information; and the cross-attention structure allows the model to establish dependencies using multi-scale information. The above structures do not involve the interaction between the two modeling processes. In order to effectively utilize the interaction between local information modeling and global information modeling, the present invention proposes a cascade framework for the interaction of local information modeling and global information modeling to enhance the extraction of discriminative pedestrian features in domain adaptive person re-identification tasks.

[0008] In summary, in order to solve the problems of limited receptive field of the model and insufficient interaction between local information modeling and global information modeling in the domain adaptive pedestrian re-identification task, the present invention proposes a domain adaptation method based on interactive cascade lightweight transformers (Domain Adaption with Interactive Cascade Lightweight Transformers, DA-ICLT). The lightweight transformer proposed in the present invention can capture long-distance dependencies and avoid the aggravation of the influence of noise labels. It deletes the multi-layer perceptron (MLP) in the transformer to enhance the generalization ability of the model and reduce the computational overhead, and uses convolutional neural networks instead of some fully connected layers to introduce translation invariance and locality. Furthermore, the interactive cascade lightweight transformer framework in the present invention can effectively enhance the expression of pedestrian features. It alternately assigns local neighborhood information and global information to pixels, so that local information modeling and global information modeling complementarily enhance the feature expression of pixels. Summary of the invention

[0009] For the domain adaptive person re-identification task, this patent proposes a domain adaptive method based on interactive cascade lightweight transformers (DA-ICLT). In this method, the problem of limited receptive field is solved through the designed lightweight transformer network, and the aggravation of the influence of noise labels is avoided; through the designed interactive cascade framework, the interactive ability of local information modeling and global information modeling is enhanced, and finally the feature expression ability of the model is improved.

[0010] The technical solution of the present invention:

[0011] A domain adaptive person re-identification method based on interactive cascaded lightweight transformers, the steps are as follows:

[0012] 1. Use CNN encoder to extract compact features from pedestrian images.

[0013] The CNN encoder consists of several convolutional neural network layers. Its output dimension is The feature map is a , where height is the height of the pedestrian image and width is the width of the pedestrian image. This feature map will be used as the input of the interactive cascade lightweight transformers.

[0014] 2. Use interactive cascaded lightweight transformers to extract recognizable pedestrian features.

[0015] The interactive cascade lightweight transformers are composed of three basic units, each of which is composed of a cascade of one layer of local lightweight transformers and two layers of global lightweight transformers. Local lightweight transformers and global lightweight transformers are two types of lightweight transformers. The local lightweight transformer and the global lightweight transformer have the same component modules, but the execution processes are different. They are both composed of an embedding module, a multi-head self-attention module, and a multi-head mixing module.

[0016] The process performed by the interactive cascade lightweight transformers consists of a sub-process performed by 1 layer of local lightweight transformers and a sub-process performed by 2 layers of global lightweight transformers.

[0017] The interactive cascade of lightweight transformers performs the following process:

[0018] (2.1) The 1-layer local lightweight transformer performs the following sub-processes:

[0019] (2.1.1) Cut the feature map into query blocks:

[0020]

[0021] Where F is the feature map of the input local lightweight transformers, and its dimension is f i represents the query block, and the subscript i is the sequence number of the query block; N q Indicates the number of query blocks; divide() indicates the operation of cutting the feature map into query blocks. N q =a×b, where a and b represent the number of times F is cut horizontally and the number of times F is cut vertically in the process of cutting the feature graph F into query blocks. i The dimension of

[0022] (2.1.2) Use the embedding module to compress the query block f i The number of channels is obtained by querying the block e i .e i The dimension of is Where d < 1024. The embedding module is a 1-layer convolutional neural network.

[0023] (2.1.3) query block e i Expand into a two-dimensional matrix s consisting of multiple one-dimensional vectors i :

[0024] s i =unfold(e i ) (2)

[0025] In the formula, unfold() represents the operation of expanding the query block into a two-dimensional matrix. This expansion operation is just to expand the last two dimensions of the query block. Expanded into one dimension Represents the query block e i The expanded two-dimensional matrix has a dimension of d×N, where s i The subscript i of is the serial number of the corresponding query block. Indicates i The j-th one-dimensional vector in The dimension of is d.

[0026] (2.1.4) Based on s i The multi-head self-attention module is used for information modeling. The process is as follows:

[0027] It consists of three mapping modules W Q , W K and W V Get the query matrix q i , the key matrix k i Sum value matrix v i :

[0028] q i =W Q s i ;k i =W K s i ;v i =W V s i (3)

[0029] W in the formula Q ,W K and W V There are three fully connected layers, and the number of neurons in each layer is d×d; i, k i and v i They are both two-dimensional matrices with a dimension of d×N, and the subscript i is the serial number of the corresponding query block.

[0030] Q i ,k i and v i Divide into H groups of query matrix, key matrix and value matrix respectively, where H is the number of heads in the multi-head self-attention mechanism. Cross-multiply the transpose of the key matrix in each group with the query matrix to obtain H groups of similarity values ​​for each pixel in the query block. Normalize each group of similarity values ​​to obtain H groups of weights. Cross-multiply the H groups of value matrices with the weights of the corresponding groups. The meaning of this operation is to aggregate each one-dimensional vector in the value matrix according to the similarity between the corresponding pixels. This operation produces H groups of aggregated results in, The dimension is The subscript i is the serial number of the corresponding query block, and the superscript m=1, 2,…, H is the corresponding group number.

[0031] (2.1.5) H sets of two-dimensional matrices Re-merge into a two-dimensional matrix, and fold the merged two-dimensional matrix into a query block u i , its dimension is The subscript i is the index of the query block. This folding operation simply folds the last dimension of the matrix into two dimensions.

[0032] (2.1.6) Will u i Send it to the multi-head mixing module to mix the attention results of different heads (groups), and the dimension is The query block r i , subscript i is the sequence number of the query block. The multi-head hybrid module is a 1-layer transposed convolutional neural network.

[0033] (2.1.7) N q query block r i Merge into one feature map r, the dimension of r is

[0034] (2.1.8) Mix the feature map r with the input feature map F:

[0035] F local =F+r (4)

[0036] Where F local The dimension of is The superscript local is used to indicate that it is the output of the local lightweight transformer, which will be used as the input of step (2.2).

[0037] (2.2) The 2-layer global lightweight transformers perform the following sub-processes:

[0038] (2.2.1) Using the embedding module to compress the number of channels of the input feature map, the dimension is Feature map E, where d<1024.

[0039] (2.2.2) Expand the feature map E into a two-dimensional matrix:

[0040] S=UNFOLD(E) (5)

[0041] In the formula, UNFOLD() represents the operation of expanding the feature map into a two-dimensional matrix. This expansion operation is just to expand the last two dimensions of the feature map. Expanded into one dimension Represents the two-dimensional matrix of the feature map E, which is composed of One-dimensional vectors, its dimension is d×N S .

[0042] (2.2.3) Take S as input, first use the multi-head self-attention module to model the information, then rearrange its output into a three-dimensional feature map, and finally use the multi-head mixing module to mix the attention results of different heads (groups). The process is the same as steps (2.1.4) to (2.1.6).

[0043] (2.2.4) Using the result of (2.2.3) as input, iterate steps (2.2.1) to (2.2.3) once.

[0044] (2.3) Take the result of (2.2.4) as input and repeat steps (2.1) to (2.2) twice. The final dimension is This feature map will be used as the input of step 3.

[0045] 3. Perform global average pooling on the input feature map to obtain a new feature map with a dimension of {1024, 1, 1}, and then reduce the dimension of the new feature map to obtain a one-dimensional feature vector with a dimension of 1024. This one-dimensional feature vector will be used as the input of step 4.

[0046] 4. At time m, the hybrid memory matrix is ​​updated using the features of the pedestrian images in the target domain training set. It is a matrix with a dimension of N t ×1024 two-dimensional matrix, where N t is the number of pedestrian images in the target domain training set. The update method is as follows:

[0047]

[0048] The parameter μ in the formula is a constant, which is used to control the speed of updating the hybrid memory matrix; is the mixed memory matrix at time m; F t The dimension is N t ×1024 two-dimensional matrix, which is composed of the one-dimensional feature vectors of the target domain training set images obtained in step 3. The superscript t indicates that it corresponds to the target domain image.

[0049] 5. Calculate the pedestrian features of the i-th pedestrian image in the target domain training set in the hybrid memory matrix and the pedestrian features of the jth pedestrian image The Euclidean distance d i,j , calculate all N t The Euclidean distance between the pedestrian features in the pictures is used to obtain the pedestrian feature distance matrix

[0050] 6. Based on the distance matrix in step 5, perform the following process: Set the parameter eps of standard density clustering to 0.6, and t Pedestrian features of Zhang's target domain training set images in the hybrid memory matrix Perform clustering to obtain the annotations for each image in the target domain training set Set the parameter eps=0.58 for strict density clustering. Perform the second clustering to get the label Set the parameter eps of relaxed density clustering to 0.62. Perform the third clustering to get the label

[0051] 7. According to Y t-standard Obtain the jth pedestrian image in the target domain training set that belongs to the same category as the i-th pedestrian image. t-loose ,Y t-tight Determine whether the two images belong to the same category in the second and third clustering. If not, classify the two images as outliers.

[0052] 8. Pseudo-label images belonging to outliers Where N o Indicates the number of outliers. For other images in the target domain training set, pseudo labels are marked according to the clustering results. Finally, the pseudo labels of the target domain training set images are obtained.

[0053] 9. Use the pseudo labels of the target domain training set and the real labels of the source domain data to supervise the model composed of the CNN encoder and the interactive cascade lightweight transformers, use formula (7) to calculate the loss value, and use the Adam gradient descent optimization algorithm to update the network parameters.

[0054]

[0055] Among them, w k 、c k and v k They represent the source domain class centroid, the target domain cluster centroid, and the target domain outlier instance features respectively. s , and They represent the number of source domain classes, target domain clusters, and target domain outliers respectively. f is the pedestrian feature output from step 3. + represents the positive prototype corresponding to f. τ is called the temperature coefficient, which is a constant. exp represents the exponential function with the natural constant e as the base,<a,b> Indicates calculating the Euclidean distance between a and b.

[0056] 10. Iterate steps 1 to 9 L times, and L in this patent is set to 50.

[0057] Beneficial effects of the present invention: The lightweight transformers designed by the present invention utilize the transformer to capture long-distance dependencies while avoiding the problem of aggravated noise label influence caused by the transformer's weak generalization ability. The interactive cascade structure proposed by the present invention gives full play to the interaction between local information modeling and global information modeling, and effectively enhances the model's ability to express pedestrian features. The present invention has achieved outstanding performance in domain adaptive pedestrian re-identification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of the pedestrian re-identification task;

[0059] Figure 2 It is the structure of CNN encoder;

[0060] Figure 3 It is a structure of interactive cascaded lightweight transformers;

[0061] Figure 4 This is a flowchart of domain adaptation (DA-ICLT) training based on interacting cascaded lightweight transformers. DETAILED DESCRIPTION

[0062] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0063] The present invention provides a pedestrian re-identification method based on domain adaptation and interactive cascade lightweight transformers. The implementation of the present invention is described in detail below with reference to the accompanying drawings, which specifically includes the following steps:

[0064] 1. Pedestrian re-identification task. The application of the present invention does not limit the source of pedestrian image data. The present invention will take the datasets Market-1501 (Zheng L, Shen L, Tian L, et al. Scalable person re-identification: A benchmark [C] / / Proceedings of the IEEE international conference on computer vision. 2015: 1116-1124.) and DukeMTMC-reID (RISTANI E, SOLERA F, ZOU R, et al. Performance measures and a data set for multi-target, multi-camera tracking [J]. Lecture Notes in Computer Science, 2016, 9914LNCS (c): 17–35.) as examples. Market-1501 consists of 32,668 images of 1,501 pedestrians taken by 6 cameras in summer. The training set contains 12,936 images of 751 pedestrians, the retrieval library contains 3,368 images of another 750 pedestrians, and the pedestrian library contains another 10,164 images of these 750 people. DukeMTMC-reID consists of 36,411 images of 1,404 pedestrians taken by 8 cameras in winter. The training set contains 16,522 images of 702 pedestrians, the retrieval library contains 2,228 images, and the pedestrian library contains 17,661 images. The pedestrian re-identification task is to find the image of the pedestrian to be retrieved in the pedestrian library, such as Figure 1 In this example, we use the Market-1501 dataset as the target domain and the DukeMTMC-reID dataset as the source domain to train the model.

[0065] 2. Use CNN encoder to extract compact features from pedestrian images.

[0066] The structure of the CNN encoder is as follows Figure 2As shown, it consists of convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 of ResNet-50 (He, Kaiming, et al. "Deepresidual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778.). The CNN encoder input dimension is a pedestrian image of {3, 256, 128}, and the CNN encoder output dimension is a feature map of {1024, 8, 4}. This feature map will be used as the input of the interactive cascade lightweight transformers.

[0067] 3. Use interactive cascaded lightweight transformers to extract recognizable pedestrian features.

[0068] The structure of the interactive cascade lightweight transformers is as follows Figure 3 As shown in the figure. It consists of 3 basic units, each of which is composed of a layer of local lightweight transformer and two layers of global lightweight transformers cascaded. Local lightweight transformer and global lightweight transformer are two types of lightweight transformers. The components of local lightweight transformer and global lightweight transformer are the same, but the execution process is different. They are both composed of embedding module, multi-head self-attention module and multi-head mixing module.

[0069] The process performed by the interactive cascade lightweight transformers consists of a sub-process performed by 1 layer of local lightweight transformers and a sub-process performed by 2 layers of global lightweight transformers.

[0070] The interactive cascade of lightweight transformers performs the following process:

[0071] (3.1) The 1-layer local lightweight transformer performs the following sub-processes:

[0072] (3.1.1) The feature map of the input local lightweight transformer is cut into 8 query blocks, and f1, f2, ... f8 are obtained. Their dimensions are all {1024, 2, 2}, and the subscripts are the query block numbers. These 8 query blocks are obtained by cutting the input feature map horizontally 4 times and vertically 2 times.

[0073] (3.1.2) Use the embedding module to compress the number of channels of the 8 query blocks, and obtain 8 compressed query blocks e1, e2, …, e8, whose dimensions are all {512, 2, 2}, and the subscript is the sequence number of the query block. The embedding module is a layer of convolutional neural network, which consists of 512 convolution kernels of size 2×2 and its stride is 1.

[0074] (3.1.3) The eight query blocks e1,…,e i ,…,e8 are expanded into 8 two-dimensional matrices s1,…,s i ,…,s8. The expansion operation is to expand the last two dimensions of the query block 2×2 into one dimension. The subscript of is the serial number of the query block it corresponds to, where and They are all one-dimensional vectors with a dimension of 512, and the superscript is the sequence number of the one-dimensional vector.

[0075] (3.1.4) Based on s i The multi-head self-attention module is used for information modeling. The process is as follows:

[0076] It consists of three mapping modules W Q , W K and W V Get the query matrix q i , the key matrix k i Sum value matrix v i :

[0077] q i =W Q s i ;k i =W K s i ;v i =W V s i (3)

[0078] W in the formula Q ,W K and W V There are three fully connected layers, and the number of neurons in each layer is 512×512; i , k i and v i They are both two-dimensional matrices with a dimension of 512×4, and the subscript i is the serial number of the corresponding query block.

[0079] Q i ,k i and v iDivide into 8 groups of query matrices, key matrices and value matrices respectively. Each group of query matrices, key matrices and value matrices is a two-dimensional matrix with a dimension of 64×4, where 8 is the number of heads in the multi-head self-attention mechanism. Cross-multiply the transpose of the key matrix in each group with the query matrix to obtain 8 groups of similarity values ​​for each pixel in the query block. Each group of similarity values ​​is stored in a two-dimensional matrix with a dimension of 4×4. Normalize each group of similarity values ​​to obtain 8 groups of weights, all of which are two-dimensional matrices with a dimension of 4×4. Cross-multiply the 8 groups of value matrices with the weights of the corresponding groups respectively. The meaning of this operation is to aggregate each one-dimensional vector in the value matrix according to the similarity between the corresponding pixels. This operation produces 8 groups of aggregation results. in, is a two-dimensional matrix with a dimension of 64×4, the subscript i is the serial number of the corresponding query block, and the superscript m=1,2,…,8 is the group number.

[0080] (3.1.5) 8 sets of two-dimensional matrices Recombine into a two-dimensional matrix with a dimension of 512×4, and fold the merged two-dimensional matrix into a query block u i , its dimension is {512,2,2}, and the subscript i is the number of the query block. This folding operation only folds the last one-dimensional matrix of length 4 into a two-dimensional matrix of 2×2.

[0081] (3.1.6) Will u i It is sent to the multi-head mixing module to mix the attention results of different heads (groups) to obtain a query block r with a dimension of {1024, 2, 2} i , where subscript i is the query block number. The multi-head hybrid module is a transposed convolutional neural network consisting of 1024 convolution kernels of size 1×1 and a stride of 1.

[0082] (3.1.7) The 8 query blocks r i Merge into a feature map r, the dimension of r is {1024,8,4}.

[0083] (3.1.8) Mix the feature map r with the input feature map F:

[0084] F local =F+r (4)

[0085] Where F local The dimension of is {1024,8,4}, and the superscript local is used to indicate that it is the output of the local lightweight transformer, which will be used as the input of step (3.2).

[0086] (3.2) The two-layer global lightweight transformer performs the following sub-processes:

[0087] (3.2.1) The embedding module is used to compress the number of channels of the input feature map, and the feature map E with a dimension of {512, 8, 4} is obtained. The embedding module is a layer of convolutional neural network, which consists of 512 convolution kernels of size 1×1 and its step size is 1.

[0088] (3.2.2) Expand the feature graph E into a two-dimensional matrix S = [S 1 ,S 2 ,…,…S 32 ]. The expansion operation simply expands the last two dimensions of the feature map (8×4) into one dimension. S consists of 32 one-dimensional vectors, and its dimension is 512×32.

[0089] (3.2.3) Based on S, the multi-head self-attention module is used to perform information modeling. The process is as follows:

[0090] It consists of three mapping modules W Q , W K and W V Get the query matrix Q, key matrix K and value matrix V:

[0091] q i =W Q Q;k i =W K K;v i =W V V (8)

[0092] W in the formula Q ,W K and W V There are three fully connected layers, and the number of neurons in each layer is 512×512; Q, K and V are all two-dimensional matrices with a dimension of 512×32.

[0093] Divide Q, K and V into 8 groups of query matrices, key matrices and value matrices respectively. Each group of query matrices, key matrices and value matrices is a two-dimensional matrix with a dimension of 64×32, where 8 is the number of heads in the multi-head self-attention mechanism. Cross-multiply the transpose of the key matrix in each group with the query matrix to obtain 8 groups of similarity values ​​for each pixel in the query block. Each group of similarity values ​​is stored in a two-dimensional matrix with a dimension of 32×32. Normalize each group of similarity values ​​to obtain 8 groups of weights, all of which are two-dimensional matrices with a dimension of 32×32. Cross-multiply the 8 groups of value matrices with the weights of the corresponding groups respectively. The meaning of this operation is to aggregate each one-dimensional vector in the value matrix according to the similarity between the corresponding pixels. This operation produces 8 groups of aggregation results w 1 ,…,w m ,…,w 8 Among them, w m is a two-dimensional matrix with a dimension of 64×32, and the superscript m=1,2,…,8 is the group number.

[0094] (3.2.4) The 8 sets of two-dimensional matrices w m The two-dimensional matrix is ​​recombined into a 512×32 two-dimensional matrix, and the merged two-dimensional matrix is ​​folded into a query block u, whose dimension is {512,8,4}. This folding operation only folds the last dimension 32 of the matrix into two-dimensional 8×4.

[0095] (3.2.5) u is sent to the multi-head mixing module to mix the attention results of different heads (groups) to obtain a query block r with a dimension of {1024,8,4}. The multi-head mixing module is a transposed convolutional neural network consisting of 1024 convolution kernels of size 1×1 and a stride of 1.

[0096] (3.2.6) Using the result of (3.2.5) as input, iterate steps (3.2.1) to (3.2.5) once.

[0097] (3.3) Take the result of (3.2.6) as input and iterate step (3.1) to step (3.2) twice. Finally, a feature map with a dimension of {1024,8,4} is obtained. This feature map will be used as the input of step 4.

[0098] 4. Perform global average pooling on the input feature map to obtain a new feature map with a dimension of {1024, 1, 1}, and then reduce the dimension of the new feature map to obtain a one-dimensional feature vector with a dimension of 1024. This one-dimensional feature vector will be used as the input of step 5.

[0099] 5. The one-dimensional feature vectors of all pedestrian images in the target domain training set form a two-dimensional pedestrian feature matrix F t , its dimension is 1024×12936. Among them, 12936 is the number of pedestrian images in the target domain training set. The hybrid memory matrix at the mth moment is updated using the features of the pedestrian images in the target domain training set. The hybrid memory matrix is ​​a two-dimensional matrix with a dimension of 1024×12936. The update method is as follows:

[0100]

[0101] 6. Calculate the pedestrian features of the i-th pedestrian image in the target domain training set in the hybrid memory matrix and the pedestrian features of the jth pedestrian image The Euclidean distance d i,j , calculate the Euclidean distance between the pedestrian features of all 12936 images, and obtain the pedestrian feature distance matrix

[0102] 7. Based on the distance matrix in step 6, perform the following process: set the parameter eps of standard density clustering to 0.6, and perform the pedestrian features in the hybrid memory matrix for the total of 12,936 target domain training set images Perform clustering to obtain the annotations for each image in the target domain training set Set the parameter eps=0.58 for strict density clustering. Perform the second clustering to get the label Set the parameter eps of relaxed density clustering to 0.62. Perform the third clustering to get the label

[0103] 8. According to Y t-standard Obtain the jth pedestrian image in the target domain training set that belongs to the same category as the i-th pedestrian image. t-loose ,Y t-tight Determine whether the two images belong to the same category in the second and third clustering. If not, classify the two images as outliers.

[0104] 9. Pseudo-label images belonging to outliers Where N o Indicates the number of outliers. For other images in the target domain training set, pseudo labels are marked according to the clustering results. Finally, the pseudo labels of the target domain training set images are obtained.

[0105] 10. Use the pseudo labels of the target domain training set and the real labels of the source domain training set to supervise the model composed of the CNN encoder and the interactive cascade lightweight transformers, use formula (7) to calculate the loss value, and use the Adam gradient descent optimization algorithm to update the network parameters.

[0106]

[0107] Among them, w k 、c k and v k They represent the source domain class centroid, the target domain cluster centroid, and the target domain outlier instance features respectively. s , and They represent the number of source domain classes, target domain clusters, and target domain outliers respectively. f is the pedestrian image feature output from step 4. + represents the positive prototype corresponding to f. τ is called the temperature coefficient, which is a constant. exp represents the exponential function with the natural constant e as the base,<a,b> Indicates calculating the Euclidean distance between a and b.

[0108] 11. Iterate steps 2 to 10 50 times. The entire domain adaptation training process is as follows Figure 4 shown.

Claims

1. A domain adaptive person re-identification method based on interactive cascaded lightweight transformers, characterized in that: Here are the steps: (1) Extracting compact features from pedestrian images using a CNN encoder The CNN encoder consists of several convolutional neural network layers, and its output dimension is The feature map is a , where height is the height of the pedestrian image and width is the width of the pedestrian image; this feature map will be used as the input of the interactive cascade lightweight transformers; (2) Extracting identifiable pedestrian features using interactive cascaded lightweight transformers The interactive cascade lightweight transformers are composed of three basic units, each of which is composed of a layer of local lightweight transformers and two layers of global lightweight transformers cascaded; the local lightweight transformers and the global lightweight transformers have the same component modules but different execution processes; they are both composed of an embedding module, a multi-head self-attention module, and a multi-head mixing module; The process performed by the interactive cascade lightweight transformers consists of a sub-process performed by 1 layer of local lightweight transformers and a sub-process performed by 2 layers of global lightweight transformers; The interactive cascade of lightweight transformers performs the following process: (2.1) The 1-layer local lightweight transformer performs the following sub-processes: (2.1.1) Cut the feature map into query blocks: Where F is the feature map of the input local lightweight transformers; f i represents the query block, and the subscript i is the sequence number of the query block; N q Indicates the number of query blocks; divide() indicates the operation of cutting the feature map into query blocks; N q =a×b, where a and b represent the number of times F is cut horizontally and the number of times F is cut vertically in the process of cutting the feature graph F into query blocks; the query block f i The dimension of (2.1.2) Use the embedding module to compress the query block f i The number of channels is obtained by querying the block e i ;e i The dimension of is Where d < 1024; the embedding module is a 1-layer convolutional neural network; (2.1.3) query block e i Expand into a two-dimensional matrix s consisting of multiple one-dimensional vectors i : s i =unfold(e i ) (2) In the formula, unfold() represents the operation of expanding the query block into a two-dimensional matrix. This expansion operation is just to expand the last two dimensions of the query block. Expanded into one dimension Represents query block e i The expanded two-dimensional matrix has a dimension of d×N, where s i The subscript i of is the serial number of the query block it corresponds to; Indicates i The j-th one-dimensional vector in The dimension of is d; (2.1.4) Based on s i The multi-head self-attention module is used for information modeling. The process is as follows: It consists of three mapping modules W Q , W K and W V Get the query matrix q i , key matrix k i Sum value matrix v i : q i =W Q s i ;k i =W K s i ;v i =W V s i (3) Where W Q , W K and W V There are three fully connected layers, and the number of neurons in each layer is d×d; i , k i and v i They are all two-dimensional matrices with a dimension of d×N, and the subscript i is the serial number of the corresponding query block; Q i , k i and v i Divide into H groups of query matrix, key matrix and value matrix respectively, where H is the number of heads in the multi-head self-attention mechanism; cross-multiply the transpose of the key matrix in each group with the query matrix to obtain H groups of similarity values ​​for each pixel in the query block; normalize each group of similarity values ​​to obtain H groups of weights; cross-multiply the H groups of value matrices with the weights of the corresponding groups respectively. The meaning of this operation is to aggregate each one-dimensional vector in the value matrix according to the similarity between the corresponding pixels. This operation produces H groups of aggregated results in, The dimension is A two-dimensional matrix, where the subscript i is the serial number of the corresponding query block, and the superscript m=1, 2, ..., H is the corresponding group number; (2.1.5) H sets of two-dimensional matrices Re-merge into a two-dimensional matrix, and fold the merged two-dimensional matrix into a query block u i , its dimension is The subscript i is the serial number of the query block; this folding operation simply folds the last dimension of the matrix into two dimensions; (2.1.6) Will u i Send it to the multi-head mixing module to mix the attention results of different heads, and the dimension is The query block r i , the subscript i is the serial number of the query block; the multi-head hybrid module is a 1-layer transposed convolutional neural network; (2.1.7) N q query block r i Merge into one feature map r, the dimension of r is (2.1.8) Mix the feature map r with the input feature map F: F local =F+r (4) In the formula, F local The dimension of is The superscript local is used to indicate that it is the output of the local lightweight transformer, which will be used as the input of step (2.2); (2.2) The 2-layer global lightweight transformers perform the following sub-processes: (2.2.1) Using the embedding module to compress the number of channels of the input feature map, the dimension is Characteristic map E, where d < 1024; (2.2.2) Expand the feature map E into a two-dimensional matrix: S=UNFOLD(E) (5) In the formula, UNFOLD() represents the operation of expanding the feature map into a two-dimensional matrix. This expansion operation is just to expand the last two dimensions of the feature map. Expanded into one dimension Represents the two-dimensional matrix of the feature map E, which is composed of One-dimensional vectors, its dimension is d×N S ; (2.2.3) Take S as input, first use the multi-head self-attention module to model the information, then rearrange its output into a three-dimensional feature map, and finally use the multi-head mixing module to mix the attention results of different heads. The process is the same as steps (2.1.4) to (2.1.6); (2.2.4) Using the result of (2.2.3) as input, iterate steps (2.2.1) to (2.2.3) once; (2.3) Take the result of (2.2.4) as input and repeat steps (2.1) to (2.2) twice; the final dimension is The feature map will be used as the input of step (3); (3) Perform global average pooling on the input feature map to obtain a new feature map with a dimension of {1024, 1, 1}, and then reduce the dimension of the new feature map to obtain a one-dimensional feature vector with a dimension of 1024; the one-dimensional feature vector will be used as the input of step (4); (4) At time m, the hybrid memory matrix is ​​updated using the features of the pedestrian images in the target domain training set. It is a matrix with a dimension of N t ×1024 two-dimensional matrix, where N t is the number of pedestrian images in the target domain training set; the update method is as follows: In the formula, μ is a constant used to control the speed of updating the hybrid memory matrix; is the mixed memory matrix at time m; F t The dimension is N t ×1024 two-dimensional matrix, which is composed of the one-dimensional feature vectors of the target domain training set images obtained in step (3), and the superscript t indicates that it corresponds to the target domain image; (5) Calculate the pedestrian features of the i-th pedestrian image in the target domain training set in the hybrid memory matrix and the pedestrian features of the jth pedestrian image The Euclidean distance d i，j , calculate all N t The Euclidean distance between the pedestrian features in the pictures is used to obtain the pedestrian feature distance matrix (6) Based on the distance matrix in step (5), the following process is performed: set the parameter eps of the standard density clustering to 0.6, and t Pedestrian features of Zhang's target domain training set images in the hybrid memory matrix Perform clustering to obtain the annotations for each image in the target domain training set Set the parameter eps=0.58 for strict density clustering. Perform the second clustering to get the label Set the parameter eps of relaxed density clustering to 0.

62. Perform the third clustering to get the label (7) According to Y t-standard Obtain the jth pedestrian image in the target domain training set that belongs to the same category as the i-th pedestrian image; according to Y t-loose , Y t-tight Determine whether the two images belong to the same category in the second and third clustering; if not, classify the two images as outliers; (8) Pseudo-label images belonging to outliers Among them, N o Indicates the number of outliers; for other images in the target domain training set, pseudo labels are marked according to the clustering results; finally, the pseudo labels of the target domain training set images are obtained (9) Use the pseudo labels of the target domain training set and the real labels of the source domain data to supervise the model composed of the CNN encoder and the interactive cascade lightweight transformers, use formula (7) to calculate the loss value, and use the Adam gradient descent optimization algorithm to update the network parameters; Among them, w k 、c k and v k Respectively represent the source domain class centroid, the target domain cluster centroid and the target domain outlier instance features; ns, and They represent the number of source domain classes, target domain clusters and target domain outliers respectively; f is the pedestrian feature output from step (3); z + represents the positive prototype corresponding to f; τ is called the temperature coefficient, which is a constant; exp() represents the exponential function with the natural constant e as the base,<a,b> Indicates calculating the Euclidean distance between a and b; (10) Iterate steps (1) to (9) L times.

Images