Shallow-deep feature fusion method and system based on token reorganization model

Abstract

The application belongs to the field of occluded pedestrian re-identification, and particularly relates to a shallow-deep feature fusion method and system based on a token reorganization model. The method comprises a token reorganization module based on an attention matrix and a shallow-deep feature learning module. The token reorganization module based on the attention matrix comprises a token screening module based on the attention matrix and a token replacement module of target relevance. The token screening module based on the attention matrix uses a classification attention selection strategy and an interactive attention strategy to divide all tokens in an image into target tokens and irrelevant tokens according to whether the tokens contain valid information. The target tokens are fed into a subsequent converter for interaction to extract important information areas in the image. The token replacement module of target relevance reconstructs the screened irrelevant tokens into target tokens. The application has carried out rich experiments on various public data sets and obtained competitive results through quantitative evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of occluded pedestrian re-identification, and particularly relates to a shallow-deep feature fusion method and system based on a Token reorganization model. BACKGROUND

[0002] Pedestrian re-identification aims to match pedestrians across cameras, which can effectively reduce the workload of workers. Due to its research value and application prospect, it has attracted widespread attention of scientific and technological workers. With the development of deep learning, satisfactory retrieval ability has been achieved on large-scale overall pedestrian re-identification datasets. However, in real scenes, pedestrians will be occluded by various objects, and the information of the occluded part will seriously damage the integrity of the pedestrian information, making it difficult to perform the pedestrian matching task.

[0003] The main challenge of occluded pedestrian re-identification is that the effective information in the feature representation extracted by the network is greatly reduced due to the occlusion of the discriminative body area by the occlusion. In early research, researchers focused on manually extracting texture and color features. However, in real scenes, there are various postures of pedestrians, complex backgrounds, and different scales, and traditional methods often have limitations. With the development of deep learning, a large number of works use convolutional neural networks to extract features from raw images. In order to make the extracted features have rich semantic information, documents use multiple convolutional layers to replace shallow networks to deepen the network to extract global features. However, this cannot solve the interference caused by the background and the occluded area, so more and more researches on local features are carried out. They apply a simple horizontal segmentation method to the feature map to roughly divide the background or occluded area and the pedestrian area, and then feed the pedestrian part (such as the head, torso, and lower body) to the subsequent network, which to some extent alleviates the influence of occlusion on pedestrian features. However, due to the different proportions of pedestrians in the image, the use of such hard segmentation methods will cause alignment and segmentation accuracy problems. In order to alleviate the above problems, people obtain prior information through external models (such as pose estimation models and human parsing models) to perform more accurate local feature segmentation. The key points simulated by the pose estimation model are integrated with the feature map, which indirectly eliminates some occlusion noise information, accurately divides the visible area features, and predicts them respectively, and the performance is improved to a certain extent. However, these methods ignore the accuracy of the prior information predicted by the external model and the domain gap problem of the data. The above research schemes focus on weakening the influence of occlusion noise, but do not consider the relevance of the visible area and the occluded area of the pedestrian, thus limiting the performance of pedestrian re-identification.

[0004] In the current study, attention mechanism has attracted extensive attention, which is embedded into the network layer to perceive important areas. With the visual transformer (ViT) as a milestone work of the application of Transformer in the field of CV, it also triggered subsequent related research. TransReID takes ViT as the backbone network for the pedestrian re-identification task, and introduces the view coding (camera ID and viewing angle information) on the encoding mechanism. The jigsaw patch module (JPM) uses shift and shuffle operation to extract local features with rich semantic information, and makes significant progress. Transformer improves the global interaction features. However, when it is used in the pedestrian re-identification task, some problems still need to be considered. The standard ViT model divides the image into image blocks, which are input into the network after flattening, and the global interaction between image blocks is carried out in the multi-head attention mechanism. However, when ViT is directly applied to the problem of pedestrian re-identification occlusion, there are two main problems to be solved. Firstly, since the ViT processes these images, the model processes all image block tokens at one time, and the image in the pedestrian re-identification occlusion dataset often contains complex background and various occlusions. The model cannot effectively capture the attention area with strong discriminative ability in the image block token, and often produces a large amount of useless calculation, which reduces the training efficiency. Secondly, all tokens in ViT are fully interacted, and with the increase of the number of stacked encoders, the similarity between tokens gradually increases. In the deep layer, only rich semantic information is contained, but discriminative information such as color, texture and contour is lacking, which leads to the inability to further improve the performance. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a shallow-deep feature fusion method and system based on a Token reorganization model, which effectively eliminates the influence of various occlusions.

[0006] The technical scheme adopted by the present application to solve the technical problems is:

[0007] The application proposes a shallow-deep feature fusion framework based on token reorganization model for pedestrian re-identification research. Unlike TransReID which processes all tokens at once, the application constructs a token reorganization module based on attention matrix, divides all tokens in the image into target tokens and irrelevant tokens according to whether they contain valid information, feeds the target tokens into the subsequent converter for interaction to extract important information areas in the image. Then several replacement strategies are developed to reconstruct the original features, filter out the tokens that contribute to identification and feed them into the subsequent encoder to reduce the interference of noise information, and enrich the target pedestrian feature representation through feature reconstruction. In addition, due to the layer-by-layer stacking of ViT, the lack of inter-layer information interaction makes the model not robust enough. A shallow-deep feature learning module is proposed based on the global interaction characteristics of Transformer, which makes the shallow features and deep semantic features interact through an efficient cross-attention module, improves the defect that the inter-layer features are not fully utilized, and fuses the converter inter-layer features to enrich the inter-layer information interaction, thereby establishing a powerful model.

[0008] Further, the image processing process of the token screening module based on the attention matrix is:

[0009] First, give the input image x of the person H×W×D , where H, W and D represent its height, width and channel number respectively, divide x into overlapping patch blocks, and flatten it into a sequence N and P represent the number and size of the patch block respectively,

[0010] Then concatenate the learnable classification code x cls with the flattened sequence, and the classification code is used as the feature representation of the final output of the network, then add the position encoding to the input sequence, so the input sequence of the Transformer encoder is represented as:

[0011]

[0012] Where Z R (N+1)×C represents the input sequence, Position embedding, l is a linear projection that maps patch to C dimension;

[0013] Then, input all tokens in Z into the multi-head self-attention module, the specific process is as follows:

[0014]

[0015] Where d is the dimension of the query vector, Q R N×C , K R N×C , V R N×C , The result is a square matrix, called attention matrix, which is then linearly combined with the Value matrix to get the final output;

[0016] The input sequence Z ∈ R (N+1)×C of a ViT, where N + 1 is the sequence length, the first token is a classification token x cls and the rest are image tokens as global feature representations.

[0017] Z cls = x cls

[0018]

[0019] The classification token interacts with the image tokens through the attention mechanism in the ViT:

[0020]

[0021] The above formula corresponds to the attention value between the classification token and the image tokens in the first row of the attention matrix in formula (2), where Q cls is the linearly mapped query vector of the classification token, K and V are the linearly mapped key matrix and value matrix of the input sequence, and the combination coefficient α ∈ R 1×N is the attention value of the classification token with respect to all image tokens, and α i can represent the importance of the i-th token.

[0022] Further, the classification attention selection strategy is specifically: in the multi-head self-attention layer, multiple heads perform the calculation of formula (5) in parallel, and there are multiple attention vectors α (h) of the classification token, where h = [1, 2, …, H] is the number of heads in the attention, and the average attention value of all heads is calculated,

[0023]

[0024] Sort from large to small, and use the top-k strategy to select the tokens with the top-k larger attention values as target tokens, and the remaining tokens as irrelevant tokens, which include backgrounds and occlusions, and get the indexes idx obj and idx occ corresponding to the target tokens and irrelevant tokens.

[0025] Further, the interaction attention strategy calculates the interaction attention value γ:

[0026]

[0027] After obtaining the interactive attention value γ, the first k tokens with high attention value are selected as target tokens according to the attention value, and the retention rate of the target tokens is set as:

[0028]

[0029] Further, the token replacement module of the target correlation includes three replacement strategies, respectively:

[0030] (1) adjacent token replacement strategy, the tokens around the irrelevant tokens are used to replace the irrelevant tokens;

[0031] (2) target token replacement strategy, the most discriminative target token is used to replace the irrelevant token;

[0032] (3) global target token replacement strategy, all target tokens are used to construct a global target token to replace the irrelevant token.

[0033] Further, the specific method of feature reconstruction is as follows: in the network model training process, the input sequence of a batch is Z∈R B×(N+1)×C , the classification tokens are separated to construct Z img ∈R B×N×C , a feature reconstruction matrix is constructed for each image in the batch

[0034]

[0035] The feature reconstruction matrix is initialized as a full 0 matrix with diagonal line 1, and the feature reconstruction is realized by changing the element value in the matrix.

[0036] Further, the shallow-deep feature learning module is specifically, the deep feature sequence is represented as Z a , the shallow feature is represented as Z b = [Z1; Z2;... Z i ], i is the number of layers of the feature to be fused, and for the shallow feature sequence Z b , the classification tokens are separated and spliced, the classification tokens of the deep feature sequence and the image block tokens are also separated, and cross attention is performed between and , the image block tokens are used as the only query, after performing cross attention, the shallow feature is fused into the deep image block sequence, and the above process is represented as follows:

[0037]

[0038] ​

[0039]

[0040] wherein W k ∈R C×C , W v ∈R C×C is a learnable linear mapping function, is a scale factor, after performing cross-attention, the output of the shallow-deep feature learning module with layer normalization and residual connection is fed into the MLP layer:

[0041]

[0042]

[0043] After cross-attention fusion, the image block token of the deep feature will pass the information learned from the shallow feature to the classification token of the self layer in the next layer encoder.

[0044] Further, the overall objective function can be written as:

[0045]

[0046]

[0047]

[0048] wherein F bn is the output after the BN layer, W is a linear projection matrix, y i is the corresponding label, C is the total number of identities, and u is a hyperparameter that adjusts the distance between positive and negative samples.

[0049] The application further provides a system for implementing the above method, comprising:

[0050] The token reorganization module based on the attention matrix, wherein the token reorganization module based on the attention matrix comprises a token screening module based on the attention matrix and a token replacement module related to the target, the token screening module based on the attention matrix uses a classification attention selection strategy and an interaction attention strategy to divide all tokens in the image into target tokens and irrelevant tokens according to whether they contain valid information, the target tokens are fed into the subsequent converter for interaction to extract important information areas in the image, and the token replacement module related to the target reconstructs the screened irrelevant tokens into target tokens.

[0051] And the shallow-deep feature learning module, after splicing the image tokens of the deep feature as the query, the classification tokens of the shallow feature as the key and the value, the two perform information interaction, and the MHSA is used to determine the amount of information that the shallow feature needs to fuse.

[0052] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0053] The application has the advantages and positive effects that:

[0054] 1. The application proposes a new framework based on Transformer to overcome the limitations of directly applying ViT to occluded pedestrian re-identification.

[0055] 2. The application proposes a token reorganization module based on attention matrix to accurately locate important information areas and improve the proportion of tokens contributing to the converter.

[0056] 3. The application proposes a shallow-deep feature learning module to integrate features of different levels of Transformer.

[0057] 4. Experimental results show that the proposed framework improves the research of ViT in the field of pedestrian re-identification and effectively eliminates the influence of various occlusions. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The overall framework of the application is shown in the figure;

[0059] Figure 2 The attention matrix omega is shown in the figure;

[0060] Figure 3 The token replacement module based on target correlation is shown in the figure;

[0061] Figure 4 The token replacement process is shown in the figure;

[0062] Figure 5 The shallow-deep feature fusion module is shown in the figure;

[0063] Figure 6 The ablation experiment results of the replacement strategy are shown in the figure;

[0064] Figure 7 The influence of hyperparameters on Rank-1 and mAP on the Occluded-Duke dataset;

[0065] Figure 8 For visualizing the results. DETAILED DESCRIPTION

[0066] The application will be further described in the following specific examples, which are only descriptive and not limiting, and cannot limit the protection scope of the application.

[0067] The overall framework of the application is shown in Figure 1 ViT takes all image patches as tokens, and the application screens target tokens and irrelevant tokens according to the contribution of token pairs to identification through the correlation between tokens, feeds the target tokens to the subsequent encoder for information interaction, prevents the propagation of irrelevant tokens in the subsequent converter, avoids the introduction of more noise by frequently selecting occluded image blocks, and effectively improves the model performance by using only image blocks with discriminability.

[0068] The token screening module based on attention matrix proposed in the application makes some improvements on the basis of ViT.

[0069] Specifically, given an input image of a person x∈R H×W×D , where H, W and D represent its height, width and channel number respectively. x is divided into overlapping patch blocks, which are flattened into a sequence N and P represent the number and size of patch blocks respectively. Then a learnable classification encoding x cls is concatenated with the flattened sequence, and the classification encoding is used as the feature representation of the final output of the network. Then position encoding is added to the input sequence. Thus the input sequence of the Transformer encoder can be represented as:

[0070]

[0071] , where Z∈R (N+1)×C represents the input sequence, represents the position embedding, and is a linear projection that maps patch to C-dimensional.

[0072] Subsequently, all tokens in x are input into a multi-head self-attention module, and the specific process is as follows:

[0073]

[0074] , where d is the dimension of the query vector, Q∈R N×C , K∈R N×C , and V∈R N×C. The result is a square matrix, which is called attention matrix. The attention matrix is then linearly combined with the Value matrix to get the final output.

[0075] The input sequence Z of ViT is R (N+1)×C where N+1 is the length of the sequence, the first token is a classification token x cls As a global feature representation, the rest of the tokens are image tokens

[0076] Z cls = x cls (3)

[0077]

[0078] The classification token interacts with the image tokens through the attention mechanism in ViT:

[0079]

[0080] The above formula corresponds to the attention value between the classification token and the image token in the first row of the attention matrix in formula (2). Where, Q cls is the linearly mapped query vector of the classification token, K and V are the linearly mapped key matrix and value matrix of the input sequence respectively. The combination coefficient α R 1×N (except for the attention value of the classification token itself) is the attention value of the classification token with respect to all image tokens. Since v i from the i-th token in V, the attention value α i (i.e. the i-th element in α) determines how much information the i-th token fuses into the classification token. Therefore, α i can represent the importance of the i-th token.

[0081] Specifically, the present application identifies the importance of the token based on the size of the attention value of the classification token with respect to other image tokens. Therefore, a classification attention selection strategy is proposed, in which multiple heads perform the calculation of formula (5) in parallel in the multi-head self-attention layer. Therefore, there are multiple attention vectors α (h) of the classification token, where h = [1, 2, …, H] is the number of heads in attention, and the present application calculates the average attention value of all heads.

[0082]

[0083] Sort from large to small, and use the top-k strategy to select the tokens with the top-k larger attention values as target tokens, and the remaining tokens as irrelevant tokens, which mainly contain background and occlusions, and get the indexes idx objidx occ .

[0084] However, the classification attention selection strategy may introduce noise information, because the selected target token may gather a large amount of irrelevant information from irrelevant tokens, so that the model cannot effectively distinguish the target token from the irrelevant token. Figure 2 For example, the attention matrix ω in the image token is as follows:

[0085] 1, 2, 3 and 4 in ω represent image tokens, and cls represents a classification token. The first row reflects the importance of the image token to the classification token. As shown in the figure, the importance of image token 3 to the classification token is the largest, and the importance of image token 1 is the smallest. By the classification attention selection strategy, the present application considers that image token 3 is a target token, and image token 1 is an irrelevant token. As shown in the fourth row, the correlation between image token 3 and image token 1 is large, which may cause image token 3 to gather a large amount of irrelevant information in image token 1, and thus noise may be introduced. Therefore, the correlation between image tokens is considered on the basis of the correlation between the classification token and the image token. Specifically, the first column of the attention matrix ω (except the attention value of the classification token itself) is represented as The meaning of is the attention score of the interaction between the context information of the image tokens. Therefore, the present application calculates the interaction attention value γ:

[0086]

[0087] After obtaining the interaction attention value γ, the top-k strategy is used to select the target token with a high attention value. The retention rate of the target token is set as:

[0088]

[0089] The method of the present application does not refer to other additional parameters, so the parameter amount of the model is not increased.

[0090] After the irrelevant token and the target token are screened by the token screening module based on the attention matrix, if the attention on the occlusion and the background area is to be reduced, one simple idea is to remove the irrelevant token, so that the perception of the target pedestrian feature by the model is enhanced. However, directly removing the irrelevant token may damage the integrity of the feature. To effectively solve this problem, the present application reconstructs the irrelevant token into a target token to effectively reduce the influence of the background and the occlusion and enhance the target pedestrian feature.

[0091] Figure 3 The architecture of the token replacement module showing the target correlation is shown. Specifically, after the target token of the input image is selected, the input sequence of the reconstruction stage is:

[0092]

[0093] where i = [1, 2, … N]. The above formula indicates that x obj is the target token, which contains the target pedestrian information in the image, x occ is the irrelevant token, which contains the background and occlusion information in the image. The present application considers feature reconstruction, and the important idea is to replace the irrelevant token. The present application designs three strategies for replacement:

[0094] (1) Neighbor token replacement strategy (NTR): the tokens around the irrelevant token are used to replace the irrelevant token.

[0095] (2) Target token replacement strategy (TTR): the most discriminative target token is used to replace the irrelevant token.

[0096] (3) Global target token replacement strategy (GTTR): all target tokens are used to construct a global target token to replace the irrelevant token.

[0097] The specific method of feature reconstruction is as follows: during the network model training process, the input sequence of a batch is Z ∈ R B×(N+1)×C , the present application separates the classification token to construct Z img ∈ R B×N×C , similar to the operation mode of matrix multiplication, the present application constructs a feature reconstruction matrix for each image in the batch

[0098]

[0099] To reduce the calculation amount, the present application initializes the feature reconstruction matrix to a full 0 matrix with the diagonal line being 1, and realizes feature reconstruction by changing the element values in the matrix. Specifically, the diagonal element is 0, which indicates that the current token needs to be replaced, and is 1, which does not replace. The column element determines which tokens are used for reconstruction. Figure 4 The specific token replacement process is shown.

[0100] where the index value is idx occThe irrelevant tokens of the matrix are assigned values of 0 at the corresponding positions on the diagonal of the matrix, and the values of the column elements are adjusted according to different replacement strategies. In the process of feature reconstruction, not only the influence of noises such as background and occlusions on the target pedestrian features is reduced, but also the representation of the global features of pedestrians is enhanced, so that the final output of the model is more discriminative.

[0101] Since each level of the network model contains features of different properties, shallow network features have more color, contour and other information, can capture more details, but have less semantic information and more noise. Deep features contain more abstract information, i.e. semantic information. The fusion of low-level features and high-level semantic features enhances feature propagation, making the network easier to train. Therefore, the features at each level of the model interact sufficiently, which can effectively improve the expression ability of the model. The present application designs a shallow-deep feature learning module based on cross attention (Cross-Attention) to enhance the information interaction ability between the level features of the Transformer. CrossViT aims to fuse information into classification tokens, while the shallow-deep feature learning module of the present application takes image blocks as an important carrier for information interaction. Since the number of layers is set from shallow to deep, the feature transmission between levels can be more effective.

[0102] Specifically, in order to effectively fuse level features, the present application uses the image tokens of deep features as query after splicing, and the classification tokens of shallow features as key and value, and the two perform information interaction. The amount of information that shallow features need to fuse is determined by MHSA. Since the classification tokens of shallow features have captured the abstract information of their own layer, interacting with the image tokens of deep features helps to introduce shallow feature information.

[0103] Figure 5 The shallow-deep feature learning module is shown. Specifically, the deep feature sequence is represented as Z a , and the shallow feature sequence is represented as Z b =[Z1; Z2;... Z i ], i is the number of features to be fused. For the shallow feature sequence Z b , the present application separates the classification tokens and splices them, as shown in equation (11).

[0104]

[0105] wherein, represents the class token in the shallow feature participating in fusion. The classification tokens and image block tokens of the deep feature sequence are also separated, and are spliced at and Cross attention is performed in the middle. The image block token contains the information of all image blocks in its own layer, which is an important carrier in the process of attention interaction. The image block token is taken as the unique query, so after the cross attention is performed, the shallow layer features can be fused into the deep layer image block sequence. The above process is represented as follows:

[0106]

[0107] where W q ∈R C×C , W k ∈R C×C , W v ∈R C×C is a learnable linear mapping function, is a scale factor. After the cross attention is performed, it is fed into the MLP layer to enhance the nonlinearity of neurons. Thus, the output of the shallow-deep layer feature learning module with layer normalization and residual connection is:

[0108]

[0109]

[0110] After the cross attention fusion, the image block token of the deep layer feature can pass the information learned from the shallow layer feature to the classification token in its own layer in the next layer encoder, so as to enhance the feature multi-scale richness.

[0111] F' = MSA (LN (Z y ))+ Z y (15)

[0112] F = MLP (LN (F))+ F (16)

[0113] Z' f is the network output feature, and the classification token thereof is used as the final feature representation.

[0114] The present application selects the cross entropy loss (Cross Entropy, CE) as the identity loss to train the classifier, and the triplet loss (Triplet Loss) narrows the distance between the same pedestrian samples and expands the distance between different pedestrians to achieve the clustering effect. The present application uses the batch normalization layer (Batch Normalization, BN) to construct the feature input classification layer, which more effectively integrates the cross entropy loss and the triplet loss to achieve better performance.

[0115]

[0116] where F bn is the output after the BN layer, W is a linear projection matrix, and yi is the corresponding label, C is the total number of identities.

[0117]

[0118] u is a hyper-parameter that adjusts the distance between positive and negative samples.

[0119] The overall objective function can be written as:

[0120]

[0121] The proposed shallow-deep feature fusion framework based on token reorganization model is evaluated on 5 large-scale data sets, and extensive ablation experiments are carried out, and the data sets include Market-1501, DukeMTMC-ReID, Occluded-Duke, Partial-ReID, Partial-iLIDS.

[0122] The Market-1501 data set is captured by 6 cameras and contains 32668 images of 1501 identities, and the images of each identity are captured by at least two cameras, and the entire data set is divided into a training set and a test set according to certain rules, wherein the training set contains 12936 images of 751 identities, and the test set contains 19732 images of 750 identities.

[0123] The DukeMTMC-ReID data set is a pedestrian re-identification subset of the DukeMTMC data set, which is a large-scale personal identity data set, recorded by 8 cameras, containing 36411 images of 1812 identities, specifically, 16522 images of 702 identities are randomly selected from the data set, and the remaining images are divided into a query set containing 2228 images and a gallery set containing 17661 images.

[0124] The Occluded-Duke data set is the largest occluded ReID data set to date, and is also the closest to the real scene, which is captured by 8 cameras, containing 15618 training set images of 708 identities, 2210 query set images of 519 identities and 17661 gallery set images of 519 identities, in order to better simulate the real scene, all images in the query set are occluded by various occlusions, and there are also a small part of this case in the gallery set.

[0125] Partial-ReID is the first data set applied to partial pedestrian re-identification, which contains 800 images of 60 identities, and each identity has 5 whole images, 5 partial images and 5 occluded images, these images have different viewpoints, backgrounds and occlusions.

[0126] Partial-iLIDS is a simulated partial pedestrian re-identification dataset based on iLIDS dataset, which is captured by non-overlapping cameras in the airport, containing 476 images of 119 identities,

[0127] The present application uses two popular performance evaluation methods in the field of pedestrian re-identification, namely Cumulative Matching Characteristics (CMC) and Mean Average Precision (mAP), to evaluate the performance of various pedestrian re-identification models. The CMC curve shows the probability of the query pedestrian appearing in different sizes of candidate lists, and the mAP index reflects the arrangement of the query pedestrian in all gallery images.

[0128] The proposed shallow-deep feature fusion framework based on token recombination model is implemented on the Pytorch 1.11.0 platform and uses a single RTX 3060 GPU for training and testing. The present application uses ViT as the basic backbone network, divides the input image into image blocks, then flattens them into image block sequences, and embeds classification tokens and position encodings for the pedestrian re-identification task. Specifically, ViT is pre-trained on the ImageNet-2k dataset. The present application uniformly adjusts the input image size to 256x128, and in the training stage, data augmentation is used to enrich data diversity before feeding the image into the network, and the image is enhanced by random horizontal inversion, image padding, random cropping, and random erasing. The present application sets the batch size to 32, and each batch contains 4 images of the same identity. Following the standard setting of ViT-base, the depth of the backbone network is L=12 layers, including m standard Transformer Layers and n Token recombination modules based on attention matrix, m and n are set to 9 and 3 respectively. The target token retention rate in the Token recombination module based on attention matrix layer is set to 0.7 to select tokens with high attention response. The network is optimized by the SGD optimizer with a momentum of 0.9 and a weight decay of 1e-4. The network training epoch is set to 160, the learning rate is initialized to 0.008 and the warm up learning strategy is used, the learning rate is linearly enhanced before epoch=5, and then the learning rate is cosine decayed. The present application found that the 160 epoch model is sufficient for convergence.

[0129] In order to fully demonstrate the performance of the method of the present application, Table 1 shows the experimental results of the shallow-deep feature fusion framework based on the token reorganization model in the Occluded-Duke dataset evaluation, and the present application is compared with four groups of mainstream occluded pedestrian re-identification methods. The first group is the research method of overall pedestrian re-identification, the second group is the method of using external models to obtain semantic information, the third group is the method of occluded data enhancement, and the fourth group is the method based on Transformer. Compared with the Adver Occluded method in the first group, the present application improves the rank-1 and mAP indicators by 24.8% and 28.1% respectively, which shows that the simple hard-cutting and alignment method cannot effectively solve the influence of occlusion, and the method proposed in the present application uses the correlation between image blocks to explore the irrelevant areas of the target and shows potential in solving the problem of occlusion interference. Compared with the most advanced method PAFM in the second group, the rank-1 and mAP indicators are improved by 14.2% and 18.0% respectively, which shows that the method of the present application can effectively alleviate the occlusion interference to accurately identify the target pedestrian without the help of external semantic information and additional clues. The third group of methods can effectively enrich the diversity of data and locate the non-occluded area, and compared with them, the model of the present application achieves excellent performance. The proposed method achieves a rank-1 indicator of 69.3% on the Occluded-Duke dataset, and compared with the advanced method AAFormer, the rank-1 indicator is improved by 2.3%. Traditional Transformer-based pedestrian re-identification research mainly captures the global feature correlation between long sequences, without considering that not all tokens in the long sequence have meaning, and some tokens contain background information and occlusion information. Based on these tokens, global interaction will appear computational redundancy and will introduce more noise information. Since the image usually contains image tokens with less information and little contribution to pedestrian re-identification, the shallow-deep feature fusion framework based on the token reorganization model proposed in the present application enables the Transformer to effectively alleviate the occlusion problem and avoid computational redundancy to enhance the learning of robust features, and the performance on the Occluded-Duke dataset is the best.

[0130] Table 1 Comparison with advanced methods on Occluded-Duke dataset

[0131]

[0132] [2].Wang Z,Zhu F,Tang S,et al.Feature Erasing and Diffusion Networkfor Occluded Person Re-Identification[J].2021.

[0133] [3]. Miao J, Wu Y, Liu P, et al. Pose-Guided Feature Alignment for Occluded Person Re-Identification [C] / / 2019 IEEE / CVF International Conference on Computer Vision (ICCV). IEEE, 2019.

[0134] [5]. Ma Z, Zhao Y, Li J. Pose-guided Inter-and Intra-part Relational Transformer for Occluded Person Re-Identification [J]. 2021.

[0135] [9]. Sun Y, Zheng L, Yang Y, et al. Beyond Part Models: Person Retrieval with Refined Part Pooling (and A Strong Convolutional Baseline) [C] / / European Conference on Computer Vision. Springer, Cham, 2017.

[0136]

[18] . He S, Luo H, Wang P, et al. TransReID: Transformer-based Object Re-Identification:, 10.13140 / RG.2.2.14420.53124 [P]. 2021.

[0137]

[19] . Zhao Y, Zhu S, Wang D, et al. Short Range Correlation Transformer for Occluded Person Re-Identification [J]. 2022.

[0138]

[30] . Yumin Suh, Jingdong Wang, Siyu Tang, et al. Part-Aligned Bilinear Representations for Person Re-identification.

[0139]

[31] . Wang G, Yang S, Liu H, et al. High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification [C] / / 2020.

[0140]

[32] . Yang J, Zhang C, Tang Y, et al. PAFM: pose-drive attention fusion mechanism for occluded person re-identification [J]. Neural Computing and Applications, 2022.

[0141]

[34] . Chen P, Liu W, Dai P, et al. Occlude Them All: Occlusion-Aware Attention Network for Occluded Person Re-ID [C] / / International Conference on Computer Vision. 2021.

[0142]

[35] . Yan C, Pang G, Jiao J, et al. Occluded Person Re-Identification With Single-Scale Global Representations [C] / / International Conference on Computer Vision. 2021.

[0143]

[36] . Jia M, Cheng X, Lu S, et al. Learning Disentangled Representation Implicitly via Transformer for Occluded Person Re-Identification [J]. 2021.

[0144]

[43] . Zhang G, Zhang P, Qi J, et al. HAT: Hierarchical Aggregation Transformers for Person Re-identification [J]. 2021.

[0145]

[44] . Zhu K, Guo H, Zhang S, et al. A former: Auto-Aligned Transformer for Person Re-Identification [J]. 2021.

[0146]

[52] . Zhao L, Xi L, Zhuang Y, et al. Deeply-Learned Part-Aligned Representations for Person Re-identification: IEEE, 10.1109 / ICCV.2017.349 [P]. 2017.

[0147]

[53] . Huang H, Li D, Zhang Z, et al. Adversarially Occluded Samples for Person Re-identification [C] / / 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018.

[0148]

[60] . He L, Liang J, Li H, et al. Deep Spatial Feature Reconstruction for Partial Person Re-identification: Alignment-Free Approach [J]. IEEE, 2018.

[0149]

[61] . Tan H, Liu X, Tian S, et al. MHSA-Net: Multi-Head Self-Attention Network for Occluded Person Re-Identification [J]. 2020.

[0150]

[62] . Zhang X, Yan Y, Xue J H, et al. Semantic-Aware Occlusion-Robust Network for Occluded Person Re-Identification [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, PP(99): 1-1.

[0151]

[63] . Li Y, He J, Zhang T, et al. Diverse Part Discovery: Occluded Person Re-identification with Part-Aware Transformer:, 10.48550 / arXiv.2106.04095 [P]. 2021.

[0152]

[64] . Hou R, Ma B, Chang H, et al. Feature Completion for Occluded Person Re-Identification [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, PP(99): 1-1.

[0153] To further evaluate the effectiveness of the shallow-deep feature fusion framework based on the token reorganization model, the present application performs comparative experiments on two partial data sets, Partial-ReID and Partial-ilIDS. Since the number of pedestrian images in these two data sets is much less than that in the Occluded-Duke, the present application uses the Market-1501 data set as the training set and the occluded images of the Partial-ReID and Partial-iLIDS data sets instead of the cropped images as the test set for evaluation. The Partial-ReID data set focuses on the recognition of body parts of pedestrians, such as arms, torsos, and lower bodies, containing less background information. The present application model focuses on reducing the interference of background and occlusion on the target pedestrian, so different hyperparameters are adopted than Occluded-Duke As can be seen from Table 2, the proposed shallow-deep feature fusion framework based on the token reorganization model is more suitable for solving the occlusion problem of pedestrian re-identification, and the rank-1 and mAP indicators evaluated on the Partial-ReID reach 86.7% and 80.3% respectively. The rank-1 indicator evaluated on the Partial-ilIDS dataset is improved by 0.8% compared with the advanced method SORN, and the converter has a higher overall recognition accuracy, so the mAP indicator is greatly improved, and the SOTA indicator score is comparable to existing methods.

[0154] Table 2 Comparison of advanced methods on Partial-ReID and Partial-iLIDS datasets

[0155]

[0156] In order to verify whether the method of the present application can exhibit generalization ability and robustness on the overall dataset, in the present application, the experimental results of the shallow-deep feature fusion framework based on the token reorganization model on the overall pedestrian re-identification dataset Market-1501 and DukeMTMC-ReID are evaluated, as shown in Table 3.

[0157] In order to better compare, the present application selects two types of mainstream methods, the first type of method based on CNN (PCB, PGFA, HOReID, ISP, SPReID) and the second type of method based on converter (Pirt, PAT, AAFormer, DRL-Net). As can be seen, the pedestrian re-identification research based on the converter essentially interacts all image tokens globally, without considering the negative influence of the tokens of non-pedestrian objects in the converter. Therefore, the method proposed in the present application can pay more attention to the features of pedestrian objects.

[0158] The experimental results are shown in Table 3. On the Market-1501 dataset, the rank-1 score of the shallow-deep feature fusion framework based on the token reorganization model reaches 95.4%, and the mAP is 88.0%, which is 1.2% and 3.1% higher than the traditional CNN-based method HOReID in rank-1 and mAP respectively. The performance is close to the converter method which focuses on overall pedestrian re-identification. This shows that the shallow-deep feature fusion framework based on the token reorganization model not only can effectively solve the occlusion problem, but also exhibits good performance on the overall problem.

[0159] Table 3 Comparison of advanced methods on Market-1501 and DukeMTMC-ReID datasets

[0160]

[0161] The present application evaluates the effectiveness of each module of the proposed shallow-deep feature fusion framework based on the token reorganization model. The present application uses a visual transformer as a baseline and adds a token filtering module based on an attention matrix, a token replacement module for target correlation and a shallow-deep feature learning module. The present application conducts an ablation experiment on the above modules on the Occluded-Duke dataset to evaluate the effectiveness of each module. The ablation experiment results of each module are shown in Table 4. The present application also conducts an ablation experiment on the hyperparameters in the token filtering module based on the attention matrix and the replacement strategy in the token replacement module for target correlation.

[0162] Table 4 Ablation experiment results of each module of the shallow-deep feature fusion framework based on the token reorganization model

[0163]

[0164] Firstly, the present application adds a token filtering module based on an attention matrix (TFM) to the baseline based on a visual transformer. The token filtering module based on an attention matrix aims to filter out target regions and irrelevant regions in the input image, and feed the target regions to the subsequent transformer to make the overall network focus on the regions with discriminative ability, and alleviate the influence of background and occlusions in the image. As shown in the results in Table 4, compared with the baseline method of index 1, the method of index 2 adding the token filtering module based on an attention matrix improves the rank-1 and mAP by 2.8% and 0.3%, respectively. This shows that the token filtering module based on an attention matrix can effectively solve the problem of image complexity in the occlusion problem, and has a positive impact on pedestrian re-identification.

[0165] The token replacement module for target correlation (TRM) is proposed based on the token filtering module based on an attention matrix, aiming to reorganize the image tokens in the features to construct more robust and discriminative features, which can also be simply inserted into other networks with a transformer as the backbone. As can be seen from the comparison between index 2 and index 3, the addition of the token replacement module for target correlation based on the Baseline+TFM improves the rank-1 by 4.9% and the mAP by 5.8%.

[0166] From the above comparison results, it can be seen that the token replacement module for target correlation effectively reduces the influence of occlusion noise on the pedestrian re-identification network, reduces the occlusion features, and at the same time, the token replacement module for target correlation enriches the original features and extracts more discriminative features, i.e. increases the proportion of target pedestrian features in the original features. In other words, the newly constructed features output from the token replacement module for target correlation will contain more target pedestrian features, i.e. discriminative features, which will help to further improve the performance of the transformer network. ​

[0167] The shallow-deep layer feature learning module focuses on the sufficient interaction between the features of each layer of the network to improve the expression ability of the model, thereby enhancing the robustness of the final features. In index 4, the shallow-deep layer feature learning module is added to the baseline, and the experimental results are greatly improved, with the rank-1 and mAP indicators reaching 66.8% and 55.6%, respectively, which are 6.1% and 3.3% higher than those in index 1. As can be seen from the comparison between index 3 and index 5, the network performance reaches the highest when the FSM module is added to the Baseline+TFM+TRM, with the rank-1 and mAP reaching 69.3% and 60.3%, respectively. This shows that with the addition of the shallow-deep layer feature learning module, the inter-layer feature information can be reasonably utilized to enhance the information interaction capability of the transformer. Therefore, the shallow-deep layer feature learning module is effective.

[0168] In order to better reconstruct the features, the present application designs three image token replacement strategies. NTR: replace the irrelevant token with the tokens around the irrelevant token; TTR: replace the irrelevant token with the most discriminative target token; GTTR: replace the irrelevant token with the global target token constructed by all target tokens. The present application conducts detailed ablation experiment analysis on the above three strategies, and the experimental results are shown in Figure 6 As can be seen, when the tokens around the irrelevant token are used to replace the irrelevant token, the rank-1 and mAP indicators reach 66.7% and 57.3%, respectively. The reason is that the combination of the tokens around the irrelevant token may cause misalignment. When strategy 3 is implemented, the network model reaches the highest, with the rank-1 and mAP reaching 69.3% and 60.3%, respectively. When all target tokens are used, each irrelevant token is replaced by the global feature of the target, making the feature more robust and complete. If strategy 2 is used, some discriminative feature representations will be inevitably reduced. Based on the above experimental analysis, strategy 3 is selected for application in the shallow-deep layer feature fusion framework network based on the token reorganization model.

[0169] Since the value of the hyperparameter in the token screening module based on the attention matrix affects the size of the target region and the irrelevant region, the present application conducts comprehensive ablation experiments on the hyperparameter . Figure 7 The experimental results of the value of are shown. As can be seen, when takes the value of 0.1, it means that the present application regards 10% of the region of the original image as the target region, i.e., the region with discriminative features, and the rank-1 and mAP indicators are only 56.2% and 45.4%, respectively. When When the value is 0.9, the original image is regarded as a target area by 90% of the area fed into the subsequent converter layer, and the performance reaches 68.7% and 58.4%, which is 10.3% and 9.4% higher than that of the prior art. When the value is 0.1, rank-1 and mAP are increased by 12.5% and 13.0%, respectively. The reason is that when the input image resolution is large, the irrelevant token often does not contain the target pedestrian features, but when the input image resolution is small, the target pedestrian features are inevitably present in the irrelevant token due to the same size of the segmented image block, and these features often have discriminative characteristics. Ultimately, the present application finds that when When the value is 0.7, the shallow-deep layer feature fusion framework based on the token reorganization model has the best performance on the Occluded-Duke dataset, with rank-1 and mAP being 69.3% and 60.3%, respectively. This shows that the The value of 0.7 is more suitable for the Occluded-Duke dataset. Due to the difference between the datasets, the proportion of the target in the image is different, and the value of the hyperparameter can be adjusted to adapt to a wide range of datasets, which also confirms that the token screening module based on the attention matrix is indeed applicable.

[0170] In order to prove that the shallow-deep layer feature fusion framework based on the token reorganization model not only can obtain excellent performance in terms of occluded pedestrian re-identification accuracy, but also has advantages in terms of time and space complexity.

[0171] The present application and two types of methods are compared in terms of complexity, wherein PGFA, HOReID, RFCnet, and FED are CNN-based methods, and the second type of methods based on Transformer include TransReID and HAT. In order to ensure the fairness of the comparison experiment, all experimental settings are consistent with the settings of the comparison paper, and the results are summarized in Table 5.

[0172] As shown in Table 5, compared with the first type of CNN-based method, the parameter size of the present application model is 58% and 65% of the HOReID and FED models, respectively, and the accuracy evaluated on Market-1501 and Occluded-Duke is higher than that of the first type of method. The present application model has a higher rank-1 index under the condition of similar parameter size and floating point calculation amount compared with TransReID. The parameter amount of the present application method is much smaller than that of the HAT method under the condition of similar performance index. Therefore, the above comparison shows that the framework of the present application is effective and efficient.

[0173] Table 5 Comparison experiment of model complexity with advanced methods

[0174]

[0175] To further verify that the token screening module based on the attention matrix proposed in the application can accurately screen out irrelevant tokens, the application uses idxocc obtained in the token screening module based on the attention matrix to generate a mask of the same size as the input image, and integrates it with the input image, Figure 8 The visual result graph of the application is shown.

[0176] Figure 8 The first row shows the original image, and the second row shows the visual image. The black image block is an irrelevant token. The application observes that most of the irrelevant tokens identified are backgrounds and occlusions, which do not actively contribute to recognition. Since the application adopts an overlapping division strategy to divide the patch block, the image size input into the network after transformation is 256x128, the patch block size in the overlapping division strategy is 16x16, and the step S is 11. Therefore, a small part of the right and lower regions of the image cannot be divided into patch blocks after overlapping division, so this small part is not covered by the black block.

[0177] The application further provides a system for implementing the above method, comprising:

[0178] The token reorganization module based on the attention matrix, wherein the token reorganization module based on the attention matrix comprises a token screening module based on the attention matrix and a token replacement module related to the target, the token screening module based on the attention matrix uses a classification attention selection strategy and an interaction attention strategy to divide all tokens in the image into target tokens and irrelevant tokens according to whether they contain valid information, the target tokens are fed into a subsequent transformer for interaction to extract important information regions in the image; and the token replacement module related to the target reconstructs the screened irrelevant tokens into target tokens.

[0179] And a shallow-deep feature learning module based on the attention matrix, which uses the image token spliced with deep features as the query, the classification token with shallow features as the key and value, and the two for information interaction to determine the amount of information that the shallow features need to fuse through MHSA.

[0180] The application further provides a computer readable storage medium having a computer program stored thereon, the computer program being used to make the computer execute the shallow-deep feature fusion method based on the Token reorganization model.

[0181] The above only describes the preferred embodiments of the application, and it should be noted that for those skilled in the art, without departing from the concept of the application, several modifications and improvements can be made, which are all within the protection scope of the application.

Claims

1. A shallow-deep feature fusion method based on a Token reorganization model, comprising: establishing an attention matrix-based Token reorganization module, wherein the attention matrix-based Token reorganization module comprises an attention matrix-based Token screening module and a target correlation Token replacement module, the attention matrix-based Token screening module uses a classification attention selection strategy and an interactive attention strategy to divide all Tokens in an image into target Tokens and irrelevant Tokens according to whether they contain valid information, the target Tokens are fed into a subsequent converter for interaction to extract important information regions in the image, and the target correlation Token replacement module reconstructs the screened irrelevant Tokens into target Tokens; establishing a shallow-deep feature learning module, using the image Tokens after splicing of deep features as query, and using the classification Tokens of shallow features as key and value, and performing information interaction between the two, and using MHSA to determine the amount of information that the shallow features need to fuse; the image processing flow of the attention matrix-based Token screening module is as follows: First, a given input person image is given where H, W and D represent its height, width and number of channels, respectively, and divided into overlapping patch blocks, flattened into a sequence N and P represent the number and size of patch blocks, respectively, Then the learnable classification codes The flattened sequence is concatenated with the classification code, which is used as the feature representation of the network's final output. Then, the positional code is added to the input sequence, so the input sequence of the Transformer encoder is represented as: ； wherein denotes the input sequence, denotes the position embedding, is a linear projection mapping the patch to C dimensions; Subsequently, the tokens are input into the multi-head self-attention module, and the specific process is as follows: All tokens are input into the multi-head self-attention module, and the specific process is as follows: ； where d is the dimension of the query vector, , , , The result of is a square matrix, which is called the attention matrix. The final output is obtained by linearly combining the attention matrix and the Value matrix. Input sequence for ViT where is the sequence length, the first token is a classification token as global feature representation, the remaining tokens are image tokens , ; ； ； the classification Tokens and the image Tokens interact through the attention mechanism in the ViT: ； The above formula corresponds to the attention value between the first row classification token and the image token in the attention matrix of formula (2), wherein, is the linearly mapped query vector of the classification token, K and V are the linearly mapped key matrix and value matrix of the input sequence respectively, and the combination coefficient is the attention value of the classification token with respect to all image tokens, can represent the importance of the i-th token.

2. The method of claim 1, wherein, The classification attention selection strategy is specifically: in the multi-head self-attention layer, multiple heads perform the calculation of formula (5) in parallel, and there are multiple classification token attention vectors , wherein is the number of heads in attention, the average attention value of all heads is calculated, ； Will Sort the tokens from largest to smallest, and use a top-k strategy to select the k largest attention values ​​as target tokens, and the remaining tokens as irrelevant tokens, including background and occlusions. Simultaneously, obtain the indices corresponding to the target tokens and irrelevant tokens. , .

3. The method of claim 2, wherein, The interaction attention strategy calculates an interaction attention value : ； In obtaining the interaction attention value After that, according to it, the top k tokens with high attention values are selected as target tokens, and the retention rate of the target tokens is set as: 。 4. The method of claim 3, wherein, the target correlation Token replacement module comprises three replacement strategies, which are: (1) adjacent Token replacement strategy, using the Tokens around the irrelevant Tokens to replace the irrelevant Tokens; (2) target Token replacement strategy, using the most discriminative target Tokens to replace the irrelevant Tokens; (3) global target Token replacement strategy, using all target Tokens to construct global target Tokens to replace the irrelevant Tokens.

5. The method of claim 4, wherein, The specific method of feature reconstruction is as follows: in the network model training process, a batch of input sequences is The classification tokens are separated to construct A feature reconstruction matrix is constructed for each image in the batch , ； reconstructing the features initialized to an all-zero matrix with ones on the diagonal, the feature reconstruction is achieved by changing the values of the elements in the matrix.

6. The method of claim 5, wherein, The shallow-deep layer feature learning module is specifically , the shallow layer feature sequence is represented as , i is the number of layers of the fused feature, for the shallow layer feature sequence , the classification tokens are separated and spliced, the classification tokens of the deep layer feature sequence and the image block tokens are also separated and cross-attention is performed between and , the image block tokens are taken as the only query, after performing cross-attention, the shallow layer feature is fused into the deep layer image block sequence, and the above process is represented as follows: ； ； ； wherein, , is a learnable linear mapping function, is a scaling factor, after performing cross-attention, the output of the shallow-deep feature learning module with layer normalization and residual connections is: ； ；； After cross-attention fusion, the image block Tokens of deep features pass the information learned from the shallow features to the classification Tokens in their own layer in the next layer encoder.

7. The method of claim 6, wherein, The overall objective function can be written as: ； ； ； where, is the output through the BN layer, is the linear projection matrix, is the corresponding label, C is the total number of identities, is a hyper-parameter that regulates the distance between positive and negative samples.

8. A system according to the method of any one of claims 1-7, comprising: an attention matrix-based Token reorganization module, wherein the attention matrix-based Token reorganization module comprises an attention matrix-based Token screening module and a target correlation Token replacement module, the attention matrix-based Token screening module uses a classification attention selection strategy and an interactive attention strategy to divide all Tokens in an image into target Tokens and irrelevant Tokens according to whether they contain valid information, the target Tokens are fed into a subsequent converter for interaction to extract important information regions in the image; the target correlation Token replacement module reconstructs the screened irrelevant Tokens into target Tokens; and a shallow-deep feature learning module, using the image Tokens after splicing of deep features as query, and using the classification Tokens of shallow features as key and value, and performing information interaction between the two, and using MHSA to determine the amount of information that the shallow features need to fuse.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which is used to make the computer execute the shallow-deep feature fusion method based on the Token reorganization model according to any one of claims 1-7.

Images