A pedestrian re-identification method based on a multi-granularity pyramid cross network

Abstract

The present application relates to a kind of pedestrian re-identification method based on multi-granularity pyramid cross network, for gradually learning the significant features of different local structures in global context.The network is mainly composed of two new designs, i.e.a multi-granularity convolution layer and a pyramid cross Transformer learning layer.The former aims to simulate human vision and study significant pedestrian features at different granularities.The latter aims to mine local information in global structure from a coarse-to-fine perspective.Considering that deep layers focus more on semantic information, the present method introduces a hierarchical aggregation module to fuse the features learned at different stages by cross attention learning, and the pedestrian features learned at shallow layers will serve as global priori of deep semantic information.The present application can maintain good versatility and robustness when pedestrian images encounter occlusion and other problems.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a pedestrian re-identification method based on a multi-granularity pyramid cross network. Background Technology

[0002] Person re-identification (Re-ID) plays a crucial role in modern intelligent surveillance technologies, such as pedestrian retrieval and behavior analysis. However, Re-ID faces challenges such as occlusion, low resolution, and variations in viewpoint / pose / domain / clothing / lighting. Therefore, it has attracted considerable attention from researchers.

[0003] In order to learn the discriminative features of pedestrian images, many attempts have been made to design efficient structures that are robust to the above challenges. Some studies (X. Qian, Y. Fu, Y.-G. Jiang, T. Xiang, and X. Xue. Multi-scale deep learning architectures for person re-identification. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pages 5399–5408, 2017; Z. Zhang, C. Lan, W. Zeng, X. Jin, and Z. Chen. Relation-aware global attention for person re-identification. In Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 3186–3195, 2020.) have proposed using the successful architectures of Convolutional Neural Networks (CNNs) in other computer vision tasks to extract robust features from pedestrian images. While CNN-based methods have achieved good results in some specific cases, their robustness is insufficient due to the limited receptive field of CNNs. Furthermore, CNN downsampling operations (such as pooling) reduce the spatial resolution of feature maps and lose fine-grained features with details, which is detrimental to the identification of pedestrians with similar appearances. More importantly, architectures that perform well in other vision domains are not well adapted to some specific challenges in pedestrian re-identification. Therefore, it is imperative to propose a specific design for pedestrian re-identification.

[0004] In recent years, Transformers have been incorporated into various computer vision tasks, including image classification, object detection, and recognition, due to their ability to model long-range dependencies. Compared to CNN-based algorithms, Transformer-based methods in Re-ID have also produced comparable matching results. However, pure Transformers cannot guarantee translation and scale invariance, which is frequently encountered in Re-ID tasks. To leverage the long-range dependency modeling capabilities of Transformers while maintaining the translation and scale invariance of CNNs, Zhang et al. (G. Zhang, P. Zhang, J. Qi, and H. Lu. Hat: Hierarchical aggregation transformers for person re-identification. In Proceedings of the 29th ACM International Conference on Multimedia, pages 516–525, 2021.) proposed an architecture consisting of a CNN for global feature learning and a Transformer for local feature learning, which achieved excellent results on publicly available large-scale Re-ID datasets. However, the input to a pure Transformer is a single-granular full feature map of a pedestrian image, which limits the Transformer's ability to extract rich local information. Previous studies (Y. Sun, L. Zheng, Y. Yang, Q. Tian, ​​and S. Wang. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proc. Eur. Conf. Comput. Vis. (ECCV), pages 480–496, 2018; M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and SCHoi. Deep learning for person re-identification: A survey and outlook. IEEE Trans. Pattern Anal. Mach. Intell., 44(6): 2872–2893, 2021.) have shown that horizontal partitioning is beneficial for networks to extract rich local information from pedestrian images. In addition, the network should also mine the discriminative semantic information implied in the various local structures in the global feature map.

[0005] Although the application of deep learning in pedestrian re-identification algorithms has yielded groundbreaking results, there is still a long way to go before it can be applied in real-world scenarios. To address the problem that different pedestrians may appear more similar than the same pedestrian due to occlusion, lighting, and pose issues, there is an urgent need to design a general, robust network structure suitable for pedestrian re-identification with strong robustness. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a pedestrian re-identification method based on a multi-granularity pyramid cross network. In view of the influence of environmental factors such as illumination, occlusion and pose on pedestrian images, a multi-granularity method is introduced. By utilizing the powerful long-range dependency modeling capability of Transformer, a discriminative multi-granularity pyramid cross network is provided. This method can maintain good versatility and robustness when pedestrian images encounter problems such as occlusion.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] This invention provides a pedestrian re-identification method based on a multi-granularity pyramid cross network, comprising the following steps:

[0009] Image information is acquired, and the preprocessed image information is input into a pre-trained multi-granularity pyramid cross-network to obtain output features. Based on the output features, an image matching the target pedestrian is retrieved from a preset image library to achieve pedestrian re-identification.

[0010] The multi-granularity pyramid cross network includes multi-level cascaded hierarchical aggregation units, each of which includes:

[0011] Multi-granularity convolutional layers are used to obtain significant pedestrian features at different granularities;

[0012] The pyramid-cross Transformer learning layer is used to capture discriminative local features from coarse to fine granular based on the input features as the output of the aggregation unit of this level. Input features other than the first level are obtained based on the output of the multi-granularity convolutional layer of this level and the output of the previous level.

[0013] As a preferred technical solution, each of the hierarchical aggregation units further includes a scale transformation layer disposed between the multi-granularity convolutional layer and the pyramid cross Transformer learning layer, for applying global max pooling to the output of the multi-granularity convolutional layer to suppress background information.

[0014] As a preferred technical solution, the multi-granularity pyramid cross network further includes a backbone network connected to each level aggregation unit, and the output features are obtained based on the output of the backbone network and the output of the last level aggregation unit.

[0015] As a preferred technical solution, the backbone network is a ResNet50 network.

[0016] As a preferred technical solution, the feature processing process of the pyramid cross-Transformer learning layer includes the following steps:

[0017] The input features are embedded to obtain an input feature map;

[0018] The input feature map is split into multiple local feature vectors, the corresponding local attention features are obtained and merged to obtain the overall feature map;

[0019] Channel MLP processing is performed on the overall feature map to obtain the discriminative local features.

[0020] As a preferred technical solution, the local attention features are obtained using the following formula:

[0021]

[0022] Among them, Y ij Let σ(·) be the local attention feature, and Q be the softmax activation function. ij To be a distinguishing feature, K is the local feature vector after splitting the input feature map. i V i Input feature map Vector after linear transformation For K i The transpose of , d is the embedding dimension, i represents the pyramid level, and j = 1, 2, ..., m represents the index of the local feature.

[0023] As a preferred technical solution, the channel MLP processing is implemented using the following formula:

[0024] Z = ζ(Norm(Y)W1) × W2 + Y

[0025] Where Z represents the discriminative saliency feature, ζ(·) is the Gelu activation function, Norm(·) represents layer normalization, Y represents the overall feature map, and W1∈R d×τd and W2∈R τd×d It is a learnable parameter, and τ is the expansion ratio.

[0026] As a preferred technical solution, the process of obtaining the pre-trained multi-granularity pyramid cross network includes the following steps:

[0027] Obtain a training sample set, and train the multi-granularity pyramid cross network based on the training sample set. When the function value of the loss function reaches the preset convergence condition, the pre-trained multi-granularity pyramid cross network is obtained.

[0028] The loss function is obtained based on validation loss, classification loss, and auxiliary loss.

[0029] As a preferred technical solution, the auxiliary loss is calculated using the following formula:

[0030]

[0031] in, To mitigate losses, To verify the loss, The loss is categorized, and S is the total number of stages.

[0032] As a preferred technical solution, in each of the hierarchical aggregation units, the multi-granularity convolutional layer is sequentially connected to the pyramid cross Transformer learning layer.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) Effectively improve versatility and robustness: This method addresses the problem that existing methods ignore the discriminative semantic information hidden in various local structures in the global feature map. It provides a multi-granularity pyramid cross network that includes multi-level hierarchical aggregation units. Each hierarchical aggregation unit includes a multi-granularity convolutional layer and a pyramid cross Transformer Network (MCTN) learning layer. Since multiple hierarchical aggregation units constitute a multi-level cascaded structure, the model can better utilize the content between different levels and further mine the potential semantic information in pedestrian images. In complex scenes with multiple cameras, this method can quickly locate all results for finding a specific pedestrian, thereby improving versatility and robustness.

[0035] (2) Good model training effect: In the training stage, in order to facilitate the learning of the network's powerful feature representation, auxiliary loss consisting of validation loss and classification loss is added at different stages of the hierarchical aggregation module. The validation loss, classification loss and auxiliary loss are used to supervise the learning of the multi-granularity cross Transformer network and improve the model training effect. Attached Figure Description

[0036] Figure 1This is a flowchart of the pedestrian re-identification method based on a multi-granularity pyramid cross network in Example 1;

[0037] Figure 2 This is a schematic diagram of a multi-granularity pyramidal cross network. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] Example 1

[0040] like Figure 1 This embodiment addresses the problem of existing methods neglecting the discriminative semantic information implicit in various local structures within the global feature map. It provides a pedestrian re-identification method based on a multi-granularity pyramid cross-network, capable of quickly locating and finding all results for a specific pedestrian in complex multi-camera scenes. This method includes the following steps:

[0041] Step S1: Input a pedestrian image I1 to be queried;

[0042] Step S2: Global features are obtained by processing the data through the backbone network ResNet50.

[0043] Step S3: Insert multi-granularity convolutional layers and pyramid cross Transformer learning layers into the first three stages of the backbone network;

[0044] Step S4: Use the multi-level aggregation module to aggregate feature maps from different stages to mine richer local semantic information;

[0045] Step S5: In the training phase, in addition to the loss of the backbone network, auxiliary loss is added to the first three phases for supervised training.

[0046] Step S6: In the testing phase, the features obtained from the backbone network and the multi-level aggregation module are fused together as the final feature representation for testing.

[0047] Step S7: Output the final pedestrian image feature representation.

[0048] like Figure 2 The multi-granularity pyramid cross network consists of five main parts:

[0049] Backbone Network: Considering the richness of the ImageNet dataset, the pre-trained ResNet architecture possesses powerful feature representation capabilities. Therefore, we adopt a ResNet50 network pre-trained on ImageNet as the backbone architecture for extracting global features from pedestrian images.

[0050] Multi-granularity convolutional layers: Multi-granularity convolutional layers simulate the process by which human vision perceives things from different angles. This layer allows for the extraction of finer-grained features, enhancing model performance.

[0051] Scale transformation: To reduce the number of model parameters and facilitate the integration of subsequent networks, global max pooling (GMP) is applied after the multi-granularity convolutional layers. GMP can suppress background information and extract the salient features of pedestrians, making the salient features of pedestrians more compact.

[0052] Pyramid Cross Transformer Learning Layer: This layer is designed as a pyramid to guide the network in discovering salient pedestrian features from a coarse to fine perspective. The attention features obtained at the coarse level are further segmented to learn finer-grained features, thereby achieving complementarity of pedestrian information at different granularities.

[0053] The hierarchical aggregation module, composed of multiple hierarchical aggregation units, is used to obtain more comprehensive pedestrian features. Since shallow layers contain more detailed information, features calculated in shallow layers can be aggregated into deeper layers, guiding them to focus more on fine-grained local features. Simultaneously, this module also helps to mine latent semantic information in shallow layers through the interaction between shallow and deep layers.

[0054] The implementation process of the three core components—multi-granularity convolutional layers, pyramid cross-transformer learning layers, and hierarchical aggregation modules—is as follows:

[0055] (1) Multi-granularity convolutional layers: As is well known, not all discriminative features in pedestrian images can be directly obtained through the backbone network, even with continuous deepening of the network. Intuitively, the human visual system typically observes things from different perspectives and granularities. Therefore, we apply multi-granularity convolutional layers, simulating the human visual system, to the first three stages of the backbone network to more comprehensively mine the semantic information contained in pedestrian images. The input data is further analyzed through three different receptive fields (i.e., granularities): 3×3, 5×5, and 7×7. To capture richer feature information, weights are not shared during the feature representation calculation at different granularities. To reduce the parameters required by the network and improve its nonlinear transformation capability, a 5×5 kernel can be decomposed into a cascade of two 3×3 kernels, and a 7×7 kernel can be decomposed into a cascade of three 3×3 kernels. The features extracted at different granularities are fused as the final feature representation of this layer. Note that the convolution operations at different granularities are transformed into residual blocks. The input data is first processed through a 1×1 convolution to compress the number of feature channels, then a 3×3 convolution is performed for feature extraction, and finally a 1×1 convolution is performed to restore the features to their original number of channels. Finally, the obtained features are concatenated with the output of the shortcut to obtain the final feature representation.

[0056] (2) Pyramid Cross Transformer Learning Layer: In fact, the performance of the Re-ID task largely depends on the semantic information of pedestrian images extracted by the neural network. By deepening the network, deeper layers can learn the semantic information of pedestrian images to a certain extent. However, shallow layers only contain detailed information and a small amount of semantic information, and a large amount of semantic information has not yet been mined. To this end, we propose a pyramid cross Transformer learning layer based on the Transformer architecture to enrich the diversity of features. The pyramid cross Transformer learning layer mainly consists of three components: input embedding, cross attention, and channel MLP.

[0057] Input Embedding: Given a feature map X, it is first processed by input embedding, similar to the block embedding of ViTs (A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis-senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,2020.). This formula can be expressed as:

[0058] X emb =Norm(InputEmb(X))

[0059] Norm(·) represents layer normalization. This indicates a sequence of length U and an embedding dimension of C. e Embedded.

[0060] Cross attention: In order to capture discriminative local information in the global structure from a coarse to fine perspective, we use a pyramid structure to explore local-global relationships.

[0061] 1) Pyramid. The pyramid contains different levels of attention computation, with each level horizontally splitting the feature map into 2i-1, where 2 is the cardinality and i is the current level. It's worth noting that the granularity of the horizontal division differs across pyramid levels. The number of divisions increases with the number of pyramid levels. Features learned at the coarse-grained level are used to guide the network in mining finer-grained local information. Furthermore, considering that overly fine divisions might compromise the integrity of local semantic information, we set the pyramid level to 3.

[0062] 2) Crossover. The feature map... Horizontally split into m local feature vectors Where i represents the pyramid level, and j = 1, 2, ..., m represents the index of the local feature. The unsegmented feature map is transformed using two different linear transformations. Transform to K i V i To obtain more representative and discriminative features, we analyzed different local vectors. By applying different linear transformations, distinctive features Q can be generated. ij The features calculated through cross-attention can be defined as:

[0063]

[0064] Where K i V i ∈R U×d Q ij ∈R U / m×d d is the embedding dimension. For K i The transpose of Q. ij ×K i T The relationship between local and global features was studied. In fact, Q ij ×K i T Similar to cosine similarity, it can therefore be used to measure relevance. The obtained cross-attention weights are normalized using the softmax activation function σ(·), and then compared with V. i Multiplication extracts local salient features from the global context. Finally, the obtained features are concatenated with the original local feature vectors to obtain the cross-attention feature representation.

[0065] 3) Merging. Each local attention feature is merged into the overall feature map of the current level, as shown below:

[0066]

[0067] The feature maps are then sent to the next layer of the pyramid, guiding the network to mine finer-grained local semantic information within the global context. Through learning the pyramid cross-transformer, the network can better discover the correlation between local and global features, highlighting discriminative features in local regions and suppressing irrelevant features.

[0068] 4) Channel MLP. The channels of the feature map contain rich details and semantic information about the same parts of a pedestrian. We preserve the channel multilayer perceptron (MLP) from the traditional Transformer to compress the features of these similar channels. The channel MLP consists of two linear transformations and a Gelu activation function ζ(·). Assuming the feature map output by cross-attention learning is Y, the process can be represented as:

[0069] Z = ζ(Norm(Y)W1) × W2 + Y,

[0070] Where Z represents the output feature of the channel MLP, and W1∈R d×τd and W2∈R τd×d It is a learnable parameter, and τ is the expansion ratio.

[0071] (3) Hierarchical Aggregation Module: Considering that shallow layers contain more untapped semantic information than deep layers, we simultaneously insert multi-granularity convolutional layers and pyramid cross-Transformer learning layers into the first three stages of the backbone network. Specifically, the feature maps calculated in each stage are not directly fed back into the backbone network for training, nor are they directly concatenated as the final feature representation. Instead, they are fused with the features extracted by the multi-granularity convolutional layer in the next stage and fed back to the pyramid cross-Transformer learning layer for pyramid cross-Transformer learning, guiding the network to discover diverse cues in pedestrian images. Through the hierarchical aggregation module, the content between different layers can be better utilized to further mine the potential semantic information in pedestrian images.

[0072] In the training phase, this invention utilizes validation loss, classification loss, and auxiliary loss to supervise the learning of multi-granularity cross-transformer networks.

[0073] Validation Loss: This loss enhances intra-class compactness and inter-class diversity, meaning it minimizes the distance between positive and negative samples using a predefined interval ξ. In other words, after training, the distance between identical pedestrians should be minimized, while the distance between pedestrians from different groups should be maximized. We use the hard triplet loss (A. Hermans, L. Beyer, and B. Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv, page 1703, 07737, 2017.) as the validation loss, defined as:

[0074]

[0075] Among them, f a Indicates the anchored sample, f p f represents positive samples with the same identity. n This represents negative samples with different identities. ||·|| represents L2-Norm, [·] + This represents the max function.

[0076] Classification Loss: Since the pedestrian re-identification task can be viewed as an image classification problem with each pedestrian's identity as the class, this paper uses the cross-entropy loss, defined as:

[0077]

[0078] Where M represents the number of pedestrian identities. and y iThese are the predicted label and the actual label for the i-th pedestrian, respectively.

[0079] Auxiliary Loss: To facilitate the learning of the network's powerful feature representations, auxiliary losses consisting of validation loss and classification loss are added at different stages of the hierarchical aggregation module. Its definition is as follows:

[0080]

[0081] Where S is the total number of stages. Finally, the total loss is:

[0082]

[0083] Here, γ is a hyperparameter used to balance the backbone network loss and auxiliary loss.

[0084] This method progressively learns salient features of different local structures within a global context. The network provided by this method mainly consists of two novel designs: a multi-granularity convolutional layer and a pyramidal cross-Transformer learning layer. The former aims to simulate human vision and study salient pedestrian features at different granularities. The latter aims to mine local information in the global structure from a coarse-to-fine perspective. Furthermore, considering that deeper layers focus on more semantic information and do not require finer-grained attention learning to avoid overfitting, while shallower layers focus on details but still contain a large amount of untapped semantic information, a hierarchical aggregation module is introduced to fuse features learned at different stages of cross-attention learning. Pedestrian features learned in the shallow layer serve as global priors for deeper semantic information. This invention addresses the influence of environmental factors such as illumination, occlusion, and pose on pedestrian images by introducing a multi-granularity method and utilizing the powerful long-range dependency modeling capabilities of the Transformer to design a discriminative network structure. Even when pedestrian images encounter occlusion or other issues, this invention maintains good versatility and robustness.

[0085] Example 2

[0086] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the pedestrian re-identification method based on multi-granularity pyramid cross network as described in Embodiment 1.

[0087] Example 3

[0088] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the pedestrian re-identification method based on a multi-granularity pyramid cross network as described in Embodiment 1.

[0089] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A pedestrian re-identification method based on a multi-granularity pyramid cross network, characterized in that, Includes the following steps: Image information is acquired, and the preprocessed image information is input into a pre-trained multi-granularity pyramid cross network to obtain output features. Based on the output features, an image matching the target pedestrian is obtained from a preset image library to achieve pedestrian re-identification. The multi-granularity pyramid cross network includes multi-level cascaded hierarchical aggregation units, each of which includes: Multi-granularity convolutional layers are used to obtain significant pedestrian features at different granularities; The pyramid-cross Transformer learning layer is used to capture discriminative local features from coarse to fine granular based on the input features as the output of the aggregation unit of this level. The input features of other levels except the first level are obtained from the output of the multi-granularity convolutional layer of this level and the output of the previous level. Each of the hierarchical aggregation units further includes a scaling layer disposed between the multi-granularity convolutional layer and the pyramid cross Transformer learning layer, for applying global max pooling to the output of the multi-granularity convolutional layer to suppress background information; The feature processing process of the pyramid cross Transformer learning layer includes the following steps: Embedding processing is performed on the input features to obtain the input feature map. The pyramid contains different levels of attention computation, with each level horizontally processing the input feature map. Split into Local feature vectors Different pyramid levels have different levels of horizontal partitioning granularity. Unsegmented feature maps are then transformed using two different linear transformations. Transform into and For different local vectors Applying different linear transformations to generate features with discriminative power. The local attention features are obtained by calculating the cross-attention using the following formula: in, For the local attention features, for Activation function for transpose, For the embedding dimension, Indicates the pyramid hierarchy. An index representing local features; Merge the local attention features to obtain the overall feature map of the current level. Then the feature map is sent to the next layer of the pyramid to guide the network to mine more fine-grained local semantic information in the global context; Channel MLP processing is performed on the feature map output by cross-attention learning to obtain the discriminative local features; The channel MLP processing is implemented using the following formula: in, For discriminative local features, for Activation function Representation layer normalization, The feature map representing the output of cross-attention learning. and These are learnable parameters. Expansion ratio; The multi-granularity pyramid cross network also includes a backbone network connected to each level of aggregation unit, and the output features are obtained based on the output of the backbone network and the output of the last level aggregation unit.

2. The pedestrian re-identification method based on a multi-granularity pyramid cross network according to claim 1, characterized in that, The process of obtaining the pre-trained multi-granularity pyramid cross network includes the following steps: Obtain a training sample set, and train the multi-granularity pyramid cross network based on the training sample set. When the function value of the loss function reaches the preset convergence condition, the pre-trained multi-granularity pyramid cross network is obtained. The loss function is obtained based on validation loss, classification loss, and auxiliary loss.

3. The pedestrian re-identification method based on a multi-granularity pyramid cross network according to claim 2, characterized in that, The backbone network is a ResNet50 network.

4. The pedestrian re-identification method based on a multi-granularity pyramid cross network according to claim 1, characterized in that, In each of the hierarchical aggregation units, the multi-granularity convolutional layer is sequentially connected to the pyramid cross Transformer learning layer.

Images