Cross-modal pedestrian re-identification method and system based on high-order structure modeling

Abstract

The application relates to a cross-modal pedestrian re-identification method and system based on high-order structure modeling, which comprises the following steps: 1) extracting VIS pictures and IR pictures of multiple pedestrians from a data set in pairs to form a VIS-IR picture pair training data set; 2) constructing a cross-modal pedestrian re-identification network model, wherein the cross-modal pedestrian re-identification network model mainly comprises a backbone network for extracting basic features of an image, a short-long range feature extraction module for extracting features in different range scales, a high-order structure learning module for learning high-order structure information in an image, and a shared space learning module for reducing the difference of pedestrians in different modalities; the model is trained by using the training data set to obtain generalizable model parameters; and 3) applying the trained cross-modal pedestrian re-identification network model to cross-modal retrieval to realize pedestrian re-identification between different modalities. The method and system are beneficial to obtaining more stable, robust and accurate cross-modal pedestrian re-identification results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a cross-modal pedestrian re-identification method and system based on high-order structure modeling. BACKGROUND

[0002] In the past few years, pedestrian re-identification (ReID) has attracted increasing attention due to its importance in surveillance and security applications. A large number of single-modal pedestrian re-identification methods based on visible light (VIS) cameras have been proposed. However, these methods can fail in low-light conditions. Unlike VIS cameras, infrared (IR) cameras are less affected by lighting changes. Recently, visible light-infrared pedestrian re-identification (VI-ReID) that can match cross-modal pedestrian images has received extensive attention, thereby alleviating the limitations of single-modal pedestrian re-identification.

[0003] A major challenge of VI-ReID is the huge modality gap between VIS and IR images. To reduce the modality gap, existing VI-ReID methods can be divided into image and feature level methods. Image level methods (Dai et al. (Dai, P.; Ji, R.; Wang, H.; Wu, Q.; and Huang, Y. 2018. Cross-modality person re-identification with generative adversarial training. In Proceedings of the International Joint Conference on Artificial Intelligence, 1-7.); Wang et al. (Wang, Z.; Wang, Z.; Zheng, Y.; Chuang, Y.; and Satoh, S. 2019. Learning to reduce dual-level discrepancy for infrared-visible person re-identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 618-626.); Wei et al. (Wei, Z.; Yang, X.; Wang, N.; and Gao, X. 2022. Rbdf: Reciprocal bidirectional framework for visible infrared person reidentification. IEEE Transactions on Cybernetics, 52(10): 10988-10998.)) usually reduce the modality gap by generating intermediate modality images or new modality images. Wei et al. (Wei, Z.; Yang, X.; Wang, N.; and Gao, X. 2022. Rbdf: Reciprocal bidirectional framework for visible infrared person reidentification. IEEE Transactions on Cybernetics, 52(10): 10988-10998.) propose a bidirectional image conversion subnetwork to generate intermediate modality images from VIS and IR modalities. However, GAN-based methods are prone to color inconsistency or image detail loss problems. Therefore, the generated images can not be reliable for subsequent retrieval.Feature-level methods map features of different modalities into a common feature space to reduce the modality discrepancy. Some methods (e.g., Ye, M.; Shen, J.; Lin, G.; Xiang, T.; Shao, L.; and Hoi, S. C. 2021b. Deep learning for person re-identification: A survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2872–2893.; Chen, C.; Ye, M.; Qi, M.; Wu, J.; Jiang, J.; and Lin, C.-W. 2022a. Structure-aware positional Transformer for visible-infrared person re-identification. IEEE Transactions on Image Processing, 31:2352–2364.; Yang, B.; Chen, J.; and Ye, M. 2023. Top-K Visual Tokens Transformer: Selecting Tokens for Visible-Infrared Person Re-Identification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 1–5.; Zhang, Y.; and Wang, H. 2023. Diverse embedding expan-sion network and low-light cross-modality benchmark for visible-infrared person re-identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2153–2162.) utilize CNN or ViT as backbone network to extract features.Other methods (e.g., Chen, C.; Ye, M.; Qi, M.; Wu, J.; Jiang, J.; and Lin, C.-W. 2022a. Structure-aware positional Transformer for visible-infrared person re-identification. IEEE Transactions on Image Processing, 31: 2352-2364.; Wan, L.; Sun, Z.; Jing, Q.; Chen, Y.; Lu, L.; and Li, Z. 2023. D2DA: Geometry-guided dual-alignment learning for RGB-infrared person re-identification. Pattern Recognition, 135: 109150.) employ off-the-shelf keypoint extractor to generate keypoint labels for pedestrian images and learn modality-agnostic features. But the keypoint extractor can introduce noisy labels, which deteriorate the discriminability of the final pedestrian features. Many VI-ReID methods (e.g., Huang, N.; Liu, J.; Luo, Y.; Zhang, Q.; and Han, J. 2023. Exploring modality-shared appearance features and modality-invariant relation features for cross-modality person re-identification. Pattern Recognition, 135: 109145. Huang, N.; Liu, K.; Liu, Y.; Zhang, Q.; and Han, J. 2022. Cross-modality person re-identification via multi-task learning. Pattern Recognition, 128: 108653.) employ contrast-based loss to directly minimize the distance between VIS and IR features to obtain a common feature space. However, due to the large modality gap between modalities, learning a reasonable common feature space is not a simple task. SUMMARY

[0004] The application aims to provide a cross-modality pedestrian re-identification method and system based on high-order structure modeling, which can obtain more stable, robust and accurate cross-modality pedestrian re-identification results.

[0005] To achieve the above object, the technical scheme adopted by the present application is: a cross-modal pedestrian re-identification method based on high-order structure modeling, comprising the following steps:

[0006] 1) Extract a plurality of pairs of visible light (VIS) pictures and infrared (IR) pictures of pedestrians from a data set to form a VIS-IR picture pair training data set;

[0007] 2) Construct a cross-modal pedestrian re-identification network model based on high-order structure modeling, which mainly includes a backbone network, a short-long range feature extraction module, a high-order structure learning module, and a shared space learning module. The backbone network is used to extract the basic features of the image, the short-long range feature extraction module is used to extract features of different range scales from the image, the high-order structure learning module is used to learn the high-order structure information in the image to better represent the features of the pedestrian, and the shared space learning module is used to reduce the differences between pedestrians in different modalities. Train the cross-modal pedestrian re-identification network model through the VIS-IR picture pair training data set to obtain generalizable model parameters;

[0008] 3) Use the trained cross-modal pedestrian re-identification network model for cross-modal retrieval to realize pedestrian re-identification between different modalities.

[0009] Further, in step 1), the data set is SYSU-MM01 / RegDB / LLCM cross-modal pedestrian re-identification data set, and K VIS pictures and K IR pictures of P pedestrians are extracted from the SYSU-MM01 / RegDB / LLCM cross-modal pedestrian re-identification data set in a paired extraction manner to form a VIS-IR picture pair training data set.

[0010] Further, in step 2), the implementation method of the cross-modal pedestrian re-identification network model is:

[0011] A) For the paired input VIS picture and IR picture, the pedestrian VIS basic features and the pedestrian IR basic features are extracted through the backbone network, and then the extracted pedestrian VIS basic features and the pedestrian IR basic features are input into the short-long range feature extraction module to extract the pedestrian VIS short-long range features and the pedestrian IR short-long range features;

[0012] B) Obtain the high-order structure information contained in the pedestrian VIS short-long range features and the pedestrian IR short-long range features through the high-order structure learning module to obtain the pedestrian VIS short-long range enhanced features and the pedestrian IR short-long range enhanced features to depict the high-order relationship between different local features of the pedestrian, and not to be affected by model collapse, thereby enhancing the feature representation;

[0013] C) input the obtained pedestrian VIS short-long range enhanced features and pedestrian IR short-long range enhanced features into a shared space learning module, learn a discriminative and reasonable shared feature space by generating intermediate features, so as to reduce the negative influence of modal difference;

[0014] D) introduce modal-range pedestrian center contrast loss to improve feature representation and reduce the difference between VIS, IR and intermediate features of the same pedestrian, and joint classification loss and triplet loss Minimize the loss value as the goal, and iteratively train the cross-modal pedestrian re-identification network model through the VIS-IR picture pair, and then obtain the trained cross-modal pedestrian re-identification network model.

[0015] Further, in step A), the backbone network includes two first stage feature extraction modules, and one second to fourth stage feature extraction module and one fifth stage feature extraction module, the short-long range feature extraction module includes a short range feature extraction module and a long range feature extraction module, the short range feature extraction module and the long range feature extraction module are respectively composed of a convolutional network and a Transformer network; the pair of VIS and IR pictures are respectively input into the first stage feature extraction module, and then simultaneously input into the second to fourth stage feature extraction module, to obtain pedestrian VIS basic features B vis and pedestrian IR basic features B ir ; then the obtained pedestrian VIS basic features B vis and pedestrian IR basic features B ir are respectively input into the short range feature extraction module and the long range feature extraction module, to obtain pedestrian VIS, IR short range features and pedestrian VIS, IR long range features The feature size is wherein H, W and C respectively represent the height, width and channel size of the feature, so as to obtain a feature set which is used as the input of the high-order structure learning module; after the high-order structure learning module, a structure enhanced feature set and the obtained feature set is input into a shared space learning module to obtain an intermediate feature set After obtaining the structure enhanced feature set and the intermediate feature set , the fifth stage feature extraction module is used to perform deep semantic association of the features.

[0016] Furthermore, in step B), to avoid the convergence of hyperedge representations when constructing higher-order relationships between pedestrians in the hypergraph model, a feature whitening operation is introduced to map features to a spherical space, reducing coupling between features and thus better establishing higher-order relationships between feature nodes. The whitened feature node f' n As shown below:

[0017]

[0018] in, Features in set Q Each feature node; μ F γ represents the mean of feature F; σ represents the lower triangular matrix obtained by Kolesky decomposition; n and β n Represents the parameters of the affine transformation;

[0019] Learn the correlation matrix using cross-correlation. To obtain the relationships between nodes, see below:

[0020] Y=ε(Ψ(F')Λ(F')Ψ(F') T Ω(F'))

[0021] in, N represents the number of nodes, which is equal to HW; M represents the number of hyperedges; C represents the number of channels; Ψ(·) represents the linear transformation layer; Λ(·) and Ω(·) represent the diagonal operation learning distance metric and the contribution function of a node to the corresponding hyperedge, respectively; ε(·) represents the step function;

[0022] Then, high-order structure-enhanced pedestrian features are obtained through hypergraph convolution. It is represented as:

[0023] R = (ID) 1 / 2 YVB -1 Y T D -1 / 2 )F'Θ+F

[0024] in, Represents the identity matrix; Represents the weight matrix; and Represents the node degree matrix and hyperedge degree matrix; Θ is a learnable parameter; corresponding to pedestrian VIS and IR short-range features. Pedestrian VIS and IR long-range features Obtain pedestrian VIS and IR short-range enhancement features Pedestrian VIS and IR long-range enhancement features Thus, a set of structural enhancement features is obtained.

[0025] Further, in step C), the features from different modalities and ranges are aligned by graph attention, and intermediate features are generated, and the differences between VIS and IR modalities are reduced by learning the intermediate features;

[0026] The generation method of the intermediate features is to establish and the relationship between the nodes as follows:

[0027]

[0028] wherein P is a similarity matrix; θ q and θ k are linear conversion layers; Softmax(·) is a Softmax function;

[0029] The similar nodes between and are aligned by graph attention to obtain which is expressed as:

[0030]

[0031] wherein GAT(·) is a graph attention operation; θ v is a linear conversion layer; λ is a hyperparameter for reducing nodes with low similarity; 1 is a unit column vector, ReLU(·) and Mean(·) are ReLU and average operation respectively;

[0032] Based on the above process, the intermediate features are generated by aligning which is expressed as:

[0033]

[0034] Similarly, the intermediate features are generated by aligning respectively, so as to obtain the intermediate feature set Then, the fifth stage feature extraction module is used to perform deep semantic association of the feature sets and ; then, the one-dimensional structure enhanced feature set and the one-dimensional intermediate feature set are obtained by performing global and local pooling on and respectively.

[0035] Further, in step D), the identity center robust to appearance changes of pedestrians is obtained by the weighted average of each person's features in one modality and specific range; for the pedestrian center feature labeled i in VIS modality and long-range feature, it is obtained by the following way:

[0036]

[0037] where K is the number of VIS features of each pedestrian;

[0038] Thus, for the VIS and IR center feature sets are obtained where P is the number of pedestrians in the training set; for the intermediate feature center feature set is obtained

[0039] the range loss is to reduce the distance between the same range VIS and IR features of the same person, while expanding the feature distance between the same range VIS and IR features of different persons, represented as:

[0040]

[0041] wherein,

[0042]

[0043] where SM represents the cosine similarity matrix of input features A and ; represents the L1 distance measurement function;

[0044] the intermediate feature loss is to reduce the difference of the intermediate features, represented as:

[0045]

[0046] the inter-modal loss is to reduce the intra-class distance and expand the inter-class distance among VIS, IR and intermediate features, represented as:

[0047]

[0048] wherein, and respectively represent the VIS, IR and intermediate modal center features corresponding to the pedestrian;

[0049] In summary, the modal-range pedestrian center contrast loss is defined as follows:

[0050]

[0051] Thus, the joint classification loss Triplet loss and the modal-range pedestrian-centric contrastive loss The loss is represented as:

[0052]

[0053] wherein, represents the classification loss, represents the triplet loss, represents the modal-range pedestrian-centric contrastive loss.

[0054] The application further provides a cross-modal pedestrian re-identification system based on intermediate modal learning under high-order structure modeling, comprising a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, when the processor executes the computer program instructions, the above-mentioned method steps can be realized.

[0055] Compared with the prior art, the application has the following beneficial effects: a cross-modal pedestrian re-identification method and system based on high-order structure modeling are provided, the method and system effectively utilize high-order relationship modeling and intermediate feature learning, can obtain generalized and robust pedestrian features, not only can effectively solve the problems of pedestrian posture change, object occlusion and the like in cross-modal pedestrian re-identification, but also can alleviate the negative effects of VIS and IR modal differences. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the architecture diagram of the cross-modal pedestrian re-identification network model based on intermediate modal learning under high-order structure modeling in the embodiments of the application. DETAILED DESCRIPTION

[0057] The application will be further described below in combination with the drawings and embodiments.

[0058] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0059] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0060] As Figure 1 shown, the embodiment provides a cross-modal pedestrian re-identification method based on high-order structure modeling, comprising the following steps:

[0061] 1) Extracting multiple pairs of visible light (VIS) pictures and infrared (IR) pictures of pedestrians from the data set to form a VIS-IR picture pair training data set.

[0062] 2) Constructing a cross-modal pedestrian re-identification network model based on intermediate modal learning under high-order structure modeling, which mainly includes a backbone network, a short-long range feature extraction module, a high-order structure learning module and a shared space learning module. The backbone network is used to extract the basic features of the image, the short-long range feature extraction module is used to extract features of different range scales from the image, the high-order structure learning module is used to learn the high-order structure information in the image to better represent the features of the pedestrian, and the shared space learning module is used to reduce the differences between pedestrians in different modalities. The cross-modal pedestrian re-identification network model is trained by the VIS-IR picture pair training data set to obtain the generalizable model parameters.

[0063] 3) The trained cross-modal pedestrian re-identification network model is used for cross-modal retrieval to realize pedestrian re-identification between different modalities.

[0064] In step 1), the data set is SYSU-MM01 / RegDB / LLCM cross-modal pedestrian re-identification data set, K VIS pictures and K IR pictures of P pedestrians are extracted from the SYSU-MM01 / RegDB / LLCM cross-modal pedestrian re-identification data set in a paired extraction manner to form a VIS-IR picture pair training data set.

[0065] In step 2), the implementation method of the cross-modal pedestrian re-identification network model is:

[0066] A) For the paired input VIS picture and IR picture, the pedestrian VIS basic feature and the pedestrian IR basic feature are extracted through the backbone network, and then the extracted pedestrian VIS basic feature and the pedestrian IR basic feature are input into the short-long range feature extraction module to extract the pedestrian VIS short-long range feature and the pedestrian IR short-long range feature.

[0067] B) The high-order structure learning module is used to obtain the high-order structure information contained in the pedestrian VIS short-long range feature and the pedestrian IR short-long range feature, to obtain the pedestrian VIS short-long range enhanced feature and the pedestrian IR short-long range enhanced feature, to depict the high-order relationship between different local features of the pedestrian, and not to be affected by the model collapse, thereby enhancing the feature representation.

[0068] C) input the obtained pedestrian VIS short-long range enhanced features and pedestrian IR short-long range enhanced features into a shared space learning module, learn a discriminative and reasonable shared feature space by generating intermediate features, so as to reduce the negative influence of modal difference.

[0069] D) introduce modal-range pedestrian center contrast loss to improve feature representation and reduce the difference between VIS, IR and intermediate features of the same pedestrian, and joint classification loss and triplet loss to minimize the loss value, and iteratively train the cross-modal pedestrian re-identification network model through VIS-IR picture pairs, thereby obtaining the trained cross-modal pedestrian re-identification network model.

[0070] In step A), the backbone network includes two first stage feature extraction modules (for extracting features of two modalities respectively), and a second to fourth stage feature extraction module and a fifth stage feature extraction module (for jointly extracting features of two modalities), the short-long range feature extraction module includes a short-range feature extraction module and a long-range feature extraction module, the short-range feature extraction module and the long-range feature extraction module are respectively composed of a convolutional network and a Transformer network; a pair of VIS and IR pictures are input into the first stage feature extraction module respectively, and then input into the second to fourth stage feature extraction module at the same time, to obtain pedestrian VIS basic features B vis and pedestrian IR basic features B ir ; then the obtained pedestrian VIS basic features B vis and pedestrian IR basic features B ir are input into the short-range feature extraction module and the long-range feature extraction module respectively, to obtain pedestrian VIS, IR short-range features and pedestrian VIS, IR long-range features The feature size is wherein H, W and C represent the height, width and channel size of the feature respectively, so as to obtain a feature set which is used as the input of the high-order structure learning module; after the high-order structure learning module, a structure enhanced feature set is obtained, and the obtained feature set is input into a shared space learning module to obtain an intermediate feature set After obtaining the structure enhanced feature set and the intermediate feature set , the fifth stage feature extraction module is used to perform deep semantic association of the features.

[0071] In step B), in order to avoid the problem of super-edge representation convergence of the hypergraph model in constructing high-order relationships of pedestrians, a feature whitening operation is introduced to map the features to a spherical space, reduce the coupling between the features, and better establish the high-order relationships between the feature nodes. The feature node f' after whitening is as follows: n As shown below:

[0072]

[0073] wherein, is each feature node of the set of features in the set F; μ F represents the mean of the features F; σ represents a lower triangular matrix obtained by Cholesky decomposition; γ n and β n represent affine transformation parameters.

[0074] The correlation learning association matrix is used to obtain the connection between nodes, as shown below:

[0075] Y = ε (Ψ (F') Λ (F') Ψ (F') T Ω (F'))

[0076] wherein, N represents the number of nodes, which is equal to HW; M represents the number of hyperedges; C represents the number of channels; Ψ (·) represents a linear conversion layer; Λ (·) and Ω (·) represent diagonal line operation learning distance measurement methods and node pair corresponding hyperedge contribution functions, respectively; ε (·) represents a step function.

[0077] Then, the pedestrian feature with high-order structure enhancement is obtained through hypergraph convolution which is represented as:

[0078] R = (I - D 1 / 2 YVB -1 Y T D -1 / 2 ) F' Θ + F

[0079] wherein, I represents an identity matrix; W represents a weight matrix; and represent a node degree matrix and a hyperedge degree matrix; Θ is a learnable parameter; corresponds to pedestrian VIS, IR short-range features and pedestrian VIS, IR long-range features to obtain pedestrian VIS, IR short-range enhanced features and pedestrian VIS, IR long-range enhanced features so as to obtain a structure-enhanced feature set

[0080] In step C), the features from different modalities and ranges are aligned by graph attention, and reliable intermediate features are generated, and the differences between VIS and IR modalities are reduced by learning intermediate features;

[0081] The generation method of the intermediate feature is to establish and the relationship between nodes as follows:

[0082]

[0083] Where P is a similarity matrix; θ q and θ k are linear transformation layers; Softmax(·) is a Softmax function.

[0084] The similar nodes between and are aligned by graph attention to obtain which is expressed as:

[0085]

[0086] Where GAT(·) is a graph attention operation; θ v is a linear transformation layer; λ is a hyperparameter for reducing low similarity nodes; 1 is a unit column vector; ReLU(·) and Mean(·) are ReLU and average operations, respectively.

[0087] Based on the above process, align with to generate intermediate features which is expressed as:

[0088]

[0089] Similarly, align to generate intermediate features so as to obtain the intermediate feature set Then, the fifth stage feature extraction module is used to perform deep semantic association of the feature sets and ; then, global and local pooling is performed on and to obtain a one-dimensional structure-enhanced feature set and a one-dimensional intermediate feature set

[0090] In step D), the identity center robust to appearance changes of pedestrians is obtained by the weighted average of each person's features in one modality and specific range; for the pedestrian center feature labeled i in VIS modality and long-range feature, it is obtained by the following way:

[0091]

[0092] where K is the number of VIS features of each pedestrian.

[0093] Thus, for get the VIS and IR center feature set where P is the number of pedestrians in the training set; for get the intermediate feature center feature set

[0094] range loss is to reduce the distance between the same range VIS and IR features of the same person, while expanding the feature distance between different people in the same range VIS and IR features, represented as:

[0095]

[0096] where,

[0097]

[0098] where SM represents the cosine similarity matrix of input features A and . represents the L1 distance measurement function.

[0099] intermediate feature loss is to reduce the difference of the intermediate features, represented as:

[0100]

[0101] inter-modal loss is to reduce the intra-class distance between VIS, IR and intermediate features and expand the inter-class distance, represented as:

[0102]

[0103] where, and respectively represent the VIS, IR, intermediate modal center features (obtained by averaging all features of the same modality of each person) corresponding to the pedestrian.

[0104] In summary, the modal-range pedestrian center contrast loss is defined as follows:

[0105]

[0106] Thus, the joint classification loss Triplet loss and modal-range pedestrian center contrast loss The loss is represented as:

[0107]

[0108] wherein, represents the classification loss, represents the triplet loss, represents the modal-range pedestrian center contrast loss.

[0109] The embodiment also provides a cross-modal pedestrian re-identification system for intermediate modal learning under high-order structure modeling, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor, when the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0110] In the embodiment, the SYSU-MM01 / RegDB / LLCM dataset is used for comparative verification under the setting of searching infrared pictures from pedestrian visible light pictures, and Table 1 shows the comparison results of the Rank-1 and mAP indexes of the method proposed in the application and other cross-modal pedestrian re-identification methods on the SYSU-MM01 / RegDB / LLCM dataset. The largest value in Table 1 is displayed in bold. As can be seen from Table 1, the method of the application has higher accuracy and robustness compared with other cross-modal pedestrian re-identification methods, and the Rank-1(%) and mAP(%) are the best.

[0111] Table 1

[0112]

[0113] In Table 1, DEEN corresponds to the method proposed by Zhang et al. (Zhang, Y.; and Wang, H. 2023. Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2153-2162.)

[0114] MAUM corresponds to the method proposed by Liu et al. (Liu, J.; Sun, Y.; Zhu, F.; Pei, H.; Yang, Y.; and Li, W. 2022a. Learning memory-augmented unidirectional metrics for cross-modality person re-identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 19366-19375.)

[0115] MMN corresponds to the method proposed by Zhang et al. (Zhang, Y.; Yan, Y.; Lu, Y.; and Wang, H. 2021. Towards a unified middle modality learning for visible-infrared person re-identification. In Proceedings of the ACM International Conference on Multimedia, 788-796.)

[0116] CAJ corresponds to the method proposed by Ye et al. (Ye, M.; Ruan, W.; Du, B.; and Shou, M. Z. 2021a. Channel Augmented Joint Learning for Visible-Infrared Recognition. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 13567-13576.)

[0117] PMT corresponds to the method proposed by Lu et al. (Lu, H.; Zou, X.; and Zhang, P. 2023. Learning progressive modality-shared Transformers for effective visible-infrared person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, 1835-1843.)

[0118] SPOT corresponds to the method proposed by Chen et al. (Chen, C.; Ye, M.; Qi, M.; Wu, J.; Jiang, J.; and Lin, C.-W. 2022a. Structure-aware positional Transformer for visible-infrared person re-identification. IEEE Transactions on Image Processing, 31:2352-2364.)

[0119] TMSE corresponds to the method proposed by Liu et al. (Liu, J.; Wang, J.; Huang, N.; Zhang, Q.; and Han, J. 2022b. Revisiting modality-specific feature compensation for visible-infrared person re-identification. IEEE Transactions on Circuits and Systems for Video Technology, 32(10):7226-7240.)

[0120] D 2 DA corresponds to the method proposed by Wan et al. (Wan, L.; Sun, Z.; Jing, Q.; Chen, Y.; Lu, L.; and Li, Z. 2023. D 2 DA: Geometry-guided dual-alignment learning for RGB-infrared person re-identification. Pattern Recognition, 135:109150.)

[0121] LbA corresponds to the method proposed by Park et al. (Park, H.; Lee, S.; Lee, J.; and Ham, B. 2021. Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 12046-12055.)

[0122] DDAG corresponds to the method proposed by Ye et al. (Ye, M.; Shen, J.; J. Crandall, D.; Shao, L.; and Luo, J. 2020. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. In Proceedings of the European Conference on Computer Vision, 229-247.)

[0123] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0124] The present application is described in terms of flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and / or block of the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0125] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowcharts and / or block diagrams flow or flows and / or block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowcharts and / or block diagrams flow or flows and / or block or blocks.Figure 1 one or more processes and / or functions described in one or more blocks. Figure 1 one or more processes and / or functions described in one or more blocks.

[0127] The above descriptions are only preferred embodiments of the present application, not intended to limit the present application to other forms described. Any person skilled in the art may make changes or modifications to the disclosed technical contents as equivalent embodiments. However, any simple modification, equivalent change and modification made to the above embodiments without departing from the technical solution of the present application, and according to the technical essence of the present application, still belong to the protection scope of the present application.

Claims

1. A cross-modality pedestrian re-identification method based on high-order structure modeling, characterized in that, The method comprises the following steps: 1) extracting visible light (VIS) pictures and infrared (IR) pictures of a plurality of pedestrians in pairs from a data set to form a VIS-IR picture pair training data set; 2) constructing a cross-modal pedestrian re-identification network model based on high-order structure modeling, the cross-modal pedestrian re-identification network model mainly comprising a backbone network, a short-long range feature extraction module, a high-order structure learning module and a shared space learning module, the backbone network being used to extract basic features of an image, the short-long range feature extraction module being used to extract features of different range scales from the image, the high-order structure learning module being used to learn high-order structure information in the image to better represent features of the pedestrian, and the shared space learning module being used to reduce differences of the pedestrian in different modalities; the cross-modal pedestrian re-identification network model is trained by using the VIS-IR picture pair training data set to obtain model parameters that can be generalized; 3) using the trained cross-modal pedestrian re-identification network model for cross-modal retrieval to realize pedestrian re-identification between different modalities; In step 2), the cross-modal pedestrian re-identification network model is implemented in the following manner: A) for the VIS picture and the IR picture input in pairs, extracting pedestrian VIS basic features and pedestrian IR basic features by using the backbone network, and then inputting the extracted pedestrian VIS basic features and the pedestrian IR basic features into the short-long range feature extraction module to extract pedestrian VIS short-long range features and pedestrian IR short-long range features; B) obtaining high-order structure information contained in the pedestrian VIS short-long range features and the pedestrian IR short-long range features by using the high-order structure learning module to obtain pedestrian VIS short-long range enhanced features and pedestrian IR short-long range enhanced features to depict high-order relationships between different local features of the pedestrian, and to enhance feature representation without being affected by model collapse; C) inputting the obtained pedestrian VIS short-long range enhanced features and the pedestrian IR short-long range enhanced features into the shared space learning module to learn a discriminative and reasonable shared feature space by generating intermediate features to reduce negative effects of the modality difference; D) Introducing modal-range pedestrian center contrast loss to improve the feature representation and reduce the difference between the same pedestrian VIS, IR and intermediate features, and joint classification loss and triplet loss , the cross-modal pedestrian re-identification network model is iteratively trained through the VIS-IR picture pair to obtain the trained cross-modal pedestrian re-identification network model. In step B), in order to avoid the problem of super-edge representation convergence of the hypergraph model in constructing high-order relationships of pedestrians, a feature whitening operation is introduced to map the features to a spherical space, reduce the coupling between the features, and better establish the high-order relationships between the feature nodes. The feature nodes after whitening are as follows: wherein, is a set of feature nodes, each feature node representing a feature of the set of features; represents a mean value of the features and represents an affine transformation parameter; Using cross-correlation learning to associate matrices Y to obtain the connections between nodes as follows: Y wherein, ; denotes the number of nodes, which is equal to ; denotes the number of hyper-edges; denotes the number of channels; denotes a linear transformation layer; and denote diagonal line operation learning distance metric method and node pair corresponding hyper-edge contribution function, respectively; denotes a step function; Then the high-order structure enhanced pedestrian features are obtained through hypergraph convolution which is expressed as: wherein, denotes an identity matrix; denotes a weight matrix; and denotes a node degree matrix and a hyperedge degree matrix; are learnable parameters; correspond to pedestrian VIS, IR short-range features and pedestrian VIS, IR long-range features , resulting in pedestrian VIS, IR short-range enhanced features and pedestrian VIS, IR long-range enhanced features , resulting in a set of structure enhanced features .

2. The high-order structure modeling based cross-modal pedestrian re-identification method according to claim 1, wherein, In step 1), the data set is a SYSU-MM01 / RegDB / LLCM cross-modal pedestrian re-identification data set, K VIS pictures and K IR pictures of P pedestrians are extracted from the SYSU-MM01 / RegDB / LLCM cross-modal pedestrian re-identification data set in pairs to form the VIS-IR picture pair training data set.

3. The high-order structure modeling based cross-modal pedestrian re-identification method according to claim 1, characterized in that, In step A), the backbone network comprises two first-stage feature extraction modules, and a second-to-fourth stage feature extraction module and a fifth stage feature extraction module, the short-long range feature extraction module comprises a short-range feature extraction module and a long-range feature extraction module, the short-range feature extraction module and the long-range feature extraction module are respectively composed of a convolutional network and a Transformer network; a pair of VIS and IR pictures are respectively input into the first-stage feature extraction module, and then are simultaneously input into the second-to-fourth stage feature extraction module to obtain pedestrian VIS basic features and pedestrian IR basic features ; then the obtained pedestrian VIS basic features and the pedestrian IR basic features are respectively input into the short-range feature extraction module and the long-range feature extraction module to obtain pedestrian VIS, IR short-range features and pedestrian VIS, IR long-range features , the feature size is , wherein , W and respectively represent the height, width and channel size of the feature, so as to obtain a feature set which is used as the input of the high-order structure learning module; The structural enhancement feature set is obtained through the high-order structure learning module ; and the obtained feature set is input into the shared space learning module to obtain an intermediate feature set ; after obtaining the structural enhancement feature set and the intermediate feature set , a deep semantic association of features is performed through a fifth stage feature extraction module.

4. The high-order structure modeling based cross-modal pedestrian re-identification method according to claim 1, wherein, In step C), the features from different modalities and ranges are aligned by using graph attention, and intermediate features are generated to reduce the difference between the VIS and IR modalities by learning the intermediate features; The generation method of the intermediate features is as follows: the inner product and the Softmax function are established and The relationship between the nodes is as follows: wherein, is a similarity matrix; and is a linear transformation layer; is a Softmax function; Aligning similar nodes between and results in which is represented as: wherein, is a graph attention operation; is a linear transformation layer; is a hyperparameter to reduce low similarity nodes; is a unit column vector, and are ReLU and averaging operation, respectively. Based on the above process, , Aligning the generated intermediate features which is expressed as: Similarly, respectively align , Generate intermediate features , so as to obtain the intermediate feature set ; then the fifth stage feature extraction module is used for deep semantic association of the feature set and ; then, through and , global and local pooling are carried out respectively to obtain the one-dimensional structure enhanced feature set and the one-dimensional intermediate feature set .

5. The high-order structure modeling based cross-modal pedestrian re-identification method according to claim 4, characterized in that, In step D), an identity center that is robust to appearance variations of pedestrians is obtained by a weighted average of the features of each person in one modality and a specific range; for the pedestrian center features labeled as in the VIS modality and long-range features, it is obtained by the following way: wherein, is the number of VIS features per pedestrian; Thus, for , the VIS and IR central feature sets are obtained where is the number of pedestrians in the training set; for , the intermediate feature central feature set is obtained ; intra-range loss is to reduce the distance between the intra-range VIS and IR features of the same person while expanding the feature distance between different persons in the intra-range VIS and IR features, expressed as: wherein, wherein, represents a cosine similarity matrix of input features ; represents an LI distance metric function; intermediate feature loss is to reduce the difference in intermediate features, denoted as: Inter-modal loss is to reduce the intra-class distance between VIS, IR and intermediate features and enlarge the inter-class distance, denoted as: wherein, , and respectively represent the VIS, IR, intermediate modality center features corresponding to the pedestrian. In summary, modal-range pedestrian center contrast loss is defined as follows: Thus, the joint classification loss , the triplet loss , and the modal-range pedestrian-centric contrastive loss are represented as: wherein, denotes the classification loss, denotes the triplet loss, denotes the modal-range pedestrian-centric contrastive loss.

6. A cross-modal pedestrian re-identification system based on high-order structure modeling, characterized in that, The computer program product comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, and when the processor executes the computer program instructions, the method steps of any one of claims 1-5 can be implemented.

Images