Cross-modal pedestrian re-identification method based on pose feature alignment

Abstract

A cross-modal pedestrian re-identification method based on pose feature alignment, comprising: 1, constructing a training set; 2, constructing a cross-modal pedestrian re-identification network, including a double-flow network, a pose estimator, a pose-guided feature enhancement block and a first generalized mean pooling layer; 3, selecting a visible light modality image, an infrared modality image, and obtaining an additional modality image through data augmentation, and inputting into the cross-modal pedestrian re-identification network to obtain the final splicing feature; 4, modal feature alignment learning is carried out on the cross-modal pedestrian re-identification network. The application enhances the feature discrimination ability by introducing the key part feature of the person, so as to eliminate the intra-class interference, and guide the cross-modal pedestrian re-identification network to simultaneously learn the cross-modal consistent feature of the same person, and the application also proposes a modal feature alignment learning method, which reduces the great difference between the modes from the aspects of feature distribution alignment and hierarchical aggregation, and also fully considers the intra-modal and inter-modal constraints.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a cross-modal person re-identification method based on pose feature alignment. Background Technology

[0002] Cross-modal person re-identification aims to retrieve the same pedestrian from different camera modalities. Given visible light and infrared images of a target person, the model needs to retrieve the same person across non-overlapping camera networks. It is more challenging than single-modal visible light person re-identification, with the main difficulties stemming from two aspects: complex issues similar to single-modal visible light person re-identification tasks, such as pose variations, viewpoint changes, and occlusion; and significant differences between the visible light and infrared modalities.

[0003] Most existing methods learn only coarse-grained features at the full image level or use rigid horizontal partitioning to obtain local features. This is easily affected by intra-class variations, such as different camera angles, shooting scenes, and lighting conditions for the same pedestrian category, and is limited to extracting semantically aligned local features. Furthermore, most existing methods use instance-level metric learning, which is unsuitable for situations with large intra-class modal differences. Other methods that reduce the Euclidean distance between feature centers of different modalities ignore intra-modal constraints. Therefore, cross-modal features in the aforementioned methods are difficult to map into a consistent space, thus limiting their performance.

[0004] Furthermore, to reduce modal differences, some methods introduce a third additional mode, similar to the infrared mode, through simple transformations or lightweight encoders, enhancing the model's robustness to modal disturbances. Building upon this, methods based on instance-level or modality-centric metric learning are employed to optimize the model. However, these methods neglect the comprehensive distribution of all instances or intramodal constraints.

[0005] Furthermore, human pose keypoint extraction is a popular task in computer vision, attracting considerable attention from researchers in recent years. Because it can extract effective human information, it has been applied in many other studies. In pedestrian re-identification, pose information indicates the location of the human body, facilitating research and finding applications in several sub-tasks, particularly occluded pedestrian re-identification. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a cross-modal pedestrian re-identification method based on pose feature alignment, which aims to eliminate interference caused by complex intra-class problems and overcome modal difference problems.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0008] This invention provides a cross-modal person re-identification method based on pose feature alignment, comprising the following steps:

[0009] S1. Acquire multiple visible light modal images and multiple infrared modal images, and use the acquired images to construct training sets, validation sets, and test sets;

[0010] S2. Construct a cross-modal pedestrian re-identification network, which includes a ResNet50 two-stream network, a pose estimator, a pose-guided feature enhancement block, and a first generalized mean pooling layer; the ResNet50 two-stream network is connected to the first generalized mean pooling layer, and the pose estimator is connected to the pose-guided feature enhancement block;

[0011] S3. Randomly select one visible light modal image and one infrared modal image from the training set, and obtain an additional modal image through data augmentation. Input the three different modal images into the ResNet50 two-stream network and the pose estimator respectively to obtain global feature maps and m key points. Input the global feature map into the first generalized mean pooling layer to obtain the global feature vector. Input the m key points and the global feature map into the pose guidance feature enhancement block, and concatenate the output of the pose guidance feature enhancement block with the global feature vector to obtain the final concatenated feature.

[0012] S4. Construct the overall loss function for the modal feature alignment learning method, and use the overall loss function to calculate the final concatenated feature x. m The total loss is calculated, and S3 to S4 are repeated until the set number of iterations is reached. The validation set is used for validation, and the set of weights with the best accuracy on the validation set is selected as the weights of the cross-modal person re-identification network to obtain the trained cross-modal person re-identification network.

[0013] S5. Test the trained cross-modal pedestrian re-identification network using the test set.

[0014] Furthermore, the ResNet50 dual-stream network in S2 includes two different convolutional blocks: convolutional block 0, convolutional block 1, convolutional block 2, convolutional block 3, and convolutional block 4.

[0015] Two different convolutional blocks of type zero are spliced ​​together and then connected to convolutional blocks of type one, type two, type three, and type four in sequence.

[0016] The pose estimator in S2 is a pre-trained pose estimator;

[0017] The pose-guided feature enhancement block in S2 includes a second generalized mean pooling layer and a max pooling layer connected to the second generalized mean pooling layer. The max pooling layer is a one-dimensional max pooling layer.

[0018] Furthermore, step S3 specifically includes the following steps:

[0019] S31. Randomly select one visible light modal image and one infrared modal image from the training set, and obtain an additional modal image through data augmentation. Input the three images of different modalities into the ResNet50 two-stream network and the pose estimator respectively to obtain global feature maps. And m key points of the portrait image;

[0020] S32, global feature map The input is fed into the first generalized mean pooling layer to obtain global features.

[0021] S33. Input m key points into the pose-guided feature enhancement block, and use the m key points to generate a two-dimensional Gaussian distribution heatmap centered on each extracted key point;

[0022] S34. Combine the two-dimensional Gaussian distribution heatmap with the global feature map in S31. By performing the Hadamard product, we can obtain the pose-guided local feature map for each key point;

[0023] S35. Input the pose-guided local feature map of each keypoint into the second generalized mean pooling layer of the pose-guided feature enhancement block to obtain the feature vector.

[0024] S36. The feature vector The input is fed into the max-pooling layer of the pose-guided feature enhancement block to obtain the enhanced local feature vector. And noise is removed in the pose-guided feature enhancement block;

[0025] S37. Remove noise from the local feature vectors. With global features The features are then concatenated to obtain the final concatenated feature x. m .

[0026] Furthermore, the additional modal image obtained through data augmentation in S31 specifically involves:

[0027] By randomly selecting one channel to introduce an additional modality into a visible person image, which can be any one of the red, green, or blue channels, an additional modality image is obtained in place of other channels.

[0028] Furthermore, S35 is specifically expressed by the following formula:

[0029]

[0030] Where GeM(.) represents the pooling operation of the first generalized mean pooling layer; This represents the heatmap signal at the j-th position on a two-dimensional Gaussian distribution heatmap.

[0031] Furthermore, the overall loss function of the modality feature alignment learning method in S4 is as follows:

[0032] L = L id +λ1L fda (V,E,T)+λ2L tmct +λ3L ic

[0033] Where L is the total loss; L id For loss of identity; L fda (V,E,T) represents the identity-aware feature distribution alignment loss; V,E,T represent the visible light feature set, the additional modality feature set, and the infrared feature set, respectively; L tmct L is the loss function for the three-mode central triplet; ic λ1 represents the in-modal center loss; λ2 and λ3 are the hyperparameters of the identity-aware feature distribution alignment loss, the hyperparameters of the trimodal center triplet loss function, and the hyperparameters of the in-modal center loss, respectively.

[0034] Furthermore, the loss of identity is specifically as follows:

[0035]

[0036] Where N is the number of samples involved in the calculation, y i Indicates the identity label of the sample. Representing an image It was identified as its identity tag y i The predicted probability output.

[0037] Furthermore, the identity-aware feature distribution alignment loss is specifically as follows:

[0038] L fda (V,E,T)=L fda (V,T)+L fda (E,T)

[0039] Among them, L fda (E,T) represents the identity-aware feature distribution alignment loss between the additional module feature set and the infrared feature set;

[0040] L fda (V,T) represents the identity-aware feature distribution alignment loss between the visible light feature set and the infrared feature set; and L fda (V,T) satisfy the following relationship:

[0041]

[0042] Among them, MMD′ 2 (V i ,T i ) = max[MMD 2 (V i ,T i )-ρ1,0]

[0043] P represents the quantity of each identity; MMD′ 2 (.) refers to the maximum mean difference loss; V i ,T i ρ1 represents the visible light feature distribution and infrared feature distribution of the i-th identity, respectively; ρ1 represents the hyperparameter.

[0044] Furthermore, the three-modal central triplet loss function is as follows:

[0045]

[0046] Where ρ2 is a hyperparameter, and [·] + =max(·,0), where p and n represent positive and negative samples respectively, and a≠i indicates that pedestrians belong to different categories. and These are represented as positive sample centers and negative sample centers, respectively.

[0047] Let i be the feature center of the i-th identity in the visible light mode of the set batch, i be the feature center of the i-th identity in the additional mode of the set batch, and i be the feature center of the i-th identity in the infrared mode of the set batch, respectively, and satisfy the following relationship:

[0048]

[0049]

[0050]

[0051] Where K represents the number of features, and v, e, and t represent the visible light mode, the additional mode, and the infrared mode, respectively; These represent the β-th visible image feature of the i-th person in the set batch, the β-th additional image feature of the i-th person in the set batch, and the β-th infrared image feature of the i-th person in the set batch, respectively.

[0052] Furthermore, the intramodal center loss is specifically as follows:

[0053]

[0054] The beneficial effects of this invention are:

[0055] This invention proposes a cross-modal person re-identification method based on pose-guided modality-invariant feature alignment learning to address the complex intra-class problem and significant modal differences between visible and infrared modalities. This invention utilizes pose-guided feature enhancement blocks to extract features from key human body parts, significantly enhancing feature representations to handle intra-class interference and explicitly guiding the person re-identification model to learn cross-modal consistent features.

[0056] Based on enhanced features, this invention proposes a modality feature alignment learning method (i.e., the process of establishing the overall loss function of the modality feature alignment learning method) to reduce the gap between modalities and force the network to learn modality-invariant feature representations. This method fully solves the problem from two aspects: feature distribution alignment and hierarchical aggregation. Attached Figure Description

[0057] Figure 1 This is a flowchart of the present invention;

[0058] Figure 2 This is a general framework diagram of the present invention;

[0059] Figure 3 This is a graph showing the search results of the present invention. Detailed Implementation

[0060] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0061] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0062] The purpose of this invention is to overcome the shortcomings of the prior art and provide a cross-modal person re-identification method based on pose feature alignment. Pose-guided modality-invariant feature alignment aims to eliminate interference from complex intra-class (i.e., same pedestrian category) problems and overcome modality difference issues. By randomly selecting one channel (red channel R, green channel G, or blue channel B) to introduce an additional modality into the visible person image, replacing other channels, the robustness of the input data to color changes is greatly improved. Therefore, three modal person images with labels are obtained, namely... Where m1∈{v,t,e} represents the visible light mode, the infrared mode, and the additional mode, Z m1 Let y be the input image, y be the label of the input image, and N be the number of samples for each modality.

[0063] like Figure 2 As shown, this invention employs a two-branch framework to learn discriminative feature representations. On one hand, pedestrian images of three different modalities are input into a pose estimator module to extract keypoints and obtain pose information. Simultaneously, these images are also input into a two-stream ResNet50 backbone network (i.e., a two-stream ResNet50 network) to extract global features. The first convolutional block is distinct, capturing shallow features specific to each modality, while the deep convolutional blocks are shared, learning deep representations shared across modalities. Then, a pose-guided feature enhancement block is introduced to extract enhanced local features x. m k Noise is removed in the pose-guided feature enhancement module. Then, the final feature representation (i.e., the final concatenated feature) x m It is a fusion of global features and enhanced local features. Based on the final feature representation, this invention introduces a modality feature alignment learning method, which designs identity-aware feature distribution alignment loss and hierarchical aggregation strategy loss to reduce the divergence between different modalities.

[0064] Reference Figure 1 and Figure 2 Specifically, this application provides a cross-modal pedestrian re-identification method based on pose feature alignment, including the following steps:

[0065] S1. Acquire multiple visible light modal images and multiple infrared modal images, and use the acquired images to construct training sets, validation sets, and test sets;

[0066] S2. Construct a cross-modal pedestrian re-identification network, which includes a ResNet50 two-stream network, a pose estimator, a pose-guided feature enhancement block, and a first generalized mean pooling layer; the ResNet50 two-stream network is connected to the first generalized mean pooling layer, and the pose estimator is connected to the pose-guided feature enhancement block;

[0067] This invention designs a posture-guided feature enhancement block (PBA) to extract fine-grained local features across different modalities. The PBA benefits the model in several ways: it provides an attention map of body parts, allowing the model to focus on information-rich and reliable body regions, and filters out noisy features caused by posture changes, viewpoint changes, and background changes. By extracting key fine-grained features through the PBA, it supplements global features, improving the model's recognition ability and explicitly guiding the model to learn cross-modal consistency features of the same person. Introducing this module yields a robust feature vector for further modal alignment learning.

[0068] S3. Randomly select one visible light modal image and one infrared modal image from the training set, and obtain an additional modal image through data augmentation. Input the three different modal images into the ResNet50 two-stream network and the pose estimator respectively to obtain global feature maps and m key points. Input the global feature map into the first generalized mean pooling layer to obtain the global feature vector. Input the m key points and the global feature map into the pose guidance feature enhancement block, and concatenate the output of the pose guidance feature enhancement block with the global feature vector to obtain the final concatenated feature.

[0069] S4. Perform modal feature alignment learning on the cross-modal pedestrian re-identification network;

[0070] Specifically, a global loss function is constructed for the modality feature alignment learning method, and the final concatenated feature x is calculated using the global loss function. m The total loss is calculated, and S3 to S4 are repeated until the set number of iterations is reached. The validation set is used for validation, and the set of weights with the best accuracy on the validation set is selected as the weights of the cross-modal person re-identification network to obtain the trained cross-modal person re-identification network.

[0071] S5. Test the trained cross-modal pedestrian re-identification network using the test set.

[0072] In some embodiments, the ResNet50 two-stream network in S2 includes two different convolutional blocks: a zero convolutional block, a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block.

[0073] Two different convolutional blocks of type zero are spliced ​​together and then connected to convolutional blocks of type one, type two, type three, and type four in sequence.

[0074] The pose estimator in S2 is a pre-trained pose estimator;

[0075] The pose-guided feature enhancement block in S2 includes a second generalized mean pooling layer and a max pooling layer connected to the second generalized mean pooling layer. The max pooling layer is a one-dimensional max pooling layer.

[0076] Generalized average pooling (GA) is a method that combines generalized pooling and average pooling. It primarily obtains each point in the output feature map by calculating the weighted average of the regions covered by the convolutional kernel as it slides across the input feature map. The weighting coefficients can be learned or set manually. Here, GA is used instead of the more widely used max pooling or average pooling. This is because GA encompasses both pooling schemes and captures discriminative features specific to the domain compared to these two methods.

[0077] In some embodiments, S3 specifically includes the following steps:

[0078] S31. Randomly select one visible light modal image and one infrared modal image from the training set, and obtain an additional modal image through data augmentation. Input the three images of different modalities into the ResNet50 two-stream network and the pose estimator respectively to obtain global feature maps. and the image of the person k={p j m key points of |j=1,2,…,m}; where m=14; the person image is a three-modal person image with labels;

[0079] S32, global feature map The input is fed into the first Generalized Mean Pooling (GeM) layer to obtain global features.

[0080] S33. Input m key points into the pose-guided feature enhancement block, and use the m key points to generate a two-dimensional Gaussian distribution heatmap centered on each extracted key point;

[0081] S34. Combine the two-dimensional Gaussian distribution heatmap with the global feature map in S31. By performing the Hadamard product, we can obtain the pose-guided local feature map for each key point;

[0082] S35. Input the local feature map of pose guidance for each key point into the second generalized mean pooling layer of the pose guidance feature enhancement block to obtain a 2048-dimensional feature vector. It corresponds to specific local features of pedestrians;

[0083] S36. The feature vector The input is fed into the max-pooling layer of the pose-guided feature enhancement block to obtain the enhanced local feature vector. This approach fuses feature vectors with information from various body parts, ignoring redundant local information. It's important to note that a one-dimensional adaptive max-pooling operation is applied here, projecting and combining n 2048-dimensional vectors into a single 2048-dimensional vector. And noise is removed in the pose-guided feature enhancement block;

[0084] S37. Remove noise from the local feature vectors. With global features The features are then concatenated to obtain the final concatenated feature x. m .

[0085] In some embodiments, obtaining an additional modal image through data augmentation in S31 specifically involves:

[0086] By randomly selecting one channel (red, green, or blue) to introduce an additional modality into a visible person image, replacing other channels, an additional modal image is obtained. This significantly improves the robustness of the input data to color changes. Therefore, three modal images of people with labels are obtained; that is...

[0087] In some embodiments, S35 is expressed by the following formula:

[0088]

[0089] Where GeM(.) represents the pooling operation of the first generalized mean pooling layer; Let represent the heatmap signal at the j-th position on a two-dimensional Gaussian distribution heatmap, and Where C', H', and W' represent the number of channels, height, and width, respectively.

[0090] In some embodiments, the overall loss function of the modality feature alignment learning method in S4 is as follows:

[0091] L = L id +λ1L fda (V,E,T)+λ2L tmct +λ3L ic

[0092] Where L is the total loss; L id For loss of identity; L fda (V,E,T) represents the identity-aware feature distribution alignment loss; V,E,T represent the visible light feature set, the additional modality feature set, and the infrared feature set, respectively; L tmct L is the loss function for the three-mode central triplet; icλ1 represents the in-modal center loss; λ2 and λ3 are the hyperparameters of the identity-aware feature distribution alignment loss, the hyperparameters of the trimodal center triplet loss function, and the hyperparameters of the in-modal center loss, respectively.

[0093] In some embodiments, this invention incorporates identity loss to jointly optimize the pose-guided modality-invariant feature alignment algorithm. The identity loss treats images with the same identity label across all modalities as belonging to the same class. This invention uses cross-entropy to calculate the identity loss, which is specifically as follows:

[0094]

[0095] Where N is the number of samples involved in the calculation, y i Indicates the identity label of the sample. Representing an image It was identified as its identity tag y i The predicted probability output.

[0096] In some embodiments, the present invention further aligns the feature distributions between the extra modality and the infrared modality to reduce modal divergence and learn modality-invariant features. The extra modality image retains structural information while ignoring the color information of the visible light modality, thereby reducing the distribution distance between the extra modality and the infrared modality. This allows the network to further mine structural relationships and exhibits stronger robustness to color changes, further narrowing the gap between the visible light and infrared modalities. Therefore, a total identity-aware feature distribution alignment loss can be obtained; the specific identity-aware feature distribution alignment loss is as follows:

[0097] L fda (V,E,T)=L fda (V,T)+L fda (E,T)

[0098] Among them, L fda (E,T) represents the identity-aware feature distribution alignment loss between the additional module feature set and the infrared feature set;

[0099] L fda (V,T) represents the identity-aware feature distribution alignment loss between the visible light feature set and the infrared feature set; and L fda (V,T) satisfy the following relationship:

[0100]

[0101] Among them, MMD′ 2 (V i ,T i ) = max[MMD 2 (V i ,Ti )-ρ1,0]

[0102] MMD 2 (V,T)=E V [k(x v ,x v ′)]+E T [k(x t ,x t ′)]-2E V,T [k(x v ,x t (1)

[0103] P represents the quantity of each identity; MMD′ 2 (.) refers to the maximum mean difference loss; V i ,T i Let be the visible light feature distribution and the infrared feature distribution of the i-th identity, respectively. i is a constraint condition with a pedestrian tag, which is to align the two modal feature distributions within an identity tag; ρ1 represents a hyperparameter, which can control the number of distribution alignments, thereby maintaining the balance between the two different modalities.

[0104] In formula (1), the first two terms are the kernel similarity of samples from the same modality, i.e., from the visible light modality and the infrared modality, respectively, and the third term calculates the kernel similarity of samples in the visible light modality and the infrared modality. By minimizing the maximum mean difference loss, the cross-modal similarity is made as close as possible to the same modal similarity, thereby keeping the two distributions consistent. Considering the identity condition and preserving the identity discrimination property, this invention uses an improved version of the maximum mean difference.

[0105] This invention employs the widely used domain difference metric, Maximum Mean Discrepancy (MMD), to align heterogeneous feature distributions. It matches the higher-order moments of two distributions and effectively determines the distance between them. The maximum mean difference is a test statistic that measures the difference between two distributions by embedding them into a reproducing kernel Hilbert space.

[0106] In some embodiments, this invention proposes a hierarchical aggregation strategy consisting of two functions: a trimodal center triplet loss function and an intramodal center loss function. The trimodal center triplet loss function increases the inter-class distance and reduces the intra-class distance in Euclidean space. In a mini-batch, the center of features for each class in each modality is calculated; the trimodal center triplet loss function is specifically as follows:

[0107]

[0108] Where ρ2 is a hyperparameter, and [•] + =max(•,0), where p and n represent positive and negative samples respectively, and a≠i indicates that pedestrians belong to different categories. and These are represented as positive sample centers and negative sample centers, respectively.

[0109] Let be the feature centers of the i-th identity of the visible light mode in the mini-batch, the feature centers of the i-th identity of the extra mode in the mini-batch, and the feature centers of the i-th identity of the infrared mode in the mini-batch, respectively, and satisfy the following relationships:

[0110]

[0111]

[0112]

[0113] Where K represents the number of features, and v, e, and t represent the visible light mode, the additional mode, and the infrared mode, respectively; Let represent the β-th visible image feature of the i-th person in the mini-batch, the β-th additional image feature of the i-th person in the mini-batch, and the β-th infrared image feature of the i-th person in the mini-batch, respectively.

[0114] While both the trimodal center triplet loss and the feature distribution alignment loss can narrow the distance between two modes, the latter statistically matches all higher-order moments and is stronger in terms of modal feature distribution alignment, while the trimodal center triplet loss can guarantee the property that inter-class features are distinguishable in all modes.

[0115] In this embodiment, the present invention proposes an intra-modal centering loss to reduce the absolute distance of the same pedestrian features within a modality. The intra-modal centering loss is specifically as follows:

[0116]

[0117] The hierarchical aggregation strategy fully considers the situation where there are significant differences between instances of different modalities within the same identity. It first uses center-based triple constraints to ensure inter-class separation, and then further handles the occurrence of anomalies by aggregating each instance of the same identity within a modality. This fully considers the constraints within and between modalities.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-modal person re-identification method based on pose feature alignment, characterized in that, Includes the following steps: S1. Acquire multiple visible light modal images and multiple infrared modal images, and use the acquired images to construct training set, validation set and test set; S2. Construct a cross-modal pedestrian re-identification network, which includes a ResNet50 two-stream network, a pose estimator, a pose-guided feature enhancement block, and a first generalized mean pooling layer; the ResNet50 two-stream network is connected to the first generalized mean pooling layer, and the pose estimator is connected to the pose-guided feature enhancement block; S3. Randomly select one visible light modal image and one infrared modal image from the training set, and obtain an additional modal image through data augmentation. Input the three different modal images into the ResNet50 two-stream network and the pose estimator respectively to obtain the global feature map and m key points respectively. The global feature map is input into the first generalized mean pooling layer to obtain the global feature vector. The m key points and the global feature map are input into the pose guidance feature enhancement block. The output of the pose guidance feature enhancement block is concatenated with the global feature vector to obtain the final concatenated feature. S4. Construct the overall loss function for the modal feature alignment learning method, and use the overall loss function to calculate the final concatenated features. The total loss is calculated, and S3 to S4 are repeated until the set number of iterations is reached. The validation set is used for validation, and the set of weights with the best accuracy on the validation set is selected as the weights of the cross-modal person re-identification network to obtain the trained cross-modal person re-identification network. S5. Test the trained cross-modal person re-identification network using the test set; S3 specifically includes the following steps: S31. Randomly select one visible light modal image and one infrared modal image from the training set, and obtain an additional modal image through data augmentation. Input the three images of different modalities into the ResNet50 two-stream network and the pose estimator respectively to obtain global feature maps. And m key points of the portrait image; S32, global feature map The input is fed into the first generalized mean pooling layer to obtain global features. ; S33. Input the m key points of the person image into the pose-guided feature enhancement block, and use the m key points to generate a two-dimensional Gaussian distribution heatmap centered on each extracted key point. S34. Combine the two-dimensional Gaussian distribution heatmap with the global feature map in S31. By performing the Hadamard product, we can obtain the pose-guided local feature map for each key point; S35. Input the pose-guided local feature map of each keypoint into the second generalized mean pooling layer of the pose-guided feature enhancement block to obtain the feature vector. ; S36. The feature vector The input is fed into the max-pooling layer of the pose-guided feature enhancement block to obtain the enhanced local feature vector. Noise is removed in the pose-guided feature enhancement block; S37. Remove noise from the local feature vectors. With global features The pieces are then stitched together to obtain the final stitched features. ; The overall loss function of the modality feature alignment learning method in S4 is as follows: in, Total loss; Loss of identity; The loss is the alignment loss for identity-aware feature distribution; These are the visible light feature set, the additional modal feature set, and the infrared feature set, respectively. The loss function is the three-modal central triplet. For in-modal center loss; , , These are the hyperparameters of the identity-aware feature distribution alignment loss, the hyperparameters of the three-modal center triplet loss function, and the hyperparameters of the in-modal center loss, respectively.

2. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The ResNet50 two-stream network in S2 includes two different convolutional blocks: a zero convolutional block, a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block. Two different convolutional blocks of type zero are spliced ​​together and then connected to convolutional blocks of type one, type two, type three, and type four in sequence. The pose estimator in S2 is a pre-trained pose estimator; The pose-guided feature enhancement block in S2 includes a second generalized mean pooling layer and a max pooling layer connected to the second generalized mean pooling layer. The max pooling layer is a one-dimensional max pooling layer.

3. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, In step S31, obtaining an additional modal image through data augmentation specifically involves: By randomly selecting one channel to introduce an additional modality into a visible person image, which can be any one of the red, green, or blue channels, an additional modality image is obtained in place of other channels.

4. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The formula for S35 is as follows: Where GeM(.) represents the pooling operation of the first generalized mean pooling layer; Represents the first two-dimensional Gaussian distribution heatmap. j The heat map signal at each location.

5. The cross-modal pedestrian re-identification method according to claim 4, characterized in that, The specific loss of identity is as follows: in, N The number of samples used in the calculation. Indicates the identity label of the sample. Representing an image It was identified as its identity tag The predicted probability output.

6. The cross-modal pedestrian re-identification method according to claim 5, characterized in that, The specific identity-aware feature distribution alignment loss is as follows: in, The identity-aware feature distribution alignment loss is used between the additional module feature set and the infrared feature set. The loss is the alignment loss between the visible light feature set and the infrared feature set for identity perception feature distribution; and The following relationship must be satisfied: in, Indicates the quantity of each identity; (.) refers to the maximum mean difference loss; The first i The visible light feature distribution of the first identity and the first i The distribution of infrared features of each identity; This represents hyperparameters.

7. The cross-modal pedestrian re-identification method according to claim 6, characterized in that, The specific three-mode central triplet loss function is as follows: in, For hyperparameters, and , p and n These are represented as positive samples and negative samples, respectively. a≠i This indicates that pedestrians belong to different categories. and These are represented as positive sample centers and negative sample centers, respectively. , , These are the first visible light modes in the specified batch. i The feature center of each identity, the first additional modality in the set batch i The feature center of each identity, the first infrared mode in the set batch i Each identity has a feature center, and each satisfies the following relation: in, Indicates the number of features, v, e, t These represent the visible light mode, the additional mode, and the infrared mode, respectively. , , These represent the first in the set batch. i The first of the individuals β The visible image feature, the first in the set batch i The first of the individuals β Additional image features, in the set batch i The first of the individuals β One infrared image feature.

8. The cross-modal pedestrian re-identification method according to claim 7, characterized in that, The specific intramodal center loss is as follows: 。

Images