Cross-modal pedestrian re-identification method based on adaptive feature aggregation

Abstract

The invention discloses a cross-modal pedestrian re-identification method based on adaptive feature aggregation, and belongs to the technical field of computer vision. According to the method, a fine-grained feature aggregation module is introduced, features with similar semantics in a feature map are effectively aggregated, meanwhile, features with far semantics are separated, fine-grained information is utilized to the maximum extent, a fine-grained feature alignment module is constructed, and distribution differences between modals are reduced by aligning and correcting semantic information of cross-modal feature vectors, so that the accuracy of the cross-modal feature vectors is improved. It is ensured that feature vectors of two modals can be effectively compared in the same semantic space, and the problems that fine-grained information is lost and feature distribution between the modals is inconsistent in the feature map pooling process are solved. According to the method, features with similar semantics in the feature map are aggregated together in an attention pooling mode through adaptive feature aggregation, features irrelevant to semantics are separated, it is guaranteed that fine-grained information in the feature map is fully utilized, and feature distribution differences in different modes can be effectively coped with.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-modal pedestrian re-identification method based on adaptive feature aggregation, belonging to the technical field of computer vision. BACKGROUND

[0002] Pedestrian re-identification is a technology that matches cross-scene pedestrian images or videos through computer vision algorithms to determine whether a specific pedestrian is contained in a monitoring picture or video sequence.

[0003] In actual monitoring environments, due to the differences in resolution, shooting angle, and distance of cameras, the quality of the collected pedestrian images is often different. Traditional single-modal pedestrian re-identification technology mainly matches the identities of pedestrians based on visible light data, and has achieved performance beyond human recognition level on multiple visible light data sets; most visible light cameras have low imaging quality in poor lighting conditions, making it difficult to provide clear pedestrian images. Modern monitoring systems usually add infrared cameras as a supplement, and automatically switch to infrared cameras for image acquisition when the ambient light is insufficient; visible light cameras generate color-rich color images, while infrared cameras generate monotonous grayscale images, and there is a significant modal difference between the two. Single-modal pedestrian re-identification technology cannot meet the actual needs of all-weather monitoring systems.

[0004] Cross-modal pedestrian re-identification technology jointly analyzes the feature information of visible light images and infrared images to achieve accurate matching under different modalities, solve the problem of failure in weak light, night, and adverse weather conditions, track the activity path of pedestrians, and is applied to the fields of maintaining social order, combating illegal activities, protecting vulnerable groups, and finding missing persons.

[0005] In cross-modal pedestrian re-identification technology, existing methods have some deficiencies, especially in feature aggregation and alignment. Traditional pedestrian re-identification methods usually rely on advanced deep learning architectures, using multiple combinations of convolutional layers, activation functions, and pooling layers to process input images layer by layer, aiming to generate feature maps containing rich spatial features and semantic information, fully reflecting the appearance characteristics of pedestrians. Since direct classification of feature maps requires a huge amount of computation, feature maps usually need to be further processed into feature vectors, and this process usually relies on average pooling as the main means. Since the average pooling operation uniformly compresses the spatial information of the feature map, discriminative features and non-discriminative features are treated equally, and features with different semantics are forced to be aggregated together, so that some fine-grained information cannot be fully utilized, resulting in limited model capture ability for fine-grained features, significantly reducing the discriminability of cross-modal features.

[0006] The existing cross-modal pedestrian re-identification technology still faces many challenges, for example, how to further reduce the modal difference between visible light and infrared features, further effective feature extraction and representation method, and improve the matching accuracy, which is the key to improve the performance of cross-modal pedestrian re-identification technology, and more efficient algorithms are needed to optimize the cross-modal pedestrian re-identification technology to provide more reliable technical support for modern monitoring systems. SUMMARY

[0007] In order to solve the problems of loss of fine-grained information in the feature map pooling process and inconsistent feature distribution between modalities in the existing cross-modal pedestrian re-identification technology, the purpose of the present application is to provide a cross-modal pedestrian re-identification method based on adaptive feature aggregation, introduce a fine-grained feature aggregation module, effectively aggregate features with similar semantics in the feature map, and separate features with distant semantics, maximize the use of fine-grained information, construct a fine-grained feature alignment module, align and correct the semantic information of the cross-modal feature vector, reduce the distribution difference between modalities, and ensure that the feature vectors of the two modalities can be effectively compared in the same semantic space.

[0008] The purpose of the present application is realized by the following technical solutions:

[0009] The cross-modal pedestrian re-identification method based on adaptive feature aggregation disclosed by the present application comprises the following steps:

[0010] Step 1: import the infrared image and the visible light image into the ResNet network, and the features after processing by the modal sharing layer Figure One into the fine-grained shared classifier to generate corresponding weight maps;

[0011] After the infrared image and the visible light image pass through the fourth layer of the ResNet network, a feature map X is generated, and the feature map X is a three-dimensional tensor with a shape of (C, H, W), wherein C is the number of channels, H is the height of the feature map, and W is the width of the feature map;

[0012] The feature map X of the infrared image and the visible light image enters the fine-grained feature aggregation module and is reshaped into a tensor X1 with a shape of (HxW, C1), and C1 is the number of channels after aggregation;

[0013] The fine-grained shared classifier is a feedforward neural network that classifies the features contained in the feature map in terms of semantics, and the fine-grained shared classifier comprises a fully connected layer, a batch normalization layer, another fully connected layer and an activation function;

[0014] The tensor X1 of the reconstructed infrared image and the visible light image is simultaneously input into the fine-grained shared classifier in the form of shape (H×W,C1) for classification. Instead of simply compressing the features uniformly, it groups features with similar semantics in the feature map into one class, promoting information sharing between infrared modal features and visible light modal features.

[0015] The feature maps X of the infrared and visible light images with input dimension C are processed by a fine-grained shared classifier and mapped to the final number of categories n, resulting in a weight map A of the infrared and visible light images with shape (H×W,n).

[0016] The sigmoid activation function is used to activate the weight graph A, so that the values ​​of the elements in the weight graph A are between 0 and 1, thus maintaining the stability of the weight graph A.

[0017] Step 2: Aggregate the weight maps of the infrared image and the visible light image with the original feature map to generate a weighted result. Then, average pool the weighted result to obtain the aggregated feature vectors of the infrared image and the visible light image, respectively.

[0018] The weight map A of the infrared and visible light images from step 1 is combined with the feature map X of the original infrared and visible light images to obtain a weighted result that includes semantic aggregation classification, providing a basis for subsequent attention pooling.

[0019] The weighted tensor X is obtained by performing matrix multiplication between the weighted graph A and the reshaped tensor X1. res As shown in equation (1):

[0020]

[0021] The combination of features and corresponding attention weights for each batch is achieved by performing matrix multiplication between the weight map A and the reshaped tensor X1.

[0022] For weighted tensor X res The mean of the first dimension is used to obtain the aggregated feature tensor X. part This enables fine-grained feature integration and improves adaptability to specific tasks.

[0023] The global features obtained from global average pooling are compared with the aggregated feature tensor X. part By splicing, the final feature X is obtained. final Feature X final It not only includes the average representation of global features, but also incorporates attention mechanisms for important regions to capture fine-grained information between modalities;

[0024] Step 3: Using a multi-classifier cross-prediction algorithm and a gradient pause strategy, aggregate the feature tensor X of the infrared and visible light images. partThe visible light classifier, the infrared classifier and the shared classifier are respectively introduced for prediction to obtain corresponding loss values;

[0025] The visible light classifier, the infrared classifier and the shared classifier are all composed of a full connection layer, which maps the dimension of the input feature to the total number of class identities;

[0026] In the training process, the aggregated feature tensor X of the infrared image and the visible light image part In addition to being input into the shared classifier for training, the infrared image and the visible light image are also respectively input into the infrared classifier and the visible light classifier for independent training, and the prediction result of the infrared image in the infrared classifier is P i,i , and the prediction result of the visible light image in the visible light classifier is P r,r , and the prediction result of the infrared image in the visible light classifier is P i,r under the premise of gradient calculation suspension, and the prediction result of the visible light image in the infrared classifier is P r,i ;

[0027] Through the training process, the visible light classifier, the infrared classifier and the shared classifier can focus on the characteristics of the respective modalities, fully utilize the information between the modalities, and learn from the feature information of different modalities, reduce the difference between the modalities, and enhance the adaptability and prediction performance;

[0028] Step 4: The final loss function is constructed by determining the loss of the prediction label to the true label, the loss of the cross-modal prediction label to the prediction label, the classification loss and the ternary loss, and the network parameters are updated;

[0029] The modal alignment loss module receives parameters P i,i , P r,r , P i,r , P r,i and the correct identity label y, and adopts a cross-entropy function to construct a loss function;

[0030] The loss of the prediction label to the true label loss1 is shown in formula (2):

[0031]

[0032] Where y i is the true class index of the sample i, is the prediction probability of the sample i for the true class y i , and B is the batch;

[0033] The loss of the modal difference loss2 is shown in formula (3):

[0034]

[0035] Where For the softmax output of mutual_pred, as shown in formula (4), sigma (pred ij ) is the softmax output of pred, as shown in formula (5), C is the total number of categories;

[0036]

[0037] Considering the accuracy of the classifier in predicting the real label and the difference between different modalities, a loss function is constructed, the first loss term (loss1) is given greater weight in the total loss, so as to emphasize the accuracy of the model in predicting the real label, promote the model to pay more attention to correctly identifying and classifying the real label in the learning process, improve the overall performance of the classifier, and introduce a regulation mechanism to balance the influence of each modality on the final prediction result in the optimization process;

[0038] According to the loss loss1 of the real label and the loss loss2 of the modal difference obtained by the predicted label, the real loss function is obtained by weighted summation, as shown in formula (6):

[0039] loss_pma=loss1*2+loss2 (6)

[0040] The prediction result output by the public classifier and the real identity label are used to determine the cross-entropy identity loss loss_id and the triple loss loss_tri, and the relationship between the feature vector and the real identity label is optimized;

[0041] The finally constructed loss function is shown in formula (7):

[0042] loss_total=loss_id+loss_tri+loss_pma*0.5 (7)

[0043] It also includes step 5: according to the cross-modal feature representation obtained by step 4 and the trained model parameters, high-precision matching and recognition of infrared and visible light pedestrian images can be realized, and the application is in the fields of intelligent monitoring, safety protection, night pedestrian recognition, smart city and intelligent transportation, and the robustness and practicality of the cross-modal pedestrian re-identification system in complex environment are improved.

[0044] Advantages:

[0045] 1. The cross-modal pedestrian re-identification method based on adaptive feature aggregation of the application can use attention pooling to aggregate features with similar semantics in the feature map together and separate features with irrelevant semantics as much as possible, so as to ensure that the fine-grained information in the feature map is fully utilized.

[0046] 2, The cross-modal pedestrian re-identification method based on adaptive feature aggregation of the application, a multi-classifier structure including a visible light classifier, an infrared classifier and a shared classifier is constructed, information sharing and complementation between modes are realized through cross-prediction, the flexibility of the model is enhanced, and the feature distribution difference under different modes can be effectively coped with. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of the cross-modal pedestrian re-identification method based on adaptive feature aggregation of the application;

[0048] Figure 2 A model framework schematic diagram of the cross-modal pedestrian re-identification method based on adaptive feature aggregation of the application;

[0049] Figure 3 A fine-grained alignment schematic diagram in the cross-modal pedestrian re-identification method based on adaptive feature aggregation of the application;

[0050] Figure 4 A whole flowchart of the cross-modal pedestrian re-identification method based on adaptive feature aggregation of the application;

[0051] Figure 5 A visual heat attention schematic diagram in the embodiment;

[0052] Figure 6 A visual feature instance distribution situation schematic diagram in the embodiment;

[0053] Figure 7 A visual retrieval result diagram in the embodiment. DETAILED DESCRIPTION

[0054] In order to better illustrate the purpose and advantages of the application, the content of the application will be further described below in combination with the drawings and examples.

[0055] Embodiment 1:

[0056] This embodiment is based on the PyTorch framework, and ResNet-50 is selected as the backbone network on a single RTX 4090 GPU, ResNet-50 has a deep residual structure and strong feature extraction capability;

[0057] Two commonly used data sets in the field of image cross-modal pedestrian re-identification are used: SYSU MM01 and RegDB;

[0058] The SYSU-MM01 dataset is the first large-scale dataset in the field of cross-modal pedestrian re-identification, and contains data of 491 pedestrians, wherein images of 395 pedestrians are used to form a training set, and the remaining 96 are used for testing, and the RegDB data is relatively small, containing 8240 images of 412 pedestrians, which are randomly divided into two parts according to the pedestrian label: one half is used for training, and the other half is used for testing;

[0059] With the infrared image and visible light image dataset as the target, the cross-modal pedestrian re-identification method based on adaptive feature aggregation is applied, and cross-modal pedestrian recognition is realized, as shown in the following formula: Figure 1 The steps include the following steps:

[0060] Step 1: The infrared image and the visible light image are introduced into the ResNet-50 network, and the features processed by the modal sharing layer are introduced into the fine-grained shared classifier to generate corresponding weight maps. Figure One

[0061] As shown in the following formula: Figure 2 In the embodiment, the pre-trained parameters of ResNet-50 on the ImageNet dataset are loaded, and ResNet-50 can directly benefit from the rich visual features of ImageNet through transfer learning, providing more robust initialization effect for subsequent training, avoiding the instability and slow convergence problem caused by random initialization, and improving the initial performance of ResNet-50.

[0062] In the embodiment, the size of the 16 infrared images and the 16 visible light images is 3x288x144, the infrared image is processed by the special ir_special_module to generate a feature map with a shape of 16x2048x18x9, the visible light image is processed by the rgb_special_module to generate a feature map with a shape of 16x2048x18x9, the channel number C is 16x2048, the height H of the feature map is 18, and the width W of the feature map is 9.

[0063] The feature map X of the infrared image and the visible light image enters the fine-grained feature aggregation module and is reshaped into a tensor X1 with a shape of (18x9, 32), and the reshaped channel number C1 is 32x2048.

[0064] The fine-grained shared classifier is a feedforward neural network, which classifies the features contained in the feature map in a semantic manner, and the fine-grained shared classifier includes a full connection layer, a batch normalization layer, another full connection layer and an activation function.

[0065] As shown in the following formula: Figure 3 ​As shown, the tensor X1 of the reconstructed infrared image and the visible light image are simultaneously input into the fine-grained shared classifier in the form of shape (H×W,C1) for classification. Instead of simply compressing the features uniformly, the features with more similar semantics in the feature map are grouped into one class, which promotes information sharing between infrared modal features and visible light modal features.

[0066] The feature maps X of the infrared and visible light images with an input dimension of 32×2048 are processed by a fine-grained shared classifier and mapped to the final number of categories n, resulting in a weight map A of the infrared and visible light images with a shape of 5184×6.

[0067] The sigmoid activation function is used to activate the weight graph A, so that the values ​​of the elements in the weight graph A are between 0 and 1, thus maintaining the stability of the weight graph A.

[0068] Step 2: Aggregate the weight maps of the infrared image and the visible light image with the original feature map to generate a weighted result. Then, average pool the weighted result to obtain the aggregated feature vectors of the infrared image and the visible light image, respectively.

[0069] The weight map A of the infrared and visible light images from step 1 is combined with the feature map X of the original infrared and visible light images to obtain a weighted result that includes semantic aggregation classification, providing a basis for subsequent attention pooling.

[0070] The weighted tensor X is obtained by performing matrix multiplication between the weighted graph A and the reshaped tensor X1. res As shown in equation (1):

[0071]

[0072] That is, a matrix operation is performed on the features with a shape of 32×162×2048 and the weight map of 32×162×6. The calculation method is to calculate the weighted sum and then divide by 162, resulting in a weighted feature of 32×6×2048. The combination of each batch of features and the corresponding attention weights is completed by matrix multiplication between the weight map A and the reshaped tensor X1.

[0073] Then the weighted feature X res Flattening the local feature vectors into 32×12288 elements (6 parts, each 2048 dimensions), we obtain the aggregated feature tensor X. part This enables fine-grained feature integration, improving adaptability to specific tasks; simultaneously, global average pooling is performed on the original features to obtain a 32×2048 global feature vector (6 parts, each 2048 dimensions);

[0074] The global features obtained from global average pooling are compared with the aggregated feature tensor X. partConcatenate to get the 32x14336 dimensional feature vector X link , X link is reshaped to 32x6x2048 and averaged over the first dimension except the batch dimension, the final feature vector X final is of shape 32x2048, and the feature X final not only contains the average representation of global features, but also fuses the attention mechanism to important regions, capturing fine-grained information between modalities;

[0075] Step 3: Through the multi-classifier cross-prediction algorithm and the gradient pause strategy, the aggregated feature tensor X part is imported into the visible light classifier, the infrared classifier and the shared classifier for prediction, respectively, to obtain the corresponding loss value;

[0076] The visible light classifier, the infrared classifier and the shared classifier are all composed of a fully connected layer, which maps the dimension of the input feature to the total number of class identities.

[0077] As shown in Figure 4 , in the training process, the aggregated feature tensor X part is input into the shared classifier for training, and is also input into the infrared classifier and the visible light classifier for independent training. The prediction result of the infrared image in the infrared classifier is P i,i , and the prediction result of the visible light image in the visible light classifier is P r,r . Under the premise of pausing gradient calculation, the prediction result of the infrared image in the visible light classifier is P i,r , and the prediction result of the visible light image in the infrared classifier is P r,i .

[0078] Through the training process, the visible light classifier, the infrared classifier and the shared classifier can focus on the characteristics of their respective modalities, fully utilize the information between modalities, and learn from the feature information of different modalities, reduce the differences between modalities, and enhance the adaptability and prediction performance.

[0079] Step 4: By determining the loss of the prediction label to the real label, the loss of the cross-modal prediction label to the prediction label, the classification loss and the triplet loss, the final loss function is constructed to update the network parameters.

[0080] The loss function contains three main parts: the identity loss calculates the difference between the prediction label and the real label through cross entropy; the modal alignment loss is also cross entropy, which measures the difference between the prediction distribution and the cross-prediction soft label.

[0081] The aggregated triplet loss optimizes the feature space distribution, in an embodiment, the distance between the same class samples and the class center is shortened, the margin is set to 0, the distance between different class centers is lengthened, the margin is set to 0.7, and the triplet loss is fused to form a final total loss;

[0082] The final total loss is a weighted sum of the identity loss, the aggregated triplet loss and the modal alignment loss, in an embodiment, the weights are 1 for the identity loss and the aggregated triplet loss, and 0.5 for the modal alignment loss;

[0083] The modal alignment loss module receives parameters P i,i , P r,r , P i,r , P r,i and correct identity label y, and adopts a cross-entropy function to construct a loss function;

[0084] The loss loss1 of the predicted label to the real label is shown in formula (2):

[0085]

[0086] Wherein, y i is the real class index of sample i, is the predicted probability of sample i for the real class y i , and B is a batch;

[0087] The loss loss2 of the modal difference is shown in formula (3):

[0088]

[0089] Wherein, is the softmax output of mutual_pred, shown in formula (4), and σ(pred ij ) is the softmax output of pred, shown in formula (5), and C is the total number of classes;

[0090]

[0091] Considering the accuracy of the classifier in predicting the real label and the difference between different modalities, a loss function is constructed, the first loss term (loss1) is given a greater weight in the total loss, so as to emphasize the accuracy of the model in predicting the real label, promote the model to pay more attention to correctly identifying and classifying the real label in the learning process, improve the overall performance of the classifier, and introduce a regulation mechanism to balance the influence of each modality on the final prediction result in the optimization process;

[0092] According to the loss loss1 of the obtained predicted label to the real label and the loss loss2 of the modal difference, a real loss function is obtained by weighted summation, as shown in formula (6):

[0093] loss_pma=loss1*2+loss2 (6)

[0094] By the prediction result output by the public classifier and the real identity label, cross-entropy identity loss loss_id and triple loss loss_tri are determined to optimize the relationship between the feature vector and the real identity label.

[0095] The finally constructed loss function is shown in formula (7):

[0096] loss_total=loss_id+loss_tri+loss_pma*0.5 (7)

[0097] In the embodiment, the model parameters are updated using the Adam optimizer with a learning rate of 0.0003, and a learning rate scheduler is enabled for dynamic adjustment. After each batch in the training process, the model is tested, and the model with the best performance is saved according to the mAP index.

[0098] The baseline method (without adding the fine-grained feature aggregation module and the fine-grained feature alignment module) has a Rank-1 of 63.80% and an mAP of 61.94% under the single search strategy of the full environment search on the SYSU-MM01 dataset, a Rank-1 of 70.45% and an mAP of 56.34% under the multiple search strategy, a Rank-1 of 68.54% and an mAP of 74.60% under the single search strategy of the indoor search, and a Rank-1 of 77.60% and an mAP of 68.07% under the multiple search strategy;

[0099] The method has a Rank-1 of 73.00% and an mAP of 69.22% under the single search strategy of the full environment search on the SYSU-MM01 dataset, a Rank-1 of 78.70% and an mAP of 64.59% under the multiple search strategy, a Rank-1 of 79.86% and an mAP of 83.26% under the single search strategy of the indoor search, and a Rank-1 of 87.13% and an mAP of 78.42% under the multiple search strategy;

[0100] Under the multiple search strategy on the SYSU-MM01 dataset, the basic performance indicators of the method are better than those of the baseline method.

[0101] As Figure 5As shown, the highlighted areas in the thermal attention map are usually concentrated on key nodes of the human body, such as the knees, shoulders and feet. Some areas are also concentrated on the significant color or texture features of clothing. The method of this invention allows the model to focus on the decision-making content for identifying pedestrians.

[0102] like Figure 6 As shown, triangles represent feature instances in visible light images, circles represent feature instances in infrared images, and shapes of the same color represent those originating from the same source. Figure 6 (A) shows the distribution of feature instances under the baseline method. It can be observed that most instances of the same identity but different modalities are relatively close to each other, while instances of different identities are relatively far apart. There are a few examples showing that instances of the same identity but different modalities are far apart. Figure 6 (B) shows the distribution of feature instances under the method of this invention. Through comparison, Figure 6 (A) Cases that are relatively far apart Figure 6 (B) exhibits a more compact distribution, and the method of the present invention can exhibit more stable performance in different modal alignments;

[0103] like Figure 7 As shown, the baseline method may misidentify false samples with similar body features as true samples in some cases. The method of the present invention has a stronger anti-interference ability than the baseline method.

[0104] It also includes step 5: Based on the optimized cross-modal feature representation obtained in step 4 and the trained model parameters, it is possible to achieve high-precision matching and recognition of infrared and visible light pedestrian images. It can be practically applied in the fields of intelligent monitoring, security, nighttime pedestrian recognition, smart cities and intelligent transportation, thereby improving the robustness and practicality of the cross-modal pedestrian re-identification system in complex environments.

[0105] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-modal person re-identification method based on adaptive feature aggregation, characterized in that: A fine-grained feature aggregation module is introduced to effectively aggregate semantically similar features in the feature map while separating semantically dissimilar features, maximizing the use of fine-grained information. A fine-grained feature alignment module is constructed to align and correct the semantic information of cross-modal feature vectors, reducing the distribution differences between modalities and ensuring that feature vectors of two modalities can be effectively compared in the same semantic space. This solves the problems of fine-grained information loss and inconsistent feature distribution between modalities during feature map pooling.

2. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 1, characterized in that: Specifically, the steps include the following: Step 1: Import the infrared image and the visible light image into the ResNet network together, and import the feature map processed by the modality sharing layer into the fine-grained shared classifier to generate the corresponding weight map. Step 2: Aggregate the weight maps of the infrared image and the visible light image with the original feature map to generate a weighted result. Then, average pool the weighted result to obtain the aggregated feature vectors of the infrared image and the visible light image, respectively. The weight map A of the infrared and visible light images from step 1 is combined with the feature map X of the original infrared and visible light images to obtain a weighted result that includes semantic aggregation classification, providing a basis for subsequent attention pooling. The weighted tensor X is obtained by performing matrix multiplication between the weighted graph A and the reshaped tensor X1. res As shown in equation (1): The combination of features and corresponding attention weights for each batch is achieved by performing matrix multiplication between the weight map A and the reshaped tensor X1. For weighted tensor X res The mean of the first dimension is used to obtain the aggregated feature tensor X. part This enables fine-grained feature integration and improves adaptability to specific tasks. The global features obtained from global average pooling are compared with the aggregated feature tensor X. part By splicing, the final feature X is obtained. final Feature X final It not only includes the average representation of global features, but also incorporates attention mechanisms for important regions to capture fine-grained information between modalities; Step 3: Using a multi-classifier cross-prediction algorithm and a gradient pause strategy, aggregate the feature tensor X of the infrared and visible light images. part The visible light classifier, infrared classifier, and shared classifier are imported separately for prediction, and the corresponding loss values ​​are obtained. Step 4: Construct the final loss function by determining the loss of predicted label against the real label, the loss of cross-modal predicted label against the predicted label, the classification loss, and the triplet loss, and update the network parameters.

3. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 2, characterized in that: In step 1, the infrared image and the visible light image are processed through the fourth layer of the ResNet network to generate a feature map X. The feature map X is a three-dimensional tensor with a shape of (C,H,W), where C is the number of channels, H is the height of the feature map, and W is the width of the feature map.

4. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 2, characterized in that: In step 1, the feature maps X of the infrared image and the visible light image enter the fine-grained feature aggregation module and are reshaped into a tensor X1 with shape (H×W,C1), where C1 is the number of channels after aggregation.

5. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 2, characterized in that: In step 2, the fine-grained shared classifier is a feedforward neural network that performs semantic classification on the features contained in the feature map. The fine-grained shared classifier includes a fully connected layer, a batch normalization layer, another fully connected layer, and an activation function. The tensor X1 of the reconstructed infrared image and the visible light image is simultaneously input into the fine-grained shared classifier in the form of shape (H×W,C1) for classification. Instead of simply compressing the features uniformly, it groups features with similar semantics in the feature map into one class, promoting information sharing between infrared modal features and visible light modal features. The feature maps X of the infrared and visible light images with input dimension C are processed by a fine-grained shared classifier and mapped to the final number of categories n, resulting in a weight map A of the infrared and visible light images with shape (H×W,n).

6. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 2, characterized in that: In step 2, the sigmoid activation function is used to activate the weight graph A, so that the values ​​of the elements in the weight graph A are between 0 and 1, thus maintaining the stability of the weight graph A.

7. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 2, characterized in that: In step 3, the visible light classifier, infrared classifier, and shared classifier are all composed of a fully connected layer, which maps the dimension of the input features to the total number of class identities. During training, the aggregated feature tensor X of infrared and visible light images... part In addition to being trained by the shared classifier, the images were also independently trained by the infrared classifier and the visible light classifier. The prediction result of the infrared image in the infrared classifier was P. i,i The prediction result of the visible light image in the visible light classifier is P. r,r With gradient calculation paused, the prediction result of the infrared image in the visible light classifier is P. i,r The prediction result of the visible light image in the infrared classifier is P. r,i ; Through the training process, the visible light classifier, infrared classifier, and shared classifier can focus on the characteristics of their respective modalities, make full use of information between modalities, and learn from each other's feature information, thereby reducing differences between modalities and enhancing adaptability and predictive performance.

8. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 2, characterized in that: In step 4, the modal alignment loss module receives parameter P. i,i P r,r P i,r P r,i And the correct identity label y, and construct the loss function using the cross-entropy function; The loss1 of the predicted label to the true label is shown in equation (2): Among them, y i For the true class index of sample i, For sample i, for the true category y i The predicted probability, where B is the batch number; The modal difference loss2 is shown in equation (3): in, The softmax output of mutual_pred is shown in equation (4), σ(pred) ij ) is the softmax output of pred, as shown in equation (5), where C is the total number of categories; Taking into account the accuracy of the classifier in predicting the true label and the differences between different modalities, a loss function is constructed, and the first loss term (loss1) is given a larger weight in the total loss in order to emphasize the accuracy of the model in predicting the true label, promote the model to pay more attention to correctly identifying and classifying the true label during the learning process, improve the overall performance of the classifier, and introduce a regulation mechanism to balance the influence of each modality on the final prediction result during the optimization process. Based on the loss1 of the predicted label to the real label and the loss2 of the modality difference, the true loss function is obtained by weighted summation, as shown in equation (6): loss_pma=loss1*2+loss2 (6) By using the prediction results output by the common classifier and the real identity labels, the cross-entropy identity loss loss_id and triplet loss loss_tri are determined, and the relationship between the feature vector and the real identity labels is optimized. The final loss function is shown in equation (7): loss_total=loss_id+loss_tri+loss_pma*0.5 (7).

9. The cross-modal person re-identification method based on adaptive feature aggregation as described in claim 1, characterized in that: Step 5: Based on the optimized cross-modal feature representation obtained in step 4 and the trained model parameters, high-precision matching and recognition of infrared and visible light pedestrian images can be achieved. It can be practically applied in intelligent monitoring, security, nighttime pedestrian recognition, smart cities and intelligent transportation, improving the robustness and practicality of the cross-modal pedestrian re-identification system in complex environments.

Images