Cross-modal pedestrian re-identification method based on transformer and adversarial learning

Abstract

A cross-modal pedestrian re-identification method based on Transformer and adversarial learning, first uses a modal-specific feature extractor to extract features from visible light images and infrared images respectively. Then, the modal-specific features extracted are adjusted by a cross-modal feature alignment module to reduce the intra-modal and inter-modal feature differences. Next, a weight-shared shared feature extractor is used to extract modal-shared features, and a multi-scale feature fusion method is adopted to use the attention mechanism of Transformer to supplement the context information of modal-shared features to modal-specific features to make up for the lack of modal-specific features in global semantics. In the adversarial learning process, the classifier and the feature extractor are alternately trained, so as to better extract modal-invariant features. The present application effectively reduces the modal difference between visible light and infrared images, significantly improves the precision and robustness of cross-modal pedestrian re-identification, and is suitable for security monitoring, intelligent transportation and other fields.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a cross-modal person re-identification method based on Transformer and adversarial learning. Background Technology

[0002] Pedestrian re-identification is a technique that uses computer vision to detect the presence of a specific pedestrian in an image or video sequence. Due to increasing public safety needs and the growing number of surveillance cameras, pedestrian re-identification technology has received increasing attention.

[0003] In the field of public safety surveillance, pedestrian re-identification technology can help surveillance systems track and identify people more effectively. Once a surveillance camera captures the whereabouts of a suspect, pedestrian re-identification technology can assist in identifying that suspect from other cameras, accurately identifying specific pedestrians even in busy crowds, thus greatly improving the capabilities of public safety surveillance. Furthermore, most cameras in modern surveillance systems have the ability to automatically switch between visible light and infrared modes under different lighting conditions. Research into cross-modal pedestrian re-identification technology enables effective identification of suspects even at night, thereby expanding the application scope and effectiveness of surveillance systems.

[0004] However, visible light images and infrared images differ significantly in their imaging mechanisms and characteristics. Infrared images typically have lower contrast because they characterize the temperature distribution of objects and reflect the differences in energy radiated outward from the target and background. Furthermore, the infrared band is affected by long transmission distances, atmospheric absorption, and scattering, leading to blurred edges. Visible light images, on the other hand, typically have higher contrast, clearly displaying the differences between the target and background. In addition, infrared images are based on the temperature of objects, thus failing to reflect the texture information of object surfaces. Visible light images, however, are based on the reflection from object surfaces, better reflecting surface texture information and demonstrating a stronger ability to reflect surface details. Therefore, research on cross-modal person re-identification primarily focuses on addressing the modal differences between these two types of images. Existing visible light and infrared person re-identification models can be categorized into models based on modality-shared feature learning and models based on modality-specific information compensation.

[0005] Models based on modal shared feature learning aim to embed features from different modalities into the same feature space and reduce cross-modal differences by using some feature-level constraints. However, due to the large modal gap between visible light and infrared images, it is difficult to directly project cross-modal images into a common space.

[0006] Models based on modality-specific information compensation attempt to compensate for missing modality-specific information in existing modalities. This involves converting visible light images to infrared images, or vice versa, to reduce appearance differences. However, significant modal differences exist between visible light and infrared images, making it challenging to generate high-quality images from one modality to another. Especially when generating visible light images from infrared images, cross-modal person re-identification may introduce more noise, such as color inconsistencies, rather than discriminative person features. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide a cross-modal person re-identification method based on Transformer and adversarial learning. The method uses Transformer to aggregate modality-shared features and modality-specific features, and uses adversarial learning to better extract modality-invariant features, thereby more effectively reducing modality differences between visible light and infrared images and improving the accuracy of cross-modal person re-identification.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A cross-modal person re-identification method based on Transformer and adversarial learning includes the following steps:

[0010] Step 1: Obtain the visible light-infrared image dataset of pedestrians, and extract modality-specific features from the visible light and infrared images respectively using a modality-specific feature extractor;

[0011] Step 2: Adjust the modality-specific features using the cross-modal feature alignment module to reduce feature differences within and between modalities;

[0012] Step 3: The modality shared features are extracted from the modality-specific features using a weight-sharing shared feature extractor, and the modality shared features and modality-specific features are combined through a multi-scale feature fusion module to enrich the final feature representation; the modality-specific feature extractor and the shared feature extractor together constitute the feature extractor;

[0013] Step 4: Adversarial learning strategy is adopted to train the classifier and feature extractor alternately. The classifier attempts to distinguish between visible light and infrared features, while the feature extractor attempts to extract modality-confusing features, making it difficult for the classifier to distinguish them, thereby better extracting modality-invariant features. The training terminates when the preset number of training rounds is reached.

[0014] Step 5: Using the trained feature extractor, perform cross-modal pedestrian re-identification with visible light or infrared pedestrian images as input.

[0015] Furthermore, in step 1 of this invention, preprocessing strategies, including random channel erasure and horizontal flipping, are first applied to the visible light and infrared images in the dataset to enrich the diversity of training samples and further increase the generalization ability of the model. Simultaneously, camera label information from the dataset is read, and the introduction of camera label information mitigates differences between cameras and strengthens the learning of modality-invariant features. Then, channel swapping enhancement is used to generate new training images as auxiliary modalities to improve robustness to color transformations. Next, features are extracted from the preprocessed visible light and infrared images using two modality-specific feature extractors, respectively. The feature extractors employ the first two residual modules of the ResNet-50 architecture.

[0016] Furthermore, the introduction of camera label information alleviates the differences between cameras and strengthens the learning of modality-invariant features. The method is as follows: during the training process, the image and its corresponding camera label are input into the network together, and an auxiliary branch for predicting the camera label is added to the network. The loss is predicted by the camera label, so that the model can notice the differences in images from different cameras, thereby extracting more discriminative features.

[0017] Furthermore, the method of using channel swapping to enhance the generation of new training images as auxiliary modalities to improve robustness to color changes is as follows: randomly select one channel (R, G, or B) of the original visible light image to replace other channels to generate a new image. The generated image is used as an auxiliary modal and is input together with the original visible light image into a visible light modal-specific feature extractor for joint training, thereby enhancing the model's robustness to color changes.

[0018] Furthermore, in step 3 of this invention, the shared feature extractor adopts the last three residual modules of ResNet-50. The multi-scale feature fusion module consists of 1×1 convolutional layers and a Transformer. The convolutional layers make the features more compact, and the Transformer supplements the modality-specific features with contextual information of the modality-shared features through an attention mechanism to compensate for the lack of global semantics in the modality-specific features.

[0019] Further, in step 4 of this invention, during the adversarial learning process, the true modality label of the visible light image is set to [1,0,0], and the true modality label of the infrared image is set to [0,1,0]. The modality confusion label of all samples from the two modalities is set to [0,0,1]. A classifier is used to transform the input features into a predicted probability distribution. Then, the classifier and feature extractor are trained alternately, so that the optimization process explicitly focuses on modality-independent perspectives and better extracts modality-invariant features.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] This invention aligns features from different modalities using a cross-modal feature alignment module, reducing intra- and inter-modal feature differences and thus improving the accuracy of cross-modal feature matching. The module uses channel consistency loss to shorten the distance between each channel feature and its corresponding channel feature center within each modality, making the features of each channel more concentrated and reducing intra-modal feature differences. Simultaneously, by increasing the distance between feature centers of different channels, it avoids features from being too similar, ensuring sufficient differentiation for each channel and improving the model's discriminative ability. Finally, modal consistency loss is used to reduce the distance between the feature centers of the visible light modality and the infrared modality with the same identity, improving the alignment effect of cross-modal features and enhancing the model's performance in multimodal scenarios.

[0022] This invention's multi-scale feature fusion module effectively fuses features from different levels using 1×1 convolutional layers and a Transformer. The 1×1 convolutional layer makes the features more compact, while the Transformer, through an attention mechanism, supplements modality-shared features with contextual information into modality-specific features, compensating for the shortcomings of modality-specific features in global semantics. Finally, the enhanced modality-specific features are fused with modality-shared features, resulting in a richer and more comprehensive feature representation, significantly improving the performance of cross-modal person re-identification.

[0023] This invention employs adversarial learning, assigning both a true modality label and a modality-confused label to each image. During the adversarial learning process, the classifier is first trained using the true modality label to correctly distinguish between visible light and infrared images. Then, the feature extractor is trained using the modality-confused label, aiming to extract modality-independent features so that the classifier classifies all samples under the modality-confused label based on these features. In this way, the optimization process explicitly focuses on modality-independent perspectives, thereby better extracting modality-invariant features and improving the accuracy of cross-modal person re-identification. Attached Figure Description

[0024] Figure 1 This is a network structure diagram of the cross-modal person re-identification method based on Transformer and adversarial learning proposed in this invention.

[0025] Figure 2 This is a schematic diagram of the cross-modal feature alignment module structure of the present invention.

[0026] Figure 3 This is a schematic diagram of the multi-scale feature fusion module structure of the present invention.

[0027] Figure 4 The Rank-10 search results of this invention are used.

[0028] Figure 5Heatmaps of features extracted using this invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other achievements obtained by those skilled in the art based on the present invention without inventive effort should fall within the scope of protection of the present invention.

[0030] Existing cross-modal person re-identification algorithms cannot effectively address the modal differences between visible light and infrared images. To solve this problem, this invention provides a cross-modal person re-identification method based on Transformer and adversarial learning.

[0031] Combination Figure 1 As shown, the method of the present invention relies on a feature extractor, a cross-modal feature alignment module, a multi-scale feature fusion module, an adversarial learning module, etc., which constitute the main part of the present invention.

[0032] refer to Figure 1 The cross-modal person re-identification method based on Transformer and adversarial learning in this invention mainly includes the following steps:

[0033] Step 1: Obtain a visible light-infrared image dataset of pedestrians as the training sample set. The training sample set contains multiple pedestrians with different identities, and each pedestrian has several infrared and visible light images. In addition, the images in the sample set were taken by multiple cameras at different angles and under different lighting conditions. These cameras cover different scenes and backgrounds, thus ensuring the diversity and generalization ability of the data.

[0034] Step 2: Preprocess the data in the training sample set.

[0035] In this embodiment, the image is first cropped to 288×144 pixels, and then horizontally flipped with a probability of 0.5 to enhance the model's generalization ability and reduce overfitting. Next, the image is normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225] to accelerate model convergence. Finally, random channel erasure is used to improve robustness to image noise.

[0036]

[0037] Where m and n represent the coordinates of the pixel, α is a fixed value, and S eThe rectangular erasure region is randomly selected. Then, channel swapping enhancement is applied to the visible light image, randomly selecting one channel (R, G, or B) to replace other channels. By focusing on one channel, a new training image is generated as an auxiliary modality, improving robustness to color transformation.

[0038]

[0039]

[0040]

[0041] Step 3: Extract modality-specific features from the visible light image and the infrared image respectively using a modality-specific feature extractor.

[0042] The modality-specific feature extractor of this invention includes an infrared feature extractor E. r and visible light feature extractor E v Infrared Feature Extractor E r Modality-specific features extracted from infrared images, visible light feature extractor E v Extract mode-specific features from visible light images. (Reference) Figure 1 Infrared Feature Extractor E r and visible light feature extractor E v Each is composed of the first two residual modules of ResNet-50, using E r and E v Infrared modal specific features F were extracted. r and specific features of visible light modes F v ,Right now:

[0043] F r =E r (X r ),F v =E v (X v )

[0044] Step 4: Adjust the modality-specific features through the cross-modal feature alignment module to reduce feature differences within and between modalities.

[0045] refer to Figure 2The cross-modal feature alignment module uses channel consistency loss to shorten the distance between each channel feature and its corresponding channel feature center within each modality, thereby making the features of each channel more concentrated and reducing feature differences within the modality. Simultaneously, by increasing the distance between feature centers of different channels, it avoids features from being too similar across channels, ensuring sufficient differentiation for each channel's features and improving the model's discriminative ability. Subsequently, modal consistency loss is used to reduce the distance between the feature centers of the visible light modality and the infrared modality with the same identity, improving the alignment effect of cross-modal features and enhancing the model's performance in multimodal scenarios.

[0046] This embodiment adjusts the modality-specific features through a cross-modal feature alignment module, and the process is as follows:

[0047] Step 41, input as Where F r and F v These are the infrared modal-specific features and visible light modal-specific features obtained through the modal-specific feature extractor, respectively. B represents the batch size, C represents the number of channels, and H and W represent the height and width, respectively.

[0048] Step 42, divide F through the equal-division channel r and F v Decomposed into three channel features: R, G, and B, that is, in C′=C / 3. This can be expressed as… m∈[r,v],j∈[R,G,B]. Then calculate the feature centers of each channel for the modality-specific features:

[0049]

[0050]

[0051] in, and These represent the feature centers of the red, green, and blue channels of the infrared modality for the i-th identity, respectively. and These represent the feature centers of the red, green, and blue channels of the visible light mode for the i-th identity, respectively. and Let represent the modal-specific features of the i-th identity and the k-th infrared sample in the red, green, and blue channels, respectively. and Let K represent the modal-specific features of the k-th visible light sample of the i-th identity in the red, green, and blue channels, respectively, where K is the number of samples of the i-th identity.

[0052] To mitigate intramodal differences, a channel consistency loss L is proposed.ccl , means as follows:

[0053]

[0054]

[0055] L ccl =L cn +L sp

[0056] Among them, L cn To compensate for the center loss of the channels, the distance from each channel to the center within the mode is reduced, L sp The channel separation loss is used to increase the distance between different channels. τ represents the corresponding boundary, which is usually set to 0.6, and N is the number of identities.

[0057] Step 43: Calculate the feature centers of infrared mode-specific features and visible light mode-specific features:

[0058]

[0059] Among them, C r,i C is the infrared modal feature center of the i-th identity. v,i Let i be the visible light modal feature center of the i-th identity. For the modality-specific features of the i-th identity and the k-th infrared sample, This defines the modality-specific features of the i-th identity and the k-th visible light sample. Finally, a modality consistency loss L is proposed. mcl To reduce the distance between the specific feature centers of the visible light mode and the specific feature centers of the infrared mode of the same identity, it is represented as follows:

[0060]

[0061] Step 5: Extract modality shared features from modality-specific features using a weight-sharing shared feature extractor. The modality-specific feature extractor and the shared feature extractor of this invention together constitute the feature extractor.

[0062] refer to Figure 1 The present invention shares a feature extractor E. s It consists of the last three residual modules of ResNet-50, using E s Extracting modal shared features F s,r and F s,v ,Right now:

[0063] F s,r =E s (F r ),F s,v =E s (Fv )

[0064] Step 6: Combine modality-shared features and modality-specific features through the multi-scale feature fusion module.

[0065] refer to Figure 3 The multi-scale feature fusion module of this invention consists of a 1×1 convolutional layer and a Transformer. The 1×1 convolutional layer makes the features more compact, while the Transformer supplements the modality-specific features with contextual information from modality-shared features through an attention mechanism, compensating for the shortcomings of modality-specific features in global semantics. Finally, the enhanced modality-specific features are fused with the modality-shared features, reducing information loss and making the final feature representation richer and more comprehensive, thereby significantly improving the effect of cross-modal person re-identification.

[0066] This invention combines modality-shared features and modality-specific features through a multi-scale feature fusion module, and the implementation process is as follows:

[0067] Step 61, input F l and F h F l This represents the modality-specific features to be incorporated, i.e., the infrared feature extractor E. r and visible light feature extractor E v Extracted mode-specific features F r or F v F h This represents the modality-shared features to be incorporated, i.e., those extracted by the shared feature extractor E. s Extracted modal shared features F s,r or F s,v F is generated through a 1×1 convolutional layer. l and F h Convert to compact representation F l ′ and F h For example, there are two 1×1 convolutional layers, represented as follows: and but:

[0068]

[0069] Step 62, assign modality-specific features F l As a query, the modality-shared feature F h Using '' as keys and values, queries, key-value matrices can be generated:

[0070] Q = W q F l ′,K=W k F h ′,V=W v Fh ′

[0071] Among them, W q W k W v These are the weight matrices used to generate the query, key, and value matrices, respectively. The attention score is then calculated.

[0072]

[0073] Where, d k It is the dimension of the key matrix, used to scale the dot product.

[0074] Step 63: Use the obtained attention scores to obtain the updated low-level features:

[0075] F l "=AV

[0076] The updated modality-specific features F l "Sharing features F with modalities" h The sum of these features yields the final characteristic:

[0077] F u =F l "+F h ′

[0078] Step 7: Adopt an adversarial learning strategy to train the classifier and feature extractor alternately. The classifier attempts to distinguish between visible light and infrared features, while the feature extractor attempts to extract modality-confusing features, making it difficult for the classifier to distinguish them, thereby better extracting modality-invariant features. Training terminates when a preset number of training rounds are reached.

[0079] The process of alternately training the classifier and feature extractor in this step can be described as follows:

[0080] Step 71, the visible light feature F obtained in the previous step v and infrared signature F r As input, the true modal label t of the visible light image v Set to [1,0,0], the true modal label t of the infrared image r Setting the modality obfuscation label t to [0,1,0] will change the modality of all images. o All are set to [0,0,1].

[0081] Step 72, refer to Figure 1 The classifier W is used to transform the input features into a predicted modality probability distribution, and then the classifier W and the feature extractor E are trained alternately.

[0082]

[0083]

[0084] Where CE(·) represents the cross-entropy loss function, N v N represents the number of visible light image samples. r This represents the number of infrared image samples.

[0085] Step 8: Using the trained feature extractor, cross-modal pedestrian re-identification can be performed with visible light or infrared pedestrian images as input.

[0086] This invention effectively reduces modal differences between visible light and infrared images, significantly improving the accuracy and robustness of cross-modal person re-identification, and is applicable to fields such as security monitoring and intelligent transportation. To further demonstrate the advantages of the proposed cross-modal person re-identification method based on Transformer and adversarial learning, extensive comparative experiments were conducted. The method was trained and evaluated separately on the SYSU-MM01 and RegDB datasets, and compared with existing cross-modal person re-identification methods.

[0087] Table 1 Comparison of the method of the present invention and existing methods on the SYSU-MM01 dataset.

[0088]

[0089] Table 2 Comparison of the method of the present invention and existing methods on the RegDB dataset.

[0090]

[0091] Dataset:

[0092] SYSU-MM01: SYSU-MM01 is an image dataset for pedestrian re-identification in visible-infrared light. The images were acquired by four RGB cameras (for bright scenes) and two infrared cameras (for dim scenes), totaling 289,145 RGB images and 16,579 infrared images of 491 pedestrians.

[0093] RegDB: RegDB is also an image dataset for pedestrian re-identification using visible light and infrared light. The dataset contains 412 distinct person identities and a total of 8240 images, including 4120 visible images and 4120 infrared images, using a dual-mode camera (i.e., visible mode and infrared mode respectively). Among these identities, 254 are female and 158 are male.

[0094] Measurement criteria:

[0095] Rank-n accuracy: This metric indicates whether a true match for the query image appears in the top n matches. Rank-1 accuracy indicates whether the first position in the match list is a true match.

[0096] Average precision (mAP): This is calculated by taking the average of the precision rates at different thresholds. mAP provides a more comprehensive reflection of the algorithm's performance under various conditions.

[0097] As can be seen from Tables 1 and 2, the cross-modal pedestrian re-identification method based on Transformer and adversarial learning proposed in this invention can effectively reduce the modal differences between visible light and infrared images, and improve the accuracy and effect of cross-modal pedestrian re-identification. Figure 4 This paper presents some retrieval results of the proposed method on a dataset. For each retrieval case, green boxes indicate correct matches to the given query, while red boxes indicate incorrect matches. The retrieval results show that the proposed method can effectively distinguish identities, and correct results typically have a high degree of matching with the image being retrieved. Figure 5 As shown, the heat map generated by the method proposed in this invention focuses more on identity-related features and has a strong anti-interference ability against factors such as lighting and occlusion.

[0098] The present invention has been disclosed in more detail above, but it is not intended to limit the present invention. All technical solutions obtained by adopting equivalent substitutions or equivalent transformations fall within the protection scope of the present invention.

Claims

1. A cross-modal person re-identification method based on Transformer and adversarial learning, characterized in that, The steps include the following: Step (1): Obtain the visible light-infrared image dataset of pedestrians, and extract modality-specific features from the visible light image and the infrared image respectively using a modality-specific feature extractor; Step (2) involves adjusting the modality-specific features through a cross-modal feature alignment module to reduce feature differences within and between modalities; Step (3) involves extracting modality-shared features from modality-specific features using a weight-sharing shared feature extractor, and combining the modality-shared features and modality-specific features through a multi-scale feature fusion module; wherein the modality-specific feature extractor and the shared feature extractor together constitute the feature extractor; Step (4): An adversarial learning strategy is adopted to train the classifier and the feature extractor alternately. The classifier attempts to distinguish between visible light and infrared features, while the feature extractor attempts to extract modality-confusing features, making it difficult for the classifier to distinguish them, thereby better extracting modality-invariant features. The training terminates when the preset number of training rounds is reached. Step (5): Using the trained feature extractor, cross-modal pedestrian re-identification is performed with visible light or infrared pedestrian images as input. In step (2), the modality-specific features are adjusted through the cross-modal feature alignment module, and the process is as follows: Step (21), input is ,in and These are infrared mode-specific features and visible light mode-specific features obtained through a mode-specific feature extractor, respectively. B Indicates batch size. C Indicates the number of channels. H and W These represent height and width, respectively. Step (22), divide the channel evenly and Decomposed into RGB three-channel features, i.e. , ,in Calculate the feature center of each channel for modal-specific features; Step (23): Calculate the feature centers of infrared modal-specific features and visible light modal-specific features: in, For the first Infrared modal feature center of each identity, For the first The visible light modal feature center of each identity, For the first The first identity Modal-specific features of an infrared sample For the first The first identity Modal-specific characteristics of a visible light sample For the first Number of samples per identity; The process for calculating the feature center of each channel for modal-specific features is as follows: in, , and They represent the first The feature centers of the infrared modal red, green, and blue channels of each identity. , and They represent the first The characteristic centers of the visible light modes red, green, and blue channels of each identity, , and They represent the first The first identity Modal-specific features of an infrared sample in the red, green, and blue channels. , and They represent the first The first identity Modal-specific features of a visible light sample in the red, green, and blue channels; Using channel consistency loss To mitigate intramodal differences, it is represented as follows: Using modal consistency loss To reduce the distance between the visible light mode feature center and the infrared mode feature center of the same identity, it is represented as follows: in To mitigate channel center loss, the distance from each channel to the center within the mode is reduced. To mitigate channel separation loss, increase the distance between different channels. Indicate the corresponding boundary, , For the number of identities.

2. The cross-modal person re-identification method based on Transformer and adversarial learning according to claim 1, characterized in that, In step (1), in the dataset, preprocessing strategies including random channel erasure and horizontal flipping are used on visible light images and infrared images; at the same time, camera label information is introduced to alleviate the differences between cameras and strengthen the learning of modality invariant features; and channel swapping is used to generate new training images as auxiliary modalities to improve robustness to color transformation.

3. The cross-modal person re-identification method based on Transformer and adversarial learning according to claim 2, characterized in that, The method of introducing camera label information to alleviate the differences between cameras and enhance the learning of modality invariant features is as follows: during the training process, the image and its corresponding camera label are input into the network together, and an auxiliary branch for predicting the camera label is added to the network. The loss is predicted by the camera label, so that the model can notice the differences between images from different cameras. The method of using channel swapping enhancement to generate new training images as auxiliary modalities to improve robustness to color transformation is as follows: randomly select one channel of the original visible light image to replace other channels to generate a new image. The generated image is used as an auxiliary modal and is input together with the original visible light image into a visible light modal-specific feature extractor for joint training.

4. The cross-modal person re-identification method based on Transformer and adversarial learning according to claim 1, characterized in that, In step (1), the modality-specific feature extractor includes an infrared feature extractor. and visible light feature extractor Infrared feature extractor Visible light feature extractor for extracting modality-specific features from infrared images. Extracting modality-specific features from visible light images and Both consist of the first two residual modules of ResNet-50; step (3) uses a shared feature extractor. Extract modality-shared features. It consists of the last three residual modules of ResNet-50.

5. The cross-modal person re-identification method based on Transformer and adversarial learning according to claim 1, characterized in that, The multi-scale feature fusion module consists of a 1×1 convolutional layer and a Transformer. The Transformer uses an attention mechanism to supplement the modality-specific features with contextual information from modality-shared features, thereby compensating for the shortcomings of modality-specific features in global semantics. The multi-scale feature fusion module combines modality-shared features and modality-specific features, as follows: Step (31), input is and ,in This indicates the modality-specific features to be incorporated. The modality-sharing features to be incorporated are represented by a 1×1 convolutional layer. and Convert to compact representation and ; Step (32), As a query, As keys and values, generate a query, key, and value matrix, and then calculate the attention score; Step (33): Use the obtained attention scores to obtain the updated low-level features. The updated low-level features and Adding them together yields the final feature. .

6. The cross-modal person re-identification method based on Transformer and adversarial learning according to claim 5, characterized in that, Step (32) generates a query, key, and value matrix representation as follows: , , The calculated attention scores are as follows: In step (33), the updated low-level features The expression is as follows: Final features The expression is as follows: in, These are the weight matrices used to generate the query, key, and value matrices, respectively. It is the dimension of the key matrix, used to scale the dot product.

7. The cross-modal person re-identification method based on Transformer and adversarial learning according to claim 1, characterized in that, In step (4), an adversarial learning strategy is adopted to alternately train the classifier and the feature extractor, as follows: Step (41), using the visible light features obtained in step (3) and infrared features As input, the true modal labels of the visible light image Set as True modal labels of infrared images Set as Modality obfuscation labels for all images All set to ; Step (42) uses classifier W to transform the input features into a predicted modality probability distribution, and then alternately trains classifier W and feature extractor E: in Represents the cross-entropy loss function. This represents the number of visible light image samples. This represents the number of infrared image samples.

Images