A semi-supervised visible-infrared pedestrian re-identification method based on multi-modal collaborative learning

Abstract

The application discloses a kind of semi-supervised visible-infrared pedestrian re-identification methods based on multi-modal collaborative learning, it is related to computer vision and multi-modal machine learning technique.Visible light image with label, unannotated infrared image and existing identity level text description are used as multi-modal training data;Image visual features and text semantic features are extracted by visual and text encoder respectively, and cross-modal joint features are generated by realizing the bidirectional gated residual modulation of text and visual features using semantic guided cross-modal gated multi-scale fusion module;Hierarchical semantic alignment module is constructed, and cross-modal collaborative semantic supervision is implemented relying on upper garment, lower garment and accessory semantics, and infrared pseudo-label is generated by using progressive two-stage optimal transport to fuse classification probability and feature affinity information;Semi-supervised collaborative training is carried out by combining multi-class supervision signals, and infrared-to-visible pedestrian retrieval is completed according to feature similarity.The application enhances the semantic consistency of multi-modal features and improves the reliability of pseudo-labels, and reduces the cost of infrared identity labeling.

Description

Technical Field

[0001] This invention relates to computer vision and multimodal machine learning technologies, specifically to a semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning, which can be applied to scenarios such as all-weather video surveillance, smart city security, and low-light pedestrian retrieval. Background Technology

[0002] In recent years, pedestrian re-identification has become an important research direction in the field of computer vision, and it has great value in applications such as video surveillance, smart cities, public safety and low-light scene retrieval. Visible-infrared pedestrian re-identification aims to match the identity of the same pedestrian across visible light images and infrared images, and is a key technology for realizing all-weather intelligent monitoring.

[0003] Visible light images typically contain color, texture, and appearance details, while infrared images mainly reflect thermal radiation information. The two differ significantly in imaging mechanisms, color representation, texture structure, and scale response. Existing fully supervised visible-infrared pedestrian re-identification methods usually require simultaneous labeling of visible light and infrared images, but the cost of infrared image acquisition and labeling is high, making it difficult to obtain accurate labels on a large scale in real-world scenarios.

[0004] Semi-supervised visible-infrared pedestrian re-identification methods typically use only visible light identity labels and are trained using unlabeled infrared samples. However, due to severe cross-modal differences, infrared pseudo-labels generated based on low-level visual features or classification probabilities are prone to noise, and the further participation of erroneous pseudo-labels in training can lead to error accumulation.

[0005] Manually labeled identity-level text descriptions contain structured semantic cues such as top color, bottom color, and backpack, which can serve as high-level semantic anchors for multimodal collaborative learning. However, existing methods often fail to fully utilize the human body hierarchy in text semantics, making it difficult to alleviate pseudo-label noise caused by fine-grained semantic confusion and cross-modal feature mismatch through cross-modal collaborative semantic supervision. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing semi-supervised visible-infrared pedestrian re-identification technologies, such as high infrared annotation costs, large pseudo-label noise, insufficient multimodal semantic alignment, and poor cross-modal retrieval performance. This invention provides a semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning. It utilizes a pre-trained visual encoder and a text encoder to extract multimodal representations, and guides cross-modal feature collaborative enhancement and infrared pseudo-label optimization through identity-level text semantic guidance. This allows for the learning of pedestrian representations with both identity discriminative power and modal consistency without requiring infrared pedestrian identity annotation, thereby reducing infrared identity annotation costs and improving cross-modal retrieval performance.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] A semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning includes the following steps:

[0009] A. Obtain semi-supervised multimodal training data; the training data includes visible light pedestrian images with identity ground truth labels, infrared pedestrian images without identity labels, and existing identity-level text descriptions that match the identities of pedestrians one by one; the identity-level text descriptions include fine-grained semantic information such as the color of the upper garment, the color of the lower garment, the items carried, and the human appearance attributes.

[0010] B. Multimodal basic feature extraction: Visual features of visible light pedestrian images and infrared pedestrian images are extracted using a pre-trained visual encoder; text token features are extracted from identity-level text descriptions using a pre-trained text encoder, and attention pooling is performed on the text token features to obtain identity-specific text semantic anchors;

[0011] C. Bidirectional gated residual modulation is achieved through a semantically guided cross-modal gated multi-scale fusion module:

[0012] C1. Generate visual gating weights based on the text semantic anchors, and use the visual gating weights to perform residual modulation on the visual features to obtain semantically modulated visual features.

[0013] C2. Generate text gating weights based on the visual features, and use the text gating weights to perform residual modulation on the text semantic anchors to obtain visually adapted text features.

[0014] D. Generate cross-modal joint features within the semantically guided cross-modal gating multi-scale fusion module: Align the visually adapted text feature space to the size of the semantically modulated visual features, then concatenate the channels and feed them into a multi-scale convolutional branch to extract multi-receptive field cross-modal collaborative features; generate adaptive weights for each scale and aggregate the multi-scale branch features by weighting, and obtain cross-modal joint features through dimension recovery and global average pooling.

[0015] E. Implement hierarchical semantic constraints through the category-aware hierarchical semantic collaborative alignment module: Divide the identity-level text description into three non-overlapping semantic sets: upper garment, lower garment, and accessories. Calculate the semantic intersection-union ratio of each pair of samples within a batch, and construct a category-aware hierarchical semantic alignment loss to constrain the hierarchical semantic consistency of multimodal features.

[0016] F. Generate unlabeled infrared sample pseudo-labels through a progressive two-stage optimal transfer pseudo-label learning module:

[0017] F1, First stage: Generate a category probability matrix based on infrared semantic modulation visual features, and fuse the category prior distribution and sample prior distribution to obtain an equilibrium probability allocation matrix;

[0018] F2, Second stage: Calculate the cosine affinity between samples based on cross-modal joint features, and combine it with the real identity labels of visible light pedestrian images to obtain the category-level affinity assignment matrix;

[0019] F3. According to the progressive fusion of the equilibrium probability allocation matrix and the category-level affinity allocation matrix in the training rounds, perform Sinkhorn-Knopp optimal transmission on the fusion matrix and output infrared sample pseudo-labels;

[0020] G. Multimodal collaborative semi-supervised model training: Using visible light real identity labels, infrared sample pseudo labels, and hierarchical text semantics as joint supervision signals, the network model is trained in a multi-stage semi-supervised manner using joint identity classification loss, triplet metric loss, and category-aware hierarchical semantic alignment loss.

[0021] H. Cross-modal pedestrian retrieval reasoning: After the model is trained, input the infrared image to be queried and the visible light image library image to extract cross-modal joint features, calculate and sort the similarity between the query features and the image library features, and output the visible-infrared cross-modal pedestrian re-identification retrieval results.

[0022] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0023] 1. Reduced data annotation costs: This invention utilizes only visible light pedestrian images with identity tags, unlabeled infrared pedestrian images, and identity-level text descriptions for semi-supervised training, eliminating the need for manual annotation of pedestrian identities in infrared images and significantly reducing the manpower and time costs required for infrared data annotation in large-scale, all-weather security scenarios.

[0024] 2. Enhance multimodal feature alignment capability: A semantically guided cross-modal gating multi-scale fusion module is used to achieve bidirectional residual modulation of text semantics and visual features, and multi-scale adaptive aggregation is combined to generate cross-modal joint features; at the same time, relying on the category-aware hierarchical semantic collaborative alignment module, fine-grained semantic constraints of upper garment, lower garment and accessories are introduced to strengthen the structured binding between text semantics and visual representation, and effectively alleviate the feature mismatch problem between visible light and infrared modalities.

[0025] 3. Improve the reliability of infrared pseudo-labels: Adopt a progressive two-stage optimal transmission pseudo-label learning scheme, dynamically fuse classification probability allocation results and cross-modal feature affinity allocation results with each training round, avoid noise pseudo-labels caused by unstable model features in the early training stage, and suppress error accumulation caused by continuous iteration of erroneous pseudo-labels.

[0026] 4. Significantly Improved Cross-Modal Retrieval Performance: Under the semi-supervised SSVI-ReID task setting, compared with the baseline method MUCG, SYSU-MM01 achieves a 13.5-fold improvement in R-1 and a 12.3-fold improvement in mAP for all-scene retrieval; a 6.8-fold improvement in R-1 and a 5.9-fold improvement in mAP for indoor retrieval; a 3.9-fold improvement in R-1 and a 6.1-fold improvement in mAP for RegDB infrared → visible light retrieval; and a 1.8-fold improvement in R-1 and a 2.6-fold improvement in mAP for the LLCM dataset. Experiments on public datasets verify that the present invention has higher retrieval accuracy and good robustness, and can meet the engineering application requirements of cross-modal pedestrian retrieval in all-weather and low-light environments. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall network architecture and training process of the present invention.

[0028] Figure 2 This is a schematic diagram of the semantically guided cross-modal gating multi-scale fusion module structure of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. This embodiment is implemented under the premise of the technical solution of the present invention, and provides specific implementation methods and complete and reproducible operation procedures. It should be noted that the protection scope of the present invention is not limited to the specific parameters and implementation cases described in this embodiment.

[0030] This invention generates cross-modal joint features through a semantically guided cross-modal gating multi-scale fusion module; implements cross-modal collaborative supervision through a category-aware hierarchical semantic collaborative alignment module; and optimizes the pseudo-labels of unlabeled infrared samples through a progressive two-stage optimal transmission pseudo-label learning module.

[0031] This embodiment uses a ResNet-50 pre-trained on ImageNet as the visual backbone network, and a frozen RoBERTa-base text encoder; the input image size for SYSU-MM01 is set to 384×192, and the input image size for RegDB and LLCM is set to 288×144. Training uses the AdamW optimizer with an initial learning rate of 3×10⁻⁶. -4 The weight decay coefficient is 1×10 -4 In rounds 40 and 70, the learning rate was adjusted to 1×10. -5 and 1×10 -6The total training rounds were 100. In SYSU-MM01, 4 identities and 4 images per identity were sampled per batch per modality; in RegDB and LLCM, 4 identities and 8 images per identity were sampled per batch per modality. Optimal transmission employed Sinkhorn-Knopp iteration until the edge distribution error was no greater than 0.1. The smoothing coefficient γ in stage 1 was set to 0.1, and progressive fusion started from round 30, increasing linearly over the next 20 rounds, with a maximum fusion weight of 0.8. The weight of the category-aware hierarchical semantic alignment loss was set to 5.0.

[0032] This embodiment specifically includes the following steps:

[0033] A. Given a semi-supervised visible-infrared pedestrian re-identification dataset, which contains visible light pedestrian images with identity labels, infrared pedestrian images without identity labels, and manually labeled identity-level text descriptions corresponding to pedestrian identities; the identity-level text descriptions include fine-grained semantic information such as the color of upper garments, the color of lower garments, and the attributes of carried objects or human appearance, which can accurately characterize the appearance differences of different pedestrians; this invention relies on identity-level fine-grained text semantics as high-level structured supervision information to make up for the visual feature confusion caused by the different imaging mechanisms and large differences in feature distribution between visible light and infrared modalities, and provides reliable prior constraints for subsequent accurate cross-modal semantic alignment.

[0034] B. Input the visible light pedestrian image and the infrared pedestrian image from step A into the pre-trained visual encoder, respectively. In this embodiment, the visual encoder uses a ResNet50 backbone network to extract visible light visual features and infrared visual features. Simultaneously, input the identity-level text description into the pre-trained text encoder to extract text token features. To obtain identity-related text semantic anchors, attention pooling is performed on the text token features to aggregate global identity semantics, remove invalid text noise, and obtain text semantic anchors with fixed dimensions and strong correlation to pedestrian identity. The pooled text semantic anchors are denoted as... .

[0035] C. Construct a semantically guided cross-modal gating multi-scale fusion module. First, the text semantic anchors obtained in step B are converted into visual gating weights to utilize text semantics to modulate the visual channel. This process is defined as follows:

[0036] (1)

[0037] in, Indicates visual gating weights, This indicates a fully connected mapping. This represents the Sigmoid activation function;

[0038] Next, text gating weights are generated based on visual features, and this process is defined as follows:

[0039] (2)

[0040] in, Indicates text gating weights, This represents a 3×3 convolution. Represents a 1×1 convolution. Indicates global average pooling;

[0041] Subsequently, the visual and textual features are subjected to bidirectional residual modulation using the aforementioned gating weights. This process is defined as follows:

[0042] (3)

[0043] (4)

[0044] in, This indicates element-wise multiplication. Represents visual features after semantic modulation. This represents the text features after visual adaptation; the bidirectional gating modulation can reduce the alignment error between text and visual features; this step constructs a bidirectional residual gating modulation mechanism that guides vision and adapts text to visual features, realizing bidirectional mutual correction between visual features and text semantics, effectively reducing the modal alignment error between fixed text semantics and dynamic visual features, and overcoming the shortcomings of traditional unidirectional semantic guidance with poor adaptability and low feature coupling.

[0045] D. In the semantically guided cross-modal gated multi-scale fusion module, the visually adapted text features obtained in step C are expanded to the same spatial size as the semantically modulated visual features, and channel concatenation is performed between them. Then, the multi-scale convolutional branch is input to extract cross-modal collaborative features under different receptive fields. This process is defined as follows:

[0046] (5)

[0047] in, This indicates a space expansion operation. Indicates channel splicing. This represents the s-th scale convolution branch, which in this embodiment corresponds to 1×1, 3×3, and 5×5 convolutions, respectively; simultaneously, scale-adaptive weights are generated through global average pooling, 1×1 convolution, and the Softmax operator. .

[0048] The weighted aggregation of multi-scale branch outputs is defined as follows:

[0049] (6)

[0050] in, This represents the aggregated multi-scale features. Finally, the feature dimensions are recovered through 1×1 convolution, and global average pooling is used to obtain the cross-modal joint features. Finally, the cross-modal joint features are used for identity classification, retrieval matching, and subsequent infrared pseudo-label generation. This step extracts fine-grained and coarse-grained pedestrian appearance features from different receptive fields through multi-scale convolution in parallel, and dynamically allocates the contribution of each scale feature with adaptive weights to avoid the problem of limited representation ability of single-scale features, and outputs cross-modal joint features with high discriminativeness and high modal consistency.

[0051] E. Construct a category-aware hierarchical semantic collaborative alignment module, which divides the identity-level text description in step A into three non-overlapping semantic sets: upper garment, lower garment, and accessories, based on human body structure; for any two samples, calculate the semantic intersection-union ratio (CIU) at the k-th semantic level, where the CIU at the k-th semantic level is denoted as . Furthermore, cross-modal collaborative semantic supervision is implemented through category-aware hierarchical semantic alignment loss, the process of which is defined as follows:

[0052] (7)

[0053] in, Indicates batch size. and Let i and j represent the text semantic distributions, respectively. express Divergence; This loss enhances the structured binding between fine-grained text semantics and visual representation; This invention decomposes disordered pedestrian text descriptions into three layers of structured semantic sets: upper garment, lower garment, and accessories. It constructs a hierarchical semantic alignment loss by using semantic intersection-union ratio weighted KL divergence, thereby achieving a structured binding between fine-grained text semantics and visual features. This effectively alleviates the confusion of similar pedestrian appearances and the problem of cross-modal feature mismatch, and significantly enhances the semantic consistency of multimodal features.

[0054] F. Construct a progressive two-stage optimal transmission pseudo-label learning module. For unlabeled infrared samples, firstly, use the infrared semantic modulation visual features obtained in step C to generate a class probability matrix, and then combine the class prior distribution and the sample prior distribution to construct the first-stage equilibrium allocation matrix. This process is defined as follows:

[0055] (8)

[0056] in, This represents the class probability matrix of the infrared samples. Represents the smoothing coefficient. and Let A and B represent the category prior distribution and the sample prior distribution, respectively. Next, L2 normalization is performed on the infrared and visible light cross-modal joint features obtained in step D. The sample-level affinity is calculated using cosine distance, and the mean affinity of samples of the same class is obtained based on the real identity label of the visible light samples to obtain the second-stage category-level affinity assignment matrix.

[0057] The second-stage category-level affinity assignment matrix has the same dimension as the first-stage category probability assignment matrix. To avoid erroneous pseudo-labels caused by unstable fusion features in the early stages of training, this invention progressively fuses the two-stage assignment results according to the training rounds. This process is defined as follows:

[0058] (9)

[0059] in, This represents the fusion weights that change with each training epoch. This represents the assignment matrix obtained from feature affinity. This represents the assignment matrix obtained from the classification probabilities; subsequently, the fusion matrix is ​​processed. The optimal transmission yields pseudo-labels for infrared samples. This invention employs a two-stage progressive fusion mechanism. In the early stages of training, the classification probability is used to ensure the basic rationality of the pseudo-labels and avoid early feature oscillation problems. In the later stages of training, the accuracy of the pseudo-labels is optimized by relying on highly stable cross-modal joint features. The progressive update strategy can effectively suppress noisy pseudo-labels, avoid the accumulation of errors during the training process, and significantly improve the reliability of pseudo-labels for unlabeled infrared samples.

[0060] G. Use the infrared pseudo-labels obtained in step F and the visible light real identity labels to carry out multi-stage semi-supervised collaborative training, and use identity classification loss, triplet metric loss and category-aware hierarchical semantic alignment loss to jointly constrain the model, so that it can simultaneously optimize identity discrimination ability, cross-modal consistency and hierarchical semantic alignment ability.

[0061] Specifically, the training loss consists of three parts: basic re-identification loss, cross-modal joint feature loss, and category-aware hierarchical semantic alignment loss. The basic re-identification loss applies to visible light and infrared visual features, including identity classification cross-entropy loss and triplet metric loss, to improve the identity discrimination ability of visual features. The cross-modal joint feature loss includes joint feature identity classification loss and joint feature triplet metric loss, used to constrain the joint features after text-visual fusion to maintain identity consistency and inter-class separability. The category-aware hierarchical semantic alignment loss applies cross-modal constraints to the semantics of upper garments, lower garments, and accessories, respectively, to improve the consistency between local clothing semantics and visual regions. In each training round, the above three parts of the loss are calculated based on the updated infrared pseudo-labels, and the trainable parameters (excluding the frozen text encoder) are updated in reverse.

[0062] H. After training is complete, input the infrared image to be queried and the visible light image library image into the trained model, extract the corresponding pedestrian features or cross-modal joint features, calculate the similarity between the query features and the image library features, and output the cross-modal pedestrian re-identification results according to the similarity ranking.

[0063] The overall network architecture of this invention embodiment is as follows: Figure 1 As shown, the network architecture has three inputs: a labeled visible light image, an unlabeled infrared image, and a pedestrian identity-level text description. The example text for the identity-level text description is "a man wearing a yellow short-sleeved shirt, a blue backpack, and blue trousers." The labeled visible light image and the unlabeled infrared image are sequentially processed through convolution and residual blocks to extract visual features. The identity-level text description is fed into a text encoder and pooled to obtain semantic features. The visible light visual features and infrared visual features share parameters with the text semantic features in two semantically guided cross-modal gated multi-scale fusion (CGMF) modules. The CGMF module outputs the visible light branch features. , , The CGMF module below outputs infrared branch features. , , Infrared branch characteristics The model is fed into a progressive two-stage optimal transfer pseudo-label learning module (PDS-OTPL), which consists of two stages: Stage 1 and Stage 2. This module iteratively updates the pseudo-labels for pedestrians in unlabeled infrared images with each training epoch (p, p+1). Three types of loss constraints are introduced during the model training phase: re-identification loss... Cross-modal joint loss Category-aware hierarchical semantic alignment loss Multiple loss functions are used to collaboratively update trainable parameters, excluding the frozen text encoder. Figure 1 The snowflake symbol in the text encoder indicates that the text encoder parameters are frozen and their weights are not updated during training.

[0064] Figure 2 This is a schematic diagram of the semantically guided cross-modal gated multi-scale fusion module (CGMF) of the present invention. The module receives text features. Original visual features Two inputs. Raw visual features. The text is input into a text gating generator via a convolutional path. Inside the text gating generator, convolutional, activation, average pooling, convolutional, and normalization layers are sequentially configured. The output features are then compressed by a compression layer to obtain visually adapted text features. Text features The input features are sequentially passed through pooling attention and linear layers before being fed into a visual gating generator. The visual gating generator contains linear layers and normalization layers. The output features are then processed through a reshaping layer to obtain semantically modulated visual features. . After spatial size expansion through repeated layers, the features are concatenated along the channel dimension to obtain the basic fused features. These basic fused features are then input into multi-scale branches consisting of 1×1, 3×3, and 5×5 convolutions, and simultaneously fed into an adaptive weight generation module. This module outputs the weights corresponding to each scale branch. The output features of each scale branch are then element-wise multiplied and fused with their corresponding weights. The resulting fused features are then processed sequentially through convolution, batch normalization, and activation layers, ultimately outputting cross-modal joint features. .

[0065] Figure 2 The two intersecting paths in the diagram represent bidirectional cross-modal gating: textual semantic features are processed through pooling attention, linear mapping, and Sigmoid normalization to generate visual gating. It is used for residual modulation of visual features; the visual features are converted into text gating through 3×3 convolution, activation, global average pooling, 1×1 convolution and sigmoid normalization. This is used for residual modulation of text features. The outer lines on both sides represent residual connections of the original features; ⊙ represents element-wise multiplication; the central fusion node represents concatenation along the channel dimension; the repeating layer represents expanding the text features to the same spatial size as the visual features; and the adaptive weights are obtained by global average pooling, 1×1 convolution, and Softmax operations.

[0066] To verify the effectiveness and superiority of the semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning proposed in this invention, performance tests were conducted on three publicly available standard datasets: SYSU-MM01, RegDB, and LLCM. Table 1 shows the performance comparison results between this invention and existing mainstream methods on the SYSU-MM01 and RegDB datasets, and Table 2 shows the performance comparison results between this invention and existing mainstream methods on the LLCM dataset.

[0067] Table 1

[0068]

[0069] Table 2

[0070]

[0071] As shown in Tables 1 and 2, the experimental data reveals that in the SSVI-ReID semi-supervised visible light + text → infrared retrieval task, the SGMT method of this invention improves upon the single-modal baseline MUCG and LLCM datasets for the same task (I → R) by 1.8 and mAP by 2.6; in the SYSU-MM01 full-scene task, it improves upon the R-1 by 13.5 and mAP by 12.3; in indoor retrieval, it improves upon the R-1 by 6.8 and mAP by 5.9; and in the RegDB dataset, it improves upon the R-1 by 3.9 and mAP by 6.1, demonstrating significant performance improvements. This invention uses only visible light labels, relying on text-layered semantic alignment and two-stage optimal transmission pseudo-label optimization, significantly outperforming existing methods without text assistance and using single-stage pseudo-labels in the SSVI-ReID trimodal semi-supervised track. The multimodal collaborative learning framework constructed in this invention can effectively mine fine-grained semantics of text, correct cross-modal feature shifts, and reduce infrared pseudo-label noise, achieving highly competitive retrieval accuracy on all standard public datasets while eliminating the cost of manual infrared image annotation, making it suitable for practical security deployment scenarios.

[0072] To enhance the guidance of textual semantics on visual features, this invention designs a semantically guided cross-modal gating multi-scale fusion module. This module transforms textual semantics into visual gating signals, while simultaneously generating textual gating signals from visual responses. Cross-modal joint features are then generated through multi-scale convolutional branches with different receptive fields and adaptive weight aggregation. To strengthen fine-grained semantic constraints, this invention designs a category-aware hierarchical semantic collaborative alignment module. This module divides identity-level textual descriptions into human body structure levels such as upper garments, lower garments, and accessories, and implements cross-modal collaborative semantic supervision through category-aware hierarchical semantic alignment loss. To fully utilize unlabeled infrared data, this invention designs a progressive two-stage optimal transmission pseudo-label learning module. This module progressively fuses classification probability allocation and cross-modal joint feature affinity allocation, and then obtains infrared sample pseudo-labels through optimal transmission, forming a multi-stage learning process that gradually transitions from classification probabilities to cross-modal joint features.

[0073] Therefore, this invention presents a multi-stage semi-supervised collaborative learning framework for visible light, infrared, and textual multimodal data. It can more accurately learn consistent feature representations between visible light and infrared modalities and generate more reliable infrared pseudo-labels using only visible light identity tags, unlabeled infrared images, and identity-level text descriptions. Experimental results on public datasets demonstrate that this invention achieves competitive retrieval performance in semi-supervised visible-infrared pedestrian re-identification tasks. The framework is robust and adaptable to all-weather, low-light, cross-modal pedestrian retrieval scenarios.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning, characterized in that, Includes the following steps: A. Acquire semi-supervised multimodal training data, the training data including visible light pedestrian images with identity labels, unlabeled infrared pedestrian images, and existing identity-level text descriptions; the identity-level text descriptions include fine-grained semantic information such as the color of the upper garment, the color of the lower garment, the items carried, or the appearance attributes of the human body; B. Visual features and text token features of visible light and infrared pedestrian images are extracted using a pre-trained visual encoder and a text encoder, respectively. Attention pooling is then performed on the text token features to obtain the pooled text semantic anchors. C. Construct a semantically guided cross-modal gating multi-scale fusion module, convert the text semantic anchors obtained in step B into visual gating weights, use text semantic modulation of the visual channel, generate text gating weights based on visual features, and use the above gating weights to perform bidirectional residual modulation on visual features and text features, thereby reducing the alignment error between text and visual features through this bidirectional gating modulation. D. Within the semantically guided cross-modal gated multi-scale fusion module, cross-modal joint features are generated through multi-scale convolutional branches and adaptive weight aggregation; E. Construct a category-aware hierarchical semantic collaborative alignment module, which divides the identity-level text description in step A into three non-overlapping semantic sets: upper garment, lower garment, and accessories, based on human body structure, and implements cross-modal collaborative semantic supervision based on the semantics of upper garment, lower garment, and accessories; F. Construct a progressive two-stage optimal transmission pseudo-label learning module, progressively integrate classification probability allocation with cross-modal feature affinity allocation, and obtain infrared sample pseudo-labels through optimal transmission; G. Use the infrared pseudo-labels obtained in step F and the visible light real identity labels to carry out multi-stage semi-supervised collaborative training, and use identity classification loss, triplet metric loss and category-aware hierarchical semantic alignment loss to jointly constrain the model, so that it can simultaneously optimize identity discrimination ability, cross-modal consistency and hierarchical semantic alignment ability. H. After training is complete, input the infrared image to be queried and the visible light image library image into the trained model, extract the corresponding pedestrian features or cross-modal joint features, calculate the similarity between the query features and the image library features, and output the visible-infrared cross-modal pedestrian re-identification results in order of similarity.

2. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step C, the text semantic anchors obtained in step B are converted into visual gating weights, and the visual channel is modulated using text semantics. This process is defined as follows: in, Indicates visual gating weights, Indicates a fully connected mapping. This represents the Sigmoid activation function. This represents the semantic anchor point of the pooled text.

3. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step C, text gating weights are generated based on visual features. This process is defined as follows: in, Indicates text gating weights, This represents the Sigmoid activation function. This represents a 3×3 convolution. This represents a 1×1 convolution, where ReLU is the ReLU activation function. Indicates global average pooling. These are the original visual features.

4. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step C, the aforementioned gating weights are used to perform bidirectional residual modulation on visual and textual features. This process is defined as follows: in, This indicates element-wise multiplication. Represents visual features after semantic modulation. Indicates text features after visual adaptation. Original visual features Indicates visual gating weights, This represents the text gating weight, which reduces the alignment error between text and visual features through bidirectional gating modulation.

5. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step D, within the semantically guided cross-modal gated multi-scale fusion module, the visually adapted text features obtained in step C are expanded to the same spatial size as the semantically modulated visual features, and channel concatenation is performed between them. Then, the multi-scale convolutional branch is used to extract cross-modal collaborative features under different receptive fields. This process is defined as follows: in, This indicates a space expansion operation. Indicates channel splicing. This represents the convolution branch at the s-th scale. Represents visual features after semantic modulation. Represents the text features after visual adaptation; scale-adaptive weights are generated through global average pooling, 1×1 convolution, and the Softmax operator.

6. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step D, the multi-scale branch outputs are weighted and aggregated, and this process is defined as follows: in, This represents the multi-scale features after aggregation. The scale-adaptive weights are those corresponding to the s-th scale branch. For element-wise multiplication operator, The cross-modal collaborative features output by the s-th scale branch are used to recover the feature dimensions of the aggregated multi-scale features through 1×1 convolution, and then global average pooling is used to obtain the cross-modal joint features. Cross-modal joint features Used for identity classification, retrieval matching, and subsequent generation of infrared pseudo-tags.

7. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step E, a category-aware hierarchical semantic collaborative alignment module is constructed. Based on human body structure, the identity-level text description from step A is divided into three non-overlapping semantic sets: upper garment, lower garment, and accessories. For any two samples, the semantic intersection-union ratio (CIU) is calculated at the k-th semantic level, and the CIU at the k-th semantic level is denoted as [the specific value is missing in the original text]. Furthermore, cross-modal collaborative semantic supervision is implemented through category-aware hierarchical semantic alignment loss, the process of which is defined as follows: in, This represents the category-aware hierarchical semantic alignment loss. Indicates batch size. This is a semantic hierarchy number. and Let i and j represent the text semantic distributions, respectively. express Divergence; this loss enhances the structured binding between fine-grained textual semantics and visual representation.

8. The semi-supervised visible-infrared pedestrian re-identification method based on multimodal collaborative learning according to claim 1, characterized in that, In step F, a progressive two-stage optimal transmission pseudo-label learning module is constructed. For unlabeled infrared samples, the infrared semantic modulation visual features obtained in step C are used to generate a class probability matrix, and the first-stage equilibrium allocation matrix is ​​constructed by combining the class prior distribution and the sample prior distribution. This process is defined as follows: in, This represents the class probability matrix of the infrared samples. Represents the smoothing coefficient. and Let these represent the class prior distribution and the sample prior distribution, respectively. Let be the diagonal matrix formed by the prior distributions of the categories. Let be the diagonal matrix formed by the prior distributions of the samples. Represents the equilibrium probability distribution matrix; The infrared and visible light cross-modal joint features obtained in step D are L2 normalized, and the sample-level affinity is calculated using cosine distance. The mean affinity of samples of the same class is obtained based on the real identity label of the visible light samples to obtain the second-stage category-level affinity assignment matrix. The second-stage category-level affinity assignment matrix has the same dimension as the first-stage category probability assignment matrix. To avoid erroneous false labels caused by unstable feature fusion in the early stages of training, the results are progressively fused in two stages according to the training rounds. This process is defined as follows: in, This represents the fusion weights that change with each training epoch. This represents the assignment matrix obtained from feature affinity. This represents the assignment matrix obtained from the classification probabilities; subsequently, the fusion matrix is ​​processed. Optimal transmission yields infrared sample pseudo-tags.

Images