Unsupervised pedestrian re-identification method based on vision-language model and enhanced branch

Abstract

The invention discloses an unsupervised pedestrian re-identification method based on a vision-language model and an enhanced branch, and the method employs a CLIP as a backbone network, and introduces an additional enhanced branch for training. Comprising the following steps: setting a text prompt with a specific identity, fixing parameters of a text encoder, and sending an image and a text into an image encoder and a text encoder; splicing the features of the last layer and the previous layer of the image encoder, and generating a pseudo tag through clustering; cluster comparison loss, image-to-text comparison loss and text-to-image comparison loss are calculated, and text prompt and an image encoder are optimized; when the camera label is available, creating an enhancement branch for each camera, and calculating the contrast loss in the camera; otherwise, creating an enhanced branch for the text, and calculating KL divergence loss; and finally, using an image encoder to obtain features for image re-identification. According to the method, through multi-layer feature fusion and branch constraint enhancement, the feature representation capability and retrieval accuracy of the model under unsupervised setting are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to an unsupervised person re-identification method based on a vision-language model and augmentation branches. Background Technology

[0002] Pedestrian re-identification, aiming to retrieve specific pedestrians across different camera viewpoints, is a key technology in intelligent video surveillance systems. While traditional supervised learning methods demonstrate excellent recognition accuracy, their heavy reliance on large amounts of expensive manually labeled data limits the scalability and deployment efficiency of models in large-scale real-world scenarios. Therefore, unsupervised pedestrian re-identification without manual labeling has become a research hotspot in recent years. Current mainstream unsupervised methods are mostly based on convolutional neural networks (CNNs), typically employing clustering-based pseudo-label generation techniques, combined with feature memory to store feature centers, and using contrastive learning to optimize feature representations. Meanwhile, pre-trained vision-language models, represented by CLIP, have demonstrated the potential to surpass traditional models in supervised tasks due to their powerful generalization ability learned from massive amounts of data.

[0003] However, effectively applying vision-language models to unsupervised person re-identification still faces numerous technical bottlenecks. Existing technologies mainly suffer from the following problems: First, most existing unsupervised methods (such as Cluster Contrast) rely solely on a single global feature for constraints, failing to fully utilize feature information from different levels of the model, resulting in insufficient feature richness. Second, existing CLIP-based re-identification methods are mostly concentrated in supervised domains; in the absence of real identity labels, it is difficult to directly utilize the image-text alignment capabilities pre-trained by CLIP to assist in clustering or feature optimization. Furthermore, existing architectures often lack flexible enhancement branch designs, failing to effectively introduce additional cross-domain or intra-camera constraints based on different prior conditions such as the availability of camera labels, thus limiting the model's discrimination accuracy in complex unsupervised environments.

[0004] Therefore, it is essential to provide an unsupervised pedestrian re-identification method based on a vision-language model and augmentation branches. Summary of the Invention

[0005] To address the shortcomings of existing unsupervised ReID technologies in terms of insufficient feature representation and incomplete utilization of multi-level semantic information, this invention mines the prior knowledge of visual language pre-trained models. By fusing multi-level features and enhancing branch constraints, it significantly improves the model's feature representation capability and retrieval accuracy under unsupervised settings.

[0006] The objective of this invention is achieved as follows:

[0007] An unsupervised person re-identification method based on a vision-language model and augmentation branch, utilizing CLIP

[0008] As the backbone network, additional augmentation branches are introduced for training. By fusing multi-layer features and augmentation branch constraints, and using an optimized image encoder to generate image features for image re-identification, the model's feature representation ability and retrieval accuracy under unsupervised settings are significantly improved. The method includes the following specific steps:

[0009] Step 1: Set identity-specific text prompts The text prompt will be the same as template A.

[0010] A photo of a person. Combined with textual description. ;

[0011] Step 2: Fix the parameters of the text encoder, and feed the image and text description into the image encoder and text encoder;

[0012] Step 3: Concatenate the output features of the last layer of the image encoder with the output features of the previous layer, and then perform a stitching operation.

[0013] Clustering is performed based on features to generate pseudo-labels;

[0014] Step 4: Calculate cluster contrast loss Image-to-text contrast loss Text-to-image comparison loss

[0015] lose To optimize text prompts and image encoders;

[0016]

[0017]

[0018]

[0019]

[0020] in The i-th image feature in a batch The positive sample proxy center, K is the number of clusters generated by clustering, and k is the index of the current cluster. It is the set temperature coefficient; It is the [CLS] token obtained from the i-th image through the image encoder. It is the [EOS] token output by the text encoder corresponding to the text description; and It projects two types of tokens onto a linear layer in the same space to obtain image features. and text features , It is the cosine similarity between the two; B is the total number of images in the current batch, and a is the index within the batch. It is the pseudo-label of the i-th image. All the fake labels in this batch are The image index set, The size of the set;

[0021] Step 5: Based on optimized text prompts, create enhanced text for when the camera tag is unavailable.

[0022] Support, and calculate KL divergence loss. Cross-domain knowledge distillation is performed by aligning the enhanced branches with the zero-shot image-text similarity distribution of the visual-language model to optimize the text branch and image encoder. When camera labels are available, an enhanced branch is created for each camera, and the intra-camera contrast loss is calculated using the feature proxy of that camera. To optimize camera branching and image encoder;

[0023]

[0024]

[0025] in These are image features from the j-th camera branch. Is with The proxy center corresponding to the pseudo-tag It is the cluster number under the j-th camera; These are image features derived from text branches. These are text features obtained through a text encoder. This is the probability distribution corresponding to the student model; These are image features obtained through the penultimate layer of the image encoder. This is the probability distribution corresponding to the teacher model;

[0026] Step 6: During the testing phase, the test set images are fed into the trained image encoder to obtain image features.

[0027] Image re-identification: For each image in the query value of the test set, find the most similar image from other cameras in the candidate image set gallery, and calculate the mAP and Rank-1 index. The higher the index, the better the effect.

[0028] The text encoder and image encoder are based on Transformer networks. The backbone network of the image encoder is specifically chosen to be ViT-B / 16, and the text encoder uses a 12-layer Transformer network.

[0029] The features used for clustering are 1280-dimensional, obtained by concatenating the 512-dimensional output features of the last layer of the image encoder with the 768-dimensional output features of the previous layer. The clustering uses the DBSCAN algorithm.

[0030] The text prompt To prevent parameter sharing between different pseudo-labels in the training set, M is set to 4. As a fuzzy description of each identity, it supplements the textual descriptions not included in the re-identification task.

[0031] The text enhancement branch is used to calculate cosine similarity with the optimized text features, and the camera enhancement branch does not share parameters across different cameras. The key feature is that all branches are initialized using the last layer parameters of the image encoder and do not participate in inference.

[0032] This invention addresses the shortcomings of existing unsupervised ReID technologies, such as insufficient feature representation capabilities and inadequate utilization of multi-level semantic information. It mines the prior knowledge of visual language pre-trained models and, by fusing multi-level features and enhancing branch constraints, makes the results of retrieving and querying the most similar cross-camera images in the candidate image set gallery more accurate. This significantly improves the model's feature representation capabilities and the mAP and Rank-1 metrics of image re-identification under unsupervised settings. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the process of the present invention;

[0034] Figure 2 This is a schematic diagram of the process of an embodiment of the present invention. Detailed Implementation

[0035] To more clearly illustrate the technical means, technical improvements, and beneficial effects of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0036] Example

[0037] See Figure 1-2 An unsupervised person re-identification method based on a vision-language model and augmentation branches.

[0038] Using CLIP as the backbone network and introducing additional enhancement branches for training, image re-identification is performed by fusing multi-layer features with enhancement branch constraints and using an optimized image encoder to generate image features. The specific steps include:

[0039] S1: For the Market1501 dataset, which has 750 identities in the training set and 751 identities in the test set.

[0040] Set identity-specific text prompts during the training phase. The text prompt will combine with the template "A photo of a person" to form a text description. M is set to 4;

[0041] S2: Fixed text encoder parameters, feed image and text descriptions into image encoder and text encoder;

[0042] S3: Combine the output features of the last layer of the image encoder with the output features of the previous layer, and then stitch them together.

[0043] Clustering is performed based on features to generate pseudo-labels;

[0044] S4: Calculate cluster contrast loss Image-to-text contrast loss Text-to-image comparison loss

[0045] lose To optimize text prompts and image encoders;

[0046]

[0047]

[0048]

[0049]

[0050] in The i-th image feature in a batch The positive sample proxy center, K is the number of clusters generated by clustering, and k is the index of the current cluster. It is the set temperature coefficient; It is the [CLS] token obtained from the i-th image through the image encoder. It is the [EOS] token output by the text encoder corresponding to the text description; and It projects two types of tokens onto a linear layer in the same space to obtain image features. and text features , It is the cosine similarity between the two; B is the total number of images in the current batch, and a is the index within the batch. It is the pseudo-label of the i-th image. All the fake labels in this batch are The image index set, The size of the set;

[0051] S5: Based on optimized text hints, it creates enhanced branches for the text when the camera tag is unavailable.

[0052] The KL divergence loss is calculated using formulas (6)(7)(8). Cross-domain knowledge distillation is performed by aligning the enhanced branches with the zero-shot image-text similarity distribution of the visual-language model to optimize the text branch and image encoder. When camera labels are available, an enhanced branch is created for each camera, and the intra-camera contrast loss is calculated using the feature proxy of that camera. To optimize camera branching and image encoder, as in formula (5);

[0053]

[0054]

[0055] in These are image features from the j-th camera branch. Is with The proxy center corresponding to the pseudo-tag It is the cluster number under the j-th camera; These are image features derived from text branches. These are text features obtained through a text encoder. This is the probability distribution corresponding to the student model; These are image features obtained through the penultimate layer of the image encoder. This is the probability distribution corresponding to the teacher model;

[0056] S6: During the testing phase, the test set images are fed into the trained image encoder to obtain image features, and then processed.

[0057] Image re-identification: For each image in the query value of the test set, find the most similar image from other cameras in the candidate image gallery, and calculate the mAP and Rank-1 metrics. The final result on the Market1501 dataset is an mAP of 90.2% and a Rank-1 of 95.7%, which is a significant improvement over the baseline method without the augmentation branch.

[0058] This invention addresses the shortcomings of existing unsupervised ReID technologies, such as insufficient feature representation capabilities and inadequate utilization of multi-level semantic information. It mines the prior knowledge of visual language pre-trained models and, by fusing multi-layered features and enhancing branch constraints, significantly improves the model's feature representation capabilities and image re-identification accuracy under unsupervised settings. The above are merely preferred embodiments of this invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An unsupervised pedestrian re-identification method based on visual-linguistic model and enhanced branch, comprising the following steps: Step 1: Set identity-specific text prompts , the text prompts and templates combine to form a text description ; Step 2: fixing the parameters of the text encoder, inputting the image and the text description into the image encoder and the text encoder; Step 3: splicing the output features of the last layer of the image encoder with the output features of the previous layer, and clustering the spliced features to generate pseudo labels; Step 5: based on the optimized text prompt, creating an enhanced branch for the text when the camera label is unavailable; Step 4: Compute cluster contrastive loss Image-to-text contrastive loss Text-to-image contrastive loss loss to optimize text hints and image encoders; ； ； ； ； wherein is the i-th image feature in the batch is the positive sample proxy center, K is the number of clusters generated by clustering, k is the index of the current cluster, is the set temperature coefficient; is the [CLS] token obtained by the image encoder for the i-th image, is the [EOS] token output by the text encoder corresponding to the text description; and is a linear layer that projects the two tokens into the same space, respectively obtaining image features and text features , is the cosine similarity between the two, B is the total number of images in the current batch, a is the intra-batch index, is the pseudo label of the i-th image, is the image index set in which all pseudo labels in the batch are , is the size of the set; Step 6: in the test stage, inputting the test set image into the trained image encoder to obtain the image features, and performing image re-identification: for each image in the query value query of the test set, finding the most similar image under other cameras in the candidate image set gallery, and calculating the mAP and Rank-1 indicators, the higher the indicators, the better the effect. Support, and calculate KL divergence loss. Cross-domain knowledge distillation is performed by aligning the enhanced branch with the zero-sample image-text similarity distribution of the visual-language model to optimize the text branch and the image encoder; When camera label is available, create augmented branch for each camera and compute intra-camera contrastive loss with the camera's feature proxy to optimize camera branch and image encoder; ； ； wherein is the image feature from the jth camera branch, is the proxy center corresponding to the pseudo label of is the number of clusters under the jth camera; is the image feature from the text branch, is the text feature obtained through the text encoder, is the probability distribution corresponding to the student model; is the image feature obtained through the second-to-last layer of the image encoder, is the probability distribution corresponding to the teacher model;​ The image encoder and the text encoder are both based on the Transformer network.

2. The unsupervised pedestrian re-identification method based on a vision-language model and an enhanced branch according to claim 1, characterized in that, The spliced features for clustering are obtained by splicing the output features of the last layer of the image encoder with the output features of the previous layer according to the channel dimension. 3.The unsupervised pedestrian re-identification method based on visual-linguistic model and enhanced branch according to claim 1, wherein, The text prompt does not share parameters between different pseudo labels in the training set.

4. The unsupervised pedestrian re-identification method based on a vision-language model and an enhanced branch according to claim 1, characterized in that, The text enhancement branch is used to calculate the cosine similarity with the optimized text features, and the camera enhancement branch does not share parameters for different cameras.

5. The unsupervised pedestrian re-identification method based on visual-linguistic model and enhanced branch according to claim 1, characterized in that, ​

Images