A pedestrian re-identification method based on a contrastive language-image pre-training model CLIP
By employing the CLIP method based on the contrastive language image pre-training model, and utilizing an image encoder with Transformer and CNN or Transformer as the backbone network, the problem of lacking text labels in person re-identification is solved. The accuracy of person re-identification is improved by using text features generated by training the image encoder, thus enhancing the accuracy of the technology application and solving the problem of lacking text label descriptions in person re-identification, achieving a higher recognition accuracy.
Patent Information
- Application Number
- CN202211173432.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-26
- Publication Date
- 2026-01-02
- Estimated Expiration
- 2042-09-26
AI Technical Summary
Existing pedestrian re-identification methods lack text label descriptions when using cross-modal language image pre-trained models, making it difficult to fully utilize the text model and affecting recognition accuracy.
The CLIP method, based on the contrastive language image pre-training model, utilizes a text encoder with Transformer as the backbone network and an image encoder with CNN or Transformer as the backbone network to perform person re-identification by generating image features. This includes contrastive loss function and cross-entropy loss function during the training phase, generating and saving text features for each identity, and then using the image encoder for recognition.
It improved the accuracy of pedestrian re-identification, and enhanced the mAP and Rank-1 metrics, especially when the Transformer was used as the image encoder as the backbone network, with an mAP of 75.8% and a Rank-1 of 89.7%.
Smart Images
Figure CN115393902B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and digital image processing technology, and in particular to a pedestrian re-identification method based on the contrastive language-image pre-training (CLIP) model. Background Technology
[0002] The goal of person re-identification is to match the same object from different camera perspectives. Past work on person re-identification typically used Convolutional Neural Networks (CNNs) as the backbone, which can lead to an overemphasis on local information and a lack of overall awareness. To address this, many methods incorporate prior knowledge into the network, such as using different branches, each focusing on different local region features; or introducing semantic segmentation results to help the network distinguish different modules and further mine features. Furthermore, recent methods have utilized attention mechanisms or employed Transformer networks as the backbone to expand the domain of interest.
[0003] Whether based on Convolutional Neural Networks (CNNs) or Transformers, pre-training is crucial for person re-identification tasks. Historically, most re-identification backbone networks were pre-trained on ImageNet classification tasks. However, recent cross-modal language image pre-training models, such as the CLIP model, have linked visual information with linguistic descriptions by changing the pre-training task, enabling image models to perceive high-level semantics from text and learn transferable features. These tasks are trained on larger datasets with image-text pairs, making them better suited for downstream tasks like image classification and segmentation. However, in person re-identification tasks, because the labels are merely indices lacking specific textual descriptions, it's difficult to easily generate the descriptive text needed by the language image model, hindering the full utilization of the text model.
[0004] Therefore, it is essential to provide a pedestrian re-identification method based on the contrastive language image pre-trained model CLIP. Summary of the Invention
[0005] The purpose of this invention is to provide a pedestrian re-identification method based on the contrastive language image pre-trained model CLIP for pedestrian re-identification tasks lacking text labels. This invention applies the language image pre-trained model to the re-identification task, which is simple and solves the problem of lacking text label descriptions in the re-identification task, thereby improving the accuracy of pedestrian re-identification.
[0006] The objective of this invention is achieved as follows:
[0007] A person re-identification method based on the contrastive language image pre-training model CLIP involves training an image encoder with a CNN or Transformer backbone using a text encoder with a Transformer backbone, and then using the image encoder to generate image features for person re-identification. The method includes the following specific steps:
[0008] Step 1: For the MSMT17 dataset with thousands of identities, during the training phase, assign a set of descriptive texts containing learnable parameters for each identity in the training set. The template for the descriptive texts is "a photo of a[X]1[X]2[X]3...[X]". M person, where [X] m (m∈1,...M) represents the corresponding learnable token parameter;
[0009] Step 2: Fix the parameters of the image encoder and text encoder, and send the image and corresponding descriptive text into the image encoder and text encoder;
[0010] Step 3: Calculate the contrast loss function L for image-to-text and text-to-image conversion. i2t and L t2i The learnable parameters in the descriptive text are trained using the following formula;
[0011] s(V i ,T i ) = V i ·T i =g I (img i )·g T (text i (a)
[0012]
[0013]
[0014] img i The [CLS] token is the classification label output by the image encoder for the i-th image, while the text... i This represents the output token [EOS] of the corresponding descriptive text after passing through the text encoder. I and g T To map the [CLS] token and [EOS] token to a linear layer in the same space, the final image feature V is obtained. i and text features T i ,s(V i ,T i ) is image feature Vi and text features T i The similarity; B is the number of images in the current batch, a is the index in the current batch, and y is the index in the current batch. i It is the identity label of the i-th image, P(y i ) represents those belonging to y in the same batch i The set of indices of all images for this identity, |P(y i | indicates the number of images contained in this set;
[0015] Step 3: Fix the text encoder and description text, generate and save the text features of each identity, and send the image to the image encoder;
[0016] Step 4: Calculate the image-to-text cross-entropy loss function and train the image encoder;
[0017]
[0018] q k =(1-∈)δ k,y +∈ / N (e)
[0019] Where N is the number of identities in the training set of the dataset, k is the index of all identities in the dataset, and q k For smooth labeling, let δ represent the expected probability that the current image belongs to the k-th identity, where δ k,y is the impulse function, which is 1 when k = y and 0 otherwise, while ∈ is a parameter that controls the smoothness of the transition.
[0020] Step 5: During the testing phase, the test set images are fed into the trained image encoder to obtain the corresponding image features and perform pedestrian re-identification. For each image in the query of the test set, the most similar image from other cameras is found in the gallery, and the mAP and Rank-1 metrics are calculated.
[0021] The text encoder, which uses a Transformer as its backbone network, is used to train an image encoder, which uses a CNN or Transformer as its backbone network. Specifically, the backbone network of the image encoder is either ResNet-50 of the CNN network or ViT-B / 16 of the Transformer network, while the backbone network used by the text encoder is an 8-layer Transformer network.
[0022] The descriptive text containing learnable parameters is not shared between different identities; it serves as a fuzzy description for each identity to supplement the text descriptions not included in the re-identification task. For a dataset containing N identities, the dimension of the text features for all identities is N×C.
[0023] The process of sending the image to the image encoder, if using Transformer as the network backbone of the image encoder, can combine the image with camera information before sending it to the image encoder. The feature is that a set of learnable tag token parameters is pre-established for each camera in the dataset, the tag token of the corresponding camera is added to the classification tag [CLS] token, and then the image is sent to the image encoder.
[0024] The image encoder is trained by calculating the image-to-text cross-entropy loss function. In addition to the image-to-text cross-entropy loss function, it also calculates the identity loss function L commonly used for person re-identification. id and triplet loss function L tri The calculation method is as follows:
[0025]
[0026] L tri =max(d p -d n +α,0) (g)
[0027] Where p k Let d be the probability predicted by the network to belong to the k-th class. p and d n Let α represent the distance to the hardest positive sample and the hardest negative sample, where L is the distance to L. tri The set threshold.
[0028] This invention utilizes a text encoder with a Transformer backbone network to train an image encoder with a CNN or Transformer backbone network: The parameters of the text encoder and image encoder are fixed, and descriptive text containing learnable parameters is assigned to each identity. The image and corresponding descriptive text are then fed into the image encoder and text encoder. Contrast loss functions between text-to-image and image-to-text are calculated, and the learnable parameters in the descriptive text are trained. The text encoder and descriptive text are fixed, and textual features for each identity are generated and saved. The image is then fed into the image encoder. The cross-entropy loss function between image and text is calculated, and the image encoder is trained. Features for person re-identification are obtained through the image encoder. This invention is the first to apply the language-image pre-trained model CLIP to the ReID task, solving the problem that the text encoder in CLIP is difficult to utilize due to the lack of labels in the ReID task. The method is simple, making the results of finding the most similar image from other cameras in the gallery for each image in the query more accurate, thus improving the mAP and Rank-1 metrics for person re-identification. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the process of the present invention;
[0030] Figure 2 This is a flowchart of an embodiment of the present invention. Detailed Implementation
[0031] To more clearly illustrate the technical means, technical improvements, and beneficial effects of this invention, the invention will be described in detail below with reference to the accompanying drawings.
[0032] Example 1
[0033] See Figure 1-2 This invention utilizes a text encoder with a Transformer as the backbone network to learn a set of fuzzy descriptive text for each identity. Text features for all identities are obtained through the descriptive text and the text encoder. An image encoder with a CNN or Transformer as the backbone network is then trained. Finally, features for person re-identification are obtained using the image encoder. The specific steps include:
[0034] S1: For the MSMT17 dataset, which has 1041 identities in the training set and 3060 identities in the test set, during the training phase, a set of descriptive texts containing learnable parameters is assigned to each of the 1041 identities. The template for the descriptive text is "a photo of a[X]1[X]2[X]3...[X]". M person., where [X] m (m∈1,...M) represents the corresponding learnable token parameter, with M set to 5;
[0035] S2: Fix the parameters of the image encoder and text encoder, and send the image and corresponding descriptive text into the image encoder and text encoder;
[0036] S3: Calculate the contrast loss function L for image-to-text and text-to-image conversion. i2t and L t2i The learnable parameters in the descriptive text are trained using the following formula;
[0037] s(V i ,T i ) = V i `T i =g I (img i )`g T (text i (a)
[0038]
[0039]
[0040] imgi The [CLS] token is the classification label output by the image encoder for the i-th image, while the text... i This represents the output token [EOS] of the corresponding descriptive text after passing through the text encoder. I and g T To map the [CLS] token and [EOS] token to a linear layer in the same space, the final image feature V is obtained. i and text features T i ,s(V i ,T i ) is image feature V i and text features T i The similarity; B is the number of images in the current batch, a is the index in the current batch, and y is the index in the current batch. i It is the identity label of the i-th image, P(y i ) represents those belonging to y in the same batch i The set of indices of all images for this identity, |P(y i | indicates the number of images contained in this set;
[0041] S3: Fixed text encoder and descriptive text, generate and save text features for each identity, and feed the image into the image encoder;
[0042] S4: Calculate the image-to-text cross-entropy loss function L i2tce The commonly used identity loss function L for pedestrian re-identification id and triplet loss function L tri Train the image encoder;
[0043]
[0044] q k =(1-∈)δ k,y +∈ / N (e)
[0045]
[0046] L tri =max(d p -d n +α,0) (g)
[0047] Where N is the number of identities in the training set of the dataset; for the MSMT17 dataset, N = 1041, k is the index of all identities in the dataset, and q k For smooth labeling, let δ represent the expected probability that the current image belongs to the k-th identity, where δ k,y is the impulse function, which is 1 when k = y and 0 otherwise, while ∈ is a parameter controlling the smoothness of the transition; pk Let d be the probability predicted by the network to belong to the k-th class. p and d n Let α represent the distance to the hardest positive sample and the hardest negative sample, where L is the distance to L. tri The threshold value is set to 0.3;
[0048] S5: The test set images are processed by the trained image encoder to obtain corresponding image features. For each image in the test set query, the most similar image from other cameras is found in the gallery, and the mAP and Rank-1 metrics are calculated. Ultimately, when using a CNN as the backbone of the image encoder, the mAP was 63.0% and the Rank-1 was 84.4%. When using a Transformer as the backbone, the mAP was 75.8% and the Rank-1 was 89.7%.
[0049] This invention applies a pre-trained language image model to the pedestrian re-identification task. The method is simple and solves the problem of lacking text label descriptions in the process, thus improving accuracy. The above are merely preferred embodiments of this invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A pedestrian re-identification method based on a contrastive language-image pre-training model (CLIP), which trains an image encoder with a CNN or a Transformer as a backbone network by using a text encoder with a Transformer as a backbone network, and generates image features by using the image encoder to perform pedestrian re-identification, characterized in that, The method comprises the following specific steps: Step 1: For the MSMT17 dataset with thousands of identities, set a set of description texts with learnable parameters for each identity in its training set during the training phase, the template of the description text is a photo of a [X]1[X]2[X]3...[X] M person, where [X] m is the corresponding learnable token parameter, m e 1,... M; Step 2: fix the parameters of the image encoder and the text encoder, and input the image and the corresponding description text into the image encoder and the text encoder; Step 3: Compute the contrastive loss function L for image-to-text and text-to-image i2t and L t2i to train the learnable parameters in the description text, which is as follows; s(V i ,T i ) = V i · T i = g I (img i ) · g T (text i ) (a) where img i is the classification token [CLS] output by the image encoder for the ith image, and text i represents the output token [EOS] of the text encoder corresponding to the description text g I and g T is a linear layer mapping the [CLS] token and the [EOS] token to the same space, and finally obtaining the image feature V i and the text feature T i , s(V i , T i ) is the similarity of the image feature V i and the text feature T i ; B is the number of images contained in the current batch, a is the index in the current batch, y i is the identity label of the ith image, P(y i ) represents the index set of all images in the same batch belonging to y i , and |P(y i )| represents the number of images contained in the set. Step 3: fix the text encoder and the description text, generate and save the text features of each identity, and input the image into the image encoder; Step 4: calculate the cross-entropy loss function of the image to the text, and train the image encoder; q k = (1 - ∈)δ k,y + ∈ / N (e) where N is the number of identities contained in the training set of the dataset, k is the index of all identities in the dataset, q k is the smooth label, representing the expected probability that the current image belongs to the kth identity, where δ k,y is the impulse function, which is 1 when k = y and 0 otherwise, and ∈ is a parameter controlling the degree of smoothness. Step 5: in the test stage, input the test set image into the trained image encoder to obtain the corresponding image feature, and perform pedestrian re-identification: for each image in the query of the test set, find the most similar image under other cameras in the gallery, and calculate the mAP and Rank-1 indexes. 2.The pedestrian re-identification method of claim 1, wherein, The text encoder using the Transformer as the backbone network is used to train the image encoder using the CNN or the Transformer as the backbone network, and the backbone network of the image encoder is specifically selected as the ResNet-50 of the CNN network or the ViT-B / 16 of the Transformer network. 3.The pedestrian re-identification method of claim 1, wherein, The description text containing the learnable parameters does not share the learnable parameters between different identities; the description text is used as the fuzzy description of each identity to supplement the text description not contained in the re-identification task.