A text-guided generalizable pedestrian re-identification method for clothes

Abstract

The application discloses a clothing generalizable pedestrian re-identification method based on text guidance, in a text domain, clothing information and clothing invariant features are described through independent phrases, avoiding their strong coupling in vision, in this way, the application can extract pure clothing invariant features from pedestrian images, and use the features to guide a visual model to perform clothing-independent pedestrian re-identification, and the application decouples clothing-related features and clothing-independent features through a text domain, and solves the feature confusion problem caused by clothing changes in traditional methods.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically relating to a text-guided, generalizable pedestrian re-identification method based on clothing. Background Technology

[0002] Person re-identification (ReID) aims to match the same pedestrians in different cameras. Existing person re-identification methods perform well when clothing is consistent, but their performance degrades significantly when clothing changes. Clothing changes can lead to significant differences in pedestrian appearance, thus affecting recognition accuracy. To address this issue, the Clothing Change Person Re-identification (CC-ReID) task has been proposed, aiming to develop a person re-identification model robust to clothing changes.

[0003] Existing CC-ReID methods are mainly divided into two categories: one category utilizes additional biometric information (such as gait and body shape) to guide the model in capturing clothing-invariant features; the other category is based on feature decoupling, attempting to directly separate clothing-invariant features from pedestrian images. However, these methods face challenges in handling the strong coupling between clothing and clothing-invariant features (such as body shape and posture), making it difficult to extract purely clothing-invariant features. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a text-guided, clothing-generalized pedestrian re-identification method. In the text domain, clothing information and clothing-invariant features are described using independent phrases, avoiding strong visual coupling between them. In this way, this invention can extract purely clothing-invariant features from pedestrian images and use these features to guide a visual model for clothing-independent pedestrian re-identification. This invention decouples clothing-related features from clothing-invariant features through the text domain, solving the feature confusion problem caused by clothing variations in traditional methods.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] Step 1: Implicit text hint generation;

[0007] Pedestrian features are decoupled into two parts: variable information related to clothing and invariant information unrelated to clothing. Two types of trainable text cues are assigned to each pedestrian to learn these two types of information respectively. In the CC-ReID task of pedestrian re-identification with clothing changes, each pedestrian has multiple sets of clothing. Multiple sets of clothing-related text cues are assigned to different clothing for each pedestrian, and these cues are aligned with the corresponding clothing images through image-text matching loss to learn clothing information. At the same time, a set of clothing-invariant cues is trained for each pedestrian. By combining these cues with cues for different clothing and aligning them with images of the same pedestrian in different clothing, clothing-invariant information is implicitly learned.

[0008] Step 2: Explicit description generation of the multimodal large language model (MLLM);

[0009] We use a multimodal large language model (MLLM) to explicitly generate clothing-independent feature descriptions for each pedestrian. Specifically, we use MLLM to generate clothing-independent feature descriptions for each pedestrian through visual question answering; these descriptions are then used to extract text features through a CLIP text encoder, serving as a guide for explicit clothing-invariant feature generation.

[0010] Step 3: Dynamic fusion;

[0011] After obtaining implicit and explicit clothing-invariant text cues, these cues are fused to generate robust clothing-invariant feature guidance. Using the attention mechanism in the Transformer architecture, the implicit cues are used as an index to select valuable information from the explicit cues for fusion, thereby generating clothing-invariant feature guidance.

[0012] Preferably, step 1 specifically comprises:

[0013] When generating clothing-related prompts, each pedestrian has C different types of clothing. First, a set of clothing-related text prompts is assigned to each type of clothing. Each Y is a learnable vector; these cues generate text descriptions using predefined templates and are fed into a frozen CLIP model to extract text features. By aligning text features with corresponding pedestrian images of clothing using image-text matching loss, clothing information is learned. This loss can be expressed as:

[0014]

[0015] in, This represents the image-to-text matching loss related to clothing, used to align the features of pedestrian images with the textual cue features of the corresponding clothing. This represents the text-to-image matching loss related to clothing, used to align the textual cue features of clothing with the corresponding pedestrian image features;

[0016] Assign each pedestrian a unique set of implicit text cues unrelated to clothing. These prompts are combined with prompts for different clothing items to generate templates, and text features are extracted. Furthermore, an image-text matching loss is used to align text features with all images of the pedestrian, ensuring that clothing-independent cues learn invariant information about the current pedestrian. This loss is expressed as:

[0017]

[0018] in, The image-to-text matching loss, which is independent of clothing, is used to align the features of pedestrian images with the features of clothing-independent text cues. This represents the text-to-image matching loss unrelated to clothing, used to align clothing-independent text cue features with corresponding pedestrian image features;

[0019] Preferably, step 2 specifically comprises:

[0020] Knowledge is extracted from an MLLM model using a natural language question formatted as: "Describe the clothing-independent features of the person in this photo, but do not describe the clothes he / she is wearing." By inputting the selected image and question into the MLLM model, detailed clothing-independent feature descriptions for each pedestrian are obtained. These descriptions are then used to extract textual features via a CLIP text encoder. Where N is the sequence length, d T It is the channel dimension; text features As explicit guidance, it is then integrated with implicit text prompts.

[0021] Preferably, step 3 specifically comprises:

[0022] For the p-th pedestrian, the implicit clothing-invariant text cue is constructed as "a [X1]". p [X2] p ,...,[X n ] p The prompt, "a photo of a person," outputs implicit text features via the CLIP text encoder. At the same time, the corresponding explicit description is also input into CLIP to obtain explicit text features. Subsequently, linear projection layers are applied to both the implicit and explicit text features, as shown below:

[0023]

[0024] Among them, FC Q FC K FC VIt is a fully connected layer used to embed implicit and explicit text hint features; Q, K, and V represent Query, Key, and Value, respectively, and then attention is calculated:

[0025]

[0026] in, d is the clothing-independent text guidance for the p-th pedestrian after fusion, and d is the channel dimension of these features;

[0027] Using equations (4) and (5), the implicit hints are used as an index to filter and integrate the effective and compatible parts from the explicit hints, thereby generating robust clothing invariant feature guidelines.

[0028] Preferably, the training strategy for the text-guided clothing-based generalizable pedestrian re-identification method is as follows:

[0029] The training process is divided into two phases; in the first phase, the CLIP model is fixed, and only the implicit text hints are optimized using image-text matching loss. and The overall loss during this training phase is expressed as:

[0030]

[0031] In the second stage, the implicit text cue and CLIP text encoder are fixed, while the dynamic fusion module and CLIP visual encoder are optimized to match clothing-invariant text guidance; the overall loss for this stage is expressed as:

[0032]

[0033] Among them, L CE and These are cross-entropy loss and triplet loss, respectively. The image-to-text matching loss is used to guide the CLIP image encoder to extract clothing-independent features from the image, making them clothing-independent text features f. T Alignment;

[0034] During the testing phase, the text encoder and text prompt generator were discarded, and only the visual encoder was used to extract the visual features of the input image for inference. The images in the image library were reordered by calculating the Euclidean distance between the query image and all images in the image library, and the image most similar to the query image was returned as the recognition result.

[0035] A computer program that causes a computer to execute the above-described clothing-generalized pedestrian re-identification method.

[0036] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to enable the electronic device to perform the above-described clothing-generalized pedestrian re-identification method.

[0037] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described clothing-generalizable pedestrian re-identification method.

[0038] A chip includes a processor for calling and running a computer program from a memory, causing a device equipped with the chip to perform the aforementioned clothing-generalizable pedestrian re-identification method.

[0039] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described clothing-generalizable pedestrian re-identification method.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention solves the feature confusion problem caused by clothing changes in traditional methods by decoupling clothing-related features (such as clothing style) from clothing-independent features (such as body shape, posture, and gender) in the text domain. Combining implicit textual prompts with explicit descriptions generated by a multimodal large language model, the dynamic fusion module uses an attention mechanism to filter key information and generate robust clothing-invariant feature guidance. Validation on mainstream datasets shows that the model significantly outperforms existing methods in recognition accuracy under clothing change scenarios, and maintains high accuracy even under complex lighting and occlusion conditions, demonstrating strong potential for engineering applications. Attached Figure Description

[0042] Figure 1 This is a framework diagram of the method of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] The core idea of ​​this invention is to decouple clothing and clothing-invariant features by using the text domain as a bridge. In the text domain, clothing information and clothing-invariant features can be described using independent phrases, avoiding their strong visual coupling. In this way, this invention can extract purely clothing-invariant features from pedestrian images and use these features to guide visual models in clothing-independent pedestrian re-identification.

[0045] To address the negative impact of clothing changes on model performance in pedestrian re-identification tasks and to better extract clothing-invariant features, this invention proposes a text-guided, clothing-generalizable pedestrian re-identification method (Tex-CG). Its structure is as follows: Figure 1 As shown, the main work of this invention includes three parts: implicit text prompt generation, explicit description generation of multimodal large language model (MLLM), and dynamic fusion.

[0046] 1. Implicit text hint generation module;

[0047] It is generally believed that pedestrian features can be decoupled into two parts: variable information related to clothing and invariant information unrelated to clothing. Based on this view, two types of trainable text cues are assigned to each pedestrian to learn these two types of information respectively. Specifically, in the CC-ReID task, each pedestrian has multiple outfits. Multiple sets of clothing-related text cues are assigned to different outfits for each pedestrian, and these cues are aligned with the corresponding outfit images using image-text matching loss to learn clothing information. At the same time, a set of clothing-invariant cues is trained for each pedestrian, and by combining it with cues for different outfits and aligning it with images of the same pedestrian in different outfits, clothing-invariant information is implicitly learned.

[0048] Specifically, when generating clothing-related prompts, each pedestrian has C different types of clothing. First, a set of clothing-related text prompts is assigned to each type of clothing. Each Y is a learnable vector. These prompts generate text descriptions using predefined templates, such as: "A picture of someone wearing [Y1], [Y2], ..., [Y]". m [Photo of the clothes.] And input the frozen CLIP model to extract text features. By aligning text features with corresponding pedestrian images of clothing using image-text matching loss, clothing information is learned. This loss can be expressed as:

[0049]

[0050] In addition to clothing-related text prompts, each pedestrian is also assigned a unique set of clothing-unrelated text prompts. These prompts are combined with prompts for different clothing items to generate a template such as "a picture of someone wearing [Y1], [Y2], ..., [Y]". m [X1] of the clothes p [X2] p ,...,[X n ] p "Photos of people" to extract text features Furthermore, a text-image matching loss is used to align text features with all images of the pedestrian, ensuring that clothing-independent cues learn invariant information about the current pedestrian. This loss can be expressed as:

[0051]

[0052] 2. Explicit description generation module for Multimodal Large Language Model (MLLM);

[0053] To generate more comprehensive and interpretable clothing-independent feature descriptions, this invention utilizes a multimodal large language model (MLLM) to explicitly generate clothing-independent feature descriptions for each pedestrian. Specifically, it uses MLLM to generate clothing-independent feature descriptions for each pedestrian through visual question answering. These descriptions are then processed by a CLIP text encoder to extract text features, which serve as guidelines for explicit clothing-invariant features.

[0054] Specifically, this invention uses a natural language question to acquire knowledge from an MLLM model. The question format is: "Please describe the clothing-independent features of the person in this photo (such as hair, skin color, posture, gender, age, backpack, etc.), but do not describe the clothes he / she is wearing." By inputting the selected image and the question into the MLLM model, this invention obtains detailed clothing-independent feature descriptions for each pedestrian. These descriptions are used to extract text features using a CLIP text encoder. Where N is the sequence length, d T These are channel dimensions. These text features As explicit guidance, it is then integrated with implicit text prompts.

[0055] 3. Dynamic fusion module;

[0056] After obtaining implicit and explicit clothing-invariant text cues, this invention designs a dynamic fusion module to merge these cues and generate robust clothing-invariant feature guidance. This module utilizes the attention mechanism in the Transformer architecture, using implicit cues as indices to select valuable information from explicit cues for fusion, thereby generating more effective clothing-invariant feature guidance.

[0057] Specifically, for the p-th pedestrian, the implicit clothing-invariant text cue is constructed as "a [X1]". p [X2] p ,...,[X n ] p The prompt, "a photo of a person," is an implicit cue feature output by the CLIP text encoder. At the same time, the corresponding explicit descriptions are also input into CLIP to obtain text features. Subsequently, the present invention applies linear projection layers to the implicit and explicit text features respectively, which can be represented as:

[0058]

[0059] Among them, FC Q FC K FC V This is a fully connected layer used to embed implicit and explicit textual cue features; Q, K, and V represent the query, key, and value, respectively. Then, this invention uses these vectors to compute attention:

[0060]

[0061] in, Let be the fused clothing-independent text guidance for the p-th pedestrian, and d be the channel dimension of these features. In this way, the present invention can use implicit cues as an index to filter and integrate effective and compatible parts from explicit cues, thereby generating robust clothing-invariant feature guidance.

[0062] 4. Training strategies;

[0063] When focusing on the training strategy, the training process of this invention is divided into two phases. In the first phase, such as... Figure 1 As shown, the CLIP model is fixed, and only the implicit text hints are optimized, using image-text matching loss. and Training is then performed. The overall loss during this phase can be expressed as:

[0064]

[0065] In the second stage, the implicit text cue and CLIP text encoder are fixed, while the dynamic fusion module and CLIP visual encoder are optimized to match clothing-invariant text guidance. The overall loss for this stage can be expressed as:

[0066]

[0067] Among them, L CE and These are cross-entropy loss and triplet loss, respectively.

[0068] During the testing phase, the text encoder and text prompt generator were discarded, and only the visual encoder was used to extract visual features from the input image for inference. By calculating the Euclidean distance between the query image and all images in the image database, this invention reorders the images in the image database and returns the image most similar to the query image as the recognition result.

[0069] Example:

[0070] This invention provides a text-guided, clothing-based, generalizable pedestrian re-identification method, the specific implementation process of which is as follows.

[0071] 1. Dataset selection;

[0072] Regarding dataset configuration, this invention uses two mainstream clothing-change person re-identification (CC-ReID) benchmark datasets for evaluation: LTCC (Long-Term Cloth-Changing) and PRCC (Person Re-Identification under Cloth Change). The LTCC dataset contains 17,119 images involving 152 pedestrian identities, of which 91 pedestrians exhibit clothing changes, totaling 416 different outfits. The training set contains 77 identities, with 46 pedestrians exhibiting clothing changes and 31 pedestrians wearing the same outfit. The test set contains 45 pedestrians with clothing changes and 30 pedestrians with the same outfit. The PRCC dataset contains 33,698 images involving 221 pedestrian identities. Each pedestrian wears the same outfit in cameras A and B, but different outfits in camera C. The training set contains 150 identities, and the test set contains 71 identities.

[0073] 2. Implementation details;

[0074] In the TeX-CG model, CLIP-ReID based on ViT-B / 16 was used as the backbone model. All pedestrian images were resized to 256×128. In the first training phase, only the learnable text cues were optimized, while the parameters of the CLIP model remained unchanged. The model was trained for 120 epochs using the Adam optimizer with an initial learning rate of 3.5×10⁻⁶. -4 The data was decayed using cosine scheduling. The batch size was set to 64, and no data augmentation methods were used. In the second training phase, MiniCPM-Llama3-V-2 5-int4 was used as a multimodal large language model (MLLM) to explicitly generate clothing-independent descriptions. Then, with the implicit text hints and CLIP text encoder frozen, the Adam optimizer was used to optimize the CLIP visual encoder and dynamic fusion module for 60 epochs. The model was warmed up for 10 epochs with a linearly increasing learning rate, starting from 1×10⁻⁶. -7 Increased to 1×10 -6 Subsequently, the learning rate was decayed by 0.1x at epochs 30 and 50. The batch size was set to 64, and the training images were augmented by random horizontal flipping, padding, cropping, and erasing.

[0075] 3. Implementation environment;

[0076] This invention uses an NVIDIA GeForce RTX 3090Ti graphics processor for training and the NVIDIA CUDA platform for accelerated training. The programming language is Python 3.8, and the deep learning framework is PyTorch 1.11.0. Other dependent libraries include NumPy 1.19.2 and Pillow 9.0.2.

[0077] 4. Model application;

[0078] During the model application phase, this invention does not perform data augmentation on the input data; it only samples the input data to a 256×128 image size. The model parameters are fixed, and the network parameters are no longer updated using the stochastic gradient descent algorithm; the network is only used as an image feature extractor.

[0079] In the actual inference process, this invention uses the CLIP-ReID model, i.e., the output features of the TeX-CG model, which deploys a dynamic fusion module. For the query sample, after feature inference by the model, this invention extracts the features of all images in the image database and stores them as a feature sequence. Subsequently, this invention calculates the Euclidean distance between the query features and the features of all images in the image database:

[0080]

[0081] Where n is the dimension of the feature. It is the feature vector of the i-th query sample. Let be the feature vector of the j-th image in the image database. By calculating the Euclidean distance, this invention obtains the distance sequence D = {d1, d2, ..., d...}. N} where N is the number of images in the image library. Subsequently, this invention reorders the images in the image library based on distance and returns the L images most similar to the query sample. If any of these images contains an image with the same ID as the query sample, the query is considered successful.

Claims

1. A text-guided, clothing-based, generalizable pedestrian re-identification method, characterized in that, Includes the following steps: Step 1: Implicit text hint generation; Pedestrian features are decoupled into two parts: variable information related to clothing and invariant information unrelated to clothing. Two types of trainable text cues are assigned to each pedestrian to learn these two types of information respectively. In the CC-ReID task of pedestrian re-identification with clothing changes, each pedestrian has multiple sets of clothing. Multiple sets of clothing-related text cues are assigned to different clothing for each pedestrian, and these cues are aligned with the corresponding clothing images through image-text matching loss to learn clothing information. At the same time, a set of clothing-invariant cues is trained for each pedestrian. By combining these cues with cues for different clothing and aligning them with images of the same pedestrian in different clothing, clothing-invariant information is implicitly learned. Step 2: Explicit description generation of the multimodal large language model (MLLM); The clothing-independent feature descriptions for each pedestrian are explicitly generated using a multimodal large language model (MLLM). Specifically, the clothing-independent feature descriptions for each pedestrian are generated using MLLM through visual question answering. These clothing-independent feature descriptions are then used to extract text features through a CLIP text encoder, which serve as a guide for explicit clothing-invariant features. Step 3: Dynamic fusion; After obtaining implicit and explicit clothing-invariant textual cues, these cues are merged to generate robust clothing-invariant feature guidelines. By leveraging the attention mechanism in the Transformer architecture, implicit cues are used as indices to select valuable information from explicit cues for fusion, thereby generating guidance on clothing-invariant features.

2. The text-guided, clothing-based, generalizable pedestrian re-identification method according to claim 1, characterized in that, Step 1 specifically involves: When generating clothing-related prompts, each pedestrian has C different types of clothing. First, a set of clothing-related text prompts is assigned to each type of clothing. Each of them It is a learnable vector; these prompts generate text descriptions using predefined templates and are fed into a frozen CLIP model to extract text features. The image-text matching loss is used to align text features with corresponding pedestrian images of clothing to learn clothing information. This loss can be expressed as: = + , (1) in, This represents the image-to-text matching loss related to clothing, used to align the features of pedestrian images with the textual cue features of the corresponding clothing. This represents the text-to-image matching loss related to clothing, used to align the textual cue features of clothing with the corresponding pedestrian image features; Assign each pedestrian a unique set of implicit text cues unrelated to clothing. These prompts are combined with prompts for different clothing items to generate templates and extract text features. Furthermore, an image-text matching loss is used to align text features with all images of the pedestrian, ensuring that clothing-independent cues learn invariant information about the current pedestrian. This loss is expressed as: = + (2) in, The image-to-text matching loss, which is independent of clothing, is used to align the features of pedestrian images with the features of clothing-independent text cues. This represents the clothing-independent text-to-image matching loss, used to align clothing-independent text cue features with corresponding pedestrian image features.

3. The text-guided, clothing-based, generalizable pedestrian re-identification method according to claim 2, characterized in that, Step 2 specifically involves: Knowledge is extracted from the MLLM using a natural language question formatted as: "Please describe the clothing-independent features of the person in this photo, but do not describe the clothes he / she is wearing"; by inputting the selected image and the question into the MLLM model, a detailed clothing-independent feature description of each pedestrian is obtained. These descriptions extract text features using the CLIP text encoder. Where N is the sequence length, It is the latitude of the passage; Text features As explicit guidance, it is then integrated with implicit text prompts.

4. The text-guided, clothing-based, generalizable pedestrian re-identification method according to claim 3, characterized in that, Step 3 specifically involves: For the A pedestrian, with invisible clothing and unchanging text prompts, is constructed as "a..." The prompt, "a photo of a person," outputs implicit text features via the CLIP text encoder. Simultaneously, the corresponding explicit description is also input into CLIP to obtain explicit text features. Subsequently, linear projection layers are applied to the implicit and explicit text features respectively, as shown below: , , (3) in, , , It is a fully connected layer used to embed implicit and explicit text hint features; , , Let Query, Key, and Value represent the query, respectively, and then calculate the attention: (4) (5) in, It is the first Text-independent guidance on clothing for individual pedestrians. These are the channel dimensions of these features; Using equations (4) and (5), the effective and compatible parts are filtered and integrated from the explicit prompts by using implicit prompts as an index, thereby generating robust clothing invariant feature guidelines.

5. The text-guided, clothing-based, generalizable pedestrian re-identification method according to claim 4, characterized in that, The training strategy for the text-guided, clothing-generalized pedestrian re-identification method is as follows: The training process is divided into two phases; in the first phase, the CLIP model is fixed, and only the implicit text hints are optimized using image-text matching loss. and The overall loss during this training phase is expressed as: (6) In the second stage, the implicit text cue and CLIP text encoder are fixed, while the dynamic fusion module and CLIP visual encoder are optimized to match clothing-invariant text guidance; the overall loss for this stage is expressed as: (7) in, and These are cross-entropy loss and triplet loss, respectively. This represents the image-to-text matching loss, used to guide the CLIP image encoder in extracting clothing-independent features from the image, making them clothing-independent text features. Alignment; During the testing phase, the text encoder and text prompt generator were discarded, and only the visual encoder was used to extract the visual features of the input image for inference. The images in the image library were reordered by calculating the Euclidean distance between the query image and all images in the image library, and the image most similar to the query image was returned as the recognition result.

6. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

8. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 5.

Images