Fine-grained unsupervised cross-modal pedestrian re-identification method based on large model semantic driving

Abstract

The invention discloses a fine-grained unsupervised cross-modal pedestrian re-identification method based on large model semantic driving, and relates to a computer vision and mode identification technology. The method comprises the following steps: inputting an unlabeled visible light-infrared pedestrian data set, generating an image text description by using a vision-language model, and analyzing the image text description into a structured semantic attribute vector through a large language model; extracting visual features of the image and semantic query embedding corresponding to attributes, and generating fine-grained features of semantic alignment through a multi-head cross attention mechanism; fusing the visual similarity and the attribute similarity to generate a cross-modal pseudo tag; combining attribute-visual alignment loss and inter-attribute decoupling loss to optimize features; in the test stage, cross-modal matching retrieval is completed based on the optimized features. Through semantic analysis and enhancement, the problem that cross-modal feature alignment is insufficient in an unsupervised scene is solved, and experiments show that compared with a mainstream method, the model performance is improved, and the method can be applied to the fields of intelligent monitoring, cross-modal retrieval and the like.

Description

Technical Field

[0001] This invention relates to computer vision and pattern recognition technology, specifically to a fine-grained unsupervised cross-modal pedestrian re-identification method based on large model semantic driving, applicable to fields such as intelligent security, video surveillance, and public safety. Background Technology

[0002] In recent years, pedestrian re-identification has become an important research direction in the field of computer vision, showing broad application prospects in key areas such as intelligent surveillance, security and criminal investigation, and finding missing persons. However, traditional pedestrian re-identification methods based on visible light images mainly rely on visible light cameras, and their performance deteriorates sharply in low-light or nighttime environments, making it difficult to meet the practical needs of all-weather monitoring. To solve this problem, visible light-infrared pedestrian re-identification has emerged, aiming to match the same pedestrian in visible light images and infrared images across different camera networks.

[0003] Despite significant progress in visible-infrared pedestrian re-identification technology, numerous challenges remain in practical deployment. Existing supervised learning methods rely on a large number of manually labeled cross-modal identity tags, which are costly and difficult to scale. Therefore, unsupervised visible-infrared pedestrian re-identification has become a more practical but also more challenging research hotspot. Traditional unsupervised methods typically rely on purely visual cues, learning feature representations by generating pseudo-labels through clustering. However, this over-reliance on visual information has serious limitations: a huge modal gap exists between visible and infrared modalities, and the visual appearance of the same pedestrian varies greatly across different modalities (e.g., loss of color information). This results in the model learning features that are often an overall average of visual cues, lacking fine-grained, modality-invariant local details (e.g., clothing style, carried items), thus severely limiting the model's ability to discriminate between difficult samples with similar appearances. Therefore, introducing textual modality as auxiliary information has become a potential solution. Although visual appearance varies with modality, the high-level semantic attributes of pedestrians (e.g., "short-sleeved shirt," "backpack") are modally invariant. However, introducing textual information into the unsupervised visible-infrared pedestrian re-identification task is not easy. On the one hand, existing visual-language models typically generate unstructured text descriptions containing a lot of noise, making it difficult to extract accurate attribute information directly. On the other hand, how to effectively guide the model to focus on specific visual regions under unsupervised conditions, and solve the feature entanglement problem to achieve fine-grained cross-modal alignment, are current technological challenges that urgently need to be overcome. Summary of the Invention

[0004] The purpose of this invention is to address the problem of existing unsupervised person re-identification methods failing to overcome the modality gap due to reliance solely on visual cues, by guiding the decoupling and alignment of visual features through structured semantic attributes. This invention provides a fine-grained unsupervised cross-modal person re-identification method based on a large-model semantically driven approach. A large language model is introduced, using textual modality as a powerful source of semantic supervision. Modality-invariant semantic attributes are extracted to bridge the significant modality gap. By constructing a structured attribute space, attributes guide the model to learn decoupled, fine-grained local features, thereby significantly improving the accuracy of cross-modal retrieval even without labels.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0006] A fine-grained unsupervised cross-modal person re-identification method based on large model semantics includes the following steps:

[0007] A. Given an unsupervised pedestrian re-identification training dataset containing visible light and infrared images;

[0008] B. For each image in the dataset in step A, generate the original text description of the image using a visual-language model, and transform the original text description into a structured semantic attribute vector using a semantic parser based on a large language model.

[0009] C. Input the images from the dataset in step A and the structured semantic attribute vectors from step B into the model to extract image visual features and attribute semantic embeddings, respectively;

[0010] D. Integrate visual similarity and attribute similarity for cross-modal clustering matching to generate cross-modal pseudo-labels;

[0011] E. Utilize attribute embedding to guide the model to focus on the corresponding visual local region through a cross-attention mechanism, and extract fine-grained features;

[0012] F. Calculate the attribute-visual alignment loss and the attribute decoupling loss to optimize the fine-grained features generated in step E, ensuring the semantic consistency and orthogonality of the features;

[0013] G. Calculate the cross-modal distance based on the decoupled fine-grained features optimized in step F, measure the similarity between the query image and the base database image using cosine distance, and output the matching results in order of similarity to complete pedestrian re-identification.

[0014] In step A, the given unsupervised pedestrian re-identification training dataset contains visible light images and infrared images. This dataset contains an unlabeled set of visible light modal images and an infrared modal image set, denoted as... Any one of these images is denoted as Due to the lack of identity labels, unsupervised learning is needed to uncover cross-modal identity associations.

[0015] In step B, the text description of the image is generated using a visual-language model. This text description is then transformed into a structured semantic attribute vector using a semantic parser based on a large language model. Specifically, for each image in the dataset... Using a pre-trained vision-language model Generate original text description The process is defined as follows:

[0016]

[0017] Build a system prompt word and user prompts The prompt template P utilizes a large language model. Unstructured text Parsed into structured attribute vectors The process is defined as follows:

[0018]

[0019] in, It is a K-dimensional tuple, where each element corresponds to a specific semantic attribute category (such as "shirt color", "items carried", etc.).

[0020] In step C, the specific steps for inputting the images from the training dataset in step A and the structured semantic attribute vectors from step B into the model to extract image visual features and attribute semantic embeddings can be as follows: input the images... Input visual Transformer backbone network, extract contents A set of visual features for each category of tokens Simultaneously, using pre-trained BERT models Generate embedding vectors for attribute values, and based on the attribute vectors... Find the corresponding embeddings and assemble them into a semantic query matrix. The process is defined as follows:

[0021]

[0022] in, Indicates the first Learnable embedding matrix for each attribute category Indicates attribute index, Embedded in the generated semantic query.

[0023] In step D, the cross-modal clustering matching, which integrates visual similarity and attribute similarity, is performed to generate cross-modal pseudo-labels. Specifically, a semantically enhanced cross-modal clustering matching method is proposed, which integrates visual similarity and the attribute similarity obtained in step B to generate cross-modal pseudo-labels. First, clustering is performed in the visible light and infrared modalities respectively to obtain visual cluster centers. and semantic clustering center Visual cluster centers are calculated from the mean of features of all instances within a cluster, while semantic cluster centers are calculated from the mode (the most frequent value) of attributes of instances within a cluster, as defined below:

[0024]

[0025] In order to correlate visible light clusters and infrared clusters Calculate fusion similarity This similarity is determined by visual similarity. Similarity to attributes Weighted composition, defined as follows:

[0026]

[0027] in, Cosine similarity to the visual prototype. For consistency scores based on modality-invariant attributes (such as gender and clothing type), The balance coefficient is based on the fusion similarity. Perform bipartite graph matching to generate pseudo-labels.

[0028] In step E, the use of attribute embedding guides the model to focus on the corresponding visual local region through a cross-attention mechanism, extracting fine-grained features; specifically, the steps may include: semantic query... As a query, visual features As Key and Value; Context Vector is computed via a multi-head cross-attention mechanism. The calculation process for the j-th attention head is defined as follows:

[0029]

[0030] in, , , Let be the projection matrix. For scaling factors; final attribute-specific features We obtain the following through residual connections and layer normalization:

[0031]

[0032] Through this process, the model learns fine-grained local visual features aligned with specific semantic attributes (such as "short-sleeved shirt" or "backpack").

[0033] In step F, the design attribute-visual alignment loss and attribute decoupling loss optimize the fine-grained features generated in step E, ensuring the semantic consistency and orthogonality of the features. The specific steps can be as follows: First, the attribute-visual alignment loss... Using contrastive learning to bring visual features closer together Its corresponding semantic query The distance between them is defined as follows:

[0034]

[0035] in, It is the fine-grained visual feature corresponding to the k-th attribute in the i-th instance (generated by step E). It is the semantic query embedding corresponding to the k-th attribute in the i-th instance (generated by step C), and sim(•) is the similarity calculation function. It is the temperature coefficient. This represents the summation of the "visual features - semantic queries" pairs corresponding to all attributes of all instances within the current batch.

[0036] Secondly, introduce decoupling loss between attributes. To force different attribute features within the same instance to be orthogonal and avoid feature entanglement, for features... The loss is defined as follows:

[0037]

[0038] in, It is the fine-grained visual feature corresponding to the k-th attribute in the i-th instance. This indicates iterating through all attributes within the same instance except for the k-th attribute. is the fine-grained visual feature corresponding to the j-th attribute in the i-th instance, and sim(•) is the similarity calculation function. It is the temperature coefficient.

[0039] The final loss function combines the basic clustering comparison loss. The semantic loss mentioned above is used to update the network parameters through backpropagation.

[0040] This invention provides a semantically enhanced unsupervised visible light-infrared pedestrian re-identification system, comprising:

[0041] Data input module: used to receive unlabeled visible light-infrared pedestrian re-identification dataset, preprocess the dataset, and then input the standardized image data into the visual-language model and the visual backbone network respectively;

[0042] Semantic parsing module: includes a text description submodule and an attribute parsing submodule. The text description submodule is used to call a pre-trained visual-language model to generate the original natural language description of the image. The attribute parsing submodule is used to parse the original natural language description into a structured semantic attribute vector based on the prompt template and the large language model.

[0043] Feature extraction module: includes a visual feature extraction unit and a semantic query generation unit. The visual feature extraction unit is used to input the image into the visual backbone network to extract a set of visual features. The semantic query generation unit is used to call the pre-trained BERT model and assemble the semantic query matrix by combining the structured semantic attribute vector.

[0044] Clustering and matching module: includes intramodal clustering unit and cross-modal matching unit. The intramodal clustering unit is used to cluster the visual features of visible light and infrared modalities respectively and calculate the visual cluster center and semantic cluster center. The cross-modal matching unit is used to weightedly fuse visual similarity and attribute similarity and generate cross-modal pseudo-labels through bipartite graph matching.

[0045] Attribute attention module: used to generate local fine-grained features by taking the semantic query matrix as the Query and the visual feature set as the Key and Value through a multi-head cross-attention mechanism;

[0046] Loss optimization module: used to calculate attribute-visual alignment loss and attribute decoupling loss, and combine them with basic clustering contrast loss to obtain the total loss, and update network parameters through backpropagation;

[0047] Matching and retrieval module: used to extract local fine-grained features between the query image and the base database image during the testing phase, calculate the feature cosine distance, and output the re-identification results in order of similarity.

[0048] Compared with the prior art, the outstanding advantages and technical effects of the present invention are as follows:

[0049] 1. Existing technologies mostly rely on single visual features, which are easily affected by modal differences. This invention uses a large language model to parse visual-language text and extract modality-invariant structured attributes, avoiding the influence of modality-specific noise. Experiments show that the r1 metric of this invention on the RegDB dataset visible to infrared task (94.05) is about 5 percentage points higher than the existing best method (SDCL's 89.21), verifying the enhanced semantic robustness.

[0050] 2. Existing unsupervised methods rely on pure visual clustering for pseudo-labels, which is prone to errors such as "different clusters with the same attribute" and "same clusters with different attributes". This invention integrates visual similarity and attribute consistency scores to correct clustering bias. Experiments show that in the SYSU-MM01 full search task, the mAP (64.71) of this method is more than 12 percentage points higher than ADCA (52.73). The improvement in pseudo-label accuracy directly leads to a leap in re-identification performance.

[0051] 3. Existing technologies suffer from semantic entanglement, which can easily lead to confusion during cross-modal matching. This invention achieves fine-grained feature learning for semantic alignment through an attribute attention module and decoupling loss. Experiments show that in the LLCM infrared-to-visible light task, the method's mINP (48.44) fills the gap in fine-grained matching compared to SEEL (-), while r1 (46.22) is nearly 3 percentage points higher than SDCL (43.4), demonstrating the advantages of the fine-grained features of this invention.

[0052] 4. The method proposed in this invention can achieve efficient and accurate unsupervised cross-modal person re-identification in real-world scenarios, while demonstrating strong adaptability to complex cross-modal scenarios and fine-grained feature matching. Experimental results show that this method significantly outperforms existing mainstream methods in unsupervised cross-modal person re-identification tasks, especially in utilizing large language models to mine semantic attributes to bridge the modality gap, providing new directions and inspirations for the research and application of unsupervised cross-modal retrieval. Attached Figure Description

[0053] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0054] Figure 2 This is a flowchart of the semantic parsing module. Detailed Implementation

[0055] The method of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. This embodiment is implemented under the premise of the technical solution of the present invention, and provides implementation methods and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.

[0056] like Figure 1 This document illustrates a flowchart of the semantically enhanced unsupervised visible light-infrared pedestrian re-identification method of the present invention. The implementation of this invention includes the following steps:

[0057] A. Data Input: Given an unsupervised visible light-infrared pedestrian re-identification dataset, which contains an unlabeled set of visible light modal images and an infrared modal image set, denoted as . Any one of these images is denoted as Due to the lack of identity labels, unsupervised learning is needed to uncover cross-modal identity associations. The dataset will be... Input the visual-language model respectively And visual backbone network.

[0058] B. For the dataset in step A For each image in the dataset, a visual-language model is used to generate the original text description of the image. Furthermore, a semantic parser based on a large language model is used to transform text descriptions into structured semantic attribute vectors; firstly, for each image in the dataset... Using a pre-trained vision-language model Generate original text description The process is defined as follows:

[0059]

[0060] Next, construct a message containing system prompts. and user prompts The prompt template P utilizes a large language model. Describing unstructured raw text Parsed into structured attribute vectors The process is defined as follows:

[0061]

[0062] in, It is a K-dimensional tuple, where each element corresponds to a specific semantic attribute category (such as "shirt color", "items carried", etc.).

[0063] C. Transfer the dataset from step A Each image in the process, along with the attribute vector from step B, is input into the visual backbone network and text encoder, respectively, to extract visual features and semantic query embeddings; specifically:

[0064] Image Input the visual backbone network for feature extraction, extracting features including A set of visual features for each category of tokens Simultaneously, using pre-trained BERT models Generate embedding vectors for attribute values, and based on the attribute vectors... Find the corresponding embeddings and assemble them into a semantic query matrix. The process is defined as follows:

[0065]

[0066] in, Indicates the first Learnable embedding matrix for each attribute category Indicates attribute index, Embedded in the generated semantic query.

[0067] D. A semantically enhanced cross-modal clustering and matching method is proposed, which integrates visual similarity with the attribute similarity obtained in step B to generate cross-modal pseudo-labels; firstly, clustering is performed in the visible light and infrared modalities respectively to obtain visual cluster centers. and semantic clustering center Visual cluster centers are calculated from the mean of features of all instances within a cluster, while semantic cluster centers are calculated from the mode (the most frequent value) of attributes of instances within a cluster, as defined below:

[0068]

[0069] In order to correlate visible light clusters and infrared clusters Calculate fusion similarity The fusion similarity Visual similarity Similarity to attributes Weighted composition, defined as follows:

[0070]

[0071] in, Cosine similarity, representing visual prototypes This represents the consistency score based on modality-invariant attributes (such as gender and clothing type). The balance coefficient is based on the fusion similarity. Perform bipartite graph matching to generate pseudo-labels.

[0072] E. Construct an attribute attention module, embedding the semantic query extracted in step C as a guide, and using a cross-attention mechanism to focus the model's attention on the visual local region corresponding to the attribute; this module will integrate the semantic query... As a query, visual features As Key and Value; Context Vector is computed via a multi-head cross-attention mechanism. The calculation process for the j-th attention head is defined as follows:

[0073]

[0074] in, , , Let be the projection matrix. The scaling factor; the final local fine-grained features. We obtain the following through residual connections and layer normalization:

[0075]

[0076] Through this process, the model learns fine-grained local visual features aligned with specific semantic attributes (such as "short-sleeved shirt" or "backpack").

[0077] F. Design attribute-visual alignment loss and attribute decoupling loss for the local fine-grained features generated in step E. Optimization is performed to ensure the semantic consistency and orthogonality of features; firstly, attribute-visual alignment loss is used. Using contrastive learning to bring visual features closer together Its corresponding semantic query The distance between them is defined as follows:

[0078]

[0079] Secondly, introduce decoupling loss between attributes. To force different attribute features within the same instance to be orthogonal and avoid feature entanglement, for features... The loss is defined as follows:

[0080]

[0081] The final loss function combines the basic clustering contrastive loss. The semantic loss mentioned above is used to update the network parameters through backpropagation.

[0082] G. Calculate the cross-modal distance based on the decoupled features optimized in step F to achieve matching and retrieval of pedestrians in visible light and infrared light; during the testing phase, extract the visual features of the query image and the base database image, use cosine distance to measure similarity, and output the re-identification results according to the similarity ranking.

[0083] An embodiment of the semantically enhanced unsupervised visible light-infrared pedestrian re-identification system of the present invention includes:

[0084] Data input module: It not only supports input of standard datasets (such as SYSU-MM01, RegDB), but is also compatible with custom unlabeled visible light-infrared pedestrian image sets, automatically distinguishes modalities and completes data format unification (such as size normalization, channel alignment).

[0085] Semantic parsing module: integrates pre-trained visual-language models (such as BLIP-2) and large language models (such as GPT-3.5), and has a built-in pedestrian attribute template library (containing 10+ categories of attributes such as clothing color and gender). It can automatically adapt to pedestrian features in different datasets and output structured K-dimensional attribute vectors.

[0086] Feature extraction module: The visual backbone network adopts a replaceable Transformer architecture (such as ViT, SwinTransformer). The BERT encoder preloads pedestrian attribute-specific embedding weights to ensure that the semantic query matrix is ​​aligned with the dimensions of the visual features (e.g., both are 768-dimensional).

[0087] Clustering matching module: Built-in interfaces for multiple clustering algorithms such as DBSCAN and K-Means, automatically calculates visual / semantic cluster centers, and generates cross-modal pseudo-labels through bipartite graph matching algorithms (such as the Hungarian algorithm).

[0088] Attribute Attention Module: Implements a multi-head cross-attention mechanism (configurable number of heads), outputs attribute-specific features through residual connections and layer normalization, and supports dynamic allocation of feature weights for different attributes (such as upper garments and carried items).

[0089] Loss optimization module: It can flexibly adjust the weight coefficients of alignment loss and decoupling loss, integrates AdamW optimizer, supports batch training and gradient pruning, and ensures stable updates of network parameters.

[0090] Matching and retrieval module: During the testing phase, it supports batch queries and real-time retrieval. While outputting similarity ranking results, it can also visually display the consistency of the matching attributes (such as matching the color of the upper garment, matching gender), improving the interpretability of the results.

[0091] Figure 2 shows an example of the semantic parsing module of the present invention, as follows:

[0092] 1. Original text description generation: For the input pedestrian image (the pedestrian image on the left in Figure 2), the pre-trained vision-language model is called to generate an original text description containing details of the image appearance, namely, "A man with short black hair, wearing a short-sleeved round-neck gray T-shirt, a pair of knee-length gray shorts, and a pair of black slippers, and he is wearing a silver watch."

[0093] 2. Inputting parsing rules and architecture: The original text description described above, along with the "standardization rules" and "preset attribute architecture" (including categories such as gender, hairstyle, clothing, and accessories), are input into the semantic parser of the large language model;

[0094] 3. Structured semantic attribute output: Based on rules and architecture, the large language model extracts corresponding information from the original text and outputs structured semantic attributes, specifically: "Gender": "Male", "Hair length": "Short", "Hair color": "Black", "Upper body color": "Gray", "Upper body type": "T-shirt", "Lower body color": "Gray", "Lower body type": "Shorts", "Shoe color": "Black", "Shoe type": "Slippers", "Headwear": "None", "Items carried": "Watch".

[0095] This module provides a unified semantic basis for cross-modal feature alignment by transforming unstructured text into standardized attribute vectors.

[0096] Table 1 shows the comparative experimental results with state-of-the-art unsupervised methods on the SYSU-MM01 and RegDB datasets.

[0097] Table 1

[0098]

[0099] As shown in Table 1, in the full search and indoor search tasks of SYSU-MM01, the core indicators of our method (AGAR-Net), such as r1 and mAP (e.g., r1=68.37 and mAP=64.71 for full search), significantly outperform existing unsupervised methods such as OTLA and ADCA, demonstrating a clear advantage. In the visible to infrared and infrared to visible light tasks of RegDB, the r1 (e.g., r1=94.65 for visible to infrared) and mAP (e.g., 86.52) of our method are much higher than other methods, resulting in a significant improvement in cross-modal matching performance.

[0100] Table 2 shows the experimental results comparing the method with the state-of-the-art unsupervised methods on the LLCM dataset.

[0101] Table 2

[0102]

[0103] As can be seen from Table 2, in the infrared to visible light and visible light to infrared tasks of LLCM, the r1 (e.g., r1=46.22 for infrared to visible light), mAP (52.01) and mINP (48.44) of this method are all better than the comparison methods such as SDCL and SEEL, and it also has the leading performance on this dataset.

[0104] The innovation of this invention lies in:

[0105] a. Introduce a semantic parser based on a large language model to extract robust modality-invariant attributes from noisy text using the reasoning capabilities of the large language model; b. Propose a semantically enhanced cross-modal clustering matching method to correct clustering errors caused by relying solely on visual features by introducing semantic consistency of text modalities, thereby improving the quality of pseudo-labels; c. Design an attribute attention module enhanced by BERT and dual semantic losses (alignment loss and decoupling loss) to achieve fine-grained, semantically aligned, and decoupled feature learning under unsupervised conditions, effectively bridging the differences between visible light and infrared modalities.

[0106] This invention utilizes a semantic parser based on a large language model to transform noisy text descriptions generated by a vision-language model into structured modality-invariant attribute vectors. To improve the quality of pseudo-labels, a semantically enhanced cross-modal clustering matching method is proposed. By fusing visual similarity and attribute consistency scores, it effectively corrects clustering matching biases caused by relying solely on visual features. Finally, a BERT-enhanced attribute attention module is constructed, using attribute embeddings as queries. Through a cross-attention mechanism, the model is guided to focus on fine-grained visual regions. Combining attribute-visual alignment loss and attribute decoupling loss, the model is forced to learn decoupled and semantically aligned local features. This method significantly improves the feature matching ability between visible light and infrared modalities under unsupervised conditions, achieving performance superior to existing mainstream methods on multiple benchmark datasets.

[0107] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving, characterized in that... Includes the following steps: A. Given an unsupervised pedestrian re-identification training dataset containing visible light and infrared images; B. For each image in the dataset in step A, generate the original text description of the image using a visual-language model, and transform the original text description into a structured semantic attribute vector using a semantic parser based on a large language model. C. Input the images from the dataset in step A and the structured semantic attribute vectors from step B into the model to extract image visual features and attribute semantic embeddings, respectively; D. Integrate visual similarity and attribute similarity for cross-modal clustering matching to generate cross-modal pseudo-labels; E. Utilize attribute embedding to guide the model to focus on the corresponding visual local region through a cross-attention mechanism, and extract fine-grained features; F. Calculate the attribute-visual alignment loss and the attribute decoupling loss to optimize the fine-grained features generated in step E, ensuring the semantic consistency and orthogonality of the features; G. Calculate the cross-modal distance based on the decoupled fine-grained features optimized in step F, measure the similarity between the query image and the base database image using cosine distance, and output the matching results in order of similarity to complete pedestrian re-identification.

2. The fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving as described in claim 1, characterized in that... In step A, the given unsupervised pedestrian re-identification training dataset contains visible light images and infrared images. This dataset contains an unlabeled set of visible light modal images and an infrared modal image set, denoted as... Any one of these images is denoted as Due to the lack of identity labels, unsupervised learning is needed to uncover cross-modal identity associations.

3. The fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving as described in claim 1, characterized in that... In step B, the original text description of the image is generated using a visual-language model. Then, a semantic parser based on a large language model transforms the original text description into a structured semantic attribute vector. Specifically, for each image in the dataset... Using a pre-trained vision-language model Generate original text description The process is defined as follows: Build a system prompt word and user prompts The prompt template P utilizes a large language model. Unstructured text Parsed into structured attribute vectors The process is defined as follows: in, It is a K-dimensional tuple, where each element corresponds to a specific semantic attribute category (such as "shirt color", "carried items", etc.).

4. The fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving as described in claim 1, characterized in that... In step C, the specific steps for inputting the images from the training dataset in step A and the structured semantic attribute vectors from step B into the model to extract image visual features and attribute semantic embeddings are as follows: Input the images... Input visual Transformer backbone network, extract contents A set of visual features for each category of tokens Simultaneously, using pre-trained BERT models Generate embedding vectors for attribute values, and based on the attribute vectors... Find the corresponding embeddings and assemble them into a semantic query matrix. The process is defined as follows: in, Indicates the first Learnable embedding matrix for each attribute category Indicates attribute index, Embedded in the generated semantic query.

5. The fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving as described in claim 1, characterized in that... In step D, the cross-modal clustering matching, which integrates visual similarity and attribute similarity, is performed to generate cross-modal pseudo-labels. Specifically, a semantically enhanced cross-modal clustering matching method is proposed, which integrates visual similarity and the attribute similarity obtained in step B to generate cross-modal pseudo-labels. First, clustering is performed in the visible light and infrared modalities respectively to obtain visual cluster centers. and semantic clustering center Visual cluster centers are calculated from the mean of features of all instances within a cluster, while semantic cluster centers are calculated from the mode (the most frequent value) of attributes of instances within a cluster, as defined below: In order to correlate visible light clusters and infrared clusters Calculate fusion similarity This similarity is determined by visual similarity. Similarity to attributes Weighted composition, defined as follows: in, Cosine similarity to the visual prototype. For consistency scores based on modality-invariant attributes (such as gender and clothing type), The balance coefficient is based on the fusion similarity. Perform bipartite graph matching to generate pseudo-labels.

6. The fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving as described in claim 1, characterized in that... In step E, the attribute embedding is used to guide the model to focus on the corresponding visual local region through a cross-attention mechanism to extract fine-grained features; The specific steps are as follows: semantic query As a query, visual features As Key and Value; Context Vector is computed via a multi-head cross-attention mechanism. The calculation process for the j-th attention head is defined as follows: in, , , Let be the projection matrix. For scaling factors; final attribute-specific features We obtain the following through residual connections and layer normalization: Through this process, the model learns fine-grained local visual features aligned with specific semantic attributes (such as "short-sleeved shirt" or "backpack").

7. The fine-grained unsupervised cross-modal person re-identification method based on large model semantic driving as described in claim 1, characterized in that... In step F, the specific steps for calculating the attribute-visual alignment loss and the attribute decoupling loss to optimize the fine-grained features generated in step E and ensure the semantic consistency and orthogonality of the features are as follows: First, the attribute-visual alignment loss... Using contrastive learning to bring visual features closer together Its corresponding semantic query The distance between them is defined as follows: in, It is the fine-grained visual feature corresponding to the k-th attribute in the i-th instance (generated by step E). It is the semantic query embedding corresponding to the k-th attribute in the i-th instance (generated by step C), and sim(•) is the similarity calculation function. It is the temperature coefficient. This represents the summation of "visual feature-semantic query" pairs corresponding to all attributes of all instances within the current batch; Secondly, introduce decoupling loss between attributes. To force different attribute features within the same instance to be orthogonal and avoid feature entanglement, for features... The loss is defined as follows: in, It is the fine-grained visual feature corresponding to the k-th attribute in the i-th instance. This indicates iterating through all attributes within the same instance except for the k-th attribute. is the fine-grained visual feature corresponding to the j-th attribute in the i-th instance, and sim(•) is the similarity calculation function. It is the temperature coefficient; The final loss function combines the basic clustering comparison loss. The semantic loss mentioned above is used to update the network parameters through backpropagation.

8. A semantically enhanced unsupervised visible light-infrared pedestrian re-identification system, characterized in that... include: Data input module: used to receive unlabeled visible light-infrared pedestrian re-identification dataset, preprocess the dataset, and then input the standardized image data into the visual-language model and the visual backbone network respectively; Semantic parsing module: includes a text description submodule and an attribute parsing submodule. The text description submodule is used to call a pre-trained visual-language model to generate the original natural language description of the image. The attribute parsing submodule is used to parse the original natural language description into a structured semantic attribute vector based on the prompt template and the large language model. Feature extraction module: includes a visual feature extraction unit and a semantic query generation unit. The visual feature extraction unit is used to input the image into the visual backbone network to extract a set of visual features. The semantic query generation unit is used to call the pre-trained BERT model and assemble the semantic query matrix by combining the structured semantic attribute vector. Clustering and matching module: includes intramodal clustering unit and cross-modal matching unit. The intramodal clustering unit is used to cluster the visual features of visible light and infrared modalities respectively and calculate the visual cluster center and semantic cluster center. The cross-modal matching unit is used to weightedly fuse visual similarity and attribute similarity and generate cross-modal pseudo-labels through bipartite graph matching. Attribute Attention Module: Used to generate local fine-grained features by taking the semantic query matrix as the Query and the visual feature set as the Key and Value through a multi-head cross-attention mechanism; Loss optimization module: used to calculate attribute-visual alignment loss and attribute decoupling loss, and combine them with basic clustering contrast loss to obtain the total loss, and update network parameters through backpropagation; Matching and retrieval module: used to extract local fine-grained features between the query image and the base database image during the testing phase, calculate the feature cosine distance, and output the re-identification results in order of similarity.

Images