Text-to-image pedestrian re-identification method based on dynamic memory enhancement

Abstract

The invention discloses a text-to-image pedestrian re-identification method based on dynamic memory enhancement, and relates to the technical field of computer vision. A dynamic memory enhancement module is designed, a prototype memory library capable of being iteratively updated is constructed based on a modern Hupfield network, multi-modal co-occurrence attribute features are dynamically stored and retrieved, sample specific noise is effectively inhibited, and a discriminative cross-modal covariance mode is reserved. Text-to-image pedestrian re-identification is reconstructed into a global-local collaborative alignment task, through an attribute fine-grained alignment module, a corresponding relation between a visual area and text features is established by utilizing single-instance contrast learning, and local fine-grained semantic alignment is realized through adaptive feature subspace reconstruction without damaging global consistency. According to the method, multi-loss function optimization strategies such as InfoLOOB loss and SDM loss are fused, the stability and discrimination of cross-modal matching are enhanced, the core problems of large intra-class variation and high inter-class similarity are effectively relieved, and more accurate and efficient text-to-image pedestrian re-identification is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a text-to-image person re-identification method based on dynamic memory enhancement. Background Technology

[0002] The core task of Text-to-Image Person Re-identification (TIReID) is to retrieve the corresponding identity image from a large-scale image database based on the natural language description of the pedestrian. This task has practical value in scenarios such as intelligent system monitoring and smart retail. It faces two major challenges: large intra-class variability, that is, the same pedestrian may have significant visual differences due to factors such as posture and lighting; and high inter-class similarity, that is, different pedestrians with similar appearance features are easily confused.

[0003] Existing TIReID methods have significant limitations: global matching methods focus on global semantics and ignore fine-grained attributes that distinguish similar pedestrians; local matching methods, while focusing on region-text correspondence, have high computational overhead and are prone to introducing noise during inference; methods based on vision-language pre-trained models (such as CLIP) suffer from selective attention bias, focusing on a few significant features while missing key information. None of the above three types of methods balance fine-grained alignment accuracy, computational efficiency, and robustness, making it difficult to meet practical retrieval needs. Summary of the Invention

[0004] The purpose of this invention is to address the problems of low cross-modal alignment accuracy caused by large intra-class variability and high inter-class similarity in text-to-image person re-identification, and to provide a text-to-image person re-identification method based on dynamic memory enhancement that balances fine-grained matching accuracy, computational efficiency and robustness. This invention is implemented based on a dynamic memory enhancement cross-modal retrieval framework.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0006] A text-to-image person re-identification method based on dynamic memory enhancement is implemented using a dynamic memory enhancement cross-modal retrieval framework. This framework sequentially includes a dynamic memory enhancement module, an attribute fine-grained alignment module, and a multi-loss function optimization module. These three modules work together to achieve a complete link from robust global feature enhancement to accurate local semantic alignment to cross-modal matching optimization. The method includes the following steps:

[0007] 1) Through the dynamic memory enhancement module, the pre-trained CLIP model is used to extract the global visual embedding of the pedestrian image and the global text embedding of the text description for the given image-text pedestrian sample pair. Based on the modern Hopfield network, the global embeddings in the training batch are aggregated into the visual memory matrix and the text memory matrix to build a dynamic memory library that is updated in real time with the training batch. Through weighted aggregation, the dynamic retrieval of multimodal co-occurrence attribute features is realized, suppressing sample-specific noise such as occlusion and illumination changes and preserving the discriminative cross-modal covariance pattern.

[0008] 2) Through the attribute fine-grained alignment module, adjective-noun attribute phrases are extracted from the text description using a large language model combined with structured prompts to generate local text embeddings. The similarity between these embeddings and the local visual embeddings of pedestrian images extracted from the CLIP model is calculated and sparsified. The visual-text fusion embeddings are obtained by aggregating the generated alignment weights. The local semantic alignment is optimized by combining FINE loss without destroying global consistency.

[0009] 3) Through the multi-loss function optimization module, InfoLOOB loss, semantic distribution matching (SDM) loss, identity (ID) loss and FINE loss are integrated to achieve end-to-end training, optimize the stability and discriminativeness of cross-modal matching, retrieve pedestrian images based on text description, and complete text-to-image pedestrian re-identification.

[0010] In step 1), the global visual embedding is extracted using the CLIP B / 16 architecture. The pedestrian image is preprocessed into non-overlapping image blocks and then linearly projected and encoded by Transformer. The [CLS] token is then used as the global visual embedding.

[0011] The global text embedding is extracted using the CLIP Xformer architecture. The pedestrian natural language description is segmented and a special token is added. After being encoded by Transformer, the [EOS] token is used as the global text embedding.

[0012] The dynamic memory database aggregates global visual embeddings within training batches into a visual memory matrix. The global text embedding is constructed using a text memory matrix T, which stores only the modality embedding of the current batch.

[0013] The weighted aggregation formula is as follows:

[0014]

[0015] In the formula, The visually enhanced embedding obtained after memory retrieval; For visual memory matrix; This is a temperature hyperparameter, with a value range of 5 to 10. Visual memory matrix The transpose of the matrix; The global visual embedding for the current query; The normalization function is used to calculate the similarity between the query and the memory embedding and to generate weighted coefficients; preferably, the temperature hyperparameter... The value range is 8.

[0016] In step 2), the FINE loss includes text-to-visual contrast loss. Visual to text contrast loss .

[0017] In step 3), the SDM loss includes image-to-text loss. Text-to-image loss This is used to align the predicted similarity distribution with the actual matching distribution;

[0018] InfoLOOB loss Semantic Distribution Matching (SDM) Loss Identity (ID) loss With FINE loss Together they constitute the overall training loss, and the formula for the overall training loss is:

[0019]

[0020] The Adam optimizer is used during training.

[0021] Compared with the prior art, the present invention has the following outstanding technical effects and advantages:

[0022] This invention addresses the low cross-modal matching accuracy problem caused by large intra-class variability and high inter-class similarity in text-to-image person re-identification. It proposes a dynamic memory-enhanced cross-modal retrieval framework, which mainly includes a dynamic memory enhancement module, an attribute fine-grained alignment module, and a multi-loss function optimization module. The dynamic memory enhancement module constructs a dynamic memory bank based on a modern Hopfield network, dynamically retrieving multi-modal co-occurrence attribute features from the global image-text embedding to suppress sample noise and preserve discriminative information. The attribute fine-grained alignment module extracts text attribute phrases through a large language model, calculates their similarity to local visual embeddings, and optimizes local semantic alignment using FINE loss. The multi-loss function optimization module integrates InfoLOOB, SDM, ID, and FINE losses to achieve end-to-end training, effectively alleviating the core challenges of text-to-image person re-identification and improving the accuracy and robustness of cross-modal person retrieval. This invention effectively suppresses sample noise such as occlusion and illumination changes through a dynamic memory enhancement module, improving the robustness of cross-modal representation; the attribute fine-grained alignment module achieves accurate semantic alignment while controlling computational overhead, making up for the shortcomings of existing methods that focus on global features; and the multi-loss function optimization strategy further enhances the discriminativeness and stability of cross-modal matching, ultimately achieving more accurate and efficient text-to-image pedestrian re-identification, adapting to practical application scenarios such as intelligent monitoring and intelligent retail. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the dynamic memory-enhanced cross-modal retrieval framework of the present invention.

[0024] Figure 2 This is a flowchart illustrating the dynamic memory enhancement module of the present invention.

[0025] Figure 3 This is a flowchart illustrating the fine-grained alignment module for attributes in this invention.

[0026] Figure 4 This is a flowchart of the pedestrian attribute text description extraction process based on LLM according to the present invention. Detailed Implementation

[0027] The following embodiments, in conjunction with the accompanying drawings, will further illustrate the present invention. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention.

[0028] The overall implementation process of this invention is based on a dynamic memory-enhanced cross-modal retrieval framework (see [link]). Figure 1This framework achieves text-to-image person re-identification through the collaborative work of a dynamic memory enhancement module, an attribute fine-grained alignment module, and a multi-loss function optimization module. The dynamic memory enhancement module constructs a dynamic memory bank based on a modern Hopfield network, dynamically retrieving multimodal co-occurrence attribute features from the global image-text embedding to suppress sample noise and retain discriminative information. The attribute fine-grained alignment module extracts text attribute phrases through a large language model, calculates their similarity to local visual embeddings, and optimizes local semantic alignment using FINE loss. The multi-loss function optimization module integrates InfoLOOB, SDM, ID, and FINE losses to achieve end-to-end training, effectively alleviating the core challenges of text-to-image person re-identification and improving the accuracy and robustness of cross-modal person retrieval.

[0029] 1. See Figure 2 The dynamic memory enhancement module specifically includes the following steps:

[0030] 1) Input Feature Preparation Part of Dynamic Memory Enhancement Module

[0031] For a given image-text pedestrian sample pair, a pre-trained CLIP model is used to extract global modality embeddings as input. On the visual side, the CLIP B / 16 architecture is used. The pedestrian image is preprocessed and segmented into non-overlapping image blocks, then linearly projected and encoded using a Transformer to extract the [CLS] token as the global visual embedding. On the text side, the CLIPXformer architecture is used. The pedestrian description text is segmented and a special token is added. After Transformer encoding, the [EOS] token is extracted as the global text embedding.

[0032] 2) Construction of Dynamic Multimodal Memory

[0033] Based on the modern Hopfield network, the global visual embeddings of N samples within a training batch are aggregated into a visual memory matrix V, and the global text embeddings are aggregated into a text memory matrix T, constructing a dynamic memory bank. This memory bank is updated in real time with each training batch, storing only the modality embeddings of the current batch, ensuring that the retrieved features are semantically closely related to the current sample, and reducing the accumulation of redundant noise and semantic consistency loss.

[0034] 3) Memory retrieval and enhancement based on modern Hopfield networks

[0035] For any query embedding (visual embedding) Or text query Taking visual queries as an example, the formula is:

[0036]

[0037] To achieve memory retrieval: Temperature condition hyperparameter (value 5-10, preferably 8) controls the sharpness of the search; Weighted coefficients are generated by calculating the similarity between the query and the embedding in the memory database. The weighted aggregation yields the result. It integrates the co-occurrence attribute features of similar samples within a batch, which can effectively suppress sample noise such as occlusion and illumination changes, while retaining cross-modal discriminative information.

[0038] 2. See Figure 3 The fine-grained attribute alignment module addresses the issue of pre-trained CLIP models emphasizing global features while neglecting fine-grained local details. Its core function is to establish accurate correspondences between text attributes and visual regions, improve cross-modal matching semantics, and control computational overhead.

[0039] Figure 4 Given an example, the input text describing a pedestrian is: "This man is walking with a black bag on his back. He is wearing a black down jacket, black pants, and black leather shoes. He is wearing glasses. He has short black hair." The text contains multi-dimensional appearance information such as the pedestrian's clothing, accessories, and hairstyle. After inputting the text, a large language model combined with structured prompts (such as "identifying adjectives and their related nouns") is used to extract attribute phrases where adjectives modify nouns. Attribute information relevant to the pedestrian's appearance is filtered out, while irrelevant semantics (such as action descriptions like "walking") are removed. The output is a list of extracted attribute phrases, for example, "black bag," "glasses," "black down jacket," "black pants," "black leather shoes," and "short black hair," each phrase corresponding to a specific appearance attribute of the pedestrian. These attribute phrases are then input into the CLIP Xformer text encoder to generate local text embeddings. (K is the number of attribute tokens, and d is the feature dimension); then, local visual embeddings of the pedestrian image are extracted from the CLIP B / 16 visual encoder. (P is the number of image patch tokens), calculate and similarity After performing min-max normalization on the similarity matrix, the sparse threshold is applied. Filter out weakly related terms, then... Obtain alignment weights Finally based on Weighted aggregation Generate visual-text fusion embeddings And introduce text-to-visual contrast loss. Visual to text contrast loss The FINE algorithm optimizes local alignment by calculating only within a single image-text pair, thus avoiding batch-level overhead and ensuring global semantic consistency.

[0040] 3. The multi-loss function optimization module of this invention achieves end-to-end training by fusing multiple loss functions, solving the problems of unstable cross-modal alignment, discrete intra-class features, and insufficient local matching accuracy in text-to-image person re-identification. This ensures that the dynamic memory enhancement framework improves retrieval accuracy while maintaining training stability. Specifically, this module integrates four key losses. First, the InfoLOOB loss calculates cross-modal contrast similarity only for negative samples, avoiding gradient saturation caused by excessively high similarity of positive samples, especially suitable for scenarios with high similarity of pedestrian samples. Second, the Semantic Distribution Matching (SDM) loss includes image-to-text loss. Text-to-image loss The system optimizes global cross-modal matching consistency by aligning the predicted similarity distribution with the actual matching distribution. Secondly, it employs an identity (ID) loss to assign identity labels to image and text embeddings, promoting more compact embedding of the same identity features in the embedding space and enhancing intra-class discriminability. Finally, it utilizes the FINE loss of the attribute fine-grained alignment module, optimizing local semantic alignment accuracy through bidirectional contrastive losses between text-to-visual and visual-to-text approaches. These four types of losses collectively constitute the overall training loss, as shown in the formula: The Adam optimizer is used during training.

[0041] To verify the effectiveness of the method of this invention, tests were conducted on three currently published text-to-image person re-identification datasets, including CUHK-PEDES, ICFG-PEDES, and RSTPReid. These correspond to Tables 1, 2, and 3, respectively. These datasets are all industry-standard cross-modal retrieval datasets, capable of comprehensively reflecting changes in pedestrian appearance and text descriptions across different scenarios. The image encoder and text encoder used in this invention are both based on the same CLIP-B / 16 pre-trained model, ensuring consistency and comparability in the comparative experiments.

[0042] I. Explanation of Comparison Methods

[0043] 1. Non-CLIP methods: Cross-modal matching is achieved based on single-modal pre-trained models (such as ResNet, ViT or BERT), including methods such as C2A2, FedSH, BEAT, PBSL, ASAMN, LCR2S, TransTPS, MGCN and so on.

[0044] 2. CLIP-type methods: Feature extraction and alignment based on the multimodal pre-trained model CLIP, including methods such as CFine, IRRA, CFAM, MACF, TBPS-CLIP, IRLT, and MMRef.

[0045] II. Evaluation Indicators

[0046] The experiment uses evaluation metrics commonly used in the cross-modal retrieval field, including:

[0047] 1. Rank-1, Rank-5, and Rank-10 accuracy: used to evaluate the hit rate of search results at different ranking positions;

[0048] 2. mAP (mean precision): Used to comprehensively evaluate the ranking quality and recall completeness of search results.

[0049] The above metrics can comprehensively measure the effectiveness of the system in actual retrieval scenarios.

[0050] III. Overview of Test Results

[0051] On the CUHK-PEDES, ICFG-PEDES, and RSTPReid datasets, the method of this invention outperforms existing comparative methods in Rank-1, Rank-5, Rank-10 accuracy and mAP. Specifically:

[0052] 1. On the CUHK-PEDES dataset: The Rank-1 yield of the method of this invention is 74.95%, and the mAP is 66.57%, which are all improved compared with various existing CLIP-based methods.

[0053] 2. On the ICFG-PEDES dataset: When faced with longer sentences and more detailed descriptions, the Rank-1 and mAP metrics of the method in this invention reach 67.06% and 40.59% respectively, and the overall performance is better than similar technical solutions.

[0054] 3. On the RSTPReid dataset: This dataset contains data collected from multiple perspectives and across cameras. The method of this invention achieved a Rank-1 success rate of 63.65% and an mAP of 49.60%, further demonstrating that the invention can adapt to application environments with significant changes in perspective.

[0055] Table 1

[0056]

[0057] Table 2

[0058]

[0059] Table 3

[0060]

[0061] This invention effectively suppresses sample noise such as occlusion and illumination changes through a dynamic memory enhancement module, improving the robustness of cross-modal representation; the attribute fine-grained alignment module achieves accurate semantic alignment while controlling computational overhead, making up for the shortcomings of existing methods that focus on global features; and the multi-loss function optimization strategy further enhances the discriminativeness and stability of cross-modal matching, ultimately achieving more accurate and efficient text-to-image pedestrian re-identification, adapting to practical application scenarios such as intelligent monitoring and intelligent retail.

[0062] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A text-to-image person re-identification method based on dynamic memory enhancement, characterized in that... This method is based on a dynamic memory-enhanced cross-modal retrieval framework, which sequentially includes a dynamic memory enhancement module, an attribute fine-grained alignment module, and a multi-loss function optimization module. The method includes the following steps: 1) Through the dynamic memory enhancement module, the pre-trained CLIP model is used to extract the global visual embedding of the pedestrian image and the global text embedding of the text description for the given image-text pedestrian sample pair. Based on the modern Hopfield network, the global embeddings in the training batch are aggregated into the visual memory matrix and the text memory matrix to build a dynamic memory library that is updated in real time with the training batch. Through weighted aggregation, the dynamic retrieval of multimodal co-occurrence attribute features is realized, suppressing sample-specific noise caused by occlusion and illumination changes and preserving the discriminative cross-modal covariance pattern. 2) Through the attribute fine-grained alignment module, adjective-noun attribute phrases are extracted from the text description using a large language model combined with structured prompts to generate local text embeddings. The similarity between these embeddings and the local visual embeddings of pedestrian images extracted from the CLIP model is calculated and sparsified. The visual-text fusion embeddings are obtained by aggregating the generated alignment weights. The local semantic alignment is optimized by combining FINE loss without destroying global consistency. 3) Through the multi-loss function optimization module, InfoLOOB loss, SDM loss, ID loss and FINE loss are integrated to achieve end-to-end training, optimize the stability and discriminativeness of cross-modal matching, retrieve pedestrian images based on text descriptions, and complete text-to-image pedestrian re-identification.

2. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 1), the global visual embedding is extracted using the CLIP B / 16 architecture. The pedestrian image is preprocessed into non-overlapping image blocks and then linearly projected and encoded by Transformer. The [CLS] token is then used as the global visual embedding.

3. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 1), the global text embedding is extracted using the CLIP Xformer architecture. The pedestrian natural language description is segmented and a special token is added before being encoded by Transformer. The [EOS] token is then used as the global text embedding.

4. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 1), the dynamic memory bank aggregates global visual embeddings within training batches into a visual memory matrix. The global text embedding is constructed using a text memory matrix T, which stores only the modality embedding of the current batch.

5. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 1), the weighted aggregation formula is: In the formula, The visually enhanced embedding obtained after memory retrieval; For visual memory matrix; This is a temperature hyperparameter, with a value range of 5 to 10. Visual memory matrix The transpose of the matrix; The global visual embedding for the current query; This is a normalization function used to calculate the similarity between the query and the memory embedding and to generate a weighted coefficient.

6. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 5, characterized in that... Temperature overparameters The value range is 8.

7. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 2), the FINE loss includes text-to-visual contrast loss. Visual to text contrast loss .

8. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 3), the SDM loss includes image-to-text loss. Text-to-image loss This is used to align the predicted similarity distribution with the true matching distribution.

9. The text-to-image person re-identification method based on dynamic memory enhancement as described in claim 1, characterized in that... In step 3), InfoLOOB loss SDM loss ID loss With FINE loss Together they constitute the overall training loss, and the formula for the overall training loss is: The Adam optimizer is used during training.

Images