Text image cross-modal pedestrian retrieval method based on identity center hierarchical semantics
By constructing a bimodal memory bank and introducing a memory anchoring learning module, the problem of insufficient utilization of cross-view information in cross-modal pedestrian retrieval of text images is solved, and cross-view identity consistency and accurate retrieval are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGNAN UNIV
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies struggle to effectively utilize cross-viewpoint information in cross-modal pedestrian retrieval of text images, resulting in a lack of cross-viewpoint identity consistency constraints in the model, making it difficult to accurately distinguish pedestrians with similar visual appearances but different identities.
A bimodal memory bank is constructed. Through basic modal intramodal memory anchoring learning, structural relation identity-centered memory anchoring learning, and unified identity-centered semantic memory anchoring learning, intra-class semantic consistency is enhanced, perspective-invariant structural layout is captured, and a modality-independent unified identity representation is generated through a cross-modal lexical exchange mechanism.
It significantly improves the model's cross-modal pedestrian retrieval performance in complex scenarios, accurately distinguishing pedestrians with similar visual appearances but different identities, thus improving the accuracy and robustness of retrieval.
Smart Images
Figure CN122049948A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a method for cross-modal pedestrian retrieval of text images based on identity-centered hierarchical semantics. Background Technology
[0002] Cross-modal pedestrian retrieval using text and images is a technique that uses natural language descriptions to retrieve matching target pedestrians from a large-scale pedestrian image database. It has broad application prospects in scenarios such as public safety and intelligent security monitoring. Due to changes in perspective, the same pedestrian presents significant visual differences from different viewpoints, making it difficult for local text descriptions to adequately characterize the pedestrian's image region from other perspectives, resulting in semantically weak positive text-image pairs. When image regions from different identities but with similar visual appearances are incorrectly aligned with text descriptions, easily confused and difficult negative text-image pairs are formed. To address this, some methods attempt to differentiate between strong and weak positive sample pairs, such as assigning different temperature coefficients or loss weights to them in contrastive learning loss, or designing semi-positive triplet losses specifically for weak positive sample pairs. However, these methods are usually limited to local perspectives and lack a continuous and stable semantic anchor to explore identity-centric structured relational semantics from all perspectives and capture perspective-invariant structural layouts.
[0003] CLIP (Contrastive Language-Image Pre-training) models are pre-trained on text-image pairs collected from large-scale networks, providing prior knowledge of cross-modal correspondences. However, their global contrastive learning objective has significant limitations in viewpoint-sensitive and attribute-rich cross-modal person retrieval tasks involving text and images. They struggle to model fine-grained local semantic alignment, making them susceptible to confusion from difficult negative sample pairs. To address this, some methods have shifted to instance-level cross-modal interaction modeling, aiming to capture more refined local visual-text correspondences. For example, random masking of words or image patches can stimulate the model's perception of local details, or a shared cross-modal decoder and difficult negative sample matching mechanism can be designed to achieve fine-grained alignment optimization. However, most existing methods are limited to modeling independent text-image pairs, failing to fully utilize cross-viewpoint information to continuously refine unified identity semantics. This results in the learned representations lacking cross-viewpoint identity consistency constraints, weakening the model's ability to discriminate difficult negative sample pairs in complex scenes. Summary of the Invention
[0004] To address the aforementioned problems and technical requirements, this inventor proposes a text-image cross-modal pedestrian retrieval method based on identity-centric hierarchical semantics. First, a bimodal memory bank is constructed to continuously accumulate and maintain visual and textual representations of the identity hierarchy from all perspectives. A basic intramodal memory anchoring learning module is introduced, which enhances intra-class semantic consistency by aligning global features with the memory anchors (positive anchors) of corresponding identities within the modality, providing a stable foundation for identity-level representation learning. Second, to address the semantic inconsistency problem in weak positive sample pairs, a structural relation identity-centric anchoring learning module is proposed. This module mines identity-centric structural relation semantics to capture perspective-invariant structural layouts. Specifically, it extracts intramodal structural relation semantics and aligns these semantics in a shared memory semantic space, thereby strengthening identity consistency modeling of structural layouts across perspectives. Finally, to alleviate confusion caused by difficult negative sample pairs, a unified identity-centric semantic anchoring learning module is designed to extract unified identity essence semantics independent of modality. It introduces a cross-modal lexical exchange mechanism to achieve bidirectional fine-grained semantic fusion, generate a unified identity representation that is independent of modality, and further align it with bimodal positive memory anchors to enhance the model's ability to extract the essential semantics of the unified identity.
[0005] The technical solution of the present invention is as follows:
[0006] A text-image cross-modal person retrieval method based on identity-centric hierarchical semantics effectively captures identity-centric hierarchical semantics in complex perspective-changing scenarios, achieving reliable text-image cross-modal person retrieval. The method includes the following steps: Step S1: Input each pedestrian image and its corresponding text description in the training set into the cross-modal pedestrian retrieval model, extract the global features, local image patch features and selection features of the image, and the global features, local word features and selection features of the text, construct the visual memory bank and the text memory bank, and use the global image features and global text features of all samples to initialize the memory anchor points of each pedestrian identity in the corresponding modal memory bank. Step S2: Based on the global features of the images and their selected features, as well as the global features of the text and their selected features, perform a trusted consensus partitioning, and assign corresponding labels to each image-text sample pair according to the partitioning results; Step S3: Enter the basic modality intra-modal memory anchoring learning, align the global image features of the current sample with the memory anchor points of the corresponding identity in the visual memory bank, and at the same time align the global text features of the current sample with the memory anchor points of the corresponding identity in the text memory bank, so as to enhance the intra-class semantic consistency within the modality and provide a stable foundation for identity representation. Step S4: Enter the structural relationship identity center memory anchoring learning. Based on the global and local image patch features of the current sample and the global and local word features of the text, generate visual and text representations of structural relationship perception. Align them with the memory anchors of the corresponding identities in the visual memory bank and the text memory bank, respectively, to capture the perspective-invariant structural layout and enhance the semantic consistency of weak positive sample pairs. Step S5: Enter the unified identity center semantic memory anchoring learning. Based on the global and local image patch features of the current sample and the global and local word features of the text, generate a unified identity representation that is independent of the modality. Align it with the memory anchor points of the corresponding identities in the visual memory bank and the text memory bank to improve the model's ability to discriminate difficult negative samples. Step S6: Based on the above three types of memory anchoring learning loss and triplet alignment loss, construct the overall training objective function, and jointly optimize the model parameters through end-to-end backpropagation to obtain the optimal cross-modal pedestrian retrieval model, realize the extraction of query text features and image features to be retrieved, and calculate the similarity score between the two to obtain the final retrieval result.
[0007] The further technical solution is that step S1 includes feature extraction for each modality selection, and the construction of a bimodal memory bank: For the global and local image patch features and the global and local word features of the current sample, based on the correlation measurement between the global features and the local features under each modality, the local features that contribute highly to the global semantics are selected, and the image selection features and text word selection features are obtained respectively, so as to retain the most discriminative fine-grained semantic information. For each pedestrian identity, the mean of the global features of all its corresponding images and texts is calculated and used as the initial memory anchor point for that identity in the visual and textual modalities, and the visual memory bank and textual memory bank are constructed accordingly. Before the end of each training batch, the memory anchors corresponding to each identity in the visual memory bank and the text memory bank are dynamically updated using a moving average strategy to continuously accumulate identity-level semantic information.
[0008] The further technical solution is that the cross-modal pedestrian retrieval model in step S1 is implemented using the visual encoder and text encoder in the CLIP model.
[0009] The further technical solution is as follows: Step S2 performs trusted consensus partitioning, and assigns corresponding labels to each image-text sample pair according to the partitioning results. Specifically, in order to alleviate potential noise interference, the first triplet alignment loss is calculated for the global features of the images and texts of all current training samples and their corresponding selected features. On this basis, a two-component Gaussian mixture model (GMM) is used to fit the loss distribution of each sample. Clean sample pairs and noisy sample pairs are distinguished according to the posterior probability, and corresponding labels are assigned.
[0010] The further technical solution is that, in step S3, the basic modality memory anchoring learning loss is defined as the sum of the visual and text basic modality memory anchoring learning losses. The visual basic modality memory anchoring learning loss is achieved by aligning the global image features of the current sample with the memory anchor points of the corresponding identity in the visual memory bank. The text basic modality memory anchoring learning loss is similar.
[0011] The further technical solution is that, in step S4, a visual representation and text representation of structural relationship perception are generated based on the global and local image patch features and the global and local word features of the current sample, including: For the global and local image patch features and the global and local word features of the current sample, the cosine similarity between the global and local features in each modality is calculated, and the structural relationship score of each modality is generated by the softmax operation. Based on the structural relationship score, the local features of the corresponding modality are weighted and aggregated. The aggregated result is then combined with the global features of the corresponding modality and layer normalization is applied to generate visual structural relationship perception representation and text structural relationship perception representation.
[0012] Furthermore, the structural relation identity center memory anchoring learning loss is the sum of the visual and text structural relation identity center memory anchoring losses. The visual structural relation identity center memory anchoring loss is achieved by aligning the visual structural relation perceptual representation with the memory anchors of the corresponding identities in the visual memory bank and the text memory bank. The text structural relation identity center memory anchoring loss is similar.
[0013] A further technical solution is that, in step S5, a unified identity representation independent of modality is generated based on the global and local image patch features and the global and local word features of the current sample, including: A cross-modal word exchange mechanism (CMTE) is adopted to perform two rounds of bidirectional exchange between the local image patch feature sequence and the local word feature sequence of the current sample, and then concatenate it with the corresponding global features to form a hybrid sequence guided by different modalities. After each exchange, a unified semantic mining module is used to extract modality-independent unified semantic representations from the two mixed sequences respectively; Based on the modality-independent unified semantic representation obtained from the second exchange and the original global features of each modality, the enhanced representations of images and text are determined, and the enhanced representations of the two are fused to obtain a modality-independent unified identity representation.
[0014] Furthermore, the unified identity center semantic memory anchoring learning loss is achieved by aligning the modality-independent unified identity representation with the memory anchors of the corresponding identity in the visual memory bank and the text memory bank.
[0015] The further technical solution is that, in step S6, an overall training objective function is constructed based on the above three types of memory anchoring learning loss and triplet alignment loss, including: For the global features of images and texts of all samples in the current training batch and their corresponding selected features, the second triplet alignment loss is calculated with the corresponding label as a constraint to suppress the negative impact of potential noisy sample pairs and further enhance the consistency of cross-modal matching. According to the preset weights, the basic modality memory anchoring loss, the structural relation identity center memory anchoring loss, the unified identity center semantic memory anchoring loss, and the second triplet alignment loss are weighted and summed to form the overall training objective function.
[0016] The beneficial technical effects of this invention are: 1. By introducing basic modal memory anchoring learning, the intra-class semantic consistency of the same identity across different camera perspectives is effectively improved, providing a stable and reliable identity hierarchy representation foundation for subsequent fine-grained cross-modal matching.
[0017] 2. By introducing structural relation identity-centered memory anchoring learning, we can mine identity-centric structural relation semantics and capture viewpoint-invariant structural layouts. Aligning visual and textual structural relation perceptual representations in a shared memory semantic space significantly alleviates the mismatch between local descriptions and image regions caused by viewpoint changes, and improves the semantic consistency of weak positive sample pairs.
[0018] 3. By introducing unified identity center semantic memory anchoring learning, it utilizes a cross-modal lexical exchange mechanism to achieve bidirectional fine-grained semantic interaction between image and text lexical sequences, generating a unified and highly discriminative identity representation. This representation is further aligned with bimodal positive memory anchors to obtain a unified identity essence semantic that is independent of modality, significantly enhancing the model's ability to distinguish difficult negative sample pairs that are visually similar but have different identities.
[0019] 4. A hierarchical and progressive memory anchoring system is constructed, progressively deepening the semantic expression of identity from intra-modal and intra-class semantic consistency and structural relationship modeling to modality-independent semantic unification. The retrieval phase does not require calling the aforementioned anchoring learning module, significantly improving the performance of cross-modal pedestrian retrieval in complex scenarios without sacrificing reasoning efficiency. Attached Figure Description
[0020] Figure 1 This is a flowchart of the text image cross-modal pedestrian retrieval method based on identity-center hierarchical semantics provided in this application; Figure 2 This is a diagram of the overall network framework for cross-modal pedestrian retrieval of text images based on hierarchical semantics of identity center, as provided in this application. Figure 3 This is a diagram of the basic intramodal memory anchoring learning module provided in this application; Figure 4 This is a diagram of the structural relationship identity-centered memory anchoring learning module provided in this application; Figure 5 This is a diagram of the semantic memory anchoring learning module of the unified identity center provided in this application; Figure 6 This is a visual comparison chart of the search results of each module provided in this application, where (a), (b), and (c) correspond to different query texts, green boxes represent correct searches, and red boxes represent incorrect searches. Detailed Implementation
[0021] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0022] Please refer to Figure 1 As shown, one embodiment of this application provides a text image cross-modal pedestrian retrieval method based on identity-centered hierarchical semantics. The identity-centered hierarchical semantic memory anchoring learning framework HICS-MAL algorithm framework in this method is as follows: Figure 2 As shown. In this embodiment, HICS-MAL uses RDE. [1] As a baseline model, it employs trusted consensus partitioning and triple alignment loss. A joint optimization model is employed. In each training epoch, a trusted consensus partition is first performed to initially assign corresponding labels to image-text pairs. Subsequently, using these labels as supervision signals, a triplet alignment loss is applied to the global features and corresponding selected features to suppress the negative impact of potentially noisy sample pairs. To further mitigate cross-viewpoint interference, this application constructs a visual and textual bimodal memory bank, which is initialized with the feature mean of all samples for each identity. During training, identity-level features across all camera views are continuously accumulated and maintained. The Basic Modal Intramodal Memory Anchoring Learning (BIM-MAL) enhances intra-class semantic consistency within modalities by aligning the global features of each modality with the memory anchor points of the corresponding identity in its corresponding modal memory bank. The Structural Relationship Identity-Centered Memory Anchoring Learning (SRIC-MAL) further mines identity-centered structural relation semantics, extracting viewpoint-invariant structural layouts to strengthen the semantic consistency of weak positive sample pairs. The Unified Identity Center Semantic Memory Anchoring Learning (UICS-MAL) employs a cross-modal lexical exchange mechanism (CMTE) to promote bidirectional fine-grained semantic interaction, extracting the essential semantics of the unified identity that are independent of modality, thereby effectively identifying easily confused and difficult negative samples. This method specifically includes the following: Step S1: Input each pedestrian image and its corresponding text description in the training set into the cross-modal pedestrian retrieval model, extract the global features, local image patch features and their selection features of the image, and the global features, local word features and their selection features of the text, construct the visual memory bank and the text memory bank, and use the global image features and global text features of all samples to initialize the memory anchor points of each pedestrian identity in the corresponding modal memory bank.
[0023] Specifically, for the input of the first Training image-text pairs Image features are obtained using the visual encoder in the CLIP model. .in, Represents the feature dimension. It is a global feature of the image. represent Local features of each image patch. For the text description, the lowercase BPE segmentation method with a vocabulary size of 49,152 is first used to segment it into a word sequence, and start markers are added to the beginning and end of the sequence. and end marker Subsequently, the tokenized sequence is input into the text encoder in the CLIP model to obtain features. ,in, and Corresponding to the marks respectively and The coding features, Considered as a sentence-level global feature. It is a local lexical-level feature.
[0024] Combination Figure 2 As shown, for the current sample image, global and local image patch features With text global and local lexical features The lexical selection module (TS) selects local features that contribute significantly to global semantics based on the correlation between global and local features across different modalities, thereby obtaining image selection features. Text lexical selection features This is to preserve the most discriminative, fine-grained semantic information.
[0025] For each pedestrian identity, the mean of the global features of all its corresponding images and texts is calculated and used as the initial memory anchor for that identity in the visual and textual modalities. Based on this, a visual memory bank and a textual memory bank are constructed for the subsequent continuous accumulation of identity-level semantic information.
[0026] Step S2: Based on the global features of the images and their selected features, as well as the global features of the text and their selected features of all samples, perform a trusted consensus partitioning, and assign corresponding labels to each image-text sample pair according to the partitioning results.
[0027] Specifically, to mitigate potential noise interference, global features of images and text from all samples in the current training round are analyzed. and their corresponding selection features The first triplet alignment loss is calculated. Based on this, a two-component Gaussian mixture model (GMM) is used to fit the loss distribution of each sample. Clean sample pairs and noisy sample pairs are distinguished according to the posterior probability, and corresponding labels are assigned. In this embodiment, the label of clean sample pairs is 1, and the label of noisy sample pairs is 0.
[0028] Step S3: Enter the basic modality intra-modal memory anchoring learning, align the global image features of the current sample with the memory anchor points of the corresponding identity in the visual memory bank, and at the same time align the global text features of the current sample with the memory anchor points of the corresponding identity in the text memory bank, so as to enhance the intra-class semantic consistency within the modality.
[0029] The basic modal intramodal memory anchoring learning process is as follows: Figure 3 As shown, for the image The visual fundamental modality intramodal memory anchoring learning loss is based on aligning the global image features of the current sample with the memory anchors corresponding to the identity in the visual memory bank, and is defined as follows:
[0030] in, It is an image The identity tag to which they belong Represents the visual memory bank and the first i Memory anchors (positive anchors) corresponding to the same identity of each sample. Represents the first in the visual memory bank j Each identity corresponds to a memory anchor point. This represents the calculation of cosine similarity. Represents the first temperature coefficient. This represents the total number of pedestrian identities in the training set.
[0031] Similarly, in the above formula Replace with , Replace with text memory , Replace with To calculate the intramodal memory anchoring learning loss of the text. Overall basic modal memory anchoring learning loss This is the sum of the learning losses for memory anchoring within the visual and textual base modalities. This mechanism enhances intra-modal consistency by bringing global features of text and images closer to positive memory anchors within their respective modalities and pushing them away from negative memory anchors.
[0032] To continuously accumulate intra-class semantic information within each modality, the memory anchors corresponding to each identity in the visual memory bank and text memory bank are dynamically updated using a moving average strategy before the end of each training batch. The update mechanism given in this embodiment is defined as follows:
[0033] in, It is a memory update parameter.
[0034] Step S4: Enter the structural relationship identity center memory anchoring learning. Based on the global and local image patch features of the current sample and the global and local word features of the text, generate visual and text representations of structural relationship perception. Align them with the memory anchors of the corresponding identities in the visual memory bank and the text memory bank, respectively, to capture the perspective-invariant structural layout and enhance the semantic consistency of weak positive sample pairs.
[0035] Structural relational identity-centered memory anchoring learning process as follows Figure 4 As shown, for the image Its global features The overall spatial structure information of pedestrians is encoded, such as the layout relationship between the torso, limbs, and footwear. To model the semantic association between local regions and the global structure, the following steps are first performed: Local features of each image patch The cosine similarity between them is used to generate visual structural relationship scores through softmax operation. :
[0036] This score reflects the degree to which a local region contributes to the global spatial structure. For example, if the torso region is partially occluded while the lower limbs region is visible, then the lower limb block... The scores will be dynamically boosted, thus supplementing the global representation with stable local cues. Conversely, scores from irrelevant background regions are effectively suppressed. Subsequently, based on the aforementioned relationship scores, the scores are adjusted accordingly. Perform weighted aggregation, and add the aggregation result to the weighted aggregation result. Layer normalization (LN) is then applied to generate a visual structural relationship perceptual representation. :
[0037] This representation explicitly models the structural contribution of each body part to the overall spatial configuration. To further enhance the model's understanding of viewpoint-invariant spatial layout, a visual structural relationship identity center memory anchoring loss is designed, based on the visual structural relationship perceptual representation. This is achieved by aligning the images with memory anchors of the corresponding identities in the visual memory bank and the text memory bank. For all images and texts in the current training batch, memory anchors with the same identities as them are retrieved from the visual memory bank and the text memory bank respectively, forming a set. and Based on this, the visual structural relationship identity-centered memory anchoring loss :
[0038] in, Represents the text memory with the first i Memory anchor points corresponding to the same identity of each sample; , It is the second temperature coefficient. As a regularization term, it is used to encourage Global features of the original image Maintain moderate differences and enhance feature diversity. This loss function drives... By approaching positive memory anchors within and across modalities while moving away from negative memory anchors, viewpoint-specific interference is suppressed, enhancing the perception of identity-invariant body component structures and establishing cross-modal consistent spatial structural semantic understanding.
[0039] Similarly, to capture context-aware language structure, this embodiment adaptively aggregates local text features and global text features. Represent the overall descriptive logic. Replace the visual features in equation (3) with text features and calculate the text structure relationship score. This results in higher weights for morphemes strongly associated with identity attributes, such as "red coat," while lower weights are given to function words like "with." This score is then used to weight and fuse the local and global features of the text to obtain a representation of the text's structural relationships. Then, in equation (5) and Replace with and Calculate the text structure relation identity center anchoring loss ,guide By moving closer to positive memory anchors within and across modalities, and further away from negative memory anchors, the model learns identity-related language structures robust to changes in perspective and phrasing. The overall structural relation identity-center anchoring loss... The sum of the memory anchoring losses for the identity center of visual and textual structural relationships is used. Through a shared memory semantic space, the visual spatial layout semantics and textual descriptive organization semantics are aligned, effectively enhancing the model's understanding of viewpoint-invariant structural relationships in weak positive sample pairs.
[0040] Step S5: Enter the unified identity center semantic memory anchoring learning. Based on the global and local image patch features of the current sample and the global and local word features of the text, generate a unified identity representation that is independent of the modality. Align it with the memory anchor points of the corresponding identities in the visual memory bank and the text memory bank to improve the model's ability to discriminate difficult negative samples.
[0041] The unified identity center semantic memory anchoring learning process is as follows: Figure 5 As shown, the cross-modal lexical exchange mechanism (CMTE) is first used to achieve bidirectional fine-grained semantic interaction, including: performing a round of bidirectional exchange between the local image patch feature sequence and the local lexical feature sequence of the current sample, and then concatenating it with the corresponding global features, that is, visual global features. Local lexical features of text By piecing together the elements, a visually guided hybrid sequence is formed. Accordingly, the global sentence-level features of the text are... With visual local block features By splicing, a text-guided hybrid sequence is formed. Subsequently, through the shared unified semantic mining module, from and Modality-independent unified semantic representations are extracted from each part. In this embodiment, the shared unified semantic mining module mainly consists of multi-head self-attention (...). ) and multilayer perceptron ( The structure is as follows:
[0042] In CMTE, global features from one modality interact with local features from another. Global text lexical units can focus on visual regions of a pedestrian's body and suppress regions irrelevant to the text description. Symmetrically, global visual lexical units aggregate key lexical units aligned with the visual content. This process achieves preliminary cross-modal semantic fusion.
[0043] To further explore the unified semantics within heterogeneous features, intra-modal semantic consolidation is performed under cross-modal guidance, i.e., a two-round bidirectional exchange is conducted. and Reassembled into The data is then input into the unified semantic mining module again to obtain secondary fusion features. Similarly, processing the corresponding text side can yield... At this point, the updated global representation and This is not a simple aggregation of original features, but rather a re-refinement based on local lexical units that have absorbed cross-modal semantics, enabling a more accurate summary of consistent, fine-grained cues across modalities. Subsequently, based on the modality-independent unified semantic representation obtained from the second exchange and the original global features of each modality, enhanced representations of the image and text are determined, i.e. Compared with original visual global features The summation and layer normalization yield the enhanced visual representation. Similarly, enhanced text representations can be obtained. Fusion enhancement representation, i.e., taking The average value yields a unified identity representation that is independent of modality. This representation effectively preserves bimodal consensus semantic cues.
[0044] To continuously strengthen Identity-aware semantic consistency, and the unified identity center semantic memory anchoring learning loss in this embodiment. based on The alignment with the memory anchors corresponding to the identity in the visual memory bank and text memory bank is implemented, as defined below:
[0045] in, , It is the third temperature coefficient. and As a regularization term, it is used to improve feature diversity. Under guidance, The model moves closer to positive bimodal memory anchors and further away from negative memory anchors. By enhancing intra-class compactness and expanding the decision boundary for identity-specific details, this loss function can be viewed as a fine-grained discriminator for hard negative samples, thereby effectively mining identity-centric unified semantics and significantly improving the model's ability to discriminate hard negative sample pairs in cross-perspective scenarios.
[0046] Step S6: Based on the above three types of memory anchoring learning loss and triplet alignment loss, construct the overall training objective function, and jointly optimize the model parameters through end-to-end backpropagation to obtain the optimal cross-modal pedestrian retrieval model, realize the extraction of query text features and image features to be retrieved, and calculate the similarity score between the two to obtain the final retrieval result.
[0047] Specifically, constructing the overall training objective function includes: calculating the second triplet alignment loss using the global features of the images and text of all samples in the current training batch and their corresponding selected features, with the corresponding labels as constraints. To suppress the negative impact of potentially noisy samples and further enhance the consistency of cross-modal matching, the memory anchoring loss within the basic modality is then weighted according to preset weights. Structural relationship identity-centered memory anchoring loss Unified Identity Center Semantic Memory Anchoring Loss and the second triplet alignment loss The weighted summation yields the overall training objective function (total loss function), defined as follows:
[0048] in, , , They are , , Weighting coefficients of the loss function.
[0049] In this embodiment, the optimal cross-modal pedestrian retrieval model is used to extract features from the query text and the image to be retrieved, and to calculate their similarity score to obtain the final retrieval result, including: During the retrieval phase, the query text and pedestrian images from the target database are input into the trained text encoder and visual encoder, respectively, to obtain global features and term selection features of the text, and global features and image selection features of the images. The cosine similarity between the global features of the query text and the global features of the images, and the cosine similarity between the term selection features of the query text and the image selection features, are calculated separately. The average of these two values is used to obtain a comprehensive similarity score, which is then used to rank the candidate images, outputting the target pedestrian image that best matches the query text.
[0050] To verify the effectiveness of this invention, experiments were conducted on the publicly available CUHK-PEDES, ICFG-PEDES, and RSTPReid datasets. CUHK-PEDES contains 40,206 images of 13,003 pedestrians with different identities, with an average of two descriptions per image. Its training set contains 34,054 images of 11,003 pedestrians and their corresponding 68,108 descriptions; the validation set consists of 3,078 images of 1,000 pedestrians and 6,158 descriptions; and the test set includes 3,074 images of 1,000 pedestrians and 6,156 descriptions. The ICFG-PEDES dataset, derived from the MSMT17 dataset, contains 54,522 images from 4,102 pedestrians with different identities, with each image accompanied by a text description. The training set contains 34,674 image-text pairs for 3,102 pedestrians, while the test set consists of 19,848 image-text pairs for 1,000 pedestrians. The RSTPReid dataset covers 20,505 images and corresponding 41,010 text descriptions for 4,101 pedestrians. Each identity has 5 images taken from different camera perspectives, and each image corresponds to two descriptions. The training set includes samples for 3,701 pedestrian identities, the validation set includes samples for 200 pedestrian identities, and the test set includes samples for 200 pedestrian identities.
[0051] During the training phase, image data is augmented using techniques such as random erasure, random horizontal flipping, and random cropping with padding. Text data is augmented using word-level random replacement, masking, and deletion operations. Input image sizes are uniformly adjusted. The maximum word length for the text input was set to 77. The batch size for each training round was set to 64. To evaluate the model's robustness in noisy environments, this experiment followed the RDE approach. [1] The proposed clean and noisy data settings involve randomly shuffling text descriptions in the training set at ratios of 20% and 50%, artificially injecting synthetic noise to simulate mismatch scenarios. (Memory update coefficients) Set it to 0.2. See Table 1 for other hyperparameter configurations: Table 1. Dataset Experimental Parameter Settings
[0052] The method proposed in this embodiment is compared with existing advanced methods. The comparison results are shown in Table 2, where the bolded values represent the best accuracy in all experiments.
[0053] Table 2. Comparison results between the present invention and existing advanced methods (%)
[0054] As shown in Table 2, the method proposed in this embodiment achieves superior retrieval performance on various datasets. This invention, by progressively deepening identity semantic expression from intra-class consistency and structural relationship modeling across modalities to modality-independent semantic unification, effectively addresses the inconsistencies between weak positive sample semantics and the challenge of easily confused difficult negative samples.
[0055] To verify the effectiveness of each learning module in this invention, an ablation experiment was conducted, and the results are shown in Table 3. Table 3. Ablation experiments for each module (%)
[0056] Baseline represents the baseline model. 1 corresponds to Baseline+BIM-MAL, 2 to Baseline+BIM-MAL+SRIC-MAL, and 3 to Baseline+BIM-MAL+UICS-MAL. The system verifies its effectiveness by progressively introducing the memory anchoring learning modules proposed in this embodiment. As shown in Table 3, when the BIM-MAL memory anchoring learning module within the basic modality is added, there is a significant improvement in retrieval accuracy on both datasets. Figure 6 As shown in (a), (b), and (c), compared to the Baseline, the correctly matched samples are ranked significantly higher in the search list after the introduction of this module, which verifies that it can effectively enhance intra-class semantic consistency to improve search performance.
[0057] Building upon this foundation, the SRIC-MAL module, which focuses on structural relationship identity center memory anchoring, was further added. As shown in Table 3, the model's retrieval accuracy was further improved across both datasets. Figure 6 As shown in (a), for example, for the first query text describing the appearance of pants and shoes, the correct retrieved images are still able to be ranked at the top despite the occlusion of these visual areas due to the change in viewing angle. These experimental results demonstrate that the module effectively alleviates the semantic mismatch problem of weak positive sample pairs by mining view-invariant structural layouts.
[0058] As shown in Table 3, when the Unified Identity Center Semantic Memory Anchoring Learning (UICS-MAL) module is added, it performs bidirectional fine-grained semantic interaction to extract modality-independent essential semantics of the unified identity, thereby suppressing interference from difficult negative samples. This module can effectively improve retrieval accuracy. Figure 6 As shown in (a) of the table, for the first query text, compared to only adding the BIM-MAL setting, the second column of its retrieved image results shows that it can effectively distinguish difficult negative samples that are semantically similar to the text description. Figure 6As shown in (c), even when there are pedestrians with high similarity but different identities, this module is able to retrieve more correct images compared to simply adding BIM-MAL.
[0059] Finally, as shown in Table 3, when these three modules are used in combination, the identity-center hierarchical semantic memory anchoring learning framework HICS-MAL proposed in this embodiment is constituted. The model trained through this framework can achieve optimal performance. Figure 6 As shown in (a), (b), and (c), HICS-MAL can still consistently rank more correct images at the top of the search results. This fully demonstrates the effectiveness of the identity-centered hierarchical semantic memory anchoring learning framework constructed in this embodiment, which can significantly improve the performance of cross-modal pedestrian retrieval in complex scenarios.
[0060] It should be noted that the various existing methods cited in Table 2 can be found in the following literature: [1]QIN Y, CHEN Y, PENG D, et al. Noisy-Correspondence Learning forText-to-Image Person Re-Identification[C] / / 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 27187-27196. [2]JIANG D, YE M. Cross-Modal Implicit Relation Reasoning andAligning for Text-to-Image Person Retrieval[C] / / 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 2787-2797. [3]SHEN W, FANG M, WANG Y, et al. Enhancing visual representation for text-based person searching[J]. Knowledge-Based Systems, 2025, 309: 112893. [4]GOU Y, LI Z, ZHANG J, et al. Instance-level feature biascalibration learning for text-to-image person re-identification[J].Knowledge-Based Systems, 2025, 315: 113251. The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.
Claims
1. A text-image cross-modal pedestrian retrieval method based on identity-centered hierarchical semantics, characterized in that, The method includes: Each pedestrian image and its corresponding text description in the training set are input into the cross-modal pedestrian retrieval model. Global features, local image patch features, and selection features of the image are extracted, as well as global features, local word features, and selection features of the text. Visual memory and text memory are constructed, and the global features of the image and the global features of the text of all samples are used to initialize the memory anchor points of each pedestrian identity in the corresponding modal memory. Based on the global features and selected features of the images and the global features and selected features of the text of all samples, a reliable consensus partition is performed, and a corresponding label is assigned to each image-text sample pair according to the partition result. Entering the basic modality intra-modal memory anchoring learning, the global image features of the current sample are aligned with the memory anchor points of the corresponding identity in the visual memory bank, and the global text features of the current sample are aligned with the memory anchor points of the corresponding identity in the text memory bank, so as to enhance the intra-class semantic consistency within the modality; Enter the structural relationship identity center memory anchoring learning, generate visual representation and text representation of structural relationship perception based on the global and local image block features and global and local word features of the current sample, and align them with the memory anchor points of the corresponding identities in the visual memory bank and the text memory bank respectively, so as to capture the perspective-invariant structural layout and enhance the semantic consistency of weak positive sample pairs. The model enters the unified identity center semantic memory anchoring learning process. Based on the global and local image patch features and the global and local word features of the current sample, a unified identity representation independent of modality is generated and aligned with the memory anchor points of the corresponding identities in the visual memory bank and the text memory bank to improve the model's ability to discriminate difficult negative samples. Based on the above three types of memory anchoring learning loss and triplet alignment loss, an overall training objective function is constructed. The model parameters are jointly optimized through end-to-end backpropagation to obtain the optimal cross-modal pedestrian retrieval model. This model extracts the features of the query text and the image to be retrieved, and calculates the similarity score between the two to obtain the final retrieval result.
2. The text image cross-modal pedestrian retrieval method according to claim 1, characterized in that, The generation of structural relationship-aware visual and textual representations based on global and local image patch features and global and local word features of the current sample includes: For the global and local image patch features and the global and local word features of the current sample, the cosine similarity between the global features and local features in each modality is calculated, and the structural relationship score of each modality is generated by the softmax operation. Based on the structural relationship score, the local features of the corresponding modality are weighted and aggregated. The aggregation result is then combined with the global features of the corresponding modality and layer normalization is applied to generate visual structural relationship perception representation and text structural relationship perception representation.
3. The text image cross-modal pedestrian retrieval method according to claim 1, characterized in that, The structural relation identity center memory anchoring learning loss is the sum of the visual and textual structural relation identity center memory anchoring losses. The visual structural relation identity center memory anchoring loss is achieved by aligning the visual structural relation perceptual representation with the memory anchors of the corresponding identities in the visual memory bank and the textual memory bank, and is defined as follows: in, It is the first i Visual structural relationship perceptual representation of each sample Represents the visual memory bank and the first i Memory anchors corresponding to the same identity of each sample Represents the text memory with the first i Memory anchor points corresponding to the same identity of each sample; and These represent sets of memory anchors with the same identity as all images and texts in the current training batch, retrieved from the visual memory bank and the text memory bank respectively. , It is the second temperature coefficient. This represents the calculation of cosine similarity; As a regularization term, it is used to encourage Global features of the original image Maintain appropriate differences and enhance feature diversity.
4. The text image cross-modal pedestrian retrieval method according to claim 1, characterized in that, The generation of a modality-independent unified identity representation based on global and local image patch features and global and local word features of the current sample includes: A cross-modal word exchange mechanism is adopted to perform two rounds of bidirectional exchange between the local image block feature sequence and the local word feature sequence of the current sample, and then concatenate it with the corresponding global features to form a hybrid sequence guided by different modalities; After each exchange, a unified semantic mining module is used to extract modality-independent unified semantic representations from the two mixed sequences respectively; Based on the modality-independent unified semantic representation obtained from the second exchange and the original global features of each modality, the enhanced representations of the image and text are determined, and the enhanced representations are fused to obtain the modality-independent unified identity representation.
5. The text image cross-modal pedestrian retrieval method according to claim 4, characterized in that, The unified identity center semantic memory anchoring learning loss is achieved by aligning the modality-independent unified identity representation with the memory anchors of the corresponding identities in the visual memory bank and the text memory bank, and is defined as follows: in, It is the first i A modality-independent unified identity representation for each sample. Represents the visual memory bank and the first i Memory anchors corresponding to the same identity of each sample Represents the text memory with the first i Memory anchor points corresponding to the same identity of each sample; and These represent sets of memory anchors with the same identity as all images and texts in the current training batch, retrieved from the visual memory bank and the text memory bank respectively. , It is the third temperature coefficient. This represents the calculation of cosine similarity; It is a visual enhancement representation. It is the global feature of the original image. It is a text-enhanced representation. It is a global feature of the original text. and As a regularization term, it is used to improve feature diversity.
6. The text image cross-modal pedestrian retrieval method according to claim 1, characterized in that, The basic modality intra-modal memory anchoring learning loss is the sum of the visual and textual basic modal intra-modal memory anchoring learning losses. The visual basic modal intra-modal memory anchoring learning loss is based on aligning the global image features of the current sample with the memory anchor points corresponding to the identity in the visual memory bank, and is defined as follows: in, It is the first i Global features of an image for each sample Represents the visual memory bank and the first i Memory anchors corresponding to the same identity of each sample Represents the first in the visual memory bank j Each identity corresponds to a memory anchor point. This represents the calculation of cosine similarity. Represents the first temperature coefficient. This represents the total number of pedestrian identities in the training set.
7. The text image cross-modal pedestrian retrieval method according to claim 1, characterized in that, The initialization of memory anchors for each pedestrian identity in the corresponding modality memory database using the global image features and global text features of all samples includes: For each pedestrian identity, the mean of the global features of all its corresponding images and texts is calculated and used as the initial memory anchor point for that identity in the visual and textual modalities, and the visual memory bank and the textual memory bank are constructed accordingly. Before the end of each training batch, the memory anchors corresponding to each identity in the visual memory bank and the text memory bank are dynamically updated using a moving average strategy to continuously accumulate identity-level semantic information.
8. The text image cross-modal pedestrian retrieval method according to claim 7, characterized in that, The update mechanism for dynamically updating the memory anchors corresponding to each identity in the visual memory bank and the text memory bank using a moving average strategy is defined as follows: in, It is a memory update parameter. , Representing the visual and textual memory banks respectively, and the first i Memory anchors corresponding to the same identity of each sample , These represent the global features of the original image and text, respectively.
9. The text image cross-modal pedestrian retrieval method according to claim 1, characterized in that, Based on the above three types of memory anchoring learning loss and triplet alignment loss, an overall training objective function is constructed, including: For the global features of images and texts of all samples in the current training batch and their corresponding selected features, triplet alignment loss is calculated with the corresponding labels as constraints to suppress the negative impact of potential noisy samples and further enhance the consistency of cross-modal matching. The overall training objective function is formed by weighting and summing the basic modality memory anchoring loss, structural relation identity center memory anchoring loss, unified identity center semantic memory anchoring loss, and the triple alignment loss according to the preset weights.
10. The text image cross-modal pedestrian retrieval method according to any one of claims 1-9, characterized in that, The cross-modal pedestrian retrieval model is implemented using the visual encoder and text encoder in the CLIP model.